Sign up to our newsletter
Don't worry, we won't give your email to any third parties
and you can unsubscribe at any time.
User-agent directive The first bit of every block of directives is the user-agent. A user-agent identifies a specific spider. The user-agent field is matched against that specific spider’s (usually longer) user-agent. For instance, the most common spider from Google has the...