robots.txt: the ultimate guide

User-agent directive The first bit of every block of directives is the user-agent. A user-agent identifies a specific spider. The user-agent field is matched against that specific spider’s (usually longer) user-agent. For instance, the most common spider from Google has the...

We do not write or claim ownership of any of the articles featured on this website. Instead we aim to highlight them so you can read the full article at the original source. To finish reading this article, follow the continue reading link below. CONTINUE READING
Author: Yoast

Yoast build & tweak websites for performance, SEO and conversion, often using their WordPress plugins

http://yoast.com/

Sign up to our newsletter

Don't worry, we won't give your email to any third parties
and you can unsubscribe at any time.