Playing with the X-Robots-Tag HTTP header

Crawler directives The robots.txt file only contains the so called ‘crawler directives’, which tells search engines where they are or aren’t allowed to go. By using the Allow directive, you can specify where search engines are allowed to crawl. Disallow...

We do not write or claim ownership of any of the articles featured on this website. Instead we aim to highlight them so you can read the full article at the original source. To finish reading this article, follow the continue reading link below. CONTINUE READING
Author: Yoast

Yoast build & tweak websites for performance, SEO and conversion, often using their WordPress plugins

http://yoast.com/

Sign up to our newsletter

Don't worry, we won't give your email to any third parties
and you can unsubscribe at any time.