Sign up to our newsletter
Don't worry, we won't give your email to any third parties
and you can unsubscribe at any time.
Crawler directives The robots.txt file only contains the so called ‘crawler directives’, which tells search engines where they are or aren’t allowed to go. By using the Allow directive, you can specify where search engines are allowed to crawl. Disallow...