Skip to content

Instantly share code, notes, and snippets.

What would you like to do?
Robots.txt file instructing crawlers to not crawl any pages on the hostname
User-agent: *
Disallow: /
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.