Skip to content

Instantly share code, notes, and snippets.

@jakebellacera

jakebellacera/robots.txt

Created September 14, 2017 23:18
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jakebellacera/b66759db5b48f14507a56657f1e635b2 to your computer and use it in GitHub Desktop.
Save jakebellacera/b66759db5b48f14507a56657f1e635b2 to your computer and use it in GitHub Desktop.
Download ZIP
Robots.txt file instructing crawlers to not crawl any pages on the hostname
Raw
robots.txt
User-agent: *
Disallow: /
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment