Skip to content

Instantly share code, notes, and snippets.

@wesort

wesort/dev.robots.txt

Last active December 12, 2019 15:40
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save wesort/95476b90c28e08de3828ef45d086a154 to your computer and use it in GitHub Desktop.
Save wesort/95476b90c28e08de3828ef45d086a154 to your computer and use it in GitHub Desktop.
Download ZIP
A sample robots.txt file for a development site
Raw
dev.robots.txt
# robotstxt.org/
User-agent: *
Disallow: /
# Allow Twitter access for validation testing
# https://cards-dev.twitter.com/validator
User-agent: Twitterbot
Disallow:
@wesort
Copy link
Author

wesort commented Dec 12, 2019

Rename the file robots.txt manually and ensure the file is ignored from git within .gitignore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment