Skip to content

Instantly share code, notes, and snippets.

@epreston

epreston/basic.robots.txt

Created April 28, 2023 19:37
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save epreston/9ec9d557b950ca548f9b23c4b4c92cb1 to your computer and use it in GitHub Desktop.
Save epreston/9ec9d557b950ca548f9b23c4b4c92cb1 to your computer and use it in GitHub Desktop.
Download ZIP
robots.txt - basic robots.txt with no configuration
Raw
basic.robots.txt
# https://www.robotstxt.org/robotstxt.html
User-agent: *
Disallow:
@epreston
Copy link
Author

not reliable but occasionally respected by search bots

should not be considered a security control

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment