Skip to content

Instantly share code, notes, and snippets.

@cmdr-rohit-bang

cmdr-rohit-bang/robots.txt

Created December 26, 2020 06:20
Show Gist options
  • Save cmdr-rohit-bang/055fcc8a8a3656d0407a72abb4338455 to your computer and use it in GitHub Desktop.
Save cmdr-rohit-bang/055fcc8a8a3656d0407a72abb4338455 to your computer and use it in GitHub Desktop.
Download ZIP
Sample robots.txt file
Raw
robots.txt
# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
User-agent: *
Disallow: /
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment