Skip to content

Instantly share code, notes, and snippets.

@bijoy26

bijoy26/robots.txt

Last active May 20, 2021 09:32
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bijoy26/cf0a74800ef74b55b92e3a16c9212679 to your computer and use it in GitHub Desktop.
Save bijoy26/cf0a74800ef74b55b92e3a16c9212679 to your computer and use it in GitHub Desktop.
Download ZIP
A sample robots.txt file for example purpose
Raw
robots.txt
# Applicable for googlebot
User-agent: googlebot
# Disallow following directories
Disallow: /img
Disalow: /data/classified/
#Only a png image file is allowed
Allow: /img/favicon/android-chrome-512x512.png
# Applicable for all user agennts
User-agent:*
# Disallow following directories
Disallow: /img/cover/
@bijoy26
Copy link
Author

bijoy26 commented Dec 19, 2019 •
edited

A demo robots.txt file created for example purpose

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment