Skip to content

Instantly share code, notes, and snippets.

@wpsmith

wpsmith/robots.txt

Created August 18, 2020 21:05
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save wpsmith/c440ef4fa1667924967ad6ba83eec94c to your computer and use it in GitHub Desktop.
Save wpsmith/c440ef4fa1667924967ad6ba83eec94c to your computer and use it in GitHub Desktop.
Download ZIP
Raw
robots.txt
User-agent: mj12bot
crawl-delay: 10
User-agent: ahrefs
crawl-delay: 10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment