Skip to content

Instantly share code, notes, and snippets.

@artikus11

artikus11/robots.txt

Created April 2, 2017 19:43
Show Gist options
  • Save artikus11/640e33653266fa20584a767f7b015640 to your computer and use it in GitHub Desktop.
Save artikus11/640e33653266fa20584a767f7b015640 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
robots.txt
User-agent: Yandex
Crawl-delay: 10 #
User-agent: *
Disallow: /search Crawl-delay: 10 #
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment