Skip to content

Instantly share code, notes, and snippets.

@dchapman1988

dchapman1988/gist:5673030

Forked from johndavid400/gist:5672719
Created May 29, 2013 19:22
Show Gist options
  • Save dchapman1988/5673030 to your computer and use it in GitHub Desktop.
Save dchapman1988/5673030 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
User-agent: Googlebot
User-agent: ArchitextSpider
User-Agent: msnbot
User-Agent: yahoo-slurp
User-Agent: teoma
User-Agent: bingbot
Disallow: /*?
Disallow: /browse
Disallow: /users/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment