Skip to content

Instantly share code, notes, and snippets.

@jemerick

jemerick/robots.txt

Created July 27, 2010 17:00
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jemerick/492514 to your computer and use it in GitHub Desktop.
Save jemerick/492514 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
robots.txt
# from http://www.last.fm/robots.txt
User-Agent: *
Disallow: /music?
Disallow: /widgets/radio?
Disallow: /show_ads.php
Disallow: /affiliate/
Disallow: /affiliate_redirect.php
Disallow: /affiliate_sendto.php
Disallow: /affiliatelink.php
Disallow: /campaignlink.php
Disallow: /delivery.php
Disallow: /music/+noredirect/
Disallow: /harming/humans
Disallow: /ignoring/human/orders
Disallow: /harm/to/self
Allow: /
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment