Skip to content

Instantly share code, notes, and snippets.

@ticidesign

ticidesign/robots.txt

Created March 16, 2018 01:02
Show Gist options
  • Save ticidesign/ec07a1f5bf13c78621d48f7eba9217f9 to your computer and use it in GitHub Desktop.
Save ticidesign/ec07a1f5bf13c78621d48f7eba9217f9 to your computer and use it in GitHub Desktop.
Download ZIP
Robots.txt that makes sure Facebook and Twitter can crawl images on your site.
Raw
robots.txt
# Disallow everything.
User-agent: *
Disallow: /
# Certain social media sites are whitelisted to allow crawlers to access page markup when links to /images are shared.
User-agent: Twitterbot
Allow: /images
User-agent: facebookexternalhit
Allow: /images
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment