Skip to content

Instantly share code, notes, and snippets.

What would you like to do?
Robots.txt that makes sure Facebook and Twitter can crawl images on your site.
# Disallow everything.
User-agent: *
Disallow: /
# Certain social media sites are whitelisted to allow crawlers to access page markup when links to /images are shared.
User-agent: Twitterbot
Allow: /images
User-agent: facebookexternalhit
Allow: /images
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment