Skip to content

Instantly share code, notes, and snippets.

@peterdalle

peterdalle/robots.txt

Created December 3, 2016 15:18
Show Gist options
  • Star 15 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save peterdalle/302303fb67c2bb73a9a09df78c59ba1d to your computer and use it in GitHub Desktop.
Save peterdalle/302303fb67c2bb73a9a09df78c59ba1d to your computer and use it in GitHub Desktop.
Download ZIP
Robots.txt that makes sure Facebook and Twitter can crawl images on your site.
Raw
robots.txt
# Disallow everything.
User-agent: *
Disallow: /
# Certain social media sites are whitelisted to allow crawlers to access page markup when links to /images are shared.
User-agent: Twitterbot
Allow: /images
User-agent: facebookexternalhit
Allow: /images
@MetaMmodern
Copy link

Thank you, it workes

@meagencycloud
Copy link

nice thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment