Robots.txt that makes sure Facebook and Twitter can crawl images on your site.
# Disallow everything. | |
User-agent: * | |
Disallow: / | |
# Certain social media sites are whitelisted to allow crawlers to access page markup when links to /images are shared. | |
User-agent: Twitterbot | |
Allow: /images | |
User-agent: facebookexternalhit | |
Allow: /images |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment