Robots.txt that makes sure Facebook and Twitter can crawl images on your site.
| # Disallow everything. | |
| User-agent: * | |
| Disallow: / | |
| # Certain social media sites are whitelisted to allow crawlers to access page markup when links to /images are shared. | |
| User-agent: Twitterbot | |
| Allow: /images | |
| User-agent: facebookexternalhit | |
| Allow: /images |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment