Skip to content

Instantly share code, notes, and snippets.

@kevinsearle

kevinsearle/gist:266642e40bbc3c46a6a019f036195c83

Last active May 31, 2024 11:33
Show Gist options
  • Save kevinsearle/266642e40bbc3c46a6a019f036195c83 to your computer and use it in GitHub Desktop.
Save kevinsearle/266642e40bbc3c46a6a019f036195c83 to your computer and use it in GitHub Desktop.
Download ZIP
AI scraper-blocking robots.txt
Raw
gistfile1.txt
User-agent: CCBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Omgilibot
Disallow: /
User-Agent: FacebookBot
Disallow: /
User-agent: Amazonbot
Disallow: /
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment