Skip to content

Instantly share code, notes, and snippets.

@lukecav

lukecav/robots.txt

Created September 29, 2023 20:58
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save lukecav/9c4a25f8ecce488fad0c1def99d80c29 to your computer and use it in GitHub Desktop.
Save lukecav/9c4a25f8ecce488fad0c1def99d80c29 to your computer and use it in GitHub Desktop.
Download ZIP
Disallow Google Extended for site content being included in AI training (Bard and Vertex AI generative APIs)
Raw
robots.txt
User-agent: Google-Extended
Disallow: /
@lukecav
Copy link
Author

lukecav commented Sep 29, 2023

https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment