Skip to content

Instantly share code, notes, and snippets.

@johnbocook

johnbocook/robots.txt Rules

Created January 10, 2023 22:33
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save johnbocook/8068c633d7ca61e2290e03665e543972 to your computer and use it in GitHub Desktop.
Save johnbocook/8068c633d7ca61e2290e03665e543972 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
robots.txt Rules
# Example 1: Block only Googlebot
User-agent: Googlebot
Disallow: /
# Example 2: Block Googlebot and Adsbot
User-agent: Googlebot
User-agent: AdsBot-Google
Disallow: /
# Example 3: Block all crawlers except AdsBot (AdsBot crawlers must be named explicitly)
User-agent: *
Disallow: /
# Sitemap Example: Full url location of sitemap
Sitemap: https://example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap.xml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment