Skip to content

Instantly share code, notes, and snippets.

@wesort

wesort/live.robots.txt

Created December 12, 2019 15:35
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save wesort/82c2262fbdcdd44b14f8da5d8ec39197 to your computer and use it in GitHub Desktop.
Save wesort/82c2262fbdcdd44b14f8da5d8ec39197 to your computer and use it in GitHub Desktop.
Download ZIP
A sample robots.txt file for a production site
Raw
live.robots.txt
# robotstxt.org/
User-agent: *
Disallow:
# Update the domain below
Host: https://example.co.uk
Sitemap: https://example.co.uk/sitemap
@wesort
Copy link
Author

wesort commented Dec 12, 2019

Rename the file robots.txt manually or with Netlify's build command

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment