Skip to content

Instantly share code, notes, and snippets.

@chrisle

chrisle/gist:2252209

Created March 30, 2012 15:15
Show Gist options
  • Star 65 You must be signed in to star a gist
  • Fork 12 You must be signed in to fork a gist
  • Save chrisle/2252209 to your computer and use it in GitHub Desktop.
Save chrisle/2252209 to your computer and use it in GitHub Desktop.
Download ZIP
CURL as GoogleBot 2.1
Raw
gistfile1.txt
curl --user-agent "Googlebot/2.1 (+http://www.google.com/bot.html)" -v $@
@Blumed
Copy link

Blumed commented Dec 10, 2020

This is using desktop and Google is moving toward mobile first. May want to update this. Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z‡ Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers?visit_id=637432331662125416-1181969658&rd=1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment