Skip to content

Instantly share code, notes, and snippets.

@leveled

leveled/curl and spoof google bot.sh

Created October 19, 2016 00:44
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save leveled/a09d7c663282c31fc824ebef2602daf5 to your computer and use it in GitHub Desktop.
Save leveled/a09d7c663282c31fc824ebef2602daf5 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
curl and spoof google bot.sh
curl -A "'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)')" 10.11.1.39/robots.txt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment