Skip to content

Instantly share code, notes, and snippets.

@paslandau

paslandau/result.txt

Created June 30, 2017 15:42
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save paslandau/d0d583975bb4dbb8f97a320ac8c5c878 to your computer and use it in GitHub Desktop.
Save paslandau/d0d583975bb4dbb8f97a320ac8c5c878 to your computer and use it in GitHub Desktop.
Download ZIP
crawling result
Raw
result.txt
https://www.aboutyou.at/ with Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
HTTP/1.1 302 Found
Location: https://m.aboutyou.at/
HTTP/1.1 200 OK
https://www.aboutyou.ch/ with Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
HTTP/1.1 302 Found
Location: https://m.aboutyou.ch/
HTTP/1.1 200 OK
https://www.aboutyou.de/ with Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
HTTP/1.1 302 Found
Location: https://m.aboutyou.de/
HTTP/1.1 200 OK
https://www.aboutyou.at/ with Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
HTTP/1.1 200 OK
https://www.aboutyou.ch/ with Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
HTTP/1.1 200 OK
https://www.aboutyou.de/ with Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
HTTP/1.1 200 OK
https://www.aboutyou.at/ with Googlebot/2.1 (+http://www.google.com/bot.html)
HTTP/1.1 200 OK
https://www.aboutyou.ch/ with Googlebot/2.1 (+http://www.google.com/bot.html)
HTTP/1.1 200 OK
https://www.aboutyou.de/ with Googlebot/2.1 (+http://www.google.com/bot.html)
HTTP/1.1 200 OK
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment