Skip to content

Instantly share code, notes, and snippets.

View paslandau's full-sized avatar
💭
¯\_(ツ)_/¯

Pascal Landau paslandau

💭
¯\_(ツ)_/¯
View GitHub Profile
@paslandau
paslandau / redirect.sh
Created June 30, 2017 15:39
Bash script to check geo- and mobile redirects for aboutyou de/at/ch when using Googlebot user agents
y="at
ch
de"; \
x="
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.google.com/bot.html)
"; \
IFS="\n"; readarray -t countries <<<"$y";\
readarray -t userAgents <<<"$x"; IFS=""; \ for y in ${userAgents[@]}; do for x in ${countries[@]}; do url="https://www.aboutyou.${x}/"; printf "\n$url with $y\n"; curl -sIL -H "user-agent: $y" $url 2>&1 | egrep 'HTTP/1.1|Location|Failed'; done; done;
@paslandau
paslandau / result.txt
Created June 30, 2017 15:42
crawling result
https://www.aboutyou.at/ with Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
HTTP/1.1 302 Found
Location: https://m.aboutyou.at/
HTTP/1.1 200 OK
https://www.aboutyou.ch/ with Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
HTTP/1.1 302 Found
Location: https://m.aboutyou.ch/
HTTP/1.1 200 OK
@paslandau
paslandau / guzzle-append-requests-on-the-fly-generator-vs-retry.php
Created April 12, 2015 08:40
Example of appending requests on the fly using an ArrayIterator as generator and comparing that approach to $event->retry() and $client->send()
<?php
/** CAUTION!!!
* You need to have a webserver running on localhost to make this work!
*/
use GuzzleHttp\Client;
use GuzzleHttp\Event\AbstractRetryableEvent;
use GuzzleHttp\Message\Request;
use GuzzleHttp\Pool;