Skip to content

Instantly share code, notes, and snippets.

@double-p

double-p/web spiders

Created August 3, 2017 15:20
Show Gist options
  • Save double-p/597545116eb89ce26c3fd6d727939209 to your computer and use it in GitHub Desktop.
Save double-p/597545116eb89ce26c3fd6d727939209 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
web spiders
# acl is_bot hdr_sub(user-agent) -f /etc/haproxy/spider.lst
Googlebot/
Bingbot/
ysearch/slurp
DuckDuckBot/
Baiduspider/
YandexBot/
sogou.com/
Exabot/
facebookexternalhit/
ia_archiver
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment