Skip to content

Instantly share code, notes, and snippets.

@yaroslav

yaroslav/nginx-limit_rate_for_crawlers.conf

Created February 15, 2009 12:08
Show Gist options
  • Save yaroslav/64690 to your computer and use it in GitHub Desktop.
Save yaroslav/64690 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
nginx-limit_rate_for_crawlers.conf
# by http://ptitov.ru/
if ($http_user_agent ~ "libwww|BlogSearch|picsearch|WebAlta|Gigabot|ping\.blo\.gs|Aport|Feedster|Blo
gPeople|Accelatech|RSSCrawler|blogoonbot|ia_archiver|ichiro|RSS_READER|RSS Crawler|blogfan|lwp|Java\
/|MaplogCrawler|sproose|Technoratibot|Moreoverbot|Jakarta|wadaino\.jp|MJ12bot|Sensis|msnbot|DepSpid|
Twiceler|Mail\.Ru|Slurp|Yandex/1\.01"){
set $limit_rate 300;
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment