Skip to content

Instantly share code, notes, and snippets.

@arosenhagen
Forked from supairish/gist:2951524
Last active February 16, 2021 11:34
Show Gist options
  • Save arosenhagen/8aaf5d7f94171778c0e9 to your computer and use it in GitHub Desktop.
Save arosenhagen/8aaf5d7f94171778c0e9 to your computer and use it in GitHub Desktop.
[nginx] limit requests from searchengine crawlers/bots to 1r/m (prevent DDOS)
http {
map $http_user_agent $limit_bots {
default '';
~*(bing|yandex|msnbot) $binary_remote_addr;
}
limit_req_zone $limit_bots zone=bots:10m rate=1r/m;
server {
location / {
limit_req zone=bots burst=5 nodelay;
}
}
}
@alanorth
Copy link

Clever use of mapping plus the $binary_remote_addr variable to apply the limit to matching user agents, but let everyone else through.

@hrvoj3e
Copy link

hrvoj3e commented Feb 16, 2021

What would happen if I used a fixed string instead of $binary_remote_addr?

My understanding is that I would put all "bots" into one key/bucket and disallow them all.
Each bot could have many IP adresses and could rotate them so $binary_remote_addr will not with that.

map $http_user_agent $limit_bots {
    default '';
    ~UptimeRobot ''; ## allow
    ~*\(.*bot.*\) 'mybotmarker';
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment