Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
[nginx] limit requests from searchengine crawlers/bots to 1r/m (prevent DDOS)
http {
map $http_user_agent $limit_bots {
default '';
~*(bing|yandex|msnbot) $binary_remote_addr;
}
limit_req_zone $limit_bots zone=bots:10m rate=1r/m;
server {
location / {
limit_req zone=bots burst=5 nodelay;
}
}
}
@alanorth

This comment has been minimized.

Copy link

@alanorth alanorth commented Nov 12, 2017

Clever use of mapping plus the $binary_remote_addr variable to apply the limit to matching user agents, but let everyone else through.

@hrvoj3e

This comment has been minimized.

Copy link

@hrvoj3e hrvoj3e commented Feb 16, 2021

What would happen if I used a fixed string instead of $binary_remote_addr?

My understanding is that I would put all "bots" into one key/bucket and disallow them all.
Each bot could have many IP adresses and could rotate them so $binary_remote_addr will not with that.

map $http_user_agent $limit_bots {
    default '';
    ~UptimeRobot ''; ## allow
    ~*\(.*bot.*\) 'mybotmarker';
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment