- Modify /etc/nginx/nginx.conf file
- Modify /etc/nginx/sites-available/site.conf file
- Create /etc/nginx/useragent.rule file
Where to find user agent strings?
https://explore.whatismybrowser.com/useragents/explore/software_name/facebook-bot/
Looking for same but for Apache2? Here: https://techexpert.tips/apache/apache-blocking-bad-bots-crawlers/
Test:
[rubin@reaper ~]$ curl -A "instagram" -I https://plrm.podcastalot.com
HTTP/2 418
server: nginx/1.18.0
date: Mon, 26 Jun 2023 06:07:25 GMT
content-type: text/html
content-length: 197
If there is the risk that a bot does not obey robots.txt, why would there be an assumption that they will obey the user agent rules and not just randomly alter legitimate browser user agents?
If you're serious about blocking access to your content I would: