Skip to content

Instantly share code, notes, and snippets.

@demyanovs
Last active October 20, 2023 20:21
Show Gist options
  • Save demyanovs/e5a1ac424e62ce01641bcc95afe0564c to your computer and use it in GitHub Desktop.
Save demyanovs/e5a1ac424e62ce01641bcc95afe0564c to your computer and use it in GitHub Desktop.
Block useless load from bots
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^Bingbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^bingbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Baiduspider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} linkdexbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} coccocbot-web [NC,OR]
RewriteCond %{HTTP_USER_AGENT} FemtosearchBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Owler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Pinterestbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} GrapeshotCrawler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} tracemyfile [NC,OR]
RewriteCond %{HTTP_USER_AGENT} CCBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} SafeDNSBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} HybridBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} GetIntent [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Feedly [NC,OR]
RewriteCond %{HTTP_USER_AGENT} FeedBurner [NC,OR]
RewriteCond %{HTTP_USER_AGENT} dotbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} BoardReader [NC,OR]
RewriteCond %{HTTP_USER_AGENT} theoldreader.com [NC,OR]
RewriteCond %{HTTP_USER_AGENT} semantic-visions.com [NC,OR]
RewriteCond %{HTTP_USER_AGENT} proximic [NC,OR]
RewriteCond %{HTTP_USER_AGENT} weborama-fetcher [NC,OR]
RewriteCond %{HTTP_USER_AGENT} trendictionbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} rogerbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} Applebot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ias_crawler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} grapeshot [NC]
RewriteRule ^.*$ - [F,L]
# http://www.bing.com/bingbot.htm
User-agent: bingbot
Disallow: /
# http://www.trendiction.com/en/publisher/bot
User-agent: trendictionbot
Disallow: /
# http://www.apple.com/go/applebot
User-agent: Applebot
Disallow: /
# http://www.grapeshot.co.uk/crawler.php
User-agent: grapeshot
Disallow: /
# http://www.semrush.com/bot.html
User-agent: SemrushBot
Disallow: /
# http://www.semrush.com/bot.html
User-agent: SemrushBot-SA
Disallow: /
# http://mj12bot.com
User-agent: MJ12bot
Disallow: /
# http://www.opensiteexplorer.org/dotbot
User-agent: dotbot
Disallow: /
# http://www.netseer.com/crawler.html
User-agent: netseer
Disallow: /
# http://www.pinterest.com/bot.html
User-agent: Pinterestbot
Disallow: /
# http://getintent.com/bot.html
User-agent: GetIntent Crawler
Disallow: /
User-agent: bidswitchbot
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: linkdexbot
Disallow: /
User-agent: coccocbot-web
Disallow: /
User-agent: FemtosearchBot
Disallow: /
User-agent: Owler
Disallow: /
User-agent: tracemyfile
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: SafeDNSBot
Disallow: /
User-agent: HybridBot
Disallow: /
User-agent: Feedly
Disallow: /
User-agent: FeedBurner
Disallow: /
User-agent: BoardReader
Disallow: /
User-agent: theoldreader.com
Disallow: /
User-agent: semantic-visions.com
Disallow: /
User-agent: proximic
Disallow: /
User-agent: weborama-fetcher
Disallow: /
User-agent: rogerbot
Disallow: /
User-agent: ias_crawler
Disallow: /
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment