Skip to content

Instantly share code, notes, and snippets.

@Gundars
Last active December 19, 2015 12:59
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Gundars/5959179 to your computer and use it in GitHub Desktop.
Save Gundars/5959179 to your computer and use it in GitHub Desktop.
Add to .htaccess file to block annoying crawlers and bots ignoring robots.txt files
# BEGIN ApacheBlockNastyBots
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ^(Aqua_Products|asterias|b2w/0\.1|BackDoorBot/1\.0|Black\ Hole|BlowFish/1\.0|Bookmark\ search\ tool|BotALot|BuiltBotTough|Bullseye/1\.0|BunnySlippers|Cegbfeieh|CheeseBot|CherryPicker|CherryPicker\ /1\.0|CherryPickerElite/1\.0|CherryPickerSE/1\.0|CopyRightCheck|cosmos|Crescent|Crescent\ Internet\ ToolPak\ HTTP\ OLE\ Control\ v\.1\.0|DittoSpyder|EmailCollector|EmailSiphon|EmailWolf|EroCrawler|ExtractorPro|FairAd\ Client|Flaming\ AttackBot|Foobot|Gaisbot|GetRight/4\.2|grub|grub-client|Harvest/1\.5|hloader|httplib|humanlinks|ia_archiver|ia_archiver/1\.6|InfoNaviRobot|Iron33/1\.0\.2|JennyBot|Kenjin\ Spider|Keyword\ Density/0\.9|larbin|LexiBot|libWeb/clsHTTP|LinkextractorPro|LinkScan/8\.1a\ Unix|LinkWalker|LNSpiderguy|lwp-trivial|lwp-trivial/1\.34|Mata\ Hari|Microsoft\ URL\ Control|Microsoft\ URL\ Control\ -\ 5\.01\.4511|Microsoft\ URL\ Control\ -\ 6\.00\.8169|MIIxpc|MIIxpc/4\.2|Mister\ PiX|moget|moget/2\.1|mozilla/4|mozilla/5|MSIECrawler|NetAnts|NetMechanic|NICErsPRO|Offline\ Explorer|Openbot|Openfind\ |Openfind\ data\ gathere|Oracle\ Ultra\ Search|PerMan|ProPowerBot/2\.14|ProWebWalker|psbot|Python-urllib|QueryN\ Metasearch|Radiation\ Retriever\ 1\.1|RepoMonkey|RepoMonkey\ Bait\ &\ Tackle/v1\.01|RMA|searchpreview|SiteSnagger|SpankBot|spanner|suzuran|Szukacz/1\.4\ |Teleport|TeleportPro|Telesoft|The\ Intraformant|TheNomad|TightTwatBot|Titan|toCrawl/UrlDispatcher|True_Robot|True_Robot/1\.0|turingos|URL\ Control|URL_Spider_Pro|URLy\ Warning|VCI|VCI\ WebViewer\ VCI\ WebViewer\ Win32|Web\ Image\ Collector|WebAuto|WebBandit|WebBandit/3\.50|WebCopier|WebEnhancer|WebmasterWorldForumBot|WebSauger|Website\ Quester|Webster\ Pro|WebStripper|WebZip|WebZip/4\.0|Wget|Wget/1\.5\.3|Wget/1\.6|WWW-Collector-E|Xenu's|Xenu's\ Link\ Sleuth\ 1\.1c|Zeus|Zeus\ 32297\ Webster\ Pro\ V2\.9\ Win32|Zeus\ Link\ Scout)$
RewriteRule .* - [F]
</IfModule>
# END ApacheBlockNastyBots
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment