Skip to content

Instantly share code, notes, and snippets.

@dvlop

dvlop/gist:fca36213ad6237891609e1e038a3bbc1

Forked from allthingsdem/gist:63b3223a7d14ac1f2457
Last active June 27, 2024 12:11
Show Gist options
  • Save dvlop/fca36213ad6237891609e1e038a3bbc1 to your computer and use it in GitHub Desktop.
Save dvlop/fca36213ad6237891609e1e038a3bbc1 to your computer and use it in GitHub Desktop.
Download ZIP
My long list of bad bots to block in htaccess, ready to copy and paste!
Raw
gistfile1.txt
# Start Bad Bot Prevention
<IfModule mod_setenvif.c>
# SetEnvIfNoCase User-Agent ^$ bad_bot
SetEnvIfNoCase User-Agent "^12soso.*" bad_bot
SetEnvIfNoCase User-Agent "^192.comAgent.*" bad_bot
SetEnvIfNoCase User-Agent "^1Noonbot.*" bad_bot
SetEnvIfNoCase User-Agent "^1on1searchBot.*" bad_bot
SetEnvIfNoCase User-Agent "^3D_SEARCH.*" bad_bot
SetEnvIfNoCase User-Agent "^3DE_SEARCH2.*" bad_bot
SetEnvIfNoCase User-Agent "^3GSE.*" bad_bot
SetEnvIfNoCase User-Agent "^50.nu.*" bad_bot
SetEnvIfNoCase User-Agent "^A1 Sitemap Generator.*" bad_bot
SetEnvIfNoCase User-Agent "^A1 Website Download.*" bad_bot
SetEnvIfNoCase User-Agent "^A6-Indexer.*" bad_bot
SetEnvIfNoCase User-Agent "^AASP.*" bad_bot
SetEnvIfNoCase User-Agent "^ABACHOBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Abonti.*" bad_bot
SetEnvIfNoCase User-Agent "^abot.*" bad_bot
SetEnvIfNoCase User-Agent "^AbotEmailSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^AboutUsBot.*" bad_bot
SetEnvIfNoCase User-Agent "^AccMonitor Compliance Server.*" bad_bot
SetEnvIfNoCase User-Agent "^accoona.*" bad_bot
SetEnvIfNoCase User-Agent "^AChulkov.NET page walker.*" bad_bot
SetEnvIfNoCase User-Agent "^Acme.Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^AcoonBot.*" bad_bot
SetEnvIfNoCase User-Agent "^acquia-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^ActiveTouristBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Ad Muncher.*" bad_bot
SetEnvIfNoCase User-Agent "^AdamM Bot,.*" bad_bot
SetEnvIfNoCase User-Agent "^adbeat_bot.*" bad_bot
SetEnvIfNoCase User-Agent "^AdMedia bot.*" bad_bot
SetEnvIfNoCase User-Agent "^adminshop.com.*" bad_bot
SetEnvIfNoCase User-Agent "^Advanced Email Extractor.*" bad_bot
SetEnvIfNoCase User-Agent "^AESOP_com_SpiderMan.*" bad_bot
SetEnvIfNoCase User-Agent "^AESpider.*" bad_bot
SetEnvIfNoCase User-Agent "^AF Knowledge Now Verity Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^aggregator:Vocus.*" bad_bot
SetEnvIfNoCase User-Agent "^ah-ha.com crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^AhrefsBot.*" bad_bot
SetEnvIfNoCase User-Agent "^AhrefsBot/5.0.*" bad_bot
SetEnvIfNoCase User-Agent "^AIBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^aiHitBot.*" bad_bot
SetEnvIfNoCase User-Agent "^aipbot.*" bad_bot
SetEnvIfNoCase User-Agent "^AISIID.*" bad_bot
SetEnvIfNoCase User-Agent "^AITCSRobot/1.1.*" bad_bot
SetEnvIfNoCase User-Agent "^Akamai-SiteSnapshot.*" bad_bot
SetEnvIfNoCase User-Agent "^AlexaWebSearchPlatform.*" bad_bot
SetEnvIfNoCase User-Agent "^AlexfDownload.*" bad_bot
SetEnvIfNoCase User-Agent "^Alexibot.*" bad_bot
SetEnvIfNoCase User-Agent "^AlkalineBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^All Acronyms Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Alpha Search Agent.*" bad_bot
SetEnvIfNoCase User-Agent "^Amerla Search Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Amfibibot.*" bad_bot
SetEnvIfNoCase User-Agent "^AmPmPPC.com.*" bad_bot
SetEnvIfNoCase User-Agent "^AMZNKAssocBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Anemone.*" bad_bot
SetEnvIfNoCase User-Agent "^Anonymous/3G bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Anonymouse.org.*" bad_bot
SetEnvIfNoCase User-Agent "^AnotherBot.*" bad_bot
SetEnvIfNoCase User-Agent "^AnswerBot.*" bad_bot
SetEnvIfNoCase User-Agent "^AnswerBus.*" bad_bot
SetEnvIfNoCase User-Agent "^AnswerChase PROve.*" bad_bot
SetEnvIfNoCase User-Agent "^AntBot.*" bad_bot
SetEnvIfNoCase User-Agent "^antibot-.*" bad_bot
SetEnvIfNoCase User-Agent "^AntiSantyWorm.*" bad_bot
SetEnvIfNoCase User-Agent "^Antro.Net.*" bad_bot
SetEnvIfNoCase User-Agent "^AONDE-Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Apache-HttpClient.*" bad_bot
SetEnvIfNoCase User-Agent "^Apache-HttpClient/4.0.1 (java 1.5).*" bad_bot
SetEnvIfNoCase User-Agent "^ApacheBench.*" bad_bot
SetEnvIfNoCase User-Agent "^Aport.*" bad_bot
SetEnvIfNoCase User-Agent "^appid: s~stremor-crawler-.*" bad_bot
SetEnvIfNoCase User-Agent "^Aqua_Products.*" bad_bot
SetEnvIfNoCase User-Agent "^AraBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Arachmo.*" bad_bot
SetEnvIfNoCase User-Agent "^Arachnophilia.*" bad_bot
SetEnvIfNoCase User-Agent "^archive.org_bot.*" bad_bot
SetEnvIfNoCase User-Agent "^aria eQualizer.*" bad_bot
SetEnvIfNoCase User-Agent "^arianna.libero.it.*" bad_bot
SetEnvIfNoCase User-Agent "^Arikus_Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Art-Online.com.*" bad_bot
SetEnvIfNoCase User-Agent "^ArtavisBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Artera.*" bad_bot
SetEnvIfNoCase User-Agent "^ASAHA Search Engine Turkey.*" bad_bot
SetEnvIfNoCase User-Agent "^ASpider/0.09.*" bad_bot
SetEnvIfNoCase User-Agent "^ASPSeek.*" bad_bot
SetEnvIfNoCase User-Agent "^ASPseek.*" bad_bot
SetEnvIfNoCase User-Agent "^asterias.*" bad_bot
SetEnvIfNoCase User-Agent "^AstroFind.*" bad_bot
SetEnvIfNoCase User-Agent "^athenusbot.*" bad_bot
SetEnvIfNoCase User-Agent "^AtlocalBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Atomic_Email_Hunter.*" bad_bot
SetEnvIfNoCase User-Agent "^attach.*" bad_bot
SetEnvIfNoCase User-Agent "^attrakt.*" bad_bot
SetEnvIfNoCase User-Agent "^Attributor.comBot.*" bad_bot
SetEnvIfNoCase User-Agent "^attributor.*" bad_bot
SetEnvIfNoCase User-Agent "^Attributor/Dejan-.*" bad_bot
SetEnvIfNoCase User-Agent "^augurfind.*" bad_bot
SetEnvIfNoCase User-Agent "^AURESYS.*" bad_bot
SetEnvIfNoCase User-Agent "^AutoBaron crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^autoemailspider.*" bad_bot
SetEnvIfNoCase User-Agent "^autowebdir.*" bad_bot
SetEnvIfNoCase User-Agent "^AVSearch-.*" bad_bot
SetEnvIfNoCase User-Agent "^axfeedsbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Axonize-bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Ayna.*" bad_bot
SetEnvIfNoCase User-Agent "^b2w.*" bad_bot
SetEnvIfNoCase User-Agent "^b2w/0.1.*" bad_bot
SetEnvIfNoCase User-Agent "^BackDoorBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BackRub/..*" bad_bot
SetEnvIfNoCase User-Agent "^BackStreet Browser.*" bad_bot
SetEnvIfNoCase User-Agent "^BackWeb.*" bad_bot
SetEnvIfNoCase User-Agent "^Baiduspider.*" bad_bot
SetEnvIfNoCase User-Agent "^BaiDuSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^BaiduSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Baiduspider-video.*" bad_bot
SetEnvIfNoCase User-Agent "^Baiduspider+.*" bad_bot
SetEnvIfNoCase User-Agent "^Bandit.*" bad_bot
SetEnvIfNoCase User-Agent "^BatchFTP.*" bad_bot
SetEnvIfNoCase User-Agent "^baypup.*" bad_bot
SetEnvIfNoCase User-Agent "^BDFetch.*" bad_bot
SetEnvIfNoCase User-Agent "^BecomeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BecomeJPBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BeetleBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Bender.*" bad_bot
SetEnvIfNoCase User-Agent "^besserscheitern-crawl.*" bad_bot
SetEnvIfNoCase User-Agent "^betaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Big Brother.*" bad_bot
SetEnvIfNoCase User-Agent "^Big Data.*" bad_bot
SetEnvIfNoCase User-Agent "^Bigado.com.*" bad_bot
SetEnvIfNoCase User-Agent "^BigCliqueBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BigCliqueBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^Bigfoot.*" bad_bot
SetEnvIfNoCase User-Agent "^BIGLOTRON.*" bad_bot
SetEnvIfNoCase User-Agent "^Bilbo.*" bad_bot
SetEnvIfNoCase User-Agent "^BilgiBetaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BilgiBot.*" bad_bot
SetEnvIfNoCase User-Agent "^bintellibot.*" bad_bot
SetEnvIfNoCase User-Agent "^bitlybot.*" bad_bot
SetEnvIfNoCase User-Agent "^BitvoUserAgent.*" bad_bot
SetEnvIfNoCase User-Agent "^Bizbot003.*" bad_bot
SetEnvIfNoCase User-Agent "^BizBot04 kirk.overleaf.com.*" bad_bot
SetEnvIfNoCase User-Agent "^BizWorks Retriever.*" bad_bot
SetEnvIfNoCase User-Agent "^Black Hole.*" bad_bot
SetEnvIfNoCase User-Agent "^Black.Hole.*" bad_bot
SetEnvIfNoCase User-Agent "^Blackbird.*" bad_bot
SetEnvIfNoCase User-Agent "^BlackMask.Net Search Engine.*" bad_bot
SetEnvIfNoCase User-Agent "^BlackWidow.*" bad_bot
SetEnvIfNoCase User-Agent "^bladder fusion.*" bad_bot
SetEnvIfNoCase User-Agent "^Blaiz-Bee.*" bad_bot
SetEnvIfNoCase User-Agent "^BLEXBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Blinkx/DFS-Fetch.*" bad_bot
SetEnvIfNoCase User-Agent "^BlitzBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^Blog Conversation Project.*" bad_bot
SetEnvIfNoCase User-Agent "^BlogMyWay.*" bad_bot
SetEnvIfNoCase User-Agent "^BlogPulseLive.*" bad_bot
SetEnvIfNoCase User-Agent "^BlogRefsBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BlogScope.*" bad_bot
SetEnvIfNoCase User-Agent "^Blogslive.*" bad_bot
SetEnvIfNoCase User-Agent "^BloobyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BlowFish.*" bad_bot
SetEnvIfNoCase User-Agent "^BLT.*" bad_bot
SetEnvIfNoCase User-Agent "^bnf.fr_bot.*" bad_bot
SetEnvIfNoCase User-Agent "^BoaConstrictor.*" bad_bot
SetEnvIfNoCase User-Agent "^BoardReader-Image-Fetcher.*" bad_bot
SetEnvIfNoCase User-Agent "^BOI_crawl_00.*" bad_bot
SetEnvIfNoCase User-Agent "^BOIA-Scan-Agent.*" bad_bot
SetEnvIfNoCase User-Agent "^BOIA.ORG-Scan-Agent.*" bad_bot
SetEnvIfNoCase User-Agent "^boitho.com-dc.*" bad_bot
SetEnvIfNoCase User-Agent "^Bookmark Buddy bookmark checker.*" bad_bot
SetEnvIfNoCase User-Agent "^Bookmark search tool.*" bad_bot
SetEnvIfNoCase User-Agent "^bosug.*" bad_bot
SetEnvIfNoCase User-Agent "^Bot Apoena.*" bad_bot
SetEnvIfNoCase User-Agent "^Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^bot.*" bad_bot
SetEnvIfNoCase User-Agent "^BOT.*" bad_bot
SetEnvIfNoCase User-Agent "^BotALot.*" bad_bot
SetEnvIfNoCase User-Agent "^BotRightHere.*" bad_bot
SetEnvIfNoCase User-Agent "^Botswana.*" bad_bot
SetEnvIfNoCase User-Agent "^bottybot.*" bad_bot
SetEnvIfNoCase User-Agent "^BpBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BRAINTIME_SEARCH.*" bad_bot
SetEnvIfNoCase User-Agent "^BrokenLinkCheck.com.*" bad_bot
SetEnvIfNoCase User-Agent "^BrowserEmulator.*" bad_bot
SetEnvIfNoCase User-Agent "^BrowserMob.*" bad_bot
SetEnvIfNoCase User-Agent "^BruinBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BSearchR&D.*" bad_bot
SetEnvIfNoCase User-Agent "^BSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^btbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Btsearch.*" bad_bot
SetEnvIfNoCase User-Agent "^Buddy.*" bad_bot
SetEnvIfNoCase User-Agent "^Buibui.*" bad_bot
SetEnvIfNoCase User-Agent "^BuildCMS crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^BuiltBotTough.*" bad_bot
SetEnvIfNoCase User-Agent "^Bullseye.*" bad_bot
SetEnvIfNoCase User-Agent "^bumblebee.*" bad_bot
SetEnvIfNoCase User-Agent "^BunnySlippers.*" bad_bot
SetEnvIfNoCase User-Agent "^BuscadorClarin.*" bad_bot
SetEnvIfNoCase User-Agent "^Buscaplus Robi.*" bad_bot
SetEnvIfNoCase User-Agent "^Butterfly.*" bad_bot
SetEnvIfNoCase User-Agent "^BuyHawaiiBot.*" bad_bot
SetEnvIfNoCase User-Agent "^BuzzBot.*" bad_bot
SetEnvIfNoCase User-Agent "^byindia.*" bad_bot
SetEnvIfNoCase User-Agent "^BYINDIA.*" bad_bot
SetEnvIfNoCase User-Agent "^BySpider.*" bad_bot
SetEnvIfNoCase User-Agent "^byteserver.*" bad_bot
SetEnvIfNoCase User-Agent "^bzBot.*" bad_bot
SetEnvIfNoCase User-Agent "^c r a w l 3 r.*" bad_bot
SetEnvIfNoCase User-Agent "^CacheBlaster.*" bad_bot
SetEnvIfNoCase User-Agent "^Caddbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Cafi.*" bad_bot
SetEnvIfNoCase User-Agent "^Camcrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^CamelStampede.*" bad_bot
SetEnvIfNoCase User-Agent "^Canon-WebRecord.*" bad_bot
SetEnvIfNoCase User-Agent "^Canon-WebRecordPro.*" bad_bot
SetEnvIfNoCase User-Agent "^CareerBot.*" bad_bot
SetEnvIfNoCase User-Agent "^cataguru.*" bad_bot
SetEnvIfNoCase User-Agent "^CatchBot.*" bad_bot
SetEnvIfNoCase User-Agent "^CazoodleBot.*" bad_bot
SetEnvIfNoCase User-Agent "^CCBot.*" bad_bot
SetEnvIfNoCase User-Agent "^CCGCrawl.*" bad_bot
SetEnvIfNoCase User-Agent "^ccubee.*" bad_bot
SetEnvIfNoCase User-Agent "^CD-Preload.*" bad_bot
SetEnvIfNoCase User-Agent "^CE-Preload.*" bad_bot
SetEnvIfNoCase User-Agent "^Cegbfeieh.*" bad_bot
SetEnvIfNoCase User-Agent "^Cerberian Drtrs.*" bad_bot
SetEnvIfNoCase User-Agent "^CERT FigleafBot.*" bad_bot
SetEnvIfNoCase User-Agent "^cfetch.*" bad_bot
SetEnvIfNoCase User-Agent "^CFNetwork.*" bad_bot
SetEnvIfNoCase User-Agent "^Chameleon.*" bad_bot
SetEnvIfNoCase User-Agent "^Charlotte.*" bad_bot
SetEnvIfNoCase User-Agent "^Check&Get.*" bad_bot
SetEnvIfNoCase User-Agent "^Checkbot.*" bad_bot
SetEnvIfNoCase User-Agent "^CheckLinks.*" bad_bot
SetEnvIfNoCase User-Agent "^Checklinks.*" bad_bot
SetEnvIfNoCase User-Agent "^CheeseBot.*" bad_bot
SetEnvIfNoCase User-Agent "^ChemieDE-NodeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^CherryPicker.*" bad_bot
SetEnvIfNoCase User-Agent "^CherryPickerElite.*" bad_bot
SetEnvIfNoCase User-Agent "^CherryPickerSE.*" bad_bot
SetEnvIfNoCase User-Agent "^Chilkat.*" bad_bot
SetEnvIfNoCase User-Agent "^ChinaClaw.*" bad_bot
SetEnvIfNoCase User-Agent "^CipinetBot.*" bad_bot
SetEnvIfNoCase User-Agent "^cis455crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^citeseerxbot.*" bad_bot
SetEnvIfNoCase User-Agent "^cizilla.com/Cizilla-.*" bad_bot
SetEnvIfNoCase User-Agent "^ClariaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Climate Ark.*" bad_bot
SetEnvIfNoCase User-Agent "^ClimateArk Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Clushbot.*" bad_bot
SetEnvIfNoCase User-Agent "^COAST scan engine.*" bad_bot
SetEnvIfNoCase User-Agent "^COAST WebMaster Pro.*" bad_bot
SetEnvIfNoCase User-Agent "^coccoc.*" bad_bot
SetEnvIfNoCase User-Agent "^CollapsarWEB.*" bad_bot
SetEnvIfNoCase User-Agent "^Collector.*" bad_bot
SetEnvIfNoCase User-Agent "^Combine.*" bad_bot
SetEnvIfNoCase User-Agent "^combine.*" bad_bot
SetEnvIfNoCase User-Agent "^COMBINE.*" bad_bot
SetEnvIfNoCase User-Agent "^Compatible.*" bad_bot
SetEnvIfNoCase User-Agent "^ConnectSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^conpilot.*" bad_bot
SetEnvIfNoCase User-Agent "^ContentSmartz.*" bad_bot
SetEnvIfNoCase User-Agent "^ContextAd Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^contype.*" bad_bot
SetEnvIfNoCase User-Agent "^cookieNET.*" bad_bot
SetEnvIfNoCase User-Agent "^CoolBot .*" bad_bot
SetEnvIfNoCase User-Agent "^CoolCheck.*" bad_bot
SetEnvIfNoCase User-Agent "^Copernic.*" bad_bot
SetEnvIfNoCase User-Agent "^Copier.*" bad_bot
SetEnvIfNoCase User-Agent "^CopyRightCheck.*" bad_bot
SetEnvIfNoCase User-Agent "^core-project.*" bad_bot
SetEnvIfNoCase User-Agent "^cosmos.*" bad_bot
SetEnvIfNoCase User-Agent "^Covario-IDS.*" bad_bot
SetEnvIfNoCase User-Agent "^Cowbot-.*" bad_bot
SetEnvIfNoCase User-Agent "^Cowdog Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^crabbyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Crawl_Application.*" bad_bot
SetEnvIfNoCase User-Agent "^crawl.UserAgent.*" bad_bot
SetEnvIfNoCase User-Agent "^Crawl.*" bad_bot
SetEnvIfNoCase User-Agent "^crawl.*" bad_bot
SetEnvIfNoCase User-Agent "^CrawlConvera.*" bad_bot
SetEnvIfNoCase User-Agent "^Crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^crawler_for_infomine.*" bad_bot
SetEnvIfNoCase User-Agent "^CRAWLER-ALTSE.VUNET.ORG-Lynx.*" bad_bot
SetEnvIfNoCase User-Agent "^crawler-upgrade-config.*" bad_bot
SetEnvIfNoCase User-Agent "^crawler.kpricorn.org.*" bad_bot
SetEnvIfNoCase User-Agent "^crawler@.*" bad_bot
SetEnvIfNoCase User-Agent "^crawler43.ejupiter.com.*" bad_bot
SetEnvIfNoCase User-Agent "^crawler4j.*" bad_bot
SetEnvIfNoCase User-Agent "^Crawly.*" bad_bot
SetEnvIfNoCase User-Agent "^CreativeCommons.*" bad_bot
SetEnvIfNoCase User-Agent "^Crescent.*" bad_bot
SetEnvIfNoCase User-Agent "^Crescent Internet ToolPak HTTP OLE Control v.1.0.*" bad_bot
SetEnvIfNoCase User-Agent "^cs-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^CSE HTML Validator.*" bad_bot
SetEnvIfNoCase User-Agent "^CSHttpClient.*" bad_bot
SetEnvIfNoCase User-Agent "^Cuasarbot.*" bad_bot
SetEnvIfNoCase User-Agent "^culsearch/culs.*" bad_bot
SetEnvIfNoCase User-Agent "^Curl.*" bad_bot
SetEnvIfNoCase User-Agent "^Custo.*" bad_bot
SetEnvIfNoCase User-Agent "^cvaulev.*" bad_bot
SetEnvIfNoCase User-Agent "^Cyberdog.*" bad_bot
SetEnvIfNoCase User-Agent "^CyberNavi_WebGet.*" bad_bot
SetEnvIfNoCase User-Agent "^CyberPatrol SiteCat Webbot.*" bad_bot
SetEnvIfNoCase User-Agent "^CyberSpyder.*" bad_bot
SetEnvIfNoCase User-Agent "^CydralSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^D1GArabicEngine.*" bad_bot
SetEnvIfNoCase User-Agent "^DA.*" bad_bot
SetEnvIfNoCase User-Agent "^DataCha0s.*" bad_bot
SetEnvIfNoCase User-Agent "^DataFountains.*" bad_bot
SetEnvIfNoCase User-Agent "^DataFountains/DMOZ Downloader.*" bad_bot
SetEnvIfNoCase User-Agent "^DataparkSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^datascape robot.*" bad_bot
SetEnvIfNoCase User-Agent "^DataSpearSpiderBot.*" bad_bot
SetEnvIfNoCase User-Agent "^DataSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Dattatec.com.*" bad_bot
SetEnvIfNoCase User-Agent "^Dattatec.com-Sitios-Top.*" bad_bot
SetEnvIfNoCase User-Agent "^Daumoa.*" bad_bot
SetEnvIfNoCase User-Agent "^DAUMOA-video.*" bad_bot
SetEnvIfNoCase User-Agent "^DAUMOA-web.*" bad_bot
SetEnvIfNoCase User-Agent "^Declumbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Deepindex.*" bad_bot
SetEnvIfNoCase User-Agent "^deepnet crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^DeepTrawl.*" bad_bot
SetEnvIfNoCase User-Agent "^dejan.*" bad_bot
SetEnvIfNoCase User-Agent "^del.icio.us-thumbnails.*" bad_bot
SetEnvIfNoCase User-Agent "^DelvuBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Der große BilderSauger.*" bad_bot
SetEnvIfNoCase User-Agent "^DiaGem.*" bad_bot
SetEnvIfNoCase User-Agent "^Diamond.*" bad_bot
SetEnvIfNoCase User-Agent "^DiamondBot.*" bad_bot
SetEnvIfNoCase User-Agent "^DiBot.*" bad_bot
SetEnvIfNoCase User-Agent "^didaxusbot.*" bad_bot
SetEnvIfNoCase User-Agent "^DigExt.*" bad_bot
SetEnvIfNoCase User-Agent "^Digger.*" bad_bot
SetEnvIfNoCase User-Agent "^DiGi-RSSBot.*" bad_bot
SetEnvIfNoCase User-Agent "^DigitalArchivesBot.*" bad_bot
SetEnvIfNoCase User-Agent "^DigOut4U.*" bad_bot
SetEnvIfNoCase User-Agent "^DIIbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Dillo.*" bad_bot
SetEnvIfNoCase User-Agent "^Dir_Snatch.exe.*" bad_bot
SetEnvIfNoCase User-Agent "^DISCo.*" bad_bot
SetEnvIfNoCase User-Agent "^DISCo Pump.*" bad_bot
SetEnvIfNoCase User-Agent "^DISCo Pump 3.0.*" bad_bot
SetEnvIfNoCase User-Agent "^DISCo Pump 3.2.*" bad_bot
SetEnvIfNoCase User-Agent "^DISCoFinder.*" bad_bot
SetEnvIfNoCase User-Agent "^Distilled-Reputation-Monitor.*" bad_bot
SetEnvIfNoCase User-Agent "^Dit.*" bad_bot
SetEnvIfNoCase User-Agent "^DittoSpyder.*" bad_bot
SetEnvIfNoCase User-Agent "^DjangoTraineeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^DKIMRepBot.*" bad_bot
SetEnvIfNoCase User-Agent "^DoCoMo.*" bad_bot
SetEnvIfNoCase User-Agent "^DOF-Verify.*" bad_bot
SetEnvIfNoCase User-Agent "^Domaincrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^domaincrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^DomainScan.*" bad_bot
SetEnvIfNoCase User-Agent "^DomainWatcher Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^DotBot.*" bad_bot
SetEnvIfNoCase User-Agent "^DotBot/1.1.*" bad_bot
SetEnvIfNoCase User-Agent "^DotSpotsBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Dow Jones Searchbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Download.*" bad_bot
SetEnvIfNoCase User-Agent "^Download Demon.*" bad_bot
SetEnvIfNoCase User-Agent "^Download Demon/3.2.0.8.*" bad_bot
SetEnvIfNoCase User-Agent "^Download Demon/3.5.0.11.*" bad_bot
SetEnvIfNoCase User-Agent "^Download Devil.*" bad_bot
SetEnvIfNoCase User-Agent "^Download Master.*" bad_bot
SetEnvIfNoCase User-Agent "^Download Ninja.*" bad_bot
SetEnvIfNoCase User-Agent "^Download Wonder.*" bad_bot
SetEnvIfNoCase User-Agent "^Downloader.*" bad_bot
SetEnvIfNoCase User-Agent "^DOY.*" bad_bot
SetEnvIfNoCase User-Agent "^dragonfly.*" bad_bot
SetEnvIfNoCase User-Agent "^Drip.*" bad_bot
SetEnvIfNoCase User-Agent "^drone.*" bad_bot
SetEnvIfNoCase User-Agent "^Drupal.*" bad_bot
SetEnvIfNoCase User-Agent "^DTAAgent.*" bad_bot
SetEnvIfNoCase User-Agent "^dtSearchSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^dumbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Dumbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Dwaar.*" bad_bot
SetEnvIfNoCase User-Agent "^Dwaarbot.*" bad_bot
SetEnvIfNoCase User-Agent "^DXSeeker.*" bad_bot
SetEnvIfNoCase User-Agent "^EAH.*" bad_bot
SetEnvIfNoCase User-Agent "^Earth Platform Indexer.*" bad_bot
SetEnvIfNoCase User-Agent "^Earth Science Educator robot.*" bad_bot
SetEnvIfNoCase User-Agent "^EARTHCOM.info.*" bad_bot
SetEnvIfNoCase User-Agent "^EARTHCOM.*" bad_bot
SetEnvIfNoCase User-Agent "^EasyDL.*" bad_bot
SetEnvIfNoCase User-Agent "^ebingbong.*" bad_bot
SetEnvIfNoCase User-Agent "^EC2LinkFinder.*" bad_bot
SetEnvIfNoCase User-Agent "^eCairn-Grabber.*" bad_bot
SetEnvIfNoCase User-Agent "^eCatch.*" bad_bot
SetEnvIfNoCase User-Agent "^eCatch/3.0.*" bad_bot
SetEnvIfNoCase User-Agent "^eChooseBot.*" bad_bot
SetEnvIfNoCase User-Agent "^EdisterBot (http://www.edister.com/bot.html)" bad_bot
SetEnvIfNoCase User-Agent "^EduGovSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^egothor.*" bad_bot
SetEnvIfNoCase User-Agent "^eidetica.com/spider.*" bad_bot
SetEnvIfNoCase User-Agent "^EirGrabber.*" bad_bot
SetEnvIfNoCase User-Agent "^Elblindo the Blind Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^ElisaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^EllerdaleBot.*" bad_bot
SetEnvIfNoCase User-Agent "^EMail Exractor.*" bad_bot
SetEnvIfNoCase User-Agent "^EmailCollector.*" bad_bot
SetEnvIfNoCase User-Agent "^EmailLeach.*" bad_bot
SetEnvIfNoCase User-Agent "^EmailSiphon.*" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf.*" bad_bot
SetEnvIfNoCase User-Agent "^EMPAS_ROBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^EnaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^endeca.*" bad_bot
SetEnvIfNoCase User-Agent "^EnigmaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Enswer Neuro Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^EntityCubeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^EroCrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^es.*" bad_bot
SetEnvIfNoCase User-Agent "^eStyleSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^eSyndiCat Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Eurosoft-Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Evaal.*" bad_bot
SetEnvIfNoCase User-Agent "^Eventware.*" bad_bot
SetEnvIfNoCase User-Agent "^Everest-Vulcan Inc..*" bad_bot
SetEnvIfNoCase User-Agent "^Exabot.*" bad_bot
SetEnvIfNoCase User-Agent "^Exabot Test.*" bad_bot
SetEnvIfNoCase User-Agent "^Exabot-Images.*" bad_bot
SetEnvIfNoCase User-Agent "^Exabot-Test.*" bad_bot
SetEnvIfNoCase User-Agent "^Exabot-XXX.*" bad_bot
SetEnvIfNoCase User-Agent "^ExaBotTest.*" bad_bot
SetEnvIfNoCase User-Agent "^ExactSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^exactseek.com.*" bad_bot
SetEnvIfNoCase User-Agent "^exooba/exooba crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Exploder.*" bad_bot
SetEnvIfNoCase User-Agent "^Express WebPictures.*" bad_bot
SetEnvIfNoCase User-Agent "^Extractor.*" bad_bot
SetEnvIfNoCase User-Agent "^ExtractorPro.*" bad_bot
SetEnvIfNoCase User-Agent "^EyeNetIE.*" bad_bot
SetEnvIfNoCase User-Agent "^ez-robot.*" bad_bot
SetEnvIfNoCase User-Agent "^Ezooms.*" bad_bot
SetEnvIfNoCase User-Agent "^f-bot test pilot.*" bad_bot
SetEnvIfNoCase User-Agent "^factbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Factbot.*" bad_bot
SetEnvIfNoCase User-Agent "^FairAd Client.*" bad_bot
SetEnvIfNoCase User-Agent "^falcon.*" bad_bot
SetEnvIfNoCase User-Agent "^Falconsbot.*" bad_bot
SetEnvIfNoCase User-Agent "^FAST Data Search Document Retriever.*" bad_bot
SetEnvIfNoCase User-Agent "^FAST ESP.*" bad_bot
SetEnvIfNoCase User-Agent "^fast-search-engine.*" bad_bot
SetEnvIfNoCase User-Agent "^fastbot crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^fastbot.de crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^FatBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Favcollector.*" bad_bot
SetEnvIfNoCase User-Agent "^Faviconizer.*" bad_bot
SetEnvIfNoCase User-Agent "^Favorites Sweeper.*" bad_bot
SetEnvIfNoCase User-Agent "^FDM 1.x.*" bad_bot
SetEnvIfNoCase User-Agent "^FDM 2.x.*" bad_bot
SetEnvIfNoCase User-Agent "^FDM 3.*" bad_bot
SetEnvIfNoCase User-Agent "^FDSE robot.*" bad_bot
SetEnvIfNoCase User-Agent "^FedContractorBot.*" bad_bot
SetEnvIfNoCase User-Agent "^fembot.*" bad_bot
SetEnvIfNoCase User-Agent "^Fetch API Request.*" bad_bot
SetEnvIfNoCase User-Agent "^fetch_ici.*" bad_bot
SetEnvIfNoCase User-Agent "^fgcrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Filangy.*" bad_bot
SetEnvIfNoCase User-Agent "^FileHound.*" bad_bot
SetEnvIfNoCase User-Agent "^FindAnISP.com_ISP_Finder.*" bad_bot
SetEnvIfNoCase User-Agent "^findlinks.*" bad_bot
SetEnvIfNoCase User-Agent "^FindWeb.*" bad_bot
SetEnvIfNoCase User-Agent "^Firebat.*" bad_bot
SetEnvIfNoCase User-Agent "^FirstGov.gov Search.*" bad_bot
SetEnvIfNoCase User-Agent "^Flaming AttackBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Flamingo_SearchEngine.*" bad_bot
SetEnvIfNoCase User-Agent "^FlashCapture.*" bad_bot
SetEnvIfNoCase User-Agent "^FlashGet.*" bad_bot
SetEnvIfNoCase User-Agent "^FlashGet WebWasher 3.2.*" bad_bot
SetEnvIfNoCase User-Agent "^FlickySearchBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Fluffy the spider.*" bad_bot
SetEnvIfNoCase User-Agent "^flunky.*" bad_bot
SetEnvIfNoCase User-Agent "^focused_crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^FollowSite.*" bad_bot
SetEnvIfNoCase User-Agent "^Foobot.*" bad_bot
SetEnvIfNoCase User-Agent "^Fooooo_Web_Video_Crawl.*" bad_bot
SetEnvIfNoCase User-Agent "^Fopper.*" bad_bot
SetEnvIfNoCase User-Agent "^Forest Conservation Portal,.*" bad_bot
SetEnvIfNoCase User-Agent "^Forest Conservation Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Forests.org Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^FormulaFinderBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Forschungsportal.*" bad_bot
SetEnvIfNoCase User-Agent "^fr_crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Francis.*" bad_bot
SetEnvIfNoCase User-Agent "^FreeWebMonitoring SiteChecker.*" bad_bot
SetEnvIfNoCase User-Agent "^FreshDownload.*" bad_bot
SetEnvIfNoCase User-Agent "^freshlinks.exe.*" bad_bot
SetEnvIfNoCase User-Agent "^FriendFeedBot.*" bad_bot
SetEnvIfNoCase User-Agent "^frodo.at.*" bad_bot
SetEnvIfNoCase User-Agent "^froGgle.*" bad_bot
SetEnvIfNoCase User-Agent "^FrontPage.*" bad_bot
SetEnvIfNoCase User-Agent "^FrontPage [NC,OR].*" bad_bot
SetEnvIfNoCase User-Agent "^Froola Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^FU-NBI/FU-NBI-.*" bad_bot
SetEnvIfNoCase User-Agent "^full_breadth_crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^FunnelBack.*" bad_bot
SetEnvIfNoCase User-Agent "^FurlBot.*" bad_bot
SetEnvIfNoCase User-Agent "^G10-Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Gaisbot.*" bad_bot
SetEnvIfNoCase User-Agent "^GaisBot.*" bad_bot
SetEnvIfNoCase User-Agent "^GalaxyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^gazz.*" bad_bot
SetEnvIfNoCase User-Agent "^generate_infomine_category_classifiers.*" bad_bot
SetEnvIfNoCase User-Agent "^genevabot.*" bad_bot
SetEnvIfNoCase User-Agent "^genieBot.*" bad_bot
SetEnvIfNoCase User-Agent "^GenieBotRD_SmallCrawl.*" bad_bot
SetEnvIfNoCase User-Agent "^Genieo.*" bad_bot
SetEnvIfNoCase User-Agent "^Geomaxenginebot.*" bad_bot
SetEnvIfNoCase User-Agent "^geometabot.*" bad_bot
SetEnvIfNoCase User-Agent "^GeonaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^GeoVisu.*" bad_bot
SetEnvIfNoCase User-Agent "^GetHTMLContents.*" bad_bot
SetEnvIfNoCase User-Agent "^Getleft.*" bad_bot
SetEnvIfNoCase User-Agent "^GetLeft.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/2.11.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/3.1.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/3.2.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/3.3.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/3.3.3.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/3.3.4.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.0.0.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.1.0.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.1.1.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.1.2.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.2.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.2b (Portuguxeas).*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.2c.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.3.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5a.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5b.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5b1.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5b2.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5b3.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5b6.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5b7.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5c.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5d.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/4.5e.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/5.0beta1.*" bad_bot
SetEnvIfNoCase User-Agent "^GetRight/5.0beta2.*" bad_bot
SetEnvIfNoCase User-Agent "^GetSmart.*" bad_bot
SetEnvIfNoCase User-Agent "^GetURL.rexx v1.05.*" bad_bot
SetEnvIfNoCase User-Agent "^GetWeb!.*" bad_bot
SetEnvIfNoCase User-Agent "^Giant.*" bad_bot
SetEnvIfNoCase User-Agent "^GigablastOpenSource.*" bad_bot
SetEnvIfNoCase User-Agent "^Gigabot.*" bad_bot
SetEnvIfNoCase User-Agent "^GigaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^GigabotSiteSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^Girafabot.*" bad_bot
SetEnvIfNoCase User-Agent "^GleameBot.*" bad_bot
SetEnvIfNoCase User-Agent "^gnome-vfs.*" bad_bot
SetEnvIfNoCase User-Agent "^Go-Ahead-Got-It.*" bad_bot
SetEnvIfNoCase User-Agent "^Go!Zilla.*" bad_bot
SetEnvIfNoCase User-Agent "^GoForIt.com.*" bad_bot
SetEnvIfNoCase User-Agent "^GOFORITBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^gold crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Goldfire Server.*" bad_bot
SetEnvIfNoCase User-Agent "^Golem/1.1.*" bad_bot
SetEnvIfNoCase User-Agent "^GoodJelly.*" bad_bot
SetEnvIfNoCase User-Agent "^Gordon-College-Google-Mini.*" bad_bot
SetEnvIfNoCase User-Agent "^goroam.*" bad_bot
SetEnvIfNoCase User-Agent "^GoSeebot.*" bad_bot
SetEnvIfNoCase User-Agent "^gotit.*" bad_bot
SetEnvIfNoCase User-Agent "^Govbot.*" bad_bot
SetEnvIfNoCase User-Agent "^GPU p2p crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Grabber.*" bad_bot
SetEnvIfNoCase User-Agent "^GrabNet.*" bad_bot
SetEnvIfNoCase User-Agent "^Grafula.*" bad_bot
SetEnvIfNoCase User-Agent "^grapeFX.*" bad_bot
SetEnvIfNoCase User-Agent "^grapeshot.*" bad_bot
SetEnvIfNoCase User-Agent "^GrapeshotCrawler/2.0.*" bad_bot
SetEnvIfNoCase User-Agent "^grbot.*" bad_bot
SetEnvIfNoCase User-Agent "^GreenYogi [ZSEBOT].*" bad_bot
SetEnvIfNoCase User-Agent "^Gromit.*" bad_bot
SetEnvIfNoCase User-Agent "^grub.*" bad_bot
SetEnvIfNoCase User-Agent "^grub crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^grub-client.*" bad_bot
SetEnvIfNoCase User-Agent "^Grub.*" bad_bot
SetEnvIfNoCase User-Agent "^Grubclient-.*" bad_bot
SetEnvIfNoCase User-Agent "^GrubNG.*" bad_bot
SetEnvIfNoCase User-Agent "^GruBot.*" bad_bot
SetEnvIfNoCase User-Agent "^gsa.*" bad_bot
SetEnvIfNoCase User-Agent "^gsa-accuracyEval.*" bad_bot
SetEnvIfNoCase User-Agent "^gsa-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^gsa-eusleg.*" bad_bot
SetEnvIfNoCase User-Agent "^GSLFbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Gulliver.*" bad_bot
SetEnvIfNoCase User-Agent "^GulperBot.*" bad_bot
SetEnvIfNoCase User-Agent "^GurujiBot.*" bad_bot
SetEnvIfNoCase User-Agent "^GVC BUSINESS crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^GVC crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^GVC SEARCH BOT.*" bad_bot
SetEnvIfNoCase User-Agent "^GVC WEB crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^GVC Weblink crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^GVC WORLD LINKS.*" bad_bot
SetEnvIfNoCase User-Agent "^gvcbot.com.*" bad_bot
SetEnvIfNoCase User-Agent "^HappyFunBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Harvest.*" bad_bot
SetEnvIfNoCase User-Agent "^Harvest/1.5.*" bad_bot
SetEnvIfNoCase User-Agent "^HarvestMan.*" bad_bot
SetEnvIfNoCase User-Agent "^Hatena Antenna.*" bad_bot
SetEnvIfNoCase User-Agent "^Hawler.*" bad_bot
SetEnvIfNoCase User-Agent "^hcat.*" bad_bot
SetEnvIfNoCase User-Agent "^hclsreport-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^HD nutch agent.*" bad_bot
SetEnvIfNoCase User-Agent "^Header_Test_Client.*" bad_bot
SetEnvIfNoCase User-Agent "^healia/healia.*" bad_bot
SetEnvIfNoCase User-Agent "^Helix.*" bad_bot
SetEnvIfNoCase User-Agent "^here will be link to crawler site.*" bad_bot
SetEnvIfNoCase User-Agent "^heritrix bot.*" bad_bot
SetEnvIfNoCase User-Agent "^heritrix.*" bad_bot
SetEnvIfNoCase User-Agent "^Heritrix.*" bad_bot
SetEnvIfNoCase User-Agent "^heritrix/3.1.0.*" bad_bot
SetEnvIfNoCase User-Agent "^heritrix/3.1.1.*" bad_bot
SetEnvIfNoCase User-Agent "^hijbul-heritrix-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^HiScan.*" bad_bot
SetEnvIfNoCase User-Agent "^HiSoftware AccMonitor Server.*" bad_bot
SetEnvIfNoCase User-Agent "^HiSoftware AccVerify.*" bad_bot
SetEnvIfNoCase User-Agent "^hitcrawler_.*" bad_bot
SetEnvIfNoCase User-Agent "^hivaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^hloader.*" bad_bot
SetEnvIfNoCase User-Agent "^HLoader.*" bad_bot
SetEnvIfNoCase User-Agent "^HMSEbot.*" bad_bot
SetEnvIfNoCase User-Agent "^HMView.*" bad_bot
SetEnvIfNoCase User-Agent "^hoge.*" bad_bot
SetEnvIfNoCase User-Agent "^holmes.*" bad_bot
SetEnvIfNoCase User-Agent "^HomePageSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^Hooblybot-Image.*" bad_bot
SetEnvIfNoCase User-Agent "^HooWWWer.*" bad_bot
SetEnvIfNoCase User-Agent "^Hostcrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^HSFT - Link Scanner.*" bad_bot
SetEnvIfNoCase User-Agent "^HSFT - LVU Scanner.*" bad_bot
SetEnvIfNoCase User-Agent "^HSlide.*" bad_bot
SetEnvIfNoCase User-Agent "^ht://check.*" bad_bot
SetEnvIfNoCase User-Agent "^htdig.*" bad_bot
SetEnvIfNoCase User-Agent "^Html Link Validator.*" bad_bot
SetEnvIfNoCase User-Agent "^HTMLParser.*" bad_bot
SetEnvIfNoCase User-Agent "^httplib.*" bad_bot
SetEnvIfNoCase User-Agent "^HTTrack.*" bad_bot
SetEnvIfNoCase User-Agent "^HTTrack [NC,OR].*" bad_bot
SetEnvIfNoCase User-Agent "^HTTrack 3.0.*" bad_bot
SetEnvIfNoCase User-Agent "^Huaweisymantecspider.*" bad_bot
SetEnvIfNoCase User-Agent "^hul-wax.*" bad_bot
SetEnvIfNoCase User-Agent "^humanlinks.*" bad_bot
SetEnvIfNoCase User-Agent "^HyperEstraier.*" bad_bot
SetEnvIfNoCase User-Agent "^Hyperix.*" bad_bot
SetEnvIfNoCase User-Agent "^ia_archiver.*" bad_bot
SetEnvIfNoCase User-Agent "^IAArchiver-.*" bad_bot
SetEnvIfNoCase User-Agent "^ibuena.*" bad_bot
SetEnvIfNoCase User-Agent "^iCab.*" bad_bot
SetEnvIfNoCase User-Agent "^ICDS-Ingestion.*" bad_bot
SetEnvIfNoCase User-Agent "^ichiro.*" bad_bot
SetEnvIfNoCase User-Agent "^iCopyright Conductor.*" bad_bot
SetEnvIfNoCase User-Agent "^IEAutoDiscovery.*" bad_bot
SetEnvIfNoCase User-Agent "^IECheck.*" bad_bot
SetEnvIfNoCase User-Agent "^iHWebChecker.*" bad_bot
SetEnvIfNoCase User-Agent "^IIITBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^iim_405.*" bad_bot
SetEnvIfNoCase User-Agent "^IlseBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Iltrovatore-Setaccio.*" bad_bot
SetEnvIfNoCase User-Agent "^IlTrovatore-Setaccio.*" bad_bot
SetEnvIfNoCase User-Agent "^IlTrovatore.*" bad_bot
SetEnvIfNoCase User-Agent "^Image Stripper.*" bad_bot
SetEnvIfNoCase User-Agent "^Image Sucker.*" bad_bot
SetEnvIfNoCase User-Agent "^ImageBot.*" bad_bot
SetEnvIfNoCase User-Agent "^imagefortress.*" bad_bot
SetEnvIfNoCase User-Agent "^ImagesHereImagesThereImagesEverywhere.*" bad_bot
SetEnvIfNoCase User-Agent "^ImageVisu.*" bad_bot
SetEnvIfNoCase User-Agent "^imds_monitor.*" bad_bot
SetEnvIfNoCase User-Agent "^imo-google-robot-intelink.*" bad_bot
SetEnvIfNoCase User-Agent "^inagist.com url crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Industry Cortex Webcrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Indy Library.*" bad_bot
SetEnvIfNoCase User-Agent "^Indy Library [NC,OR].*" bad_bot
SetEnvIfNoCase User-Agent "^indylabs_marius.*" bad_bot
SetEnvIfNoCase User-Agent "^InelaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Inet32 Ctrl.*" bad_bot
SetEnvIfNoCase User-Agent "^inetbot.*" bad_bot
SetEnvIfNoCase User-Agent "^info seeker.*" bad_bot
SetEnvIfNoCase User-Agent "^InfoLink.*" bad_bot
SetEnvIfNoCase User-Agent "^infomine.ucr.edu.*" bad_bot
SetEnvIfNoCase User-Agent "^INFOMINE.*" bad_bot
SetEnvIfNoCase User-Agent "^InfoNaviRobot.*" bad_bot
SetEnvIfNoCase User-Agent "^Informant.*" bad_bot
SetEnvIfNoCase User-Agent "^Infoseek Sidewinder.*" bad_bot
SetEnvIfNoCase User-Agent "^InfoSeek Sidewinder.*" bad_bot
SetEnvIfNoCase User-Agent "^Infoseek SideWinder.*" bad_bot
SetEnvIfNoCase User-Agent "^InfoTekies.*" bad_bot
SetEnvIfNoCase User-Agent "^InfoUSABot.*" bad_bot
SetEnvIfNoCase User-Agent "^INGRID.*" bad_bot
SetEnvIfNoCase User-Agent "^INGRID/0.1.*" bad_bot
SetEnvIfNoCase User-Agent "^Inktomi.*" bad_bot
SetEnvIfNoCase User-Agent "^InsightsCollector.*" bad_bot
SetEnvIfNoCase User-Agent "^InsightsWorksBot.*" bad_bot
SetEnvIfNoCase User-Agent "^InspireBot.*" bad_bot
SetEnvIfNoCase User-Agent "^InsumaScout.*" bad_bot
SetEnvIfNoCase User-Agent "^Intelix.*" bad_bot
SetEnvIfNoCase User-Agent "^Intelliseek.*" bad_bot
SetEnvIfNoCase User-Agent "^InterGET.*" bad_bot
SetEnvIfNoCase User-Agent "^Internet Ninja.*" bad_bot
SetEnvIfNoCase User-Agent "^Internet Ninja 4.0.*" bad_bot
SetEnvIfNoCase User-Agent "^Internet Ninja 5.0.*" bad_bot
SetEnvIfNoCase User-Agent "^Internet Ninja 6.0.*" bad_bot
SetEnvIfNoCase User-Agent "^INTERNET RADIO crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^InternetLinkAgent.*" bad_bot
SetEnvIfNoCase User-Agent "^Interseek.*" bad_bot
SetEnvIfNoCase User-Agent "^IOI.*" bad_bot
SetEnvIfNoCase User-Agent "^ip-web-crawler.com.*" bad_bot
SetEnvIfNoCase User-Agent "^IPAdd Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^IpselonBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Ipselonbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Iria.*" bad_bot
SetEnvIfNoCase User-Agent "^IRLbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Iron33/1.0.2.*" bad_bot
SetEnvIfNoCase User-Agent "^Isara/Isara-.*" bad_bot
SetEnvIfNoCase User-Agent "^iSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^iSiloX.*" bad_bot
SetEnvIfNoCase User-Agent "^IstellaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^its-learning crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^IU_CSCI_B659_class_crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^iVia Page Fetcher.*" bad_bot
SetEnvIfNoCase User-Agent "^iVia Site Checker.*" bad_bot
SetEnvIfNoCase User-Agent "^iVia.*" bad_bot
SetEnvIfNoCase User-Agent "^JadynAve.*" bad_bot
SetEnvIfNoCase User-Agent "^JadynAveBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Jakarta.*" bad_bot
SetEnvIfNoCase User-Agent "^Jakarta Commons-HttpClient.*" bad_bot
SetEnvIfNoCase User-Agent "^Jbot.*" bad_bot
SetEnvIfNoCase User-Agent "^JemmaTheTourist.*" bad_bot
SetEnvIfNoCase User-Agent "^JennyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Jetbot.*" bad_bot
SetEnvIfNoCase User-Agent "^JetBrains Omea Pro.*" bad_bot
SetEnvIfNoCase User-Agent "^JetCar.*" bad_bot
SetEnvIfNoCase User-Agent "^Jim.*" bad_bot
SetEnvIfNoCase User-Agent "^JoBo.*" bad_bot
SetEnvIfNoCase User-Agent "^JobSpider_BA.*" bad_bot
SetEnvIfNoCase User-Agent "^JOC.*" bad_bot
SetEnvIfNoCase User-Agent "^JOC Web Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^JoeDog.*" bad_bot
SetEnvIfNoCase User-Agent "^JoyScapeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^JSpyda.*" bad_bot
SetEnvIfNoCase User-Agent "^Junut Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^JustView.*" bad_bot
SetEnvIfNoCase User-Agent "^Jyxobot.*" bad_bot
SetEnvIfNoCase User-Agent "^K.S.Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^KakcleBot.*" bad_bot
SetEnvIfNoCase User-Agent "^kalooga.*" bad_bot
SetEnvIfNoCase User-Agent "^KaloogaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^KATATUDO-Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^kbeta1.*" bad_bot
SetEnvIfNoCase User-Agent "^KeepNI web site monitor.*" bad_bot
SetEnvIfNoCase User-Agent "^Kenjin.Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Keybot Translation-Search-Machine.*" bad_bot
SetEnvIfNoCase User-Agent "^KeywenBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Keyword Density/0.9.*" bad_bot
SetEnvIfNoCase User-Agent "^Keyword.Density.*" bad_bot
SetEnvIfNoCase User-Agent "^kinjabot.*" bad_bot
SetEnvIfNoCase User-Agent "^Kitenga-crawler-bot.*" bad_bot
SetEnvIfNoCase User-Agent "^KiwiStatus.*" bad_bot
SetEnvIfNoCase User-Agent "^kmbot-.*" bad_bot
SetEnvIfNoCase User-Agent "^kmccrew Bot Search.*" bad_bot
SetEnvIfNoCase User-Agent "^Knight.*" bad_bot
SetEnvIfNoCase User-Agent "^KnowItAll.*" bad_bot
SetEnvIfNoCase User-Agent "^Knowledge Engine.*" bad_bot
SetEnvIfNoCase User-Agent "^Knowledge.com.*" bad_bot
SetEnvIfNoCase User-Agent "^KoepaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Koninklijke.*" bad_bot
SetEnvIfNoCase User-Agent "^KrOWLer.*" bad_bot
SetEnvIfNoCase User-Agent "^KSbot.*" bad_bot
SetEnvIfNoCase User-Agent "^kuloko-bot.*" bad_bot
SetEnvIfNoCase User-Agent "^kulturarw3.*" bad_bot
SetEnvIfNoCase User-Agent "^KummHttp.*" bad_bot
SetEnvIfNoCase User-Agent "^Kurzor.*" bad_bot
SetEnvIfNoCase User-Agent "^Kyluka crawl.*" bad_bot
SetEnvIfNoCase User-Agent "^L.webis.*" bad_bot
SetEnvIfNoCase User-Agent "^Labhoo.*" bad_bot
SetEnvIfNoCase User-Agent "^labourunions411.*" bad_bot
SetEnvIfNoCase User-Agent "^Lachesis.*" bad_bot
SetEnvIfNoCase User-Agent "^lachesis.*" bad_bot
SetEnvIfNoCase User-Agent "^Lament.*" bad_bot
SetEnvIfNoCase User-Agent "^LamerExterminator.*" bad_bot
SetEnvIfNoCase User-Agent "^LapozzBot.*" bad_bot
SetEnvIfNoCase User-Agent "^larbin.*" bad_bot
SetEnvIfNoCase User-Agent "^LARBIN-EXPERIMENTAL.*" bad_bot
SetEnvIfNoCase User-Agent "^LBot.*" bad_bot
SetEnvIfNoCase User-Agent "^LeapTag.*" bad_bot
SetEnvIfNoCase User-Agent "^LeechFTP.*" bad_bot
SetEnvIfNoCase User-Agent "^LeechGet.*" bad_bot
SetEnvIfNoCase User-Agent "^LetsCrawl.com.*" bad_bot
SetEnvIfNoCase User-Agent "^LexiBot.*" bad_bot
SetEnvIfNoCase User-Agent "^LexxeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^lftp.*" bad_bot
SetEnvIfNoCase User-Agent "^libcrawl.*" bad_bot
SetEnvIfNoCase User-Agent "^libiViaCore.*" bad_bot
SetEnvIfNoCase User-Agent "^libWeb/clsHTTP.*" bad_bot
SetEnvIfNoCase User-Agent "^libwww-perl.*" bad_bot
SetEnvIfNoCase User-Agent "^likse.*" bad_bot
SetEnvIfNoCase User-Agent "^Linguee Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Link Checker.*" bad_bot
SetEnvIfNoCase User-Agent "^Link Validator.*" bad_bot
SetEnvIfNoCase User-Agent "^link_checker.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkAlarm.*" bad_bot
SetEnvIfNoCase User-Agent "^linkbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Linkbot.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkCheck by Siteimprove.com.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkCheck Scanner.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkChecker.*" bad_bot
SetEnvIfNoCase User-Agent "^linkdex.com.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkextractorPro.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkLint.*" bad_bot
SetEnvIfNoCase User-Agent "^linklooker.*" bad_bot
SetEnvIfNoCase User-Agent "^Linkman.*" bad_bot
SetEnvIfNoCase User-Agent "^Links SQL.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkScan.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkScan/8.1a.Unix.*" bad_bot
SetEnvIfNoCase User-Agent "^LinksManager.com_bot.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkSweeper.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkWalker.*" bad_bot
SetEnvIfNoCase User-Agent "^LiteFinder.*" bad_bot
SetEnvIfNoCase User-Agent "^LitlrBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Little Grabber at Skanktale.com.*" bad_bot
SetEnvIfNoCase User-Agent "^Livelapbot.*" bad_bot
SetEnvIfNoCase User-Agent "^LM Harvester.*" bad_bot
SetEnvIfNoCase User-Agent "^LMQueueBot.*" bad_bot
SetEnvIfNoCase User-Agent "^LNSpiderguy.*" bad_bot
SetEnvIfNoCase User-Agent "^LoadTimeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^LocalcomBot.*" bad_bot
SetEnvIfNoCase User-Agent "^locust.*" bad_bot
SetEnvIfNoCase User-Agent "^LolongBot.*" bad_bot
SetEnvIfNoCase User-Agent "^LookBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Lsearch/sondeur.*" bad_bot
SetEnvIfNoCase User-Agent "^lssbot.*" bad_bot
SetEnvIfNoCase User-Agent "^LT Scotland Checklink.*" bad_bot
SetEnvIfNoCase User-Agent "^lwp-request.*" bad_bot
SetEnvIfNoCase User-Agent "^lwp-trivial.*" bad_bot
SetEnvIfNoCase User-Agent "^lwp-trivial/1.34.*" bad_bot
SetEnvIfNoCase User-Agent "^LWP::Simple.*" bad_bot
SetEnvIfNoCase User-Agent "^LWP/natweb-bad-link-mailer.*" bad_bot
SetEnvIfNoCase User-Agent "^Lycos_Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Lydia Entity Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^LynnBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Lytranslate.*" bad_bot
SetEnvIfNoCase User-Agent "^Mag-Net.*" bad_bot
SetEnvIfNoCase User-Agent "^Magnet.*" bad_bot
SetEnvIfNoCase User-Agent "^magpie-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Magus Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Mail.Ru_Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Mail.RU_Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Mail.Ru_Bot/2.0.*" bad_bot
SetEnvIfNoCase User-Agent "^Mail.Ru.*" bad_bot
SetEnvIfNoCase User-Agent "^Mail.RU.*" bad_bot
SetEnvIfNoCase User-Agent "^MAINSEEK_BOT.*" bad_bot
SetEnvIfNoCase User-Agent "^Mammoth.*" bad_bot
SetEnvIfNoCase User-Agent "^Map robot.*" bad_bot
SetEnvIfNoCase User-Agent "^MarkWatch.*" bad_bot
SetEnvIfNoCase User-Agent "^MaSagool.*" bad_bot
SetEnvIfNoCase User-Agent "^masidani_bot_.*" bad_bot
SetEnvIfNoCase User-Agent "^Mass Downloader.*" bad_bot
SetEnvIfNoCase User-Agent "^Mass Downloader/2.2.*" bad_bot
SetEnvIfNoCase User-Agent "^Mata Hari.*" bad_bot
SetEnvIfNoCase User-Agent "^Mata.Hari.*" bad_bot
SetEnvIfNoCase User-Agent "^matentzn at cs dot man dot ac dot uk.*" bad_bot
SetEnvIfNoCase User-Agent "^maxamine.com--robot.*" bad_bot
SetEnvIfNoCase User-Agent "^maxamine.com-robot.*" bad_bot
SetEnvIfNoCase User-Agent "^maxomobot.*" bad_bot
SetEnvIfNoCase User-Agent "^McBot.*" bad_bot
SetEnvIfNoCase User-Agent "^medrabbit.*" bad_bot
SetEnvIfNoCase User-Agent "^Megite.*" bad_bot
SetEnvIfNoCase User-Agent "^MemacBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Memo.*" bad_bot
SetEnvIfNoCase User-Agent "^MendeleyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Mercator-.*" bad_bot
SetEnvIfNoCase User-Agent "^mercuryboard_user_agent_sql_injection.nasl.*" bad_bot
SetEnvIfNoCase User-Agent "^metacarta.*" bad_bot
SetEnvIfNoCase User-Agent "^Metaeuro Web Search.*" bad_bot
SetEnvIfNoCase User-Agent "^Metager2.*" bad_bot
SetEnvIfNoCase User-Agent "^metager2-verification-bot.*" bad_bot
SetEnvIfNoCase User-Agent "^MetaGloss.*" bad_bot
SetEnvIfNoCase User-Agent "^metal crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^metaquerier.cs.uiuc.edu.*" bad_bot
SetEnvIfNoCase User-Agent "^METASpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Metaspinner.*" bad_bot
SetEnvIfNoCase User-Agent "^MetaURI API/2.0 metauri.com.*" bad_bot
SetEnvIfNoCase User-Agent "^MetaURI API/2.0 +metauri.com.*" bad_bot
SetEnvIfNoCase User-Agent "^MFcrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^MFHttpScan.*" bad_bot
SetEnvIfNoCase User-Agent "^MIDown tool.*" bad_bot
SetEnvIfNoCase User-Agent "^MIIxpc.*" bad_bot
SetEnvIfNoCase User-Agent "^MIIxpc/4.2.*" bad_bot
SetEnvIfNoCase User-Agent "^mini-robot.*" bad_bot
SetEnvIfNoCase User-Agent "^minibot.*" bad_bot
SetEnvIfNoCase User-Agent "^miniRank.*" bad_bot
SetEnvIfNoCase User-Agent "^Mirror.*" bad_bot
SetEnvIfNoCase User-Agent "^Missigua Locator.*" bad_bot
SetEnvIfNoCase User-Agent "^Mister PiX.*" bad_bot
SetEnvIfNoCase User-Agent "^Mister Pix II 2.01.*" bad_bot
SetEnvIfNoCase User-Agent "^Mister Pix II 2.02a.*" bad_bot
SetEnvIfNoCase User-Agent "^Mister PiX version.dll.*" bad_bot
SetEnvIfNoCase User-Agent "^Mister.PiX.*" bad_bot
SetEnvIfNoCase User-Agent "^Miva.*" bad_bot
SetEnvIfNoCase User-Agent "^MJ12bot.*" bad_bot
SetEnvIfNoCase User-Agent "^mnoGoSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^MnoGoSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^mod_accessibility.*" bad_bot
SetEnvIfNoCase User-Agent "^moduna.com.*" bad_bot
SetEnvIfNoCase User-Agent "^moget.*" bad_bot
SetEnvIfNoCase User-Agent "^moget/2.1.*" bad_bot
SetEnvIfNoCase User-Agent "^MojeekBot.*" bad_bot
SetEnvIfNoCase User-Agent "^MonkeyCrawl.*" bad_bot
SetEnvIfNoCase User-Agent "^MOSES.*" bad_bot
SetEnvIfNoCase User-Agent "^mowserbot.*" bad_bot
SetEnvIfNoCase User-Agent "^MQbot.*" bad_bot
SetEnvIfNoCase User-Agent "^MSE360.*" bad_bot
SetEnvIfNoCase User-Agent "^MSIndianWebcrawl.*" bad_bot
SetEnvIfNoCase User-Agent "^MSMOBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^msnbot-products.*" bad_bot
SetEnvIfNoCase User-Agent "^MSNPTC.*" bad_bot
SetEnvIfNoCase User-Agent "^MSRBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^MT-Soft.*" bad_bot
SetEnvIfNoCase User-Agent "^MultiText.*" bad_bot
SetEnvIfNoCase User-Agent "^My_Little_SearchEngine_Project.*" bad_bot
SetEnvIfNoCase User-Agent "^my-heritrix-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^MyApp.*" bad_bot
SetEnvIfNoCase User-Agent "^MYCOMPANYBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^mycrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^MyEngines-US-Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^MyFamilyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Myra.*" bad_bot
SetEnvIfNoCase User-Agent "^nabot.*" bad_bot
SetEnvIfNoCase User-Agent "^nabot_.*" bad_bot
SetEnvIfNoCase User-Agent "^Najdi.si.*" bad_bot
SetEnvIfNoCase User-Agent "^Nambu.*" bad_bot
SetEnvIfNoCase User-Agent "^NAMEPROTECT.*" bad_bot
SetEnvIfNoCase User-Agent "^NASA Search.*" bad_bot
SetEnvIfNoCase User-Agent "^NatchCVS.*" bad_bot
SetEnvIfNoCase User-Agent "^National Park Service Dan Buan.*" bad_bot
SetEnvIfNoCase User-Agent "^naver.*" bad_bot
SetEnvIfNoCase User-Agent "^naverbookmarkcrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^NaverBot_dloader.*" bad_bot
SetEnvIfNoCase User-Agent "^NaverBot-.*" bad_bot
SetEnvIfNoCase User-Agent "^NaverBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Navroad.*" bad_bot
SetEnvIfNoCase User-Agent "^NearSite.*" bad_bot
SetEnvIfNoCase User-Agent "^NEC-MeshExplorer.*" bad_bot
SetEnvIfNoCase User-Agent "^NeoScioCrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^NerdByNature.Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^NerdyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Nerima-crawl-.*" bad_bot
SetEnvIfNoCase User-Agent "^Nessus.*" bad_bot
SetEnvIfNoCase User-Agent "^NESSUS::SOAP.*" bad_bot
SetEnvIfNoCase User-Agent "^nestReader.*" bad_bot
SetEnvIfNoCase User-Agent "^Net Vampire.*" bad_bot
SetEnvIfNoCase User-Agent "^Net Vampire/3.0.*" bad_bot
SetEnvIfNoCase User-Agent "^Net::Trackback.*" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts.*" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts/1.10.*" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts/1.23.*" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts/1.24.*" bad_bot
SetEnvIfNoCase User-Agent "^NetAnts/1.25.*" bad_bot
SetEnvIfNoCase User-Agent "^NetCarta CyberPilot Pro.*" bad_bot
SetEnvIfNoCase User-Agent "^Netcraft.*" bad_bot
SetEnvIfNoCase User-Agent "^NetID.com Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^NetMechanic.*" bad_bot
SetEnvIfNoCase User-Agent "^Netprospector.*" bad_bot
SetEnvIfNoCase User-Agent "^NetResearchServer.*" bad_bot
SetEnvIfNoCase User-Agent "^NetSeer.*" bad_bot
SetEnvIfNoCase User-Agent "^NetSeer crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^NetSeer crawler/2.0.*" bad_bot
SetEnvIfNoCase User-Agent "^NetShift=.*" bad_bot
SetEnvIfNoCase User-Agent "^NetSongBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Netsparker.*" bad_bot
SetEnvIfNoCase User-Agent "^NetSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^NetSrcherP.*" bad_bot
SetEnvIfNoCase User-Agent "^NetZIP.*" bad_bot
SetEnvIfNoCase User-Agent "^NewMedhunt.*" bad_bot
SetEnvIfNoCase User-Agent "^news bot .*" bad_bot
SetEnvIfNoCase User-Agent "^News_Search_App.*" bad_bot
SetEnvIfNoCase User-Agent "^NewsGatherer.*" bad_bot
SetEnvIfNoCase User-Agent "^Newsgroupreporter.*" bad_bot
SetEnvIfNoCase User-Agent "^NewsTroveBot.*" bad_bot
SetEnvIfNoCase User-Agent "^NextGenSearchBot.*" bad_bot
SetEnvIfNoCase User-Agent "^nextthing.org.*" bad_bot
SetEnvIfNoCase User-Agent "^NG.*" bad_bot
SetEnvIfNoCase User-Agent "^nicebot.*" bad_bot
SetEnvIfNoCase User-Agent "^NICErsPRO.*" bad_bot
SetEnvIfNoCase User-Agent "^niki-bot.*" bad_bot
SetEnvIfNoCase User-Agent "^NimbleCrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^nimbus-1.*" bad_bot
SetEnvIfNoCase User-Agent "^ninetowns.*" bad_bot
SetEnvIfNoCase User-Agent "^Ninja.*" bad_bot
SetEnvIfNoCase User-Agent "^NjuiceBot.*" bad_bot
SetEnvIfNoCase User-Agent "^NLese.*" bad_bot
SetEnvIfNoCase User-Agent "^NLESE USEPA.*" bad_bot
SetEnvIfNoCase User-Agent "^Nogate.*" bad_bot
SetEnvIfNoCase User-Agent "^Norbert the Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^NoteworthyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^NPbot.*" bad_bot
SetEnvIfNoCase User-Agent "^NPBot.*" bad_bot
SetEnvIfNoCase User-Agent "^NPBot-.*" bad_bot
SetEnvIfNoCase User-Agent "^NRCan intranet crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^NSDL_Search_Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^nu_tch-princeton/Nu_tch.*" bad_bot
SetEnvIfNoCase User-Agent "^nuggetize.com BOT.*" bad_bot
SetEnvIfNoCase User-Agent "^nuSearch Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^NuSearch Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Nusearch Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Nutch.*" bad_bot
SetEnvIfNoCase User-Agent "^nutch.*" bad_bot
SetEnvIfNoCase User-Agent "^nutch1/huntsman.*" bad_bot
SetEnvIfNoCase User-Agent "^NutchCVS.*" bad_bot
SetEnvIfNoCase User-Agent "^NutchOrg.*" bad_bot
SetEnvIfNoCase User-Agent "^NWSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Nymesis.*" bad_bot
SetEnvIfNoCase User-Agent "^nys-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^ObjectsSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^oBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Obvius external linkcheck.*" bad_bot
SetEnvIfNoCase User-Agent "^Ocelli.*" bad_bot
SetEnvIfNoCase User-Agent "^Octopus.*" bad_bot
SetEnvIfNoCase User-Agent "^ODP entries t_st.*" bad_bot
SetEnvIfNoCase User-Agent "^oegp v..*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/1.2.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/1.4.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/1.6.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/1.7.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/1.9.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/2.0.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/2.1.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/2.3.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/2.4.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Explorer/2.5.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline Navigator.*" bad_bot
SetEnvIfNoCase User-Agent "^Offline.Explorer.*" bad_bot
SetEnvIfNoCase User-Agent "^OGspider.*" bad_bot
SetEnvIfNoCase User-Agent "^OmiExplorer_Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^OmniExplorer_Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^OmniFind.*" bad_bot
SetEnvIfNoCase User-Agent "^omnifind.*" bad_bot
SetEnvIfNoCase User-Agent "^OmniWeb.*" bad_bot
SetEnvIfNoCase User-Agent "^OnetSzukaj.*" bad_bot
SetEnvIfNoCase User-Agent "^online link validator.*" bad_bot
SetEnvIfNoCase User-Agent "^OOZBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^Openbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Openfind.*" bad_bot
SetEnvIfNoCase User-Agent "^Openfind data gatherer.*" bad_bot
SetEnvIfNoCase User-Agent "^OpenIntelligenceData.*" bad_bot
SetEnvIfNoCase User-Agent "^OpenISearch.*" bad_bot
SetEnvIfNoCase User-Agent "^OpenLink Virtuoso RDF crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^OpenSearchServer_Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^OpiDig.*" bad_bot
SetEnvIfNoCase User-Agent "^optidiscover.*" bad_bot
SetEnvIfNoCase User-Agent "^Oracle Secure Enterprise Search.*" bad_bot
SetEnvIfNoCase User-Agent "^Oracle Ultra Search.*" bad_bot
SetEnvIfNoCase User-Agent "^OrangeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^ORISBot.*" bad_bot
SetEnvIfNoCase User-Agent "^ornl_crawler_1.*" bad_bot
SetEnvIfNoCase User-Agent "^ORNL_Mercury.*" bad_bot
SetEnvIfNoCase User-Agent "^osis-project.jp.*" bad_bot
SetEnvIfNoCase User-Agent "^OsO.*" bad_bot
SetEnvIfNoCase User-Agent "^OutfoxBot.*" bad_bot
SetEnvIfNoCase User-Agent "^OutfoxMelonBot.*" bad_bot
SetEnvIfNoCase User-Agent "^OWLER-BOT.*" bad_bot
SetEnvIfNoCase User-Agent "^owsBot.*" bad_bot
SetEnvIfNoCase User-Agent "^ozelot.*" bad_bot
SetEnvIfNoCase User-Agent "^P3P Client.*" bad_bot
SetEnvIfNoCase User-Agent "^page_verifier.*" bad_bot
SetEnvIfNoCase User-Agent "^PageBitesHyperBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Pagebull.*" bad_bot
SetEnvIfNoCase User-Agent "^PageDown.*" bad_bot
SetEnvIfNoCase User-Agent "^PageFetcher.*" bad_bot
SetEnvIfNoCase User-Agent "^PageGrabber.*" bad_bot
SetEnvIfNoCase User-Agent "^PagePeeker.*" bad_bot
SetEnvIfNoCase User-Agent "^PageRank Monitor.*" bad_bot
SetEnvIfNoCase User-Agent "^pamsnbot.htm.*" bad_bot
SetEnvIfNoCase User-Agent "^Panopy Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^panscient.com.*" bad_bot
SetEnvIfNoCase User-Agent "^Pansophica.*" bad_bot
SetEnvIfNoCase User-Agent "^Papa Foto.*" bad_bot
SetEnvIfNoCase User-Agent "^PaperLiBot.*" bad_bot
SetEnvIfNoCase User-Agent "^PaperLiBot/2.1.*" bad_bot
SetEnvIfNoCase User-Agent "^parasite.*" bad_bot
SetEnvIfNoCase User-Agent "^parsijoo.*" bad_bot
SetEnvIfNoCase User-Agent "^Pathtraq.*" bad_bot
SetEnvIfNoCase User-Agent "^Pattern.*" bad_bot
SetEnvIfNoCase User-Agent "^Patwebbot.*" bad_bot
SetEnvIfNoCase User-Agent "^pavuk.*" bad_bot
SetEnvIfNoCase User-Agent "^PaxleFramework.*" bad_bot
SetEnvIfNoCase User-Agent "^PBBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^pcBrowser.*" bad_bot
SetEnvIfNoCase User-Agent "^pd-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^penthesila.*" bad_bot
SetEnvIfNoCase User-Agent "^perform_crawl.*" bad_bot
SetEnvIfNoCase User-Agent "^PerMan.*" bad_bot
SetEnvIfNoCase User-Agent "^PerMan Surfer.*" bad_bot
SetEnvIfNoCase User-Agent "^personal ultimate crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^PHP version tracker.*" bad_bot
SetEnvIfNoCase User-Agent "^PHPCrawl.*" bad_bot
SetEnvIfNoCase User-Agent "^PhpDig.*" bad_bot
SetEnvIfNoCase User-Agent "^PicoSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^PIENO robot.*" bad_bot
SetEnvIfNoCase User-Agent "^pipBot.*" bad_bot
SetEnvIfNoCase User-Agent "^pipeLiner.*" bad_bot
SetEnvIfNoCase User-Agent "^Pita.*" bad_bot
SetEnvIfNoCase User-Agent "^pixfinder.*" bad_bot
SetEnvIfNoCase User-Agent "^PiyushBot.*" bad_bot
SetEnvIfNoCase User-Agent "^plaNETWORK Bot Search.*" bad_bot
SetEnvIfNoCase User-Agent "^Plucker.*" bad_bot
SetEnvIfNoCase User-Agent "^Plukkie.*" bad_bot
SetEnvIfNoCase User-Agent "^Plumtree 6.0.*" bad_bot
SetEnvIfNoCase User-Agent "^Pockey.*" bad_bot
SetEnvIfNoCase User-Agent "^Pockey-GetHTML.*" bad_bot
SetEnvIfNoCase User-Agent "^PoCoHTTP.*" bad_bot
SetEnvIfNoCase User-Agent "^pogodak.ba.*" bad_bot
SetEnvIfNoCase User-Agent "^Pogodak.co.yu.*" bad_bot
SetEnvIfNoCase User-Agent "^Poirot.*" bad_bot
SetEnvIfNoCase User-Agent "^polybot.*" bad_bot
SetEnvIfNoCase User-Agent "^Pompos.*" bad_bot
SetEnvIfNoCase User-Agent "^Poodle predictor.*" bad_bot
SetEnvIfNoCase User-Agent "^PopScreenBot.*" bad_bot
SetEnvIfNoCase User-Agent "^PostPost.*" bad_bot
SetEnvIfNoCase User-Agent "^PrivacyFinder.*" bad_bot
SetEnvIfNoCase User-Agent "^ProjectWF-java-test-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^ProPowerBot/2.14.*" bad_bot
SetEnvIfNoCase User-Agent "^ProWebWalker.*" bad_bot
SetEnvIfNoCase User-Agent "^Proxem WebSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^proximic.*" bad_bot
SetEnvIfNoCase User-Agent "^PROXY crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^psbot.*" bad_bot
SetEnvIfNoCase User-Agent "^PsBot.*" bad_bot
SetEnvIfNoCase User-Agent "^psbot-page.*" bad_bot
SetEnvIfNoCase User-Agent "^PSS-Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^psycheclone.*" bad_bot
SetEnvIfNoCase User-Agent "^pub-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^pucl.*" bad_bot
SetEnvIfNoCase User-Agent "^pulseBot (pulse Web Miner).*" bad_bot
SetEnvIfNoCase User-Agent "^Pump.*" bad_bot
SetEnvIfNoCase User-Agent "^PWeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Python-urllib.*" bad_bot
SetEnvIfNoCase User-Agent "^pythonic-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^PythonWikipediaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^q1.*" bad_bot
SetEnvIfNoCase User-Agent "^QEAVis agent.*" bad_bot
SetEnvIfNoCase User-Agent "^QFKBot.*" bad_bot
SetEnvIfNoCase User-Agent "^qualidade.*" bad_bot
SetEnvIfNoCase User-Agent "^Qualidator.com Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^QuepasaCreep.*" bad_bot
SetEnvIfNoCase User-Agent "^QueryN Metasearch.*" bad_bot
SetEnvIfNoCase User-Agent "^QueryN.Metasearch.*" bad_bot
SetEnvIfNoCase User-Agent "^quest.durato.*" bad_bot
SetEnvIfNoCase User-Agent "^Quintura-Crw.*" bad_bot
SetEnvIfNoCase User-Agent "^QunarBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Qweery_robot.txt_CheckBot.*" bad_bot
SetEnvIfNoCase User-Agent "^QweeryBot.*" bad_bot
SetEnvIfNoCase User-Agent "^r2iBot.*" bad_bot
SetEnvIfNoCase User-Agent "^R6_CommentReader.*" bad_bot
SetEnvIfNoCase User-Agent "^R6_FeedFetcher.*" bad_bot
SetEnvIfNoCase User-Agent "^R6_VoteReader.*" bad_bot
SetEnvIfNoCase User-Agent "^RaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Radian6.*" bad_bot
SetEnvIfNoCase User-Agent "^radian6_linkcheck.*" bad_bot
SetEnvIfNoCase User-Agent "^Radiation Retriever 1.1.*" bad_bot
SetEnvIfNoCase User-Agent "^RAMPyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^RankurBot/Rankur.*" bad_bot
SetEnvIfNoCase User-Agent "^Rational SiteCheck.*" bad_bot
SetEnvIfNoCase User-Agent "^RcStartBot.*" bad_bot
SetEnvIfNoCase User-Agent "^RealDownload.*" bad_bot
SetEnvIfNoCase User-Agent "^RealDownload/4.0.0.40.*" bad_bot
SetEnvIfNoCase User-Agent "^RealDownload/4.0.0.41.*" bad_bot
SetEnvIfNoCase User-Agent "^RealDownload/4.0.0.42.*" bad_bot
SetEnvIfNoCase User-Agent "^Reaper.*" bad_bot
SetEnvIfNoCase User-Agent "^REBI-Shoveler.*" bad_bot
SetEnvIfNoCase User-Agent "^REBI-shoveler.*" bad_bot
SetEnvIfNoCase User-Agent "^Recorder.*" bad_bot
SetEnvIfNoCase User-Agent "^RedBot.*" bad_bot
SetEnvIfNoCase User-Agent "^RedCarpet.*" bad_bot
SetEnvIfNoCase User-Agent "^ReGet.*" bad_bot
SetEnvIfNoCase User-Agent "^RepoMonkey.*" bad_bot
SetEnvIfNoCase User-Agent "^RepoMonkey Bait & Tackle/v1.01.*" bad_bot
SetEnvIfNoCase User-Agent "^research robot.*" bad_bot
SetEnvIfNoCase User-Agent "^Riddler.*" bad_bot
SetEnvIfNoCase User-Agent "^RIIGHTBOT/RIIGHT-.*" bad_bot
SetEnvIfNoCase User-Agent "^RiseNetBot.*" bad_bot
SetEnvIfNoCase User-Agent "^RiverGlassScanner.*" bad_bot
SetEnvIfNoCase User-Agent "^RiverglassScanner.*" bad_bot
SetEnvIfNoCase User-Agent "^RMA.*" bad_bot
SetEnvIfNoCase User-Agent "^RoboPal.*" bad_bot
SetEnvIfNoCase User-Agent "^Robosourcer.*" bad_bot
SetEnvIfNoCase User-Agent "^Robot.*" bad_bot
SetEnvIfNoCase User-Agent "^ROBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^robot.*" bad_bot
SetEnvIfNoCase User-Agent "^robotek.*" bad_bot
SetEnvIfNoCase User-Agent "^robots.*" bad_bot
SetEnvIfNoCase User-Agent "^Robozilla.*" bad_bot
SetEnvIfNoCase User-Agent "^rogerBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Rome Client.*" bad_bot
SetEnvIfNoCase User-Agent "^Rondello.*" bad_bot
SetEnvIfNoCase User-Agent "^Rotondo.*" bad_bot
SetEnvIfNoCase User-Agent "^Roverbot.*" bad_bot
SetEnvIfNoCase User-Agent "^RPT-HTTPClient.*" bad_bot
SetEnvIfNoCase User-Agent "^rtgibot.*" bad_bot
SetEnvIfNoCase User-Agent "^RufusBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Runnk online rss reader.*" bad_bot
SetEnvIfNoCase User-Agent "^Runnk RSS aggregator.*" bad_bot
SetEnvIfNoCase User-Agent "^s~stremor-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^S2Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^SafariBookmarkChecker.*" bad_bot
SetEnvIfNoCase User-Agent "^SafetyNet Robot 0.1.*" bad_bot
SetEnvIfNoCase User-Agent "^SaladSpoon/ShopSalad.*" bad_bot
SetEnvIfNoCase User-Agent "^Sapienti/Indexer.*" bad_bot
SetEnvIfNoCase User-Agent "^SBIder.*" bad_bot
SetEnvIfNoCase User-Agent "^SBL-BOT.*" bad_bot
SetEnvIfNoCase User-Agent "^SCFCrawler/Nutch-1.8.*" bad_bot
SetEnvIfNoCase User-Agent "^Scich.*" bad_bot
SetEnvIfNoCase User-Agent "^ScientificCommons.org.*" bad_bot
SetEnvIfNoCase User-Agent "^ScollSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^ScooperBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Scooter.*" bad_bot
SetEnvIfNoCase User-Agent "^ScoutJet.*" bad_bot
SetEnvIfNoCase User-Agent "^ScrapeBox.*" bad_bot
SetEnvIfNoCase User-Agent "^Scrapy.*" bad_bot
SetEnvIfNoCase User-Agent "^SCrawlTest.*" bad_bot
SetEnvIfNoCase User-Agent "^Scrubby.*" bad_bot
SetEnvIfNoCase User-Agent "^scSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Scumbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Search Publisher.*" bad_bot
SetEnvIfNoCase User-Agent "^search x-bot.*" bad_bot
SetEnvIfNoCase User-Agent "^Search-Channel.*" bad_bot
SetEnvIfNoCase User-Agent "^Search-Engine-Studio.*" bad_bot
SetEnvIfNoCase User-Agent "^search.KumKie.com.*" bad_bot
SetEnvIfNoCase User-Agent "^search.msn.com/msnbot.htm.*" bad_bot
SetEnvIfNoCase User-Agent "^search.updated.com.*" bad_bot
SetEnvIfNoCase User-Agent "^search.usgs.gov.*" bad_bot
SetEnvIfNoCase User-Agent "^Searcharoo.NET.*" bad_bot
SetEnvIfNoCase User-Agent "^SearchBlox.*" bad_bot
SetEnvIfNoCase User-Agent "^searchbot.*" bad_bot
SetEnvIfNoCase User-Agent "^SearchBot.*" bad_bot
SetEnvIfNoCase User-Agent "^searchengine.*" bad_bot
SetEnvIfNoCase User-Agent "^searchhippo.com.*" bad_bot
SetEnvIfNoCase User-Agent "^SearchIt-Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^searchmarking.*" bad_bot
SetEnvIfNoCase User-Agent "^searchmarks.*" bad_bot
SetEnvIfNoCase User-Agent "^searchmee_v.*" bad_bot
SetEnvIfNoCase User-Agent "^Searchmee! Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^searchmining.*" bad_bot
SetEnvIfNoCase User-Agent "^SearchnowBot_v1.*" bad_bot
SetEnvIfNoCase User-Agent "^searchpreview.*" bad_bot
SetEnvIfNoCase User-Agent "^SearchSpider.com.*" bad_bot
SetEnvIfNoCase User-Agent "^SearQuBot/SearQuBot v1.0.*" bad_bot
SetEnvIfNoCase User-Agent "^SEB Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Seekbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Seeker.lookseek.com.*" bad_bot
SetEnvIfNoCase User-Agent "^SeeqBot.*" bad_bot
SetEnvIfNoCase User-Agent "^seeqpod-vertical-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Selflinkchecker.*" bad_bot
SetEnvIfNoCase User-Agent "^Semager.*" bad_bot
SetEnvIfNoCase User-Agent "^semanticdiscovery.*" bad_bot
SetEnvIfNoCase User-Agent "^Semantifire1.*" bad_bot
SetEnvIfNoCase User-Agent "^semisearch.*" bad_bot
SetEnvIfNoCase User-Agent "^SemrushBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SEOENGWorldBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SeznamBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SeznamBot/2.0.*" bad_bot
SetEnvIfNoCase User-Agent "^SeznamBot/3.2-test1.*" bad_bot
SetEnvIfNoCase User-Agent "^ShablastBot.*" bad_bot
SetEnvIfNoCase User-Agent "^ShadowWebAnalyzer.*" bad_bot
SetEnvIfNoCase User-Agent "^Shareaza.*" bad_bot
SetEnvIfNoCase User-Agent "^Shelob.*" bad_bot
SetEnvIfNoCase User-Agent "^sherlock.*" bad_bot
SetEnvIfNoCase User-Agent "^ShopWiki.*" bad_bot
SetEnvIfNoCase User-Agent "^ShowLinks.*" bad_bot
SetEnvIfNoCase User-Agent "^ShowyouBot.*" bad_bot
SetEnvIfNoCase User-Agent "^siclab.*" bad_bot
SetEnvIfNoCase User-Agent "^silk.*" bad_bot
SetEnvIfNoCase User-Agent "^Siphon.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteArchive.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteCheck-sitecrawl by Siteimprove.com.*" bad_bot
SetEnvIfNoCase User-Agent "^sitecheck.internetseer.com.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteFinder.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteGuardBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteOrbiter.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteSnagger.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteSucker.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteSweeper.*" bad_bot
SetEnvIfNoCase User-Agent "^SiteXpert.*" bad_bot
SetEnvIfNoCase User-Agent "^SkimBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SkimWordsBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SkreemRBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Skywalker.*" bad_bot
SetEnvIfNoCase User-Agent "^Sleipnir.*" bad_bot
SetEnvIfNoCase User-Agent "^slow-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^SlySearch.*" bad_bot
SetEnvIfNoCase User-Agent "^smart-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^SmartDownload.*" bad_bot
SetEnvIfNoCase User-Agent "^SmartDownload/1.2.76 (Win32; Apr 1 1999)" bad_bot
SetEnvIfNoCase User-Agent "^SmartDownload/1.2.77 (Win32; Aug 17 1999)" bad_bot
SetEnvIfNoCase User-Agent "^SmartDownload/1.2.77 (Win32; Feb 1 2000)" bad_bot
SetEnvIfNoCase User-Agent "^SmartDownload/1.2.77 (Win32; Jun 19 2001)" bad_bot
SetEnvIfNoCase User-Agent "^Smarte Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^smartwit.com.*" bad_bot
SetEnvIfNoCase User-Agent "^Snake.*" bad_bot
SetEnvIfNoCase User-Agent "^snap.com beta crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Snapbot.*" bad_bot
SetEnvIfNoCase User-Agent "^SnapPreviewBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Snappy.*" bad_bot
SetEnvIfNoCase User-Agent "^snookit/Snookit.*" bad_bot
SetEnvIfNoCase User-Agent "^Snooper/b97_01.*" bad_bot
SetEnvIfNoCase User-Agent "^Snoopy.*" bad_bot
SetEnvIfNoCase User-Agent "^SocSciBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SOFT411 Directory.*" bad_bot
SetEnvIfNoCase User-Agent "^sogou.*" bad_bot
SetEnvIfNoCase User-Agent "^Sogou.*" bad_bot
SetEnvIfNoCase User-Agent "^sohu agent.*" bad_bot
SetEnvIfNoCase User-Agent "^sohu-search.*" bad_bot
SetEnvIfNoCase User-Agent "^Sokitomi crawl.*" bad_bot
SetEnvIfNoCase User-Agent "^Solbot/1.0 LWP/5.07.*" bad_bot
SetEnvIfNoCase User-Agent "^sootle.*" bad_bot
SetEnvIfNoCase User-Agent "^Sosospider.*" bad_bot
SetEnvIfNoCase User-Agent "^Space Bison.*" bad_bot
SetEnvIfNoCase User-Agent "^Space Fung.*" bad_bot
SetEnvIfNoCase User-Agent "^SpaceBison.*" bad_bot
SetEnvIfNoCase User-Agent "^SpankBot.*" bad_bot
SetEnvIfNoCase User-Agent "^spanner.*" bad_bot
SetEnvIfNoCase User-Agent "^Spanner/1.0 (Linux 2.0.27 i586)" bad_bot
SetEnvIfNoCase User-Agent "^Spatineo Monitor Controller.*" bad_bot
SetEnvIfNoCase User-Agent "^Spatineo Serval Controller.*" bad_bot
SetEnvIfNoCase User-Agent "^Spatineo Serval GetMapBot.*" bad_bot
SetEnvIfNoCase User-Agent "^special_archiver.*" bad_bot
SetEnvIfNoCase User-Agent "^Speedy Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^SpeedySpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Sphere Scout.*" bad_bot
SetEnvIfNoCase User-Agent "^Sphere Scout&v.*" bad_bot
SetEnvIfNoCase User-Agent "^Sphider.*" bad_bot
SetEnvIfNoCase User-Agent "^Sphider2.*" bad_bot
SetEnvIfNoCase User-Agent "^spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Spider.TerraNautic.net.*" bad_bot
SetEnvIfNoCase User-Agent "^SpiderEngine.*" bad_bot
SetEnvIfNoCase User-Agent "^SpiderKU.*" bad_bot
SetEnvIfNoCase User-Agent "^SpiderMan.*" bad_bot
SetEnvIfNoCase User-Agent "^Spinn3r.*" bad_bot
SetEnvIfNoCase User-Agent "^Spinne.*" bad_bot
SetEnvIfNoCase User-Agent "^sportcrew-Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^spyder3.microsys.com.*" bad_bot
SetEnvIfNoCase User-Agent "^SQ Webscanner.*" bad_bot
SetEnvIfNoCase User-Agent "^sqlmap.*" bad_bot
SetEnvIfNoCase User-Agent "^Squid-Prefetch.*" bad_bot
SetEnvIfNoCase User-Agent "^SquidClamAV_Redirector.*" bad_bot
SetEnvIfNoCase User-Agent "^Sqworm.*" bad_bot
SetEnvIfNoCase User-Agent "^SrevBot.*" bad_bot
SetEnvIfNoCase User-Agent "^sslbot.*" bad_bot
SetEnvIfNoCase User-Agent "^SSM Agent.*" bad_bot
SetEnvIfNoCase User-Agent "^StackRambler.*" bad_bot
SetEnvIfNoCase User-Agent "^StarDownloader.*" bad_bot
SetEnvIfNoCase User-Agent "^statbot.*" bad_bot
SetEnvIfNoCase User-Agent "^statcrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^statedept-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Steeler.*" bad_bot
SetEnvIfNoCase User-Agent "^STEGMANN-Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^stero.*" bad_bot
SetEnvIfNoCase User-Agent "^Stripper.*" bad_bot
SetEnvIfNoCase User-Agent "^Stumbler.*" bad_bot
SetEnvIfNoCase User-Agent "^suchclip.*" bad_bot
SetEnvIfNoCase User-Agent "^Sucker.*" bad_bot
SetEnvIfNoCase User-Agent "^SumeetBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SumitBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SummizeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SummizeFeedReader.*" bad_bot
SetEnvIfNoCase User-Agent "^Sunrise XP.*" bad_bot
SetEnvIfNoCase User-Agent "^SuperBot.*" bad_bot
SetEnvIfNoCase User-Agent "^superbot.com.*" bad_bot
SetEnvIfNoCase User-Agent "^SuperBot/3.0 (Win32)" bad_bot
SetEnvIfNoCase User-Agent "^SuperBot/3.1 (Win32)" bad_bot
SetEnvIfNoCase User-Agent "^SuperHTTP.*" bad_bot
SetEnvIfNoCase User-Agent "^SuperLumin Downloader.*" bad_bot
SetEnvIfNoCase User-Agent "^SuperPagesBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Supybot.*" bad_bot
SetEnvIfNoCase User-Agent "^SURF.*" bad_bot
SetEnvIfNoCase User-Agent "^Surfbot.*" bad_bot
SetEnvIfNoCase User-Agent "^SurfControl.*" bad_bot
SetEnvIfNoCase User-Agent "^SurveyBot.*" bad_bot
SetEnvIfNoCase User-Agent "^suzuran.*" bad_bot
SetEnvIfNoCase User-Agent "^SWEBot.*" bad_bot
SetEnvIfNoCase User-Agent "^swish-e.*" bad_bot
SetEnvIfNoCase User-Agent "^SygolBot.*" bad_bot
SetEnvIfNoCase User-Agent "^SynapticWalker.*" bad_bot
SetEnvIfNoCase User-Agent "^Syntryx ANT Scout Chassis Pheromone.*" bad_bot
SetEnvIfNoCase User-Agent "^SystemSearch-robot.*" bad_bot
SetEnvIfNoCase User-Agent "^Szukacz.*" bad_bot
SetEnvIfNoCase User-Agent "^Szukacz/1.4.*" bad_bot
SetEnvIfNoCase User-Agent "^T-H-U-N-D-E-R-S-T-O-N-E.*" bad_bot
SetEnvIfNoCase User-Agent "^Tailrank.*" bad_bot
SetEnvIfNoCase User-Agent "^tAkeOut.*" bad_bot
SetEnvIfNoCase User-Agent "^Talkro Web-Shot.*" bad_bot
SetEnvIfNoCase User-Agent "^TAMU_CRAWLER.*" bad_bot
SetEnvIfNoCase User-Agent "^TapuzBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Tarantula.*" bad_bot
SetEnvIfNoCase User-Agent "^targetblaster.com.*" bad_bot
SetEnvIfNoCase User-Agent "^TargetYourNews.com bot.*" bad_bot
SetEnvIfNoCase User-Agent "^TAUSDataBot.*" bad_bot
SetEnvIfNoCase User-Agent "^taxinomiabot.*" bad_bot
SetEnvIfNoCase User-Agent "^TeamSoft WinInet Component.*" bad_bot
SetEnvIfNoCase User-Agent "^Tecomi Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^TeezirBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport Pro.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport Pro/1.29.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport Pro/1.29.1590.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport Pro/1.29.1634.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport Pro/1.29.1718.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport Pro/1.29.1820.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport Pro/1.29.1847.*" bad_bot
SetEnvIfNoCase User-Agent "^TeleportPro.*" bad_bot
SetEnvIfNoCase User-Agent "^Telesoft.*" bad_bot
SetEnvIfNoCase User-Agent "^Teradex Mapper.*" bad_bot
SetEnvIfNoCase User-Agent "^TERAGRAM_CRAWLER.*" bad_bot
SetEnvIfNoCase User-Agent "^TerrawizBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TestBot.*" bad_bot
SetEnvIfNoCase User-Agent "^testbot.*" bad_bot
SetEnvIfNoCase User-Agent "^testBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^testing of bot.*" bad_bot
SetEnvIfNoCase User-Agent "^TextBot.*" bad_bot
SetEnvIfNoCase User-Agent "^thatrobotsite.com.*" bad_bot
SetEnvIfNoCase User-Agent "^The Dyslexalizer.*" bad_bot
SetEnvIfNoCase User-Agent "^The Intraformant.*" bad_bot
SetEnvIfNoCase User-Agent "^The.Intraformant.*" bad_bot
SetEnvIfNoCase User-Agent "^TheNomad.*" bad_bot
SetEnvIfNoCase User-Agent "^Theophrastus.*" bad_bot
SetEnvIfNoCase User-Agent "^theusefulbot.*" bad_bot
SetEnvIfNoCase User-Agent "^TheUsefulbot_.*" bad_bot
SetEnvIfNoCase User-Agent "^ThumbBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Thumbnail.CZ robot.*" bad_bot
SetEnvIfNoCase User-Agent "^thumbshots-de-bot.*" bad_bot
SetEnvIfNoCase User-Agent "^tigerbot.*" bad_bot
SetEnvIfNoCase User-Agent "^TightTwatBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TinEye.*" bad_bot
SetEnvIfNoCase User-Agent "^Titan.*" bad_bot
SetEnvIfNoCase User-Agent "^TITAN.*" bad_bot
SetEnvIfNoCase User-Agent "^to-dress_ru_bot_.*" bad_bot
SetEnvIfNoCase User-Agent "^to-night-Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^toCrawl/UrlDispatcher.*" bad_bot
SetEnvIfNoCase User-Agent "^Topicalizer.*" bad_bot
SetEnvIfNoCase User-Agent "^topicblogs.*" bad_bot
SetEnvIfNoCase User-Agent "^Toplistbot.*" bad_bot
SetEnvIfNoCase User-Agent "^TopServer PHP.*" bad_bot
SetEnvIfNoCase User-Agent "^topyx-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Touche.*" bad_bot
SetEnvIfNoCase User-Agent "^TourlentaScanner.*" bad_bot
SetEnvIfNoCase User-Agent "^TPSystem.*" bad_bot
SetEnvIfNoCase User-Agent "^TRAAZI.*" bad_bot
SetEnvIfNoCase User-Agent "^TranSGeniKBot.*" bad_bot
SetEnvIfNoCase User-Agent "^travel-search.*" bad_bot
SetEnvIfNoCase User-Agent "^TravelBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TravelLazerBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Treezy.*" bad_bot
SetEnvIfNoCase User-Agent "^TREX.*" bad_bot
SetEnvIfNoCase User-Agent "^TridentSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Trovator.*" bad_bot
SetEnvIfNoCase User-Agent "^True_Robot.*" bad_bot
SetEnvIfNoCase User-Agent "^tScholarsBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TSM Translation-Search-Machine.*" bad_bot
SetEnvIfNoCase User-Agent "^TsWebBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TulipChain.*" bad_bot
SetEnvIfNoCase User-Agent "^turingos.*" bad_bot
SetEnvIfNoCase User-Agent "^TurnitinBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TurnitinBot/1.5.*" bad_bot
SetEnvIfNoCase User-Agent "^TutorGigBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TweetedTimes Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^TweetmemeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TwengaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TwengaBot-Discover.*" bad_bot
SetEnvIfNoCase User-Agent "^Twiceler.*" bad_bot
SetEnvIfNoCase User-Agent "^Twikle.*" bad_bot
SetEnvIfNoCase User-Agent "^twinuffbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Twisted PageGetter.*" bad_bot
SetEnvIfNoCase User-Agent "^Twitturls.*" bad_bot
SetEnvIfNoCase User-Agent "^Twitturly .*" bad_bot
SetEnvIfNoCase User-Agent "^TygoBot.*" bad_bot
SetEnvIfNoCase User-Agent "^TygoProwler.*" bad_bot
SetEnvIfNoCase User-Agent "^Typhoeus.*" bad_bot
SetEnvIfNoCase User-Agent "^U.S. Government Printing Office.*" bad_bot
SetEnvIfNoCase User-Agent "^uberbot.*" bad_bot
SetEnvIfNoCase User-Agent "^ucb-nutch.*" bad_bot
SetEnvIfNoCase User-Agent "^UdmSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^UFAM-crawler-.*" bad_bot
SetEnvIfNoCase User-Agent "^Ultraseek.*" bad_bot
SetEnvIfNoCase User-Agent "^UnChaos.*" bad_bot
SetEnvIfNoCase User-Agent "^unchaos_crawler_.*" bad_bot
SetEnvIfNoCase User-Agent "^UnisterBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Unitek UniEngine.*" bad_bot
SetEnvIfNoCase User-Agent "^UniversalSearch.*" bad_bot
SetEnvIfNoCase User-Agent "^UnwindFetchor.*" bad_bot
SetEnvIfNoCase User-Agent "^UofTDB_experiment.*" bad_bot
SetEnvIfNoCase User-Agent "^updated.*" bad_bot
SetEnvIfNoCase User-Agent "^URL Control.*" bad_bot
SetEnvIfNoCase User-Agent "^url_gather.*" bad_bot
SetEnvIfNoCase User-Agent "^URL-Checker.*" bad_bot
SetEnvIfNoCase User-Agent "^URLAppendBot.*" bad_bot
SetEnvIfNoCase User-Agent "^URLBlaze.*" bad_bot
SetEnvIfNoCase User-Agent "^urlchecker.*" bad_bot
SetEnvIfNoCase User-Agent "^urlck/1.2.3.*" bad_bot
SetEnvIfNoCase User-Agent "^UrlDispatcher.*" bad_bot
SetEnvIfNoCase User-Agent "^URLSpiderPro.*" bad_bot
SetEnvIfNoCase User-Agent "^URLy Warning.*" bad_bot
SetEnvIfNoCase User-Agent "^URLy.Warning.*" bad_bot
SetEnvIfNoCase User-Agent "^USAF AFKN K2SPIDER.*" bad_bot
SetEnvIfNoCase User-Agent "^usasearch.*" bad_bot
SetEnvIfNoCase User-Agent "^USS-Cosmix.*" bad_bot
SetEnvIfNoCase User-Agent "^USyd-NLP-Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^Vacobot.*" bad_bot
SetEnvIfNoCase User-Agent "^Vacuum.*" bad_bot
SetEnvIfNoCase User-Agent "^VadixBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Vagabondo.*" bad_bot
SetEnvIfNoCase User-Agent "^Validator.*" bad_bot
SetEnvIfNoCase User-Agent "^Valkyrie/1.0 libwww-perl/0.40" bad_bot
SetEnvIfNoCase User-Agent "^vBSEO.*" bad_bot
SetEnvIfNoCase User-Agent "^VCI.*" bad_bot
SetEnvIfNoCase User-Agent "^VCI WebViewer VCI WebViewer Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^VerbstarBot.*" bad_bot
SetEnvIfNoCase User-Agent "^VeriCiteCrawler/Nutch-2.2.1" bad_bot
SetEnvIfNoCase User-Agent "^Verifactrola.*" bad_bot
SetEnvIfNoCase User-Agent "^Verity-URL-Gateway.*" bad_bot
SetEnvIfNoCase User-Agent "^Vermut.*" bad_bot
SetEnvIfNoCase User-Agent "^vermut.*" bad_bot
SetEnvIfNoCase User-Agent "^versus crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^versus.integis.ch.*" bad_bot
SetEnvIfNoCase User-Agent "^viasarchivinginformation.html.*" bad_bot
SetEnvIfNoCase User-Agent "^VIP.*" bad_bot
SetEnvIfNoCase User-Agent "^VIPr.*" bad_bot
SetEnvIfNoCase User-Agent "^virus_detector.*" bad_bot
SetEnvIfNoCase User-Agent "^virus-detector.*" bad_bot
SetEnvIfNoCase User-Agent "^VisBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Vishal For CLIA.*" bad_bot
SetEnvIfNoCase User-Agent "^VisWeb.*" bad_bot
SetEnvIfNoCase User-Agent "^Vital Search'n Urchin.*" bad_bot
SetEnvIfNoCase User-Agent "^vlad.*" bad_bot
SetEnvIfNoCase User-Agent "^vlsearch.*" bad_bot
SetEnvIfNoCase User-Agent "^VMBot.*" bad_bot
SetEnvIfNoCase User-Agent "^VocusBot.*" bad_bot
SetEnvIfNoCase User-Agent "^VoidEYE.*" bad_bot
SetEnvIfNoCase User-Agent "^VoilaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Vortex.*" bad_bot
SetEnvIfNoCase User-Agent "^voyager-hc.*" bad_bot
SetEnvIfNoCase User-Agent "^voyager-partner-deep.*" bad_bot
SetEnvIfNoCase User-Agent "^voyager.*" bad_bot
SetEnvIfNoCase User-Agent "^VSE.*" bad_bot
SetEnvIfNoCase User-Agent "^vspider.*" bad_bot
SetEnvIfNoCase User-Agent "^W3C_Unicorn.*" bad_bot
SetEnvIfNoCase User-Agent "^W3C-WebCon.*" bad_bot
SetEnvIfNoCase User-Agent "^w3m.*" bad_bot
SetEnvIfNoCase User-Agent "^w3search.*" bad_bot
SetEnvIfNoCase User-Agent "^wacbot.*" bad_bot
SetEnvIfNoCase User-Agent "^wastrix.*" bad_bot
SetEnvIfNoCase User-Agent "^Water Conserve Portal.*" bad_bot
SetEnvIfNoCase User-Agent "^Water Conserve Spider.*" bad_bot
SetEnvIfNoCase User-Agent "^WatzBot.*" bad_bot
SetEnvIfNoCase User-Agent "^wauuu engine/Wauuu.*" bad_bot
SetEnvIfNoCase User-Agent "^Wavefire.*" bad_bot
SetEnvIfNoCase User-Agent "^Waypath.*" bad_bot
SetEnvIfNoCase User-Agent "^Wazzup.*" bad_bot
SetEnvIfNoCase User-Agent "^Wazzup1.0.4800.*" bad_bot
SetEnvIfNoCase User-Agent "^wbdbot.*" bad_bot
SetEnvIfNoCase User-Agent "^Web CEO Online robot.*" bad_bot
SetEnvIfNoCase User-Agent "^web crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Web Downloader.*" bad_bot
SetEnvIfNoCase User-Agent "^Web Image Collector.*" bad_bot
SetEnvIfNoCase User-Agent "^Web Link Validator.*" bad_bot
SetEnvIfNoCase User-Agent "^Web Magnet.*" bad_bot
SetEnvIfNoCase User-Agent "^Web Site Downloader.*" bad_bot
SetEnvIfNoCase User-Agent "^Web Sucker.*" bad_bot
SetEnvIfNoCase User-Agent "^web-agent.*" bad_bot
SetEnvIfNoCase User-Agent "^Web-Sniffer.*" bad_bot
SetEnvIfNoCase User-Agent "^Web.Image.Collector.*" bad_bot
SetEnvIfNoCase User-Agent "^WebaltBot.*" bad_bot
SetEnvIfNoCase User-Agent "^WebAuto.*" bad_bot
SetEnvIfNoCase User-Agent "^WebAuto/3.40.*" bad_bot
SetEnvIfNoCase User-Agent "^WebBandit.*" bad_bot
SetEnvIfNoCase User-Agent "^webbandit.*" bad_bot
SetEnvIfNoCase User-Agent "^WebBandit/3.50.*" bad_bot
SetEnvIfNoCase User-Agent "^WebBOT.*" bad_bot
SetEnvIfNoCase User-Agent "^webbot.*" bad_bot
SetEnvIfNoCase User-Agent "^WebBot.*" bad_bot
SetEnvIfNoCase User-Agent "^webbul-bot.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCapture.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCapture 2.0.*" bad_bot
SetEnvIfNoCase User-Agent "^webcheck 1.*" bad_bot
SetEnvIfNoCase User-Agent "^Webclipping.com.*" bad_bot
SetEnvIfNoCase User-Agent "^webcollage.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier v.2.2.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier v2.5.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier v2.6.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier v2.7a.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier v2.8.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier v3.0.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier v3.0.1.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier v3.2.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopier v3.2a.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCopy.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCorp.*" bad_bot
SetEnvIfNoCase User-Agent "^webcrawl.net.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^webcrawler.*" bad_bot
SetEnvIfNoCase User-Agent "^WebCrawler/3.0 Robot libwww/5.0a.*" bad_bot
SetEnvIfNoCase User-Agent "^WebDownloader for X.*" bad_bot
SetEnvIfNoCase User-Agent "^Webdup.*" bad_bot
SetEnvIfNoCase User-Agent "^WebEMailExtrac.*" bad_bot
SetEnvIfNoCase User-Agent "^WebEnhancer.*" bad_bot
SetEnvIfNoCase User-Agent "^WebFetch.*" bad_bot
SetEnvIfNoCase User-Agent "^webfetch.*" bad_bot
SetEnvIfNoCase User-Agent "^WebFetcher/0.8,.*" bad_bot
SetEnvIfNoCase User-Agent "^WebGather.*" bad_bot
SetEnvIfNoCase User-Agent "^WebGo IS.*" bad_bot
SetEnvIfNoCase User-Agent "^webGobbler.*" bad_bot
SetEnvIfNoCase User-Agent "^WebImages.*" bad_bot
SetEnvIfNoCase User-Agent "^Webinator-search2.fasthealth.com.*" bad_bot
SetEnvIfNoCase User-Agent "^Webinator-WBI.*" bad_bot
SetEnvIfNoCase User-Agent "^WebIndex.*" bad_bot
SetEnvIfNoCase User-Agent "^WebIndexer.*" bad_bot
SetEnvIfNoCase User-Agent "^weblayers/0.0.*" bad_bot
SetEnvIfNoCase User-Agent "^WebLeacher.*" bad_bot
SetEnvIfNoCase User-Agent "^WeblexBot.*" bad_bot
SetEnvIfNoCase User-Agent "^WebLinker/0.0 libwww-perl/0.1.*" bad_bot
SetEnvIfNoCase User-Agent "^webLyzard.*" bad_bot
SetEnvIfNoCase User-Agent "^WebmasterCoffee.*" bad_bot
SetEnvIfNoCase User-Agent "^WebmasterWorld Extractor.*" bad_bot
SetEnvIfNoCase User-Agent "^WebmasterWorldForumBot.*" bad_bot
SetEnvIfNoCase User-Agent "^WebMiner.*" bad_bot
SetEnvIfNoCase User-Agent "^WebMoose/0.0.0000.*" bad_bot
SetEnvIfNoCase User-Agent "^WeBot.*" bad_bot
SetEnvIfNoCase User-Agent "^WebPix.*" bad_bot
SetEnvIfNoCase User-Agent "^WebReaper.*" bad_bot
SetEnvIfNoCase User-Agent "^WebRipper.*" bad_bot
SetEnvIfNoCase User-Agent "^WebSauger.*" bad_bot
SetEnvIfNoCase User-Agent "^WebSauger 1.20b.*" bad_bot
SetEnvIfNoCase User-Agent "^WebSauger 1.20j.*" bad_bot
SetEnvIfNoCase User-Agent "^WebSauger 1.20k.*" bad_bot
SetEnvIfNoCase User-Agent "^Webscan.*" bad_bot
SetEnvIfNoCase User-Agent "^websearchbench.*" bad_bot
SetEnvIfNoCase User-Agent "^WebSite.*" bad_bot
SetEnvIfNoCase User-Agent "^Website Downloader.*" bad_bot
SetEnvIfNoCase User-Agent "^Website Explorer.*" bad_bot
SetEnvIfNoCase User-Agent "^Website eXtractor.*" bad_bot
SetEnvIfNoCase User-Agent "^Website Quester.*" bad_bot
SetEnvIfNoCase User-Agent "^Website.Quester.*" bad_bot
SetEnvIfNoCase User-Agent "^websitemirror.*" bad_bot
SetEnvIfNoCase User-Agent "^WebSpear.*" bad_bot
SetEnvIfNoCase User-Agent "^websphinx.test.*" bad_bot
SetEnvIfNoCase User-Agent "^WebSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Webster.*" bad_bot
SetEnvIfNoCase User-Agent "^Webster Pro.*" bad_bot
SetEnvIfNoCase User-Agent "^Webster.Pro.*" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper.*" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper/2.03.*" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper/2.10.*" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper/2.12.*" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper/2.13.*" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper/2.15.*" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper/2.16.*" bad_bot
SetEnvIfNoCase User-Agent "^WebStripper/2.19.*" bad_bot
SetEnvIfNoCase User-Agent "^WebTrafficExpress.*" bad_bot
SetEnvIfNoCase User-Agent "^WebTrends Link Analyzer.*" bad_bot
SetEnvIfNoCase User-Agent "^WebVac.*" bad_bot
SetEnvIfNoCase User-Agent "^webwalk.*" bad_bot
SetEnvIfNoCase User-Agent "^WebWalker.*" bad_bot
SetEnvIfNoCase User-Agent "^WebWalker/1.10.*" bad_bot
SetEnvIfNoCase User-Agent "^WebWasher.*" bad_bot
SetEnvIfNoCase User-Agent "^Webwasher.*" bad_bot
SetEnvIfNoCase User-Agent "^WebWatch.*" bad_bot
SetEnvIfNoCase User-Agent "^WebWhacker.*" bad_bot
SetEnvIfNoCase User-Agent "^WebXM.*" bad_bot
SetEnvIfNoCase User-Agent "^WebZIP.*" bad_bot
SetEnvIfNoCase User-Agent "^Weddings.info Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^wenbin/search.*" bad_bot
SetEnvIfNoCase User-Agent "^WEP Search.*" bad_bot
SetEnvIfNoCase User-Agent "^WEPA.*" bad_bot
SetEnvIfNoCase User-Agent "^WeRelateBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget.*" bad_bot
SetEnvIfNoCase User-Agent "^wget.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.4.0.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.5.2.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.5.3.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.6.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.7.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.8.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.8.1.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.8.1+cvs.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.8.2.*" bad_bot
SetEnvIfNoCase User-Agent "^Wget/1.9-beta.*" bad_bot
SetEnvIfNoCase User-Agent "^Whacker.*" bad_bot
SetEnvIfNoCase User-Agent "^Whirlpool Web Engine.*" bad_bot
SetEnvIfNoCase User-Agent "^WhoWhere Robot.*" bad_bot
SetEnvIfNoCase User-Agent "^Widow.*" bad_bot
SetEnvIfNoCase User-Agent "^WikiaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Wikio.*" bad_bot
SetEnvIfNoCase User-Agent "^wikiwix-bot-.*" bad_bot
SetEnvIfNoCase User-Agent "^WinHTTP Example.*" bad_bot
SetEnvIfNoCase User-Agent "^WinHttp.WinHttpRequest..*" bad_bot
SetEnvIfNoCase User-Agent "^WIRE.*" bad_bot
SetEnvIfNoCase User-Agent "^wired-digital-newsbot/1.5.*" bad_bot
SetEnvIfNoCase User-Agent "^WISEbot.*" bad_bot
SetEnvIfNoCase User-Agent "^WISENutbot.*" bad_bot
SetEnvIfNoCase User-Agent "^wish-la.*" bad_bot
SetEnvIfNoCase User-Agent "^wish-project.*" bad_bot
SetEnvIfNoCase User-Agent "^wisponbot.*" bad_bot
SetEnvIfNoCase User-Agent "^WMCAI-robot.*" bad_bot
SetEnvIfNoCase User-Agent "^wminer.*" bad_bot
SetEnvIfNoCase User-Agent "^WMSBot.*" bad_bot
SetEnvIfNoCase User-Agent "^woriobot.*" bad_bot
SetEnvIfNoCase User-Agent "^worldshop.*" bad_bot
SetEnvIfNoCase User-Agent "^WorQmada.*" bad_bot
SetEnvIfNoCase User-Agent "^Wotbox.*" bad_bot
SetEnvIfNoCase User-Agent "^wume_crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^WWW Collector.*" bad_bot
SetEnvIfNoCase User-Agent "^WWW-Collector-E.*" bad_bot
SetEnvIfNoCase User-Agent "^WWW-Mechanize.*" bad_bot
SetEnvIfNoCase User-Agent "^WWWOFFLE.*" bad_bot
SetEnvIfNoCase User-Agent "^wwwrobot.*" bad_bot
SetEnvIfNoCase User-Agent "^wwwster.*" bad_bot
SetEnvIfNoCase User-Agent "^WWWWanderer v3.0.*" bad_bot
SetEnvIfNoCase User-Agent "^wwwxref.*" bad_bot
SetEnvIfNoCase User-Agent "^Wysigot.*" bad_bot
SetEnvIfNoCase User-Agent "^X-clawler.*" bad_bot
SetEnvIfNoCase User-Agent "^X-crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Xaldon.*" bad_bot
SetEnvIfNoCase User-Agent "^Xaldon WebSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Xaldon WebSpider 2.5.b3.*" bad_bot
SetEnvIfNoCase User-Agent "^Xaldon_WebSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Xenu.*" bad_bot
SetEnvIfNoCase User-Agent "^Xenu's.*" bad_bot
SetEnvIfNoCase User-Agent "^Xenu's Link Sleuth 1.1c.*" bad_bot
SetEnvIfNoCase User-Agent "^Xerka MetaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Xerka WebBot.*" bad_bot
SetEnvIfNoCase User-Agent "^XGET/0.7.*" bad_bot
SetEnvIfNoCase User-Agent "^xirq.*" bad_bot
SetEnvIfNoCase User-Agent "^XmarksFetch.*" bad_bot
SetEnvIfNoCase User-Agent "^xqrobot.*" bad_bot
SetEnvIfNoCase User-Agent "^Y!J-BRI.*" bad_bot
SetEnvIfNoCase User-Agent "^Y!J-BRJ/YATS crawler.*" bad_bot
SetEnvIfNoCase User-Agent "^Y!J-BRW.*" bad_bot
SetEnvIfNoCase User-Agent "^Y!J-BSC.*" bad_bot
SetEnvIfNoCase User-Agent "^Y!J-PSC.*" bad_bot
SetEnvIfNoCase User-Agent "^Y!J-SRD.*" bad_bot
SetEnvIfNoCase User-Agent "^Y!J.*" bad_bot
SetEnvIfNoCase User-Agent "^Y!TunnelPro.*" bad_bot
SetEnvIfNoCase User-Agent "^yacy.net.*" bad_bot
SetEnvIfNoCase User-Agent "^yacybot.*" bad_bot
SetEnvIfNoCase User-Agent "^Yandex.*" bad_bot
SetEnvIfNoCase User-Agent "^YandexBot.*" bad_bot
SetEnvIfNoCase User-Agent "^YandexSomething.*" bad_bot
SetEnvIfNoCase User-Agent "^Yanga WorldSearch Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^yarienavoir.net.*" bad_bot
SetEnvIfNoCase User-Agent "^Yasaklibot.*" bad_bot
SetEnvIfNoCase User-Agent "^yBot.*" bad_bot
SetEnvIfNoCase User-Agent "^YebolBot.*" bad_bot
SetEnvIfNoCase User-Agent "^yellowJacket.*" bad_bot
SetEnvIfNoCase User-Agent "^yes.*" bad_bot
SetEnvIfNoCase User-Agent "^YesupBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Yeti.*" bad_bot
SetEnvIfNoCase User-Agent "^YioopBot.*" bad_bot
SetEnvIfNoCase User-Agent "^yolinkBot.*" bad_bot
SetEnvIfNoCase User-Agent "^yoogliFetchAgent.*" bad_bot
SetEnvIfNoCase User-Agent "^Yoono.*" bad_bot
SetEnvIfNoCase User-Agent "^yoono.*" bad_bot
SetEnvIfNoCase User-Agent "^Yoriwa.*" bad_bot
SetEnvIfNoCase User-Agent "^YottaCars_Bot.*" bad_bot
SetEnvIfNoCase User-Agent "^you-dir.*" bad_bot
SetEnvIfNoCase User-Agent "^Z-Add Link Checker.*" bad_bot
SetEnvIfNoCase User-Agent "^zagrebin.*" bad_bot
SetEnvIfNoCase User-Agent "^Zao.*" bad_bot
SetEnvIfNoCase User-Agent "^zedzo.digest.*" bad_bot
SetEnvIfNoCase User-Agent "^zedzo.validate.*" bad_bot
SetEnvIfNoCase User-Agent "^zermelo.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 11389 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 11652 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 18018 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 26378 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 30747 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 32297 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 39206 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 41641 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 44238 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 51070 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 51674 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 51837 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 63567 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 6694 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 82016 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 82900 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 84842 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 90872 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 94934 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 95245 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 95351 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus 97371 Webster Pro V2.9 Win32.*" bad_bot
SetEnvIfNoCase User-Agent "^Zeus Link Scout.*" bad_bot
SetEnvIfNoCase User-Agent "^zibber-v.*" bad_bot
SetEnvIfNoCase User-Agent "^zimeno.*" bad_bot
SetEnvIfNoCase User-Agent "^Zing-BottaBot.*" bad_bot
SetEnvIfNoCase User-Agent "^ZipppBot.*" bad_bot
SetEnvIfNoCase User-Agent "^ZoomSpider.*" bad_bot
SetEnvIfNoCase User-Agent "^Zotag Search.*" bad_bot
SetEnvIfNoCase User-Agent "^ZuiBot.*" bad_bot
SetEnvIfNoCase User-Agent "^ZumBot.*" bad_bot
SetEnvIfNoCase User-Agent "^Zyborg.*" bad_bot
SetEnvIfNoCase User-Agent "^ZyBorg.*" bad_bot
SetEnvIfNoCase User-Agent "^Zyte.*" bad_bot
<Limit GET POST PUT>
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Limit>
</IfModule>
# End Bad Bot Prevention
@ADDISON74
Copy link

ADDISON74 commented Jul 2, 2022 •
edited
Loading

Thank you for the long list. A few issues can be improved:

  1. There are a lot duplicate lines.

  2. As long as you use SetEnvInfNoCase directive the following lines do the same thing:

SetEnvIfNoCase User-Agent "^Yoono.*" bad_bot
SetEnvIfNoCase User-Agent "^yoono.*" bad_bot
  1. A lot of them can be grouped and removed. The third line in the following example covers the first two.
SetEnvIfNoCase User-Agent "^LinkCheck by Siteimprove.com.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkCheck Scanner.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkChecker.*" bad_bot
  1. I would removed the versions for some bots to match their names without versions. I would also shorten the whole name of the bot by limiting it to 1 - 2 words. For example:
SetEnvIfNoCase User-Agent "^NetSeer crawler/2.0.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport Pro/1.29.1847.*" bad_bot

becomes

SetEnvIfNoCase User-Agent "^NetSeer.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport.*" bad_bot
  1. For a better readability SetEnvIfNoCase directive can be replaced by BrowserMatchNoCase directive. For example:
SetEnvIfNoCase User-Agent "^NPbot.*" bad_bot

becomes

BrowserMatchNocase ^NPbot(.*) bad_bot

As a personal observation your list seems to be created in 2014 and it is a very long one that can decrease your webserver performance. I would start with a short new list based on this online generator https://www.htaccessredirect.net/block-bots. Then I will check daily my webserver log with this command.

awk -F'"' '/GET/ {print $6}' /path_to_webserver_log/file_name.log | cut -d' ' -f1 | sort | uniq -c | sort -rn

The output provides all the USER_AGENTS that crawl your website and for how many times. Add in the .htaccess file any of them and in a month the bandwidth will decrease a lot.

If you want to know more about some bots you find in the webserver log here is a long list https://user-agents.net/bots.

@drekzikko
Copy link

Thank you for the long list. A few issues can be improved:

  1. There are a lot duplicate lines.
  2. As long as you use SetEnvInfNoCase directive the following lines do the same thing:
SetEnvIfNoCase User-Agent "^Yoono.*" bad_bot
SetEnvIfNoCase User-Agent "^yoono.*" bad_bot
  1. A lot of them can be grouped and removed. The third line in the following example covers the first two.
SetEnvIfNoCase User-Agent "^LinkCheck by Siteimprove.com.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkCheck Scanner.*" bad_bot
SetEnvIfNoCase User-Agent "^LinkChecker.*" bad_bot
  1. I would removed the versions for some bots to match their names without versions. I would also shorten the whole name of the bot by limiting it to 1 - 2 words. For example:
SetEnvIfNoCase User-Agent "^NetSeer crawler/2.0.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport Pro/1.29.1847.*" bad_bot

becomes

SetEnvIfNoCase User-Agent "^NetSeer.*" bad_bot
SetEnvIfNoCase User-Agent "^Teleport.*" bad_bot
  1. For a better readability SetEnvIfNoCase directive can be replaced by BrowserMatchNoCase directive. For example:
SetEnvIfNoCase User-Agent "^NPbot.*" bad_bot

becomes

BrowserMatchNocase ^NPbot(.*) bad_bot

As a personal observation your list seems to be created in 2014 and it is a very long one that can decrease your webserver performance. I would start with a short new list based on this online generator https://www.htaccessredirect.net/block-bots. Then I will check daily my webserver log with this command.

awk -F'"' '/GET/ {print $6}' /path_to_webserver_log/file_name.log | cut -d' ' -f1 | sort | uniq -c | sort -rn

The output provides all the USER_AGENTS that crawl your website and for how many times. Add in the .htaccess file any of them and in a month the bandwidth will decrease a lot.

If you want to know more about some bots you find in the webserver log here is a long list https://user-agents.net/bots.

are u done with that? can u please drop the .htaccess please

@drekzikko
Copy link

@ADDISON74 please

@askiiart
Copy link

askiiart commented Aug 8, 2023

OpenAI bot: GPTBot

@super-duper-dan
Copy link

I would add Bytespider to the list.

@askiiart
Copy link

askiiart commented Aug 28, 2023 •
edited
Loading

I would add Bytespider to the list.

IIRC, and if Google hasn't failed me, Bytespider doesn't respect robots.txt. Instead, I use this nginx config to block it:

if ($http_user_agent ~ (Bytespider)) {
    return 418;
}

A more appropriate HTTP status code would probably be 403, but 418 is funnier.

@danci1973
Copy link

A couple of suggestions based on my logs (including percentage of all requests in last 14 days):

For comparison - all Google bots combined (which, unfortunately, we have to live with) account for 1,21%.

@askiiart
Copy link

And here's a link for blocking all AI scrapers: https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/

@rayanmaia
Copy link

The facebookexternalhit bot was simply destroying my server, extremely auto CPU consumption.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment