Last active
March 25, 2024 04:46
-
-
Save solaceten/3a0ac3bfbb90d540f81dee7263ede36d to your computer and use it in GitHub Desktop.
Block those bad bots in CSF and Mod Sec
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Block Bad Bots in CSF and MOD SEC - easy | |
| # 1) install CSF and Mod Sec | |
| # 2) Set up the mod sec rules - instructions here (note these are for Direct admin, they will likely work on other systems too) | |
| # https://www.vpsbasics.com/security/how-to-block-bad-bots-using-modsecurity-with-directadmin/ | |
| # 3) get bad bots list from Ultimate Bad Bot | |
| # 3.1) Go Here: https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/tree/master/_generator_lists | |
| # 3.2) Open the latest bad-user-agents-htaccess.list and view the "raw" | |
| # 3.3) copy to your bad-botlist.txt file | |
| # 4) install - then restart csf and ldf and watch your logs... | |
| # hint: if you set LF_MODSEC in the csf firewall ( e.g. set it as LF_MODSEC = 5 and LF_MODSEC_PERM = 86400 (which is 24hrs) then CSF will add persistent offenders to the block list.... | |
| # 5) for additional direct protection, you can use this method - but they downside of using it is it doesn't show enough detail for tracking, in the error logs: | |
| # https://www.geekytuts.net/block-bad-bots-cpanel-globally-apache/ | |
| # Add the blackbots - Manually added 'favourites' | |
| # | |
| AhrefsBot | |
| Baiduspider | |
| Baiduspider-video | |
| Baiduspider-image | |
| Baiduspider-ads | |
| Baiduspider-cpro | |
| Baiduspider-news | |
| Baiduspider-favo | |
| Petalbot | |
| Yandex | |
| # | |
| # | |
| # BLACKBOTS list from Ultimate Bad Bot list - copy and paste manually | |
| # https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/_generator_lists/bad-user-agents-htaccess.list | |
| # | |
| 360Spider | |
| 80legs | |
| Abonti | |
| Aboundex | |
| Acunetix | |
| ADmantX | |
| AhrefsBot | |
| AIBOT | |
| AiHitBot | |
| Aipbot | |
| Alexibot | |
| Alligator | |
| AllSubmitter | |
| Anarchie | |
| Apexoo | |
| ASPSeek | |
| Asterias | |
| Attach | |
| autoemailspider | |
| BackDoorBot | |
| Backlink-Ceck | |
| backlink-check | |
| BacklinkCrawler | |
| BackStreet | |
| BackWeb | |
| Badass | |
| Bandit | |
| Barkrowler | |
| BatchFTP | |
| Battleztar Bazinga | |
| BBBike | |
| BDFetch | |
| BetaBot | |
| Bigfoot | |
| Bitacle | |
| Blackboard | |
| Black Hole | |
| BlackWidow | |
| BLEXBot | |
| Blow | |
| BlowFish | |
| Boardreader | |
| Bolt | |
| BotALot | |
| Brandprotect | |
| BrandProtect | |
| Brandwatch | |
| Bubing | |
| Buddy | |
| BuiltBotTough | |
| BuiltWith | |
| Bullseye | |
| BunnySlippers | |
| BuzzSumo | |
| Calculon | |
| CATExplorador | |
| CazoodleBot | |
| CCBot | |
| Cegbfeieh | |
| CheeseBot | |
| CherryPicker | |
| ChinaClaw | |
| Chlooe | |
| Claritybot | |
| Cliqzbot | |
| Cogentbot | |
| cognitiveseo | |
| Collector | |
| com.plumanalytics | |
| Copier | |
| CopyRightCheck | |
| Copyscape | |
| Cosmos | |
| Craftbot | |
| CrazyWebCrawler | |
| CRAZYWEBCRAWLER | |
| Crescent | |
| CSHttp | |
| Curious | |
| Custo | |
| DatabaseDriverMysqli | |
| DataCha0s | |
| DBLBot | |
| demandbase-bot | |
| Demon | |
| Deusu | |
| Devil | |
| Digincore | |
| DIIbot | |
| Dirbuster | |
| Disco | |
| Discobot | |
| Discoverybot | |
| DittoSpyder | |
| DomainAppender | |
| DomainCrawler | |
| DomainSigmaCrawler | |
| DomainStatsBot | |
| Dotbot | |
| Download\ Demon | |
| Download\ Devil | |
| Download\ Wonder | |
| Dragonfly | |
| Drip | |
| DTS\ Agent | |
| EasyDL | |
| Ebingbong | |
| eCatch | |
| ECCP/1.0 | |
| Ecxi | |
| EirGrabber | |
| EMail\ Collector | |
| EMail\ Extractor | |
| EMail\ Siphon | |
| EMail\ Wolf | |
| EroCrawler | |
| Evil | |
| Exabot | |
| Express\ WebPictures | |
| Extractor | |
| ExtractorPro | |
| Extreme\ Picture\ Finder | |
| EyeNetIE | |
| Ezooms | |
| FDM | |
| FHscan | |
| Fimap | |
| Firefox/7.0 | |
| FlashGet | |
| Flunky | |
| Foobot | |
| fq | |
| Freeuploader | |
| FrontPage | |
| Fyrebot | |
| GalaxyBot | |
| Genieo | |
| GermCrawler | |
| Getintent | |
| GetRight | |
| GetWeb | |
| Gigablast | |
| Gigabot | |
| G-i-g-a-b-o-t | |
| G\-i\-g\-a\-b\-o\-t | |
| Go-Ahead-Got-It | |
| Gotit | |
| GoZilla | |
| Go!Zilla | |
| Grabber | |
| GrabNet | |
| Grafula | |
| GrapeFX | |
| GrapeshotCrawler | |
| GridBot | |
| GT::WWW | |
| HaosouSpider | |
| Harvest | |
| Havij | |
| HEADMasterSEO | |
| Heritrix | |
| Hloader | |
| HMView | |
| HTMLparser | |
| HTTP::Lite | |
| HTTrack | |
| Humanlinks | |
| HybridBot | |
| Iblog | |
| IDBot | |
| Id-search | |
| IlseBot | |
| Image\ Fetch | |
| Image\ Stripper | |
| Image\ Sucker | |
| Indy\ Library | |
| InfoNaviRobot | |
| InfoTekies | |
| instabid | |
| Intelliseek | |
| InterGET | |
| Internet\ Ninja | |
| InternetSeer | |
| internetVista\ monitor | |
| Iria | |
| IRLbot | |
| Iskanie | |
| JamesBOT | |
| Jbrofuzz | |
| JennyBot | |
| JetCar | |
| JikeSpider | |
| JOC\ Web\ Spider | |
| Joomla | |
| Jorgee | |
| JustView | |
| Jyxobot | |
| Kenjin\ Spider | |
| Keyword\ Density | |
| Kozmosbot | |
| Lanshanbot | |
| Larbin | |
| LeechFTP | |
| LeechGet | |
| LexiBot | |
| Lftp | |
| LibWeb | |
| Libwhisker | |
| Lightspeedsystems | |
| Likse | |
| Linkdexbot | |
| LinkextractorPro | |
| LinkpadBot | |
| LinkScan | |
| LinksManager | |
| LinkWalker | |
| LinqiaMetadataDownloaderBot | |
| LinqiaRSSBot | |
| LinqiaScrapeBot | |
| Lipperhey | |
| Litemage_walker | |
| Lmspider | |
| LNSpiderguy | |
| Ltx71 | |
| lwp-request | |
| LWP::Simple | |
| lwp-trivial | |
| Magnet | |
| Mag-Net | |
| magpie-crawler | |
| Mail.ru | |
| Majestic12 | |
| MarkMonitor | |
| MarkWatch | |
| Masscan | |
| Mass\ Downloader | |
| Mata\ Hari | |
| Meanpathbot | |
| mediawords | |
| MegaIndex.ru | |
| Metauri | |
| MFC_Tear_Sample | |
| Microsoft\ Data\ Access | |
| Microsoft\ URL\ Control | |
| MIDown\ tool | |
| MIIxpc | |
| Mister\ PiX | |
| MJ12bot | |
| Mojeek | |
| Morfeus\ Fucking\ Scanner | |
| MSFrontPage | |
| MSIE\ 6.0 | |
| MSIECrawler | |
| Msrabot | |
| MS\ Web\ Services\ Client\ Protocol | |
| Musobot | |
| Name\ Intelligence | |
| Nameprotect | |
| Navroad | |
| NearSite | |
| Needle | |
| Nessus | |
| NetAnts | |
| Netcraft | |
| netEstate\ NE\ Crawler | |
| NetLyzer | |
| NetMechanic | |
| NetSpider | |
| Nettrack | |
| Net\ Vampire | |
| Netvibes | |
| NetZIP | |
| NextGenSearchBot | |
| Nibbler | |
| NICErsPRO | |
| Niki-bot | |
| Nikto | |
| NimbleCrawler | |
| Ninja | |
| Nmap | |
| NPbot | |
| Nutch | |
| Octopus | |
| Offline\ Explorer | |
| Offline\ Navigator | |
| Openfind | |
| OpenLinkProfiler | |
| Openvas | |
| OrangeBot | |
| OrangeSpider | |
| OutfoxBot | |
| PageAnalyzer | |
| Page\ Analyzer | |
| PageGrabber | |
| Page\ Grabber | |
| page\ scorer | |
| PageScorer | |
| Panscient | |
| Papa\ Foto | |
| Pavuk | |
| pcBrowser | |
| PECL::HTTP | |
| PeoplePal | |
| PHPCrawl | |
| Picscout | |
| Picsearch | |
| PictureFinder | |
| Pimonster | |
| Pi-Monster | |
| Pixray | |
| PleaseCrawl | |
| plumanalytics | |
| Pockey | |
| POE-Component-Client-HTTP | |
| Probethenet | |
| ProPowerBot | |
| ProWebWalker | |
| Proximic | |
| Psbot | |
| Pump | |
| PyCurl | |
| QueryN\ Metasearch | |
| Qwantify | |
| RankActive | |
| RankActiveLinkBot | |
| RankFlex | |
| RankingBot | |
| RankingBot2 | |
| Rankivabot | |
| RankurBot | |
| RealDownload | |
| Reaper | |
| RebelMouse | |
| Recorder | |
| RedesScrapy | |
| ReGet | |
| RepoMonkey | |
| Ripper | |
| RocketCrawler | |
| Rogerbot | |
| SalesIntelligent | |
| SBIder | |
| ScanAlert | |
| Scanbot | |
| Scrapy | |
| Screaming | |
| Screaming\ Frog\ SEO\ Spider | |
| ScreenerBot | |
| Searchestate | |
| SearchmetricsBot | |
| Semrush | |
| SemrushBot | |
| SEOkicks | |
| SEOkicks-Robot | |
| SEOlyticsCrawler | |
| Seomoz | |
| SEOprofiler | |
| seoscanners | |
| SEOstats | |
| Siphon | |
| SISTRIX | |
| SISTRIX\ Crawler | |
| Sitebeam | |
| SiteExplorer | |
| Siteimprove | |
| SiteLockSpider | |
| SiteSnagger | |
| SiteSucker | |
| Site\ Sucker | |
| Sitevigil | |
| Slackbot-LinkExpanding | |
| SlySearch | |
| SmartDownload | |
| Snake | |
| Snapbot | |
| Snoopy | |
| SocialRankIOBot | |
| Sogou\ web\ spider | |
| Sosospider | |
| Sottopop | |
| SpaceBison | |
| Spammen | |
| SpankBot | |
| Spanner | |
| Spbot | |
| Spinn3r | |
| SputnikBot | |
| Sqlmap | |
| Sqlworm | |
| Sqworm | |
| Steeler | |
| Stripper | |
| Sucker | |
| Sucuri | |
| SuperBot | |
| SuperHTTP | |
| Surfbot | |
| SurveyBot | |
| Suzuran | |
| Swiftbot | |
| sysscan | |
| Szukacz | |
| T0PHackTeam | |
| T8Abot | |
| tAkeOut | |
| Teleport | |
| TeleportPro | |
| Telesoft | |
| Telesphoreo | |
| Telesphorep | |
| The\ Intraformant | |
| TheNomad | |
| TightTwatBot | |
| Titan | |
| Toata | |
| Toweyabot | |
| Trendictionbot | |
| True_Robot | |
| Turingos | |
| TurnitinBot | |
| Turnitin\ Bot | |
| Turnitin\ Robot | |
| TwengaBot | |
| Twice | |
| Typhoeus | |
| UnisterBot | |
| URLy.Warning | |
| URLy\ Warning | |
| Vacuum | |
| Vagabondo | |
| VB\ Project | |
| VCI | |
| VeriCiteCrawler | |
| VidibleScraper | |
| Virusdie | |
| VoidEYE | |
| Voil | |
| Voltron | |
| Wallpapers/3.0 | |
| WallpapersHD | |
| WASALive-Bot | |
| WBSearchBot | |
| Webalta | |
| WebAuto | |
| Web\ Auto | |
| WebBandit | |
| Web\ Bandit | |
| WebCollage | |
| Web\ Collage | |
| WebCopier | |
| Web\ Copier | |
| WEBDAV | |
| WEBDAV\ Client | |
| WebEnhancer | |
| Web\ Enhancer | |
| WebFetch | |
| Web\ Fetch | |
| WebFuck | |
| Web\ Fuck | |
| WebGo\ IS | |
| WebImageCollector | |
| Web\ Image\ Collector | |
| WebLeacher | |
| WebmasterWorldForumBot | |
| webmeup-crawler | |
| WebPix | |
| Web\ Pix | |
| WebReaper | |
| Web\ Reaper | |
| WebSauger | |
| Web\ Sauger | |
| Webshag | |
| WebsiteExtractor | |
| Website\ Extractor | |
| WebsiteQuester | |
| Website\ Quester | |
| Webster | |
| WebStripper | |
| Web\ Stripper | |
| WebSucker | |
| Web\ Sucker | |
| WebWhacker | |
| Web\ Whacker | |
| WebZIP | |
| WeSEE | |
| Whack | |
| Whacker | |
| Whatweb | |
| Widow | |
| WinHTTrack | |
| WiseGuys\ Robot | |
| WISENutbot | |
| Wonderbot | |
| Woobot | |
| Wotbox | |
| Wprecon | |
| WPScan | |
| WWW-Collector-E | |
| WWW-Mechanize | |
| WWW::Mechanize | |
| WWWOFFLE | |
| x09Mozilla | |
| x22Mozilla | |
| Xaldon_WebSpider | |
| Xaldon\ WebSpider | |
| Xenu | |
| YoudaoBot | |
| Zade | |
| Zermelo | |
| Zeus | |
| Zgrab | |
| Zitebot | |
| ZmEu | |
| ZumBot | |
| ZyBorg |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment