Skip to content

Instantly share code, notes, and snippets.

@juslintek
Created April 10, 2018 00:37
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save juslintek/e21f07a5ec563cc5395b310e51083152 to your computer and use it in GitHub Desktop.
Save juslintek/e21f07a5ec563cc5395b310e51083152 to your computer and use it in GitHub Desktop.
Ka blokuoja servas
map $http_user_agent $bad_bot {
default 0;
"~*Baidu" 1;
"~*slurp" 1;
"~*Spaidu" 1;
"~*YandexBot" 1;
"~*YandexImages" 1;
"~*80legs" 1;
"~*360Spider" 1;
"~*Aboundex" 1;
"~*Abonti" 1;
"~*AhrefsBot" 1;
"~*AIBOT" 1;
"~*aipbot" 1;
"~*aiHitBot" 1;
"~*Alexibot" 1;
"~*Alligator" 1;
"~*AllSubmitter" 1;
"~*Anarchie" 1;
"~*Apexoo" 1;
"~*asterias" 1;
"~*ASPSeek" 1;
"~*attach" 1;
"~*autoemailspider" 1;
"~*BackDoorBot" 1;
"~*BackStreet" 1;
"~*BackWeb" 1;
"~*Badass" 1;
"~*Bandit" 1;
"~*BatchFTP" 1;
"~*Battleztar\ Bazinga" 1;
"~*BBBike" 1;
"~*BDFetch" 1;
"~*Bigfoot" 1;
"~*Bitacle" 1;
"~*Black\ Hole" 1;
"~*BlackWidow" 1;
"~*BLEXBot" 1;
"~*BlowFish" 1;
"~*Blow" 1;
"~*boardreader" 1;
"~*BotALot" 1;
"~*Brandprotect" 1;
"~*Brandwatch" 1;
"~*Buddy" 1;
"~*BuiltBotTough" 1;
"~*BuiltWith" 1;
"~*Bullseye" 1;
"~*BunnySlippers" 1;
"~*calculon\ spider" 1;
"~*CCBot" 1;
"~*Cegbfeieh" 1;
"~*CheeseBot" 1;
"~*CherryPicker" 1;
"~*ChinaClaw" 1;
"~*Claritybot" 1;
"~*Cliqzbot" 1;
"~*Cogentbot" 1;
"~*CPython" 1;
"~*Collector" 1;
"~*Copier" 1;
"~*CopyRightCheck" 1;
"~*Copyscape" 1;
"~*Cosmos" 1;
"~*craftbot@yahoo.com" 1;
"~*CrazyWebCrawler" 1;
"~*Crescent" 1;
"~*CSHttp" 1;
"~*Custo" 1;
"~*DBLBot" 1;
"~*Demon" 1;
"~*deusu" 1;
"~*Devil" 1;
"~*Disco" 1;
"~*DIIbot" 1;
"~*dirbuster" 1;
"~*discobot" 1;
"~*discoverybot" 1;
"~*DittoSpyder" 1;
"~*DomainAppender" 1;
"~*DomainSigmaCrawler" 1;
"~*dotbot" 1;
"~*Download\ Demon" 1;
"~*Download\ Devil" 1;
"~*Download\ Wonder" 1;
"~*dragonfly" 1;
"~*Drip" 1;
"~*DTS\ Agent" 1;
"~*EasyDL" 1;
"~*ebingbong" 1;
"~*eCatch" 1;
"~*ECCP/1.0" 1;
"~*EirGrabber" 1;
"~*EMail\ Collector" 1;
"~*EmailCollector" 1;
"~*EMail\ Extractor" 1;
"~*EMail\ Siphon" 1;
"~*EmailSiphon" 1;
"~*EMail\ Wolf" 1;
"~*EmailWolf" 1;
"~*EroCrawler" 1;
"~*Evil" 1;
"~*Exabot" 1;
"~*Express\ WebPictures" 1;
"~*Extractor" 1;
"~*ExtractorPro" 1;
"~*EyeNetIE" 1;
"~*ezooms" 1;
"~*fimap" 1;
"~*FlashGet" 1;
"~*flunky" 1;
"~*Foobot" 1;
"~*freeuploader" 1;
"~*FrontPage" 1;
"~*GalaxyBot" 1;
"~*Genieo" 1;
"~*getintent" 1;
"~*GetRight" 1;
"~*GetWeb" 1;
"~*gigablast" 1;
"~*gigabot" 1;
"~*Go-Ahead-Got-It" 1;
"~*gotit" 1;
"~*Go!Zilla" 1;
"~*Grabber" 1;
"~*GrabNet" 1;
"~*Grafula" 1;
"~*grapeFX" 1;
"~*GrapeshotCrawler" 1;
"~*GridBot" 1;
"~*HaosouSpider" 1;
"~*Harvest" 1;
"~*Havij" 1;
"~*Heritrix" 1;
"~*hloader" 1;
"~*HMView" 1;
"~*HTTrack" 1;
"~*HTMLparser" 1;
"~*Huaweisymantecspider" 1;
"~*humanlinks" 1;
"~*iblog" 1;
"~*IlseBot" 1;
"~*imagefetch" 1;
"~*Image\ Fetch" 1;
"~*Image\ Stripper" 1;
"~*Image\ Sucker" 1;
"~*Indy\ Library" 1;
"~*InfoNaviRobot" 1;
"~*InfoTekies" 1;
"~*Intelliseek" 1;
"~*Internet\ Ninja" 1;
"~*internetVista\ monitor" 1;
"~*InterGET" 1;
"~*Iria" 1;
"~*iskanie" 1;
"~*JamesBOT" 1;
"~*JennyBot" 1;
"~*JetCar" 1;
"~*jbrofuzz" 1;
"~*JikeSpider" 1;
"~*JOC" 1;
"~*JOC\ Web\ Spider" 1;
"~*Joomla" 1;
"~*JustView" 1;
"~*Jyxobot" 1;
"~*Kenjin\ Spider" 1;
"~*Keyword\ Density" 1;
"~*kraken" 1;
"~*lanshanbot" 1;
"~*larbin" 1;
"~*LeechFTP" 1;
"~*LeechGet" 1;
"~*LexiBot" 1;
"~*lftp" 1;
"~*libWeb" 1;
"~*libwhisker" 1;
"~*likse" 1;
"~*linkdexbot" 1;
"~*LinkpadBot" 1;
"~*LinkextractorPro" 1;
"~*LinkScan" 1;
"~*LinqiaMetadataDownloaderBot" 1;
"~*LinqiaScrapeBot" 1;
"~*LinqiaRSSBot" 1;
"~*lmspider" 1;
"~*LNSpiderguy" 1;
"~*LinkWalker" 1;
"~*ltx71" 1;
"~*lwp-request" 1;
"~*LWP::Simple" 1;
"~*magpie-crawler" 1;
"~*Magnet" 1;
"~*Mag-Net" 1;
"~*mail.ru" 1;
"~*Mail.RU_Bot" 1;
"~*majestic12" 1;
"~*MarkMonitor" 1;
"~*MarkWatch" 1;
"~*Mass\ Downloader" 1;
"~*Mata\ Hari" 1;
"~*MegaIndex.ru" 1;
"~*Metauri" 1;
"~*Microsoft\ Data\ Access" 1;
"~*Microsoft\ URL\ Control" 1;
"~*MIDown\ tool" 1;
"~*MIIxpc" 1;
"~*Mister\ PiX" 1;
"~*MJ12bot" 1;
"~*mojeek" 1;
"~*MSIECrawler" 1;
"~*MSIE\ 6.0" 1;
"~*MS\ Web\ Services\ Client\ Protocol" 1;
"~*msrabot" 1;
"~*musobot" 1;
"~*Name\ Intelligence" 1;
"~*Nameprotect" 1;
"~*Navroad" 1;
"~*NearSite" 1;
"~*needle" 1;
"~*nessus" 1;
"~*NetAnts" 1;
"~*Netcraft" 1;
"~*netEstate\ NE\ Crawler" 1;
"~*NetMechanic" 1;
"~*NetSpider" 1;
"~*nettrack" 1;
"~*Net\ Vampire" 1;
"~*NetZIP" 1;
"~*NextGenSearchBot" 1;
"~*nibbler" 1;
"~*NICErsPRO" 1;
"~*niki-bot" 1;
"~*nikto" 1;
"~*NimbleCrawler" 1;
"~*Ninja" 1;
"~*Nmap" 1;
"~*NPbot" 1;
"~*Nutch" 1;
"~*Octopus" 1;
"~*Offline\ Explorer" 1;
"~*Offline\ Navigator" 1;
"~*omgilibot" 1;
"~*OpenLinkProfiler" 1;
"~*Openfind" 1;
"~*Openvas" 1;
"~*OrangeBot" 1;
"~*OrangeSpider" 1;
"~*OutfoxBot" 1;
"~*PageAnalyzer" 1;
"~*Page\ Analyzer" 1;
"~*PageGrabber" 1;
"~*Page\ Grabber" 1;
"~*Papa\ Foto" 1;
"~*pavuk" 1;
"~*pcBrowser" 1;
"~*PECL::HTTP" 1;
"~*Picscout" 1;
"~*Picsearch" 1;
"~*pimonster" 1;
"~*Pi-Monster" 1;
"~*Pixray" 1;
"~*Pockey" 1;
"~*POE-Component-Client-HTTP" 1;
"~*ProPowerBot" 1;
"~*ProWebWalker" 1;
"~*proximic" 1;
"~*psbot" 1;
"~*Pump" 1;
"~*PyCurl" 1;
"~*QueryN\ Metasearch" 1;
"~*Qwantify" 1;
"~*RankActiveLinkBot" 1;
"~*Rankivabot" 1;
"~*RealDownload" 1;
"~*Reaper" 1;
"~*Ripper" 1;
"~*Recorder" 1;
"~*ReGet" 1;
"~*RepoMonkey" 1;
"~*Rogerbot" 1;
"~*ScanAlert" 1;
"~*Scrapy" 1;
# "~*Screaming\ Frog\ SEO\ Spider" 1;
"~*ScreenerBot" 1;
"~*searchestate" 1;
"~*SearchmetricsBot" 1;
"~*SemrushBot" 1;
"~*SEOkicks" 1;
"~*SEOkicks-Robot" 1;
"~*SEOlyticsCrawler" 1;
"~*seomoz" 1;
"~*SEOstats" 1;
"~*SISTRIX" 1;
"~*Siphon" 1;
"~*sitebeam" 1;
"~*SiteExplorer" 1;
"~*Siteimprove" 1;
"~*SiteLockSpider" 1;
"~*SiteSucker" 1;
"~*Site\ Sucker" 1;
"~*SiteSnagger" 1;
"~*sitevigil" 1;
"~*Slackbot-LinkExpanding" 1;
"~*SlySearch" 1;
"~*SmartDownload" 1;
"~*Snake" 1;
"~*Snapbot" 1;
"~*Snoopy" 1;
"~*SocialRankIOBot" 1;
"~*Sogou\ web\ spider" 1;
"~*Sosospider" 1;
"~*SpaceBison" 1;
"~*spammen" 1;
"~*spanner" 1;
"~*SpankBot" 1;
"~*Spinn3r" 1;
"~*spbot" 1;
"~*sqlmap" 1;
"~*Sqlworm" 1;
"~*Sqworm" 1;
"~*Stripper" 1;
"~*Sucker" 1;
"~*Sucuri" 1;
"~*SuperBot" 1;
"~*SuperHTTP" 1;
"~*Surfbot" 1;
"~*SurveyBot" 1;
"~*suzuran" 1;
"~*Swiftbot" 1;
"~*SISTRIX\ Crawler" 1;
"~*Szukacz" 1;
"~*T8Abot" 1;
"~*tAkeOut" 1;
"~*Teleport" 1;
"~*TeleportPro" 1;
"~*Telesoft" 1;
"~*Telesphoreo" 1;
"~*Telesphorep" 1;
"~*The\ Intraformant" 1;
"~*TheNomad" 1;
"~*TightTwatBot" 1;
"~*Titan" 1;
"~*T0PHackTeam" 1;
"~*Toweyabot" 1;
"~*Trendictionbot" 1;
"~*True_Robot" 1;
"~*Turingos" 1;
"~*TurnitinBot" 1;
"~*Turnitin\ Robot" 1;
"~*Turnitin\ Bot" 1;
"~*Twice" 1;
"~*URLy.Warning" 1;
"~*URLy\ Warning" 1;
"~*Vacuum" 1;
"~*VB\ Project" 1;
"~*VCI" 1;
"~*VeriCiteCrawler" 1;
"~*VidibleScraper" 1;
"~*VoidEYE" 1;
"~*Voil" 1;
"~*WASALive-Bot" 1;
"~*WBSearchBot" 1;
"~*WebAuto" 1;
"~*WebBandit" 1;
"~*Web\ Bandit" 1;
"~*WebCopier" 1;
"~*Web\ Copier" 1;
"~*WEBDAV" 1;
"~*WEBDAV\ Client" 1;
"~*WebEnhancer" 1;
"~*Web\ Enhancer" 1;
"~*WebFetch" 1;
"~*Web\ Fetch" 1;
"~*WebFuck" 1;
"~*Web\ Fuck" 1;
"~*WebGo\ IS" 1;
"~*Web\ Image\ Collector" 1;
"~*WebLeacher" 1;
"~*WebmasterWorldForumBot" 1;
"~*WebPix" 1;
"~*Web\ Pix" 1;
"~*WebReaper" 1;
"~*Web\ Reaper" 1;
"~*WebSauger" 1;
"~*Web\ Sauger" 1;
"~*webshag" 1;
"~*WebsiteExtractor" 1;
"~*Website\ Extractor" 1;
"~*WebsiteQuester" 1;
"~*Website\ Quester" 1;
"~*Webster" 1;
"~*WebStripper" 1;
"~*Web\ Stripper" 1;
"~*WebSucker" 1;
"~*Web\ Sucker" 1;
"~*WebWhacker" 1;
"~*Web\ Whacker" 1;
"~*WebZIP" 1;
"~*WeSEE" 1;
"~*Whack" 1;
"~*Whacker" 1;
"~*whatweb" 1;
"~*Widow" 1;
"~*WinHTTrack" 1;
"~*WiseGuys\ Robot" 1;
"~*WISENutbot" 1;
"~*wonderbot" 1;
"~*woobot" 1;
"~*Wotbox" 1;
"~*WWWOFFLE" 1;
"~*WWW::Mechanize" 1;
"~*WWW-Collector-E" 1;
"~*Xaldon_WebSpider" 1;
"~*Xaldon\ WebSpider" 1;
"~*Xenu" 1;
"~*Y!J-ASR" 1;
"~*YoudaoBot" 1;
"~*Zade" 1;
"~*Zeus" 1;
"~*zgrab" 1;
"~*zitebot" 1;
"~*ZmEu" 1;
"~*ZumBot" 1;
"~*Zyborg" 1;
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment