Skip to content

Instantly share code, notes, and snippets.

@hrbrmstr
Created August 1, 2017 02:37
Show Gist options
  • Save hrbrmstr/7bfbc4a3b4cadb3a32d2e1b72440bf6a to your computer and use it in GitHub Desktop.
Save hrbrmstr/7bfbc4a3b4cadb3a32d2e1b72440bf6a to your computer and use it in GitHub Desktop.
robots.txt user agent strings from June 2017 robots.txt Common Crawl
useragent n
* 494648
IRLbot 378368
bingbot 100948
MJ12bot 94377
msnbot 66142
AhrefsBot 51999
Yandex 50260
Slurp 49160
SemrushBot 45041
sogouspider 42866
Sogouwebspider 41080
MJ12bot/v1.4.3 39158
MSNBot 24619
Y!J-ASR/0.1crawler 19286
MSNBOT 19285
Googlebot 16963
spbot 15224
Baiduspider 12954
psbot 12081
Bingbot 8647
Mail.Ru 8213
BLEXBot 8124
CCBot 7384
Mail.RU_Bot 7156
Mediapartners-Google 6199
WebAlta 6031
WebAltaCrawler 6027
Rambler 6021
YandexBot 5308
008 5154
Googlebot-Image 4795
ia_archiver 4503
dotbot 4444
TurnitinBot 4292
Gigabot 4102
Exabot 4091
mantam 4085
Sosospider 3836
asterias 3739
YahooPipes1.0 3620
WebCopier 3557
SearchmetricsBot 3475
YandexImages 3256
WebStripper 3229
moget 3223
Teleport 3142
SiteSnagger 3064
Applebot 3063
twiceler 3061
OfflineExplorer 3020
Mediapartners-Google* 2983
LinkWalker 2945
rogerbot 2915
XoviBot 2915
EmailSiphon 2888
Telesoft 2847
CherryPicker 2845
EmailCollector 2802
WebsterPro 2802
EmailWolf 2764
Wget 2760
WebSauger 2758
WebAuto 2746
NetAnts 2746
Zeus 2743
WWW-Collector-E 2736
InfoNaviRobot 2736
TheNomad 2736
BuiltBotTough 2735
lwp-trivial 2735
BunnySlippers 2735
LinkextractorPro 2734
NICErsPRO 2733
MIIxpc 2733
BotALot 2732
LexiBot 2732
SpankBot 2732
EroCrawler 2731
WebBandit 2729
ProWebWalker 2729
CheeseBot 2729
DittoSpyder 2727
True_Robot 2727
VCI 2727
JennyBot 2726
WebEnhancer 2726
RepoMonkey 2726
Openfind 2726
spanner 2724
humanlinks 2722
Crescent 2722
suzuran 2722
hloader 2722
cosmos 2718
CopyRightCheck 2718
RMA 2718
httplib 2716
WebsiteQuester 2715
turingos 2713
ExtractorPro 2712
WebZip 2700
MisterPiX 2700
MataHari 2700
URLyWarning 2699
BlowFish/1.0 2698
toCrawl/UrlDispatcher 2698
Bullseye/1.0 2697
libWeb/clsHTTP 2697
KenjinSpider 2697
WebBandit/3.50 2695
WebImageCollector 2694
Szukacz/1.4 2694
TheIntraformant 2693
SemrushBot-SA 2693
CherryPickerElite/1.0 2692
CherryPickerSE/1.0 2691
True_Robot/1.0 2690
moget/2.1 2689
BackDoorBot/1.0 2689
LinkScan/8.1aUnix 2688
Harvest/1.5 2688
lwp-trivial/1.34 2688
MIIxpc/4.2 2686
CrescentInternetToolPakHTTPOLEControlv.1.0 2686
QueryNMetasearch 2685
ProPowerBot/2.14 2681
WebZip/4.0 2677
Zeus32297WebsterProV2.9Win32 2673
VCIWebViewerVCIWebViewerWin32 2671
Wget/1.5.3 2652
Foobot 2650
RepoMonkeyBait&Tackle/v1.01 2650
Wget/1.6 2643
Xenu's 2634
Yahoo!Slurp 2619
HTTrack 2606
Xenu'sLinkSleuth1.1c 2600
baiduspider 2594
MicrosoftURLControl-5.01.4511 2537
MicrosoftURLControl-6.00.8169 2532
Baiduspider-image 2527
TightTwatBot 2458
Baiduspider-video 2435
Openfinddatagathere 2427
Yeti 2414
MegaIndex.ru/2.0 2384
MegaIndex.ru 2369
WebmasterWorldForumBot 2307
Titan 2245
SeznamBot 2232
Cegbfeieh 2211
BlackHole 2184
Alexibot 2161
WebReaper 2127
NetMechanic 2105
googlebot 1964
DotBot 1946
proximic 1919
archive.org_bot 1892
Teoma 1890
HTTrack3.0 1868
msnbot-media 1855
ConveraCrawler 1847
QuepasaCreep 1766
findlinks 1765
sistrix 1763
BacklinkCrawler 1755
YoudaoBot 1753
Jetbot 1746
discobot 1717
SiteExplorer 1710
VoilaBot 1704
SuperBot 1688
HuaweiSymantecSpider 1662
Twitterbot 1653
WinHTTrack 1638
DeuSu 1634
SEOkicks-Robot 1627
Google 1627
NetAttache 1626
webcopy 1619
webmirror 1617
websiteextractor 1613
DISCoPump3.1 1613
WebStripper/2.02 1612
SuperBot/2.6 1611
NetAttacheLight1.1 1595
semalt.com 1574
Gigabot/3.0 1572
linkdexbot 1571
Nutch 1561
sitebot 1552
eDintornicrawler 1551
MLBot 1518
grapeshot 1485
Findxbot 1476
DomainCrawler 1473
TeleportPro 1456
Qwantify 1437
MSIECrawler 1398
Ezooms 1388
Aboundex 1381
Yisouspider 1374
360Spider 1359
Googlebot-Mobile 1324
ShopWiki 1262
NaverBot 1261
LNSpiderguy 1179
CazoodleBot 1176
Twiceler 1173
Riddler 1147
Speedy 1137
MJ12Bot 1130
Facebot 1124
ahrefs 1118
panscient.com 1085
KeywordDensity/0.9 1078
YandexImageResizer 1070
Adsbot-Google 1051
NextGenSearchBot 1042
TweetmemeBot 1036
larbin 1014
ichiro 1012
BecomeBot 996
googlebot-image 925
yahoo-mmcrawler 915
Claritybot 908
Linguee 887
Mozilla/4.0(compatible;BullsEye;Windows95) 878
WeSEE 863
Wotbox 854
yeti 846
naverbot 835
noxtrumbot 833
Scrubby 814
Abonti 814
yacybot 802
Python-urllib 794
baidu 775
coccoc 769
Robozilla 760
gigabot 756
Gaisbot 756
Yahoo 729
Spinn3r 728
AdsBot-Google 725
SafeDNSBot 722
grub-client 715
SputnikBot 714
libwww 710
TwengaBot 708
yahoo-blogs/v3.9 703
BingPreview 694
SMTBot 693
MSNbot 685
istellabot 682
Aboundexbot 673
WBSearchBot 670
EtaoSpider 665
Microsoft.URL.Control 663
Flamingo_SearchEngine 654
Aqua_Products 651
teoma 636
Openbot 634
bing 631
YandexMobileBot 630
Cliqzbot 623
URL_Spider_Pro 616
MojeekBot 608
slurp 606
googlebot-mobile 598
Laserlikebot 598
Kraken 597
adbeat_bot 597
b2w/0.1 597
Blekkobot 596
PaperLiBot 594
Yahoo-Newscrawler 593
Bookmarksearchtool 592
URLControl 591
Uptimebot 590
ZeusLinkScout 588
Iron33/1.0.2 588
AdIdxBot 587
searchpreview 583
Copernic 583
QuerySeekerSpider 582
twitterbot 581
naver 581
RadiationRetriever1.1 579
TweetedTimesBot 579
Mozilla/4.0(compatible;MSIE4.0;WindowsNT) 577
MicrosoftURLControl 577
GetRight/4.2 577
FlamingAttackBot 576
JikeSpider 576
Mozilla/4.0(compatible;MSIE4.0;Windows95) 576
ScoutJet 576
PerMan 573
FairAdClient 573
Mozilla/4.0(compatible;MSIE4.0;Windows98) 572
CFNetwork 570
uMBot-LN 569
TinEye-bot 569
Plukkie 565
UptimeRobot 565
g2reader-bot 560
Msnbot 558
PiplBot 558
Girafabot 556
OracleUltraSearch 556
Vegi 555
heritrix 555
Sogou 555
Pinterestbot 554
wonderbot 554
VigLink 554
Leikibot 553
coccocbot 553
GetIntent 552
uipbot 552
Superfeedr 551
trendiction 551
Dragonbot 550
avkzarabotok 550
_zbot 550
feedbot 550
metabot.ru 550
Clickagy 550
calculon 550
contxbot 550
YaK 550
mcsbot 550
Tapatalk 550
moatbot 550
Seoterritory 550
Taboolabot 550
RSSMicro 550
Ocarinabot 550
Slack-ImgProxy 550
HybridBot 550
Feedspotbot 550
Zombiebot 550
Discordbot 550
GnowitNewsbot 543
Mediatoolkitbot 543
oBot 542
Multiviewbot 541
Treato-Bot 541
CharlieBot 540
wget 540
zarabotok--doma 540
Sottopop 540
focusbot 540
Pvblcbot 540
ZoominfoBot 540
bot-pge.chlooe.com 540
NPBot 530
Ezooms/1.0 528
yandex 526
magpie-crawler 525
Purebot 525
Goodzer 514
MSNBot-Media 508
NerdByNature.Bot 505
YodaoBot 504
BaiduImagespider 501
aipbot 497
Baiduspider+ 492
Baiduspider/2.0 491
gsa-crawler 489
Baidu 481
LinkpadBot 480
mozilla/4 476
mozilla/5 469
007ac9 462
80legs 459
TwengaBot-Discover 457
GoogleBot 455
grub 454
daumoa 449
Mozilla/4.0(compatible;MSIE4.0;WindowsXP) 449
Mozilla/4.0(compatible;MSIE4.0;Windows2000) 449
RavenCrawler 447
Ocelli 443
BLP_bbot/0.1 442
MSSearch4.0Robot 440
BingBot 439
msn 437
Snapbot 431
trovitBot 430
Mozilla/4.0(compatible;MSIE4.0;WindowsME) 427
YahooSeeker 423
turnitinbot 419
SemrushBot-SI 417
NerdyBot 410
EasouSpider 409
sitecheck.internetseer.com 408
WebZIP 408
Mozilla/4.0(compatible;MSIE4.01;WindowsNT;MSSearch4.0Robot)Microsoft 407
SurveyBot 405
ScanAlert 405
ips-agent 404
Steeler 403
ZoomSpider 403
YahooYSMcm 389
Scooter 388
OmniExplorer_Bot 385
SuperPagesUrlVerifyBot 383
Fasterfox 381
nutch 380
YandexBot/3.0 379
AdvBot 378
Genieo/1.0 377
LCC 377
Xenu 376
MSNBot/Bingbot 373
Dolphin 368
Voyager/1.0 365
Riddlerbot 359
SISTRIXCrawler 358
mj12bot 358
Fetch 356
InternetSeer.com 355
ccbot 350
TwengaBot-2.0 343
AppleBot 341
linko 341
SuperPagesBot 340
YandexFavicons 340
YPBot 340
Yahoo-MMCrawler 338
DownloadNinja 338
wotbox 337
e-SocietyRobot 336
ContextAd 335
Zealbot 335
AskJeeves 335
linkdex.com 334
ezooms 333
Y!J-BRW 332
AugustBot 332
SearQuBot 330
Seznamscreenshot-generator 330
ZyBORG 323
Java/1.6.0_04 322
Jyxobot 322
SISTRIX 315
linkdex 312
IconSurf 309
duggmirror 308
Zao 307
ICC-Crawler 305
Yahoo-slurp 298
BotRightHere 298
UbiCrawler 296
R6_CommentReader 296
* #anyrobot 294
megaIndex.ru 290
netEstateNECrawler 289
Charlotte 289
DotBot/1.1 289
betaBot 287
aiHitBot 285
BaiduMobaider 285
R6_FeedFetcher 284
BUbiNG 282
WebCopierv3.2a 277
checks.panopta.com 277
linkdex.com/v2.0 276
TurnitinBot/1.5 275
Healthbot 275
Healthbot/Health_and_Longevity_Project_(HealthHaven.com) 275
//www.baidu.com/search/spider.html) 274
AdvBot/2.0 274
Gimme60bot 273
FindTheBest.com 273
SpeedySpider 272
BLP_bbot 271
Synapse 270
WebZIP/4.21 268
DOC 267
WebZIP/5.0 267
Openfinddatagatherer 266
envolk 265
MegaIndex 265
WebCapture2.0 264
//www.baidu.com/search/spider.html 264
HaosouSpider 262
WebCopierv.2.2 261
lmspider 258
Bing 254
Adidxbot 253
Twitterbot/1.0 252
CamelStampede 251
Rakutenbot 250
IstellaBot 249
msnbot-products 247
YesupBot 246
voyager 245
MSRBOT 245
mxbot 243
meanpathbot 240
msnbot-news 239
KSCrawler 238
MSR-ISRCCrawler 238
yahoo-slurp 237
AhrefsBot/5.1 236
Fbot 233
nu_tch-princeton 232
//ahrefs.com/robot/ 232
abby 231
MSUbot 231
SheenBot 231
Lycos_Spider 231
Kwaclebot 231
LargeSmallCrawler 229
UnwindFetchor/1.0 229
FASTEnterpriseCrawler6 229
PostRank 229
DotSpotsBot 229
WillowInternetCrawlerbyTwotrees 228
inagist.comurlcrawler 228
Jaxified 228
agbot 228
Domaincrawler1.0 228
Hailoobot 228
//labs.topsy.com/butterfly/)Gecko/2009032608Firefox/3.0.8 228
diribot 228
YahooMobile/1.0 228
Influencebot/0.9 228
CyberPatrolSiteCatWebbot 228
YisouSpider 224
k2spider 219
//megaindex.com/crawler) 217
robozilla 213
//yandex.com/bots 212
CompSpyBot 212
Trident 211
MJ12bot/v1.4.5 211
Mozilla/4.0+(compatible;+T-H-U-N-D-E-R-S-T-O-N-E) 211
ZyBorg 210
ahrefsbot 208
//www.sogou.com/docs/help/webmasters.htm#07) 205
scrubby 205
Vagabondo 203
Wotbox/2.01 202
exabot 202
Mail.RU_Bot/2.0 201
applebot 200
//www.wotbox.com/bot/ 200
ia_archiver/1.6 200
StackRambler 199
SBIder 196
OrangeBot 196
//yandex.com/bots) 192
AhrefBot 192
seznambot 192
becomebot 191
MaxPointCrawler 189
SolomonoBot 186
Exabot-Thumbnails 185
VegeBot 184
//go.mail.ru/help/robots 182
megaindex.com 182
BPImageWalker 181
Java/1.4.1_04 181
SurveyBot_IgnoreIP 181
htdig 178
discoverybot 178
Nutch-1.4 177
adidxbot 176
//www.majestic12.co.uk/bot.php?+) 175
//deusu.de/robot.html 175
linkdexbot/2.2 175
//www.profound.net/domainappender) 175
MSSearch6.0Robot 174
//ltx71.com/) 173
spbot/5.0.2 173
Sogou+web+spider/4.0 172
//go.mail.ru/help/robots) 172
//ltx71.com/ 172
//ahrefs.com/robot/) 172
//www.wotbox.com/bot/) 171
yoozBot-2.2 171
MegaIndex.ru/ 170
DeuSu/5.0.2 170
SMTBot/1.0 170
DataparkSearch 170
Mail.RU_Bot/Img/2.0 170
GrapeshotCrawler/2.0 169
DuckDuckGo 169
//www.genieo.com/webfilter.html) 169
Applebot/0.1 169
//www.linkdex.com/bots/) 169
Mail.RU_Bot/Robots/2.0 169
spiderman 169
maxpoint.crawler 168
//api.slack.com/robots 168
Siteimprove.com 168
Cliqzbot/1.0 168
Googlebot-image 168
Java/1.6.0_21 168
maxpoint.crawler+at+maxpointinteractive+dot+com 167
//www.proximic.com/info/spider.php) 167
//www.apple.com/go/applebot) 167
//www.similartech.com/smtbot) 167
Spider/5.1 167
//www.opensiteexplorer.org/dotbot,+help@moz.com) 167
//www.grapeshot.co.uk/crawler.php 167
majestic12.co.uk/bot.php 167
//cliqz.com/company/cliqzbot) 167
//www.archive.org/details/archive.org_bot) 167
//sur.ly/bot.htm 167
safedns.com/searchbot 167
//www.haosou.com 167
//www.profound.net/domainappender 167
Nutch-1.10 167
//yooz.ir;+info@yooz.ir 167
//megaindex.com/crawler 167
go.mail.ru/ 167
//duckduckgo.com 167
curl/7.19.7+(x86_64-unknown-linux-gnu)+libcurl/7.19.7+NSS/3.12.7.0+zlib/1.2.3+libidn/1.18+libssh2/1.2.2 167
MojeekBot/0.6 167
www.proximic.com/info/spider.php 167
//www.similartech.com/smtbot 167
www.apple.com/go/applebot 167
//safesearch.avira.com,+safesearch-abuse@avira.com) 167
orange.com 167
www.baidu.com/search/spider.html 167
awooo 167
www.dataparksearch.org/bot 167
Slack-ImgProxy+1.136 167
//www.dataprovider.com/) 167
libcurl/7.19.7 167
//cliqz.com 167
SurdotlyBot/1.0 167
Xenu+Link 167
LinkCheck+by+Siteimprove.com 167
//yooz.ir;+info@yooz.ir) 167
js-crawler 167
//inet-sochi.ru/gday 167
//www.mojeek.com/bot.html) 167
Simple/6.13+libwww-perl/6.13 167
www.genieo.com/webfilter.html 167
ZoomInformation+Bot 167
Nutch-1.9 167
Sleuth/1.3.8 167
SafeSearch+microdata+crawler 167
NRLCorpusBuilder/Nutch-1.9 167
//archive.org/details/archive.org_bot 167
//api.slack.com/robots) 167
HETZNER-RZ-NBG-NET 167
//www.linkdex.com/bots/ 167
NextGenSearchBot.aspx 167
//www.zoominfo.com/About/misc/NextGenSearchBot.aspx) 167
//archive.org/details/archive.org_bot) 167
crawler/js-crawler 167
zlib/1.2.3 167
//www.zoominfo.com 167
//www.haosou.com/help/help_3_2.html) 167
//www.opensiteexplorer.org/dotbot 167
www.sogou.com 167
libssh2/1.2.2 167
safesearch.avira.com 167
nutch-1.4/Nutch-1.4 167
luis@cybo.com 167
//www.safedns.com/searchbot) 167
//www.mojeek.com/bot.html 167
//www.megaindex.ru/ 167
yooz.ir 167
curl/7.29.0 167
support.orangebot@orange.com 167
DomainAppender+/1.0 167
Dataprovider/6.92 167
OrangeBot/2.0 167
ContentVerification/5.1 167
Xenu+Link+Sleuth/1.3.8 167
Screaming+Frog+SEO 167
Mozilla/5.0+(compatible;+OrangeBot/2.0;+support.orangebot@orange.com) 167
www.baidu.com 167
UXCrawlerBot 167
Mozilla/5.0+(compatible;+MSIE+10.0;+Windows+NT+6.1;+Trident/6.0)+LinkCheck+by+Siteimprove.com 167
www.proximic.com 167
//em7.sciencelogic.com 167
//duckduckgo.com) 167
//sur.ly/bot.html) 167
xSocks+v0.1 167
libidn/1.18 167
Semalt.com 167
//OpenLinkProfiler.org/bot 167
Screaming+Frog+SEO+Spider/5.1 167
//www.dataprovider.com/ 167
tutorgigbot 167
maxpoint.crawler@maxpointinteractive.com 167
//www.grapeshot.co.uk/crawler.php) 167
//deusu.de/robot.html) 167
curl/7.43.0 167
//OpenLinkProfiler.org/bot+) 167
seekbot 166
NA 164
looksmart 164
SWEBot 164
GrapeshotCrawler 157
Jetbot/1.0 156
//www.almaden.ibm.com/cs/crawler 155
HatenaAntenna 154
dumbot 154
curl 153
Enterprise_Search 152
facebookexternalhit 152
sootle 152
WellsSearchII0.0 151
EdisterBot 151
heritrix/1.10.0 151
lemurwebcrawler 150
WebVac 150
memorybot 149
google-hoteladsverifier 149
FreeFind 149
Fatbot 149
netseer 148
Java/1.6.0_11 148
Genieo 147
PagePeeker 147
CatchBot 147
Yahoo-Blogs 146
es 146
Enterprise_Search/1.0 145
sogou 144
plukkie 144
Updownerbot 144
Eluta.ca 143
Java/1.6.0_20 141
sentibot 140
Java/1.6.0_16 139
Java/1.6.0_15 139
Java/1.6.0_18 139
Java/1.6.0_24 139
FunWebProducts 139
Java/1.6.0_25 139
Java/1.6.0_17 139
Java/1.6.0_22 139
Stanford 139
Java/1.6.0 138
blp_bbot/0.1 138
Ultraseek 138
StanfordCompSci 137
GoldfireServer 137
Googlebot-Docs 137
gsa-crawler+(Enterprise;+T3-HR8MK3S756ETJ;+google-support@extended-content.com) 137
DowJonesSearchbot 136
lipperhey 135
RPT-HTTPClient 134
dotbot/1.0 133
Sogouwebspider/4.0 133
szukacz 132
blexbot 132
Shim-Crawler 132
HubSpot 131
FAST 130
Insieve+Bot 130
mail.ru 130
xovibot 129
SiteBot 128
OpenindexSpider 128
ScreamingFrogSEOSpider 128
SEOkicks 127
smtbot 126
wbsearchbot 125
publiclibraryarchive.org 125
msnbot-NewsBlogs 125
seokicks-robot 125
Heritrix 124
linkdexbot/2.0 124
AddThis.com 123
megaindex.ru 123
libwww-perl 121
SlySearch 121
MotoMinerBot 120
MSN 119
semrushbot 119
siteexplorer 118
Googlebot-News 118
linkdexbot/2.1 117
DISCoPump 117
Yanga 117
advbot 116
netestatenecrawler 116
exblanguagecrawler 116
aboundexbot 116
feedbooster 116
mixbot 116
bubing 116
searchmetricsbot 116
easouspider 116
bpimagewalker/2.0 116
bingbot/2.0 116
abonti 116
unisterbot 116
megaindex.ru/2.0 116
linkpadbot 116
screenerbot 116
Webinator 115
ZumBot 114
Yetibot 114
EC2LinkFinder 114
CrescentInternetToolPak 113
voltron 112
vspider 112
SindiceBot 112
XoviBot/2.0 112
YandexNews 112
worldwebheritage.org 111
seoscanners.net 111
Exabot* 109
PerManSurfer 109
aihitbot 108
MFCFoundationClassLibrary 108
WebEMailExtractor 108
Netprospector 108
LNSpiderGuy 108
AdamaticSolutions 107
Control 107
Microsoft 107
google 106
worldwebheritage.org/1.0 106
JamesBOT 106
oodlebot 105
Baiduspider-news 105
Echo 105
Yahoo!-AdCrawler 104
Mozilla 104
Mozilla/4.0+(compatible;MSIE4.0;Windows98) 103
BaiDuSpider* 103
happyfunbot 103
NewsTroveBot 103
gigabot* 103
GenieBot 103
ipselonbot 103
UnisterBot 102
cuil 102
Jeeves 102
Orthogaffe 100
voilabot 99
Bender 97
Java 97
suggybot 94
findfiles.net 93
Huaweisymantecspider 92
GetRight 92
BaiDuSpider 92
ip-web-crawler.com 91
omgilibot 91
ECCP 91
Baiduspider-favo 91
Baiduspider-cpro 91
TinEye 90
worldwebheritage.org/1.1 90
YakazBot 90
BackDoorBot 90
uniplace-bot 89
Mail.ru 89
FyberSpider 88
HubSpotWebcrawler 88
LexxeBot 87
Harvest 87
Bullseye 87
BlackWidow 86
UnwindFetchor 86
HubSpotLinksCrawler1.0 86
Pixray-Seeker 85
Sogouspider 85
LSSRocketCrawler 85
AhrefsBot/4.0 85
Baiduspider-ads 85
BlowFish 84
AhrefsBotisnotallowedanymore 84
uniplace-index 82
Custo 82
FlashGet 81
Webalta 81
ltx71 81
Googlebot-Video 81
eCatch 80
AppCodesCrawler 80
WebFetch 80
vebidoobot 79
Go-Ahead-Got-It 79
DuckDuckBot 79
Voluniabot 79
WWWOFFLE 79
Surfbot 79
MsnBot 79
CrazyWebCrawler-Spider 79
ReGet 78
VoidEYE 78
HypeStat 78
JetCar 78
HMView 78
Rogerbot 78
NearSite 78
RealDownload 78
NetSpider 78
SmartDownload 78
SuperHTTP 78
ImageWalker 78
pcBrowser 78
//www.hubspot.com/ 77
YandexMobileBot/3.0 77
Teemer 76
LinkpadBot/1.06 76
NetZIP 75
DomainRe-AnimatorBot 75
www.integromedb.org/Crawler 75
pavuk 75
* 75
LeechFTP 74
TruliaBot 74
Grafula 74
EirGrabber 74
LWNutch 74
costam 74
gonzo 73
Semrush 73
Widow 73
SemrushBot/1.1~bl 73
Dotbot 73
EyeNetIE 73
Navroad 72
ChinaClaw 72
picmole 72
InterGET 72
Mp3Bot 72
WebLeacher 72
GrabNet 72
Octopus 72
PageGrabber 72
tAkeOut 72
WebWhacker 72
Meridian-crawler 71
hap-crawler 71
OpenHoseBot 71
fast 71
searchmetrics-bot 71
KaloogaBot 70
PropsmartCrawler 70
BDFetch 70
yandexbot 70
Go!Zilla 70
SimilarPages 70
MediaPartners-Google 70
Mozilla* 69
Zing-BottaBot 69
jobs.de-Robot 69
Presto 69
Yahoo!slurp 69
SimilarPages/Nutch-1.0-dev 69
contype 69
YandexMedia 68
FriendFeedBot 68
FlipboardProxy 68
bot/1.0 67
//www.baidu.com/search/spider.htm) 66
spider 66
msnbot-MM 66
CrystalSemanticsBot 66
TosCrawler 66
egothor 66
SEOkicksRobot 66
woriobot 66
Intelliseek 65
ExBLanguageCrawler 65
SogouOrionspider 64
Sogouinstspider 64
iCjobs 63
BoardReader 63
Psbot 63
RookeeBot 63
SogouNewsSpider 63
AhrefsBot/5.0 62
lb-spider 62
Sogouspider2 62
WebmasterCoffee 62
FrontPage 61
Aport 61
rookeebot 61
Szukacz 60
W3C-checklink 60
Sogoublog 60
YandexPagechecker 60
yetibot 59
AppleNewsBot 59
MaxPointCrawler/Nutch-1.1 59
ProPowerBot 58
Mozilla/4.0(compatible;MSIE6.0;WindowsNT;MSSearch4.0Robot) 58
yisouspider 58
ia_archiver-web.archive.org 58
LinkScan 57
+Baiduspider/2.0 57
zibber-v0.1(www.zibb.com/crawler/) 57
AhrefsBot/3.1 56
search17 56
DISCo 56
*" 56
/ 56
+Baiduspider 56
InfoPath 55
StorygizeBot 55
YandexDirect 55
LinkisBot 55
HoaxyBot 55
bot 55
UptimeRobot/2.0 54
WikioFeedBot 54
xovi 54
bdbrandprotect 54
Comodo-Certificates-Spider 54
GSLFbot 54
NetResearchServer 53
Yeti-Mobile 53
schrein 53
ContentCrawler 53
AfiliasWebMiningTool 53
ReverseGet 53
DCPbot 53
lex 53
OpidooBOT 53
COMODOSSLChecker 53
kalooga 52
ImageBot 52
Exabot/3.0 52
CuriousGeorge 52
ShowyouBot 52
FatBot 52
crawl.yahoo.net 52
IDBot 52
Eurobot 52
JenkersBot 52
cityrevier 51
Snoopy 51
Diffbot 51
WijuBot 51
niki-bot 51
*bingbot.* 51
Kemvibot 51
spbot/4.4.2 50
Xenu’sLinkSleuth1.1c 50
Xenu’s 50
Swiftbot 50
appie 50
Pinterest 50
scooter 50
antibot 49
BLEXBot/1.0 49
MajesticSEO 49
Naverbot 49
ChangeDetection 49
Unister 49
thunderstone 49
Spiderlytics 49
zibber-v0.1 49
Go-http-client/1.1 48
SogouPicSpider 48
semantic-visions.comcrawler 48
Sqworm 48
GingerCrawler 48
ImageWalker/2.0 48
Flash+Processor 48
SiteExplorer/1.0b 48
DoCoMo 47
//www.cuil.com/twiceler/robot.html) 47
GermCrawler 47
WebmasterWorldExtractor 47
YahooSeeker/M1A1-R2D2 47
Powermarks 47
AddThis.comrobottech.support@clearspring.com 47
MJ12bot/v1.4.0 47
it2media-domain-crawler 46
BaiduSpider 46
BSpider 46
gsa-crawler-www 46
cityreview 46
Scrapy 45
GetWeb! 45
URLy.Warning 45
NimbleCrawler 45
adsbot-google 45
trendictionbot 45
OOZBOT 45
DomainAppender 44
boitho.com-dc 44
YandexMetrika 44
OfflineExplorer/1.9 44
FASTEnterpriseCrawler 44
NPbot 44
SimplePie 44
LocalcomBot 44
ChinasoSpider 44
CherryPicker/1.0 44
Toweyabot 44
Flipboard 43
ObjectsSearch 43
Baiduspider/5.0 43
newsbot 43
Radian6 43
Baiduspider/3.0 43
SiteLockSpider 43
Baiduspider/4.0 43
yetbot 43
baiduspider+ 43
SapphireWebCrawler 42
OpenHoseBot/2.1 42
omgilibot/0.4 42
pompos 42
careerbot 42
msnbot-UDiscovery/2.0b 42
Xaldon\WebSpider 42
fr-crawler/1.1 42
ca-crawler/1.0 42
SiteSucker 42
MemoryBot 41
Offline\Navigator 41
//www.majestic12.co.uk/bot.php?+ 41
Express\WebPictures 41
Sosospider+ 41
MIDown\tool 41
Web\Image\Collector 41
Offline\Explorer 41
Kemvibot/1.0 41
JOC\Web\Spider 41
Mass\Downloader 41
Image\Sucker 41
Papa\Foto 41
Website\Quester 41
RediffNewsBot 41
WebSucker 41
WebGo\IS 41
Spiderbot 41
Web\Sucker 41
Internet\Ninja 41
sosospider 41
Image\Stripper 41
Mister\PiX 41
Download\Demon 41
Net\Vampire 41
Indy\Library 41
SentiBot 41
GwdangSpider 40
Lachesis 40
HuihuiSpider 40
LinkedInBot 40
ASPSeek 40
autoemailspider 40
Daumoa 40
WochachaSpider 40
IsraBot 40
Gigabot/2.0 39
Yahoo! 39
YandexBlogs 39
vscooter 39
AhrefsBot/2.0 39
AboutUsBot 39
yolinkBot 39
stress-agent 39
Wget/1.8.2 39
Infohelfer 39
ArchitextSpider 39
IndyLibrary 38
Xenu'sLinkSleuth 38
Generic 38
x28-job-bot 38
kulturarw3 38
YangaWorldSearchBot 38
penthesilea 38
NinjaBot 38
newsregistry 38
msnbot-UDiscovery 38
um-LN 38
Grabber 38
DoCoMo/2.0 37
Reaper 37
DigExt 37
All 37
craftbot@yahoo.com 37
YYSpider 37
Unister* 36
Yahoo!SlurpChina 36
ahrefs.com 36
AbachoBOT 36
Qualidator* 36
MicrosoftURLControl–6.00.8169 36
4SeoHuntBot 36
Simple 36
JobboerseBot 36
MicrosoftURLControl–5.01.4511 36
MIA 36
ipsAgent 36
WebofantBot 36
Roverbot 36
BPImageWalker* 36
*thunderstone* 36
WASALive-Bot 36
YodaoBot/1.0 36
MassDownloader 35
//www.spidersoft.com) 35
NPBot-1/2.0 35
Webster.Pro 35
WebStripper/2.16 35
LingueeBot 35
FAST-WebCrawler 35
Yandexbot 35
RogerBot 35
Yahoo-Slurp 35
OpenWebIndex 35
YaDirectFetcher 35
RealDownload/4.0.0.41 34
Magnet 34
RealDownload/4.0.0.40 34
NetAnts/1.25 34
Website.Quester 34
RealDownload/4.0.0.42 34
Wget/1.7 34
LinkScan/8.1a.Unix 34
GetRight/4.5 34
WebStripper/2.19 34
GetRight/4.5b 34
GetRight/4.5b6 34
Yandex.com 34
Baiduspider-mobile 34
EasyDL 34
ImageSucker 34
DISCoFinder 34
GetRight/5.0beta1 34
JOCWebSpider 34
NetVampire 34
eCatch/3.0 34
MIDowntool 34
Keyword.Density 34
semanticdiscovery 34
GetRight/4.3 34
WebGoIS 34
SuperHTTP/1.0 34
Webster 34
GetRight/4.2c 34
GetRight/4.1.0 34
yacy 34
Drip 34
Wget/1.5.2 34
OfflineExplorer/1.7 34
Twiceler-0.9 34
TeleportPro/1.29 34
Siphon 34
GetRight/4.5b3 34
GetRight/3.1 34
GetRight/3.3 34
ImageStripper 34
WebStripper/2.13 34
GetRight/4.5a 34
OfflineExplorer/1.6 34
GetRight/5.0beta2 34
GetRight/4.5d 34
GetRight/4.5e 34
OfflineExplorer/1.4 34
GetRight/4.5c 34
Mag-Net 34
GetRight/4.5b7 34
OfflineExplorer/1.2 34
BatchFTP 33
Pockey 33
//www.sogou.com/docs/help/webmasters.htm 33
BackWeb 33
//www.archive.org/details/archive.org_bot 33
msrbot 33
Mail.RU 33
AskPeterBot 33
Buddy 33
bingbot-media 33
//www.exabot.com/go/robot 33
stalker 33
Pump 33
webcrawler 33
Zade 33
SpaceBison 33
Vacuum 33
//www.linkpad.ru 33
//www.proximic.com/info/spider.php 33
Ninja 33
Whacker 33
Galbot 33
Collector 33
Recorder 33
Sucker 33
Bandit 33
JustView 33
MFC_Tear_Sample 33
Stripper 33
SEOENGWorldBot 33
Snake 33
Mozilla/5.0(compatible;Ezooms/1.0;ezooms.bot[at]gmail[dot]com) 33
Iria 33
Xaldon_WebSpider 33
Copier 33
Zeusbot 33
fyberspider 32
PGImageCrawler 32
HTTrack[NC,OR] 32
JobdiggerSpider 32
netEstateRSScrawler 32
search-one-scgov 32
Alexabot 32
Alexa 32
Accelobot 32
XenuLinkSleuth/1.3.8 32
Browsershots 32
yahoo 32
Plukkie/1.5 32
NG/2.0 32
MisterPixII2.02a 31
Spinne 31
MassDownloader/2.2 31
DISCoPump3.2 31
Go!Zilla(www.gozilla.com) 31
Zeus11652WebsterProV2.9Win32 31
NetVampire/3.0 31
TeleportPro/1.29.1847 31
SmartDownload/1.2.77(Win32;Feb12000) 31
crawler4j 31
WebCopierv3.0.1 31
OfflineExplorer/2.0 31
DomainsDB.netMetaCrawler 31
WebSauger1.20j 31
Zeus97371WebsterProV2.9Win32 31
BoardTracker 31
psbot/0.1 31
WebCopierv2.8 31
WebCopierv2.7a 31
DownloadDemon/3.2.0.8 31
WebAuto/3.40(Win98;I) 31
OfflineExplorer/2.4 31
Zeus82016WebsterProV2.9Win32 31
WebSauger1.20b 31
WebSauger1.20k 31
FrontPage[NC,OR] 31
Bilbo 31
Mister.PiX 31
MisterPixII2.01 31
Zeus30747WebsterProV2.9Win32 31
NetZipDownloader1.0Win32(Nov121998) 31
OfflineExplorer/2.5 31
MisterPiXversion.dll 31
TeleportPro/1.29.1634 31
NetZip-Downloader/1.0.62(Win32;Dec71998) 31
InternetNinja5.0 31
GetRight/4.2b(Portuguxeas) 31
Zeus63567WebsterProV2.9Win32 31
FlashGetWebWasher3.2 31
InternetNinja6.0 31
InternetNinja4.0 31
Zeus90872WebsterProV2.9Win32 31
Zeus44238WebsterProV2.9Win32 31
Tagoobot 31
Zeus11389WebsterProV2.9Win32 31
WebCopierv2.5 31
Mata\Hari 30
DTS\Agent 30
thesubot 30
httrack 30
ScreenerBot 30
Keyword\Density 30
Kenjin\Spider 30
Black.Hole 30
likse 30
Uptimebot/1.0 30
InterfaxBot 30
TMCrawler 30
gotit 30
genieBot 30
Black\Hole 30
The\Intraformant 30
URLy\Warning 30
Gulliver 30
mozilla/3 30
occ-crawler 30
Microsoft\URL\Control 30
Download\Wonder 30
libWeb 30
attach 30
lftp 30
PycURL 30
JOC 30
//www.bing.com) 30
AhrefsBot/3.0 30
Spiderbot/Nutch-1.7 29
Pompos 29
The.Intraformant 29
OrangeBot-Collector 29
TeleportPro/1.29.1590 29
www.deadlinkchecker.com 29
YandexCalendar 29
MSNBOT/0.1 29
holmes 29
//www.innerprise.net/usp-spider.asp) 29
ExpressWebPictures 29
ApocalXExplorerBot 29
DownloadDemon 29
Wget/1.8 28
XaldonWebSpider 28
Wget/1.9-beta 28
NetAnts/1.23 28
GetRight/3.3.3 28
libcrawl 28
ScreamingFrogSEOSpider/4.1 28
Website\eXtractor 28
GetRight/4.0.0 28
Mata.Hari 28
GetRight/2.11 28
msnbot<BR> 28
Offline.Explorer 28
WebStripper/2.15 28
WebStripper/2.10 28
Wget/1.8.1 28
GetRight/4.5b2 28
repparser 28
GalaxyBot 28
SurdotlyBot 28
PhpDig 28
OfflineNavigator 28
GetRight/3.2 28
JakartaCommons-HttpClient 28
007ac9Crawler 28
YandexMarket 28
bitlybot 28
sindice-site-manager 28
Scanmine 28
NetAnts/1.24 28
NetAnts/1.10 28
PapaFoto 28
CybEye.com 28
Genio 28
SimplePie/1.3.1 28
Teleport\Pro 28
GetRight/3.3.4 28
GetRight/4.5b1 28
sp_auditbot 28
InternetNinja 28
OdklBot 28
WebStripper/2.03 28
WocBot 28
WebStripper/2.12 28
GetRight/4.1.1 28
webfetch/2.1.0 28
GetRight/4.1.2 28
metajobbot 27
KeywordDensity 27
robot 27
sukibot_heritrix 27
HTTrackWebsiteCopier 27
facebookexternalhit/1.1 27
YamanaLab-Robot 27
VocusBot 27
ZmEu 27
Euripbot 27
DBLBot 27
SEMrushBot 27
OpenLinkProfiler 27
RepoMonkeyBait&amp;Tackle/v1.01 27
accelobot 27
SiteBot/0.1 27
//help.soso.com/webspider.htm) 27
ScSpider 27
NutchCVS/0.7.1 27
smrjbot/0.0.20 26
deepcrawl 26
smrjbot 26
scrapybot 26
QueryN.Metasearch 26
nabot 26
Altavista 26
psycheclone 26
Slackbot-LinkExpanding 26
expo9 26
DIIbot 26
CydralSpider 26
Dumbot 26
SEOdiver 26
swish-e 26
TeleportPro/1.29.1718 26
Web.Image.Collector 26
Unknownrobot 26
seoscanners 26
CoolBot 26
XenuLinkSleuth 26
OfflineExplorer/2.1 25
Zeus84842WebsterProV2.9Win32 25
Wget/1.8.1+cvs 25
Zeus95245WebsterProV2.9Win32 25
XaldonWebSpider2.5.b3 25
WebCapture 25
WebsiteeXtractor 25
SuperBot/3.1(Win32) 25
Kenjin.Spider 25
MSNBot-media 25
Zeus82900WebsterProV2.9Win32 25
Zeus94934WebsterProV2.9Win32 25
cfetch 25
Zeus39206WebsterProV2.9Win32 25
crystalsemanticsbot 25
WebCopierv3.2 25
Zeus51837WebsterProV2.9Win32 25
LexxeBot/1.0 25
Zeus51070WebsterProV2.9Win32 25
WebCopierv2.6 25
TeleportPro/1.29.1820 25
Zeus6694WebsterProV2.9Win32 25
SmartDownload/1.2.77(Win32;Aug171999) 25
BruinBot 25
SensisWebCrawler 25
WebCopierv3.0 25
DownloadDemon/3.5.0.11 25
Go!Zilla3.3(www.gozilla.com) 25
Zeus18018WebsterProV2.9Win32 25
Zeus51674WebsterProV2.9Win32 25
SogouSpider 25
IndyLibrary[NC,OR] 25
gonzo* 25
SuperBot/3.0(Win32) 25
RufusBot 25
ExpressWebPictures(www.express-soft.com) 25
HooWWWer 25
Mozilla/4.0(compatible;MSIE5.0;WindowsNT;DigExt;DTSAgent 25
Microsoft.URL 25
Zeus95351WebsterProV2.9Win32 25
OfflineExplorer/2.3 25
ScoutAbout 25
SmartDownload/1.2.77(Win32;Jun192001) 25
Go!Zilla3.5(www.gozilla.com) 25
SightupBot 25
Zeus41641WebsterProV2.9Win32 25
Zeus71129WebsterProV2.9Win32 25
DISCoPump3.0 25
SmartDownload/1.2.76(Win32;Apr11999) 25
Zeus26378WebsterProV2.9Win32 25
WebReaper[info@webreaper.net] 24
Yahoo-MMAudVid 24
zitebot 24
InterfaxScanBot 24
GozaikBot 24
ICCrawler-iCjobs 24
MSSearch5.0Robot 24
alexa 24
Googlebot* 24
thumbshots-de-bot 24
larbin_2.6.2larbin2.6.2@unspecified.mail 24
CamontSpider 24
SearchDaimon.com-dc 24
NetSeercrawler 24
Amfibibot 23
WebViewer 23
WebReaper[webreaper@otway.com] 23
TutorGig 23
majestic12 23
Bitacle 23
//www.picmole.com) 23
FindLinks 23
lwp-request 23
BizInformation 23
iCCrawler 23
larbin_2.6.2(larbin2.6.2@unspecified.mail) 23
Larbin 23
WebEMailExtrac.* 23
Zeusbot/Nutch-1.0-dev 23
larbin_2.6.2(listonATccDOTgatechDOTedu) 23
larbin_2.6.2(vitalbox1@hotmail.com) 23
//www.seoprofiler.com/bot/) 23
larbin_2.6.2larbin@correa.org 23
mlbot 23
larbin(samualt9@bigfoot.com) 23
Rankivabot 23
alexabot 23
larbin_2.6.2(kabura@sushi.com) 23
larbin_2.6.2listonATccDOTgatechDOTedu 23
nicebot 23
Feedfetcher-Google 23
LiteFinder 23
crawl 23
LivelapBot 23
roadrunner 23
larbin_2.6.2kabura@sushi.com 23
mozilla 23
larbinsamualt9@bigfoot.com 23
//www.accelobot.com) 23
btbot 23
//garlik.com/,crawler@garlik.com) 23
URLAppendBot 22
Purebot/1.1 22
purebot 22
Ahrefs 22
Soso 22
Touche 22
NetStat.RuAgent 22
GameSpyHTTP/1.0 22
ConveraCrawler/0.9e 22
HTTP 22
Mercator 22
AdsBot-Google-Mobile-Apps 22
Yahoo!-MMCrawler/3.x 22
gigablast 22
WebReapervWebReaperv7.3-www,otway.com/webreaper 22
HenryTheMiragoRobot 22
YandexDirectDyn 22
UrlDispatcher 22
WebReaperv9.1-www.otway.com/webreaper 22
WebsiteQuester-www.asona.org 22
Snappy 22
WebReaperv9.7-www.webreaper.net 22
CherryPickerSE 22
MMCrawler/3.x 22
radian 22
twiceler. 22
mirago 22
WebsiteQuester-www.esalesbiz.com/extra/ 22
Tasapspider 22
Yahoo-NewsCrawler 21
NutchCVS 21
Mail.Ru/1.0 21
Birubot 21
Caliperbot/1.0 21
MyEngines-Bot 21
toCrawl 21
Typhoeus 21
CherryPickerElite 21
//crawler.sistrix.net/) 21
OutfoxBot 21
pimonster 21
bot* 21
YadirectBot 21
Veooz 21
//www.thefind.com/crawler) 21
EzoomsRobot 21
SEOlyticsCrawler 21
Lycos_Spider_(T-Rex) 21
DomainSigmaCrawler 21
Pixray 21
Cowbot 21
BLEXbot 21
CloudFlare-AlwaysOnline 21
Lycos 21
ApptusBot 21
Igentia 21
Moni 20
Haste 20
dcbspider 20
GoForIt 20
YandexImages/3.0 20
JemmaTheTourist 20
"*" 20
IlseBot 20
Searchspider 20
MMCrawler 20
Butterfly 20
Spider_Monkey 20
Georgios 20
Pi-Monster 20
gridBOT 20
obot 20
flatlandbot 20
HaoSouSpider 20
aranhabot 20
Inktomi 20
speedy 20
MSFrontPage 20
Molbsy 20
ADmantX 20
Iron33 20
KnowItAll 20
Adidxbox 20
DealOzBot 20
USyd-NLP-Spider 20
woobot 20
sna 20
VegeBot#allVegeBotservices 20
TAMU_CS_IRL_CRAWLER 20
YRSPider 20
Nikto 20
BNSBot 20
foobar 20
iCcrawler 20
InfoPath.2 20
ezooms.bot@gmail.com 20
sohu-search 20
Vegibot 20
LinkdexBot 20
NPT 20
VSE/1.0 20
Pimonster 20
WWW 20
IXECrawler 20
SuperGet 20
rj302004.crawl.yahoo.net 19
cXensebot 19
WotBox 19
YandexVideoParser 19
MicrosoftURL 19
abayonne-256-1-89-49.w90-45.abo.wanadoo.fr 19
spider23.picsearch.com 19
search.live.com 19
solomono 19
Netcraft 19
um-FC 19
MuscatFerret 19
RankActiveLinkBot 19
WebTrends 19
rj302006.crawl.yahoo.net 19
Y!J-ASR 19
spider25.picsearch.com 19
AlexaBot 19
rj302005.crawl.yahoo.net 19
ACONTBOT 19
crawler.bloglines.com 19
METASpider 19
rj302007.crawl.yahoo.net 19
endecawebcrawler 19
ImageScapeRobot 19
atoulouse-257-1-118-62.w82-125.abo.wanadoo.fr 19
VeriCiteCrawler 19
msnbot/2.0b 19
WinHttp 19
MnoGoSearch 19
rj302008.crawl.yahoo.net 19
onlinehome-server.info 19
ImageScapeRobot(lim@cs.leidenuniv.nl) 19
RadiationRetriever 19
CrazyWebCrawler 19
//www.asona.org) 19
MetaURI 18
Picsearch 18
JoBo 18
GetFoundBot 18
aboundex/0.3 18
BotOnParade 18
FlickBot 18
webcopier 18
SpiderJack 18
w2gbot 18
//www.SearchEngineWorld.combot 18
FatBot2.0 18
dloader 18
Memo 18
MixBot 18
Intelix 18
Mirror 18
Slurp.so/1.0 18
SkimBot 18
Slurp/3.0-AU 18
//www.WebmasterWorld.combot 18
Detectify 18
//www.seokicks.de/robot.html) 18
PHP 18
Yasni 18
yahoobot 18
Slurp/2.0-OwlWeekly 18
linguatools 18
webvac 18
Shelob 18
Isidorus 18
bumblebee 18
Slurp/2.0j 18
W3C_Validator 17
Website 17
DownloadWonder 17
Perl 17
Extreme\Picture\Finder 17
our\agent 17
Heretrix 17
Vintage 17
httpscraper 17
LinqiaMetadataDownloaderBot 17
SocialRankIOBot 17
Openfind\data\gatherer 17
nexuscache 17
fireball 17
GetSmart 17
Xara 17
Webhook 17
Webdownloader 17
Siteimprove 17
Alligator 17
ColdFusion 17
FileHound 17
larbin_2.6.2vitalbox1@hotmail.com 17
anarchie 17
Link 17
Y!TunnelPro 17
LinqiaScrapeBot 17
Webster\Pro 17
Qualidator 17
DA 17
Xaldon 17
onestop 17
FAST\WebCrawler 17
WellsSearchII 17
Net\Probe 17
Konqueror 17
80bot 17
WebCopy 17
ZBot 17
PHPot 17
Sqworm/2.9.85-BETA(beta_release;20011115-775;i686-pc-linux 17
slysearch 17
Missigua 17
HTTPviewer 17
Fetch\API\Request 17
Bot\mailto 17
QueryN 17
laycat 17
NetZip 17
Copyscape 17
PingALink\Monitoring\Services 17
Voyager 17
semager 17
httpfetcher 17
Mozilla/4.0(compatible;NetcraftWebServerSurvey) 17
Web\Downloader 17
PHP\version 17
Rico 17
Stealer 17
Websucker 17
B2w 17
Libby_ 17
clsHTTP 17
Websites 17
ClariaBot 17
Webminer 17
TwengaBot/2.0 17
Google-bot 17
googlebot-news 17
BackStreet 17
FreeFind.com 17
YandexBlog 17
AISearchBot 17
WebReaperv9.8-www.webreaper.net 17
fr-crawler 17
lwp\request 17
GigablastOpenSource 17
HitboxDoctor 17
Downloader 17
NationalDirectory\WebSpider 17
Zookabot 17
GosoSpider 17
 * 17
pingdom 17
libwwwperl 17
WinHttpRequest 17
COAST\WebMaster 17
Ping 17
Mewsoft\Search\Engine 17
HTTPapp 17
Diamond 17
Python\urllib 17
Seekbot 17
PerlLWP 17
sogouwebspider 17
DISCo\Pump 17
Vayala 17
Metacarta 17
Webmirror 17
Wells 17
PageAnalyzer 17
RAMPyBot 17
Wildsoft\Surfer 17
Jonzilla 17
HTTPTrack 17
Nmap 16
Metauri 16
berlin-fu-cow 16
008/0.85 16
w3mir 16
SEOstats 16
AllSubmitter 16
MarkMonitor 16
Voil 16
Anarchie 16
MarkWatch 16
Sqlworm 16
HTMLparser 16
WebsiteExtractor 16
Meltwater 16
Joomla 16
WebEmailExtrac 16
Brandprotect 16
Extractor 16
Devil 16
YioopBot 16
InfoTekies 16
Turingos 16
grub-client-1.2.1 16
vobsub 16
Havij 16
Demon 16
AskTbORJ 16
Badass 16
BuiltWith 16
WebMirror 16
T0PHackTeam 16
Evil 16
Telesphorep 16
T8Abot 16
WhitevectorCrawler 16
8LEGS 16
Yandex* 16
Twice 16
InfoSpiders 16
ECCP/1.0 16
Nameprotect 16
Mechanize 16
AIBOT 16
Bigfoot 16
Googlebot-Image/1.o 16
LeechGet 16
bingbot-mobile 16
Clewwa-Bot 16
BBBike 16
Trendictionbot 16
msnbot-media/1.1 16
Apexoo 16
LinqiaRSSBot 16
Openvas 16
Ripper 16
WEBDAV 16
WebFuck 16
Blow 16
GetWeb 16
teleport 16
PyCurl 16
Brandwatch 16
WebFetcher 16
WebCatcher 16
Picscout 16
Cosmos 16
POE-Component-Client-HTTP 16
WebPix 16
Whack 16
VidibleScraper 16
musobot 16
WISENutbot 16
OrangeSpider 16
voyager/1.0 16
Telesphoreo 16
Disco 16
Covario-IDS/1.0 16
UrlPouls 16
Sucuri 16
YahooSlurp 16
SpiderBot 16
GridBot 16
RepoMonkey* 16
CSHttp 16
Cogentbot 16
CPython 16
abot 15
FurlBot 15
Appie 15
Ingrid 15
MRSPUTNIK 15
Pipl 15
iskanie 15
Accoona 15
DSurf 15
YandexCatalog 15
MailSweeper 15
FairShare 15
yoozBot 15
python-requests 15
A6-Indexer 15
123People 15
xGet 15
PangusoSpider 15
AntenneHatena 15
facebot 15
nutch-solr-integration 15
Mail 15
Yahoo-Blogs/v3.9 15
EliteSysEntry 15
AlvinetSpider 15
Templeton 15
WebWalk 15
Mediapartners 15
munky 15
seoengbot 15
//www.flamingosearch.com/bot) 15
taptubot 15
SnapPreviewBot 15
Seznambot 15
GurujiBot 15
atSpider 15
VIREL 14
Discobot 14
MyFamilyBot 14
grapeFX 14
Pingdom.com_bot 14
yanga 14
WorldBrewBot 14
Y!J-BSC 14
trovit 14
facebookexternalhit/1.0 14
Webwhacker 14
//www.opensiteexplorer.org/dotbot,help@moz.com) 14
GetBot 14
Zyborg 14
jobrapido 14
TurnitinBot/2.1 14
vilainrobot 14
//www.cuill.com/twiceler/robot.html 14
Ruby 14
gonzo1 14
twengabot 14
moget* 14
PhpDig* 14
woozweb-monitoring 14
seoscanners.net/1 14
IntegromeDB 14
Wget/1.10.2 14
YahooPipes2.0 14
yodaobot 14
Webzip 14
HappyFunBot 14
MentorMateSpider 14
deusu 14
sprobot 14
MyOnID 14
IssueCrawler 14
Java/1.6.0_10 14
AdsBot-Google-Mobile 14
KFSW-Bot 14
polybot 14
LightningDownload 14
penthesilea* 14
DiamondBot 14
intelium_bot 14
MSIE 14
inktomi 14
knowaboutBot 14
EMail\Wolf 13
Summify 13
datagnionbot 13
Web\Fuck 13
needle 13
boardreader 13
ebingbong 13
mojeek 13
Go-http-client 13
Shai'Hulud 13
GetUrl 13
java 13
lanshanbot 13
nettrack 13
WebDownloader 13
ScooperBot 13
sqlmap 13
UltraSeek 13
KDDExploror 13
Web\Reaper 13
MSSearch 13
iblog 13
postrank 13
Page\Analyzer 13
searchestate 13
CyberPatrol 13
LapozzBot 13
QueryN\Metasearch 13
Page\Grabber 13
EMail\Collector 13
Web\Whacker 13
Battleztar\Bazinga 13
jbrofuzz 13
gonzo1P 13
webcollage 13
MSIE\6.0 13
calculon\spider 13
Yandex 13
imagefetch 13
Web\Fetch 13
MS\Web\Services\Client\Protocol 13
Website\Extractor 13
HavIndex 13
seomoz 13
msrabot 13
whatweb 13
gonzo2 13
Web\Bandit 13
nessus 13
spammen 13
nikto 13
sitebeam 13
nibbler 13
Web\Pix 13
VB\Project 13
freeuploader 13
Microsoft\Data\Access 13
Sogou\web\spider 13
webshag 13
//pubget.com/help/bot) 13
80legs.com 13
gonzo2P 13
Turnitin\Bot 13
Site\Sucker 13
MaxPointCrawler/Nutch-1.10 13
EMail\Siphon 13
Holmes 13
Web\Stripper 13
zgrab 13
Web\Enhancer 13
Collage 13
libwhisker 13
ssearcher100 13
WEBDAV\Client 13
getintent 13
PBWF 13
Lipperhey 13
PagesInventory 13
WiseGuys\Robot 13
Web\Sauger 13
squirrobot 13
internetVista\monitor 13
Landau-Media-Spider 13
MSNBOT_Mobile 13
Screaming\Frog\SEO\Spider 13
dirbuster 13
convera 13
Name\Intelligence 13
Image\Fetch 13
ESIRover 13
memoryBot 13
Web\Copier 13
InternetExplore 13
Turnitin\Robot 13
Download\Devil 13
sitevigil 13
Acoon 13
MarketwireBot 13
dragonfly 13
fimap 13
FuelMyRoute.com 13
netEstate\NE\Crawler 13
SISTRIX\Crawler 13
Sitebot 13
memoryBot* 13
kraken 13
EMail\Extractor 13
flunky 13
Spaidu 13
Y!J-MBS/1.0 12
LipperheySEOService 12
//www.sogou.com/docs/help/webmasters.htm#07 12
Covario 12
SemanticScholarBot 12
SafeSearchmicrodatacrawler 12
Yahoo-MMCrawler/3.x 12
Y!J-SRD/1.0 12
citeseerxbot 12
cb/nutch 12
Sogou+web+spider 12
Libwww-perl 12
crawly 12
Webbot 12
^Pixray 12
swebot 12
DepSpid 12
googlespider 12
linksmanager_bot 12
uipbot/1.0(uipbot@semasio.net) 12
WebCopierv3.3 12
Vortex 12
SogouPicSpider/3.0 12
Moreoverbot 12
Exabot/2.0 12
linksmanager 12
*#directedtoallspiders 12
ZipppBo 12
Slurp/2.0-Kite-Hourly 12
Googlebot/2.1 12
Pingdom 12
Trendiction-Bot 12
AhrefsBot/1.0 12
NetSprint 12
phpcrawl 12
MantraAgent 12
VoilaBot* 12
lssrocketcrawler 12
^echobot 12
Bot 12
GrubNG 12
CyberAlert 12
ICCrawler-ICjobs 12
httpunit 12
iisbot 12
FatBot/2.0 12
QihooBot 12
MicrosoftURLControl* 12
Nutch* 12
OmnitureTestAndTargetCrawl 12
BlogPulseLive 12
TerrawizBot 12
Yandex/1.01.001 12
DTAAgent 12
^Lydia 12
MVAClient 12
myurlcrawler 12
Teoma#ASK.com 12
//www.google.com/bot.html) 12
BusinessWireBot 12
WebAltaCrawler/2.0 12
Socialradarbot/2.0 12
InfoSeekSidewinder/9.0 12
OpenXSpider 11
JaxifiedBot 11
PageBitesHyperBot 11
sbider 11
Ruky-Bot 11
AppleWebKit 11
Cityreview 11
SogouOrionspider/3.0 11
infometrics-bot 11
Yasaklibot 11
askpeter_jeanie 11
JobRoboter 11
PostPost 11
Sputnik 11
uMBot-FC 11
vkShare 11
sistrixcrawler 11
iearthworm/1.0 11
echobot 11
Sidewinder 11
jyxobot 11
mogimogi 11
ContextAdBot 11
Nebullabot 11
searchlink 11
Yandex/1.01.001(compatible;Win16;P) 11
newslookup-bot 11
KoepaBot 11
EsperanzaBot 11
LinkChecker 11
ExaBot 11
Peew 11
AnyApexBot 11
website-datenbank.de 11
YandexBot3.0 11
iearthworm 11
Mail.RU_Bot/Fast/2.0 11
askpeter_bot 11
SistrixCrawler 11
TwengaBot1.x 11
SimpleCrawler 11
MJ12bot#onlythenewsservice 11
VYU2 11
Scoutjet 11
Shoularobot 11
BOT 11
bingspider 11
FAST-WebCrawler3.6 11
NewsGator 11
AddSugarSpiderBot 11
wf84 11
Sogou-Test-Spider/4.0 11
envolk[ITS]spider 11
RexyoBot1.11 11
Orbiter 11
Eurobot/1.0 11
FAST-WebCrawler3.7 11
AddThis 11
Yahoo* 11
NetSeerCrawler 11
blekkobot 11
NutchOrg 11
//wortschatz.uni-leipzig.de/findlinks/) 11
ArchiveTeamArchiveBot 11
Netluchs 11
Acunetix 11
lwp* 11
SearchSight 11
LDSpider 11
WomlpeFactory 11
heise-IT-Markt-Crawler 11
Twitter 11
SetLinks 11
updated 11
Qseero 11
Nymesis 11
SocialShare 11
MooseBot 11
newslebot 11
FAST-WebCrawler3.8 11
Urlfilebot 11
Sogouheadspider/3.0 11
SetLinksbot 11
EmailSearch 11
NG-Search 11
truwoGPS 11
dloader(NaverRobot) 11
yoogliFetchAgent 11
FAST-WebCrawler3.x 11
pixraybot 11
Grapeshot 11
FDSErobot 11
Maxthon 11
silk 11
Mozilla/5.0(compatible;Ezooms/1.0;ezooms.bot@gmail.com) 11
g2crawler 11
TheSuBot 11
ConveraCrawler/0.9d 11
mabontland 11
PrivacyFinder 11
oegp 11
SetLinksbot2.0 11
STIBcrawler 11
Y!J 11
Francis 11
gyffu_bot 10
mediajam-bot 10
Sosoimagespider 10
Linknzbot 10
LDRbot 10
Gigablast 10
Sistrix 10
aipbot* 10
PHPCrawl 10
NetinfoBot 10
WikioImagesBot 10
WiseGuysRobot 10
MicrosoftURLControl-6.01.9782 10
"Mozilla/5.0(Java)outbrain" 10
//search.msn.com/msnbot.htm) 10
MarketBrewBot 10
Yeti/1.0 10
Yet 10
megaindex 10
BPImageWalker/2.0 10
focuseekbot 10
HubSpotPageFetchingBot 10
aipbot/1.0 10
Newsbot 10
Websquash.com 10
SwayyAPI 10
//ahrefs.com/robot 10
duckduckbot 10
BufferBot 10
PHP/ 10
Linknzbot* 10
Friendlyrobot 10
Sogouwebspider/3.0 10
EveryoneSocialBot 10
findxbot 10
Moozilla 10
MissiguaLocator 10
LWP 10
JakartaCommons 10
SeznamBot/3.0 10
changedetection 10
BadBot 10
LWP* 10
AcademicBot 10
omniexplorer_bot 10
L.webis 10
//help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) 10
admantx 10
*#allowallbots 10
lssbot 10
ia_archiver* 10
PictureBot 10
//www.baidu.com/search/spider_jp.html) 10
WebCrawler 10
y!j-bsc 10
HTTrack3.0x 10
SinaWeiboBot 10
RightIntelBot 10
y!j/1.0 10
EMailExractor 10
DynamicTemplates/2.0 10
NutchCVS/0.06-dev 10
MorningPaper 10
Y!J-BSC/1.0 10
sReleaseBot 10
htdig/3.1.5 10
ZoomBot 10
Ahrefsbot 10
Esribot 10
siteexplorer.info 10
facebookexternalhit/* 10
8484BostonProjectv1.0 10
NG\1.x(Exalead) 10
montastic-monitor 10
dtSearchSpider 10
Linknzbot2004 10
EnaBot 10
fr_crawler 10
//www.cuill.com/twiceler/robot.html) 10
WhoWhere 10
xintellibot 10
Lipperhey-Kaus-Australis 10
WX_mail/2.000 10
YahooYSMcm/2.0.0 10
robotleapit.com 10
wombatbot 10
Yandex.GazetaBot 10
Ask 9
//www.google.com/adsbot.html) 9
Web.Imge.Collector 9
netresearchserver 9
*libwww* 9
OpenISearch 9
Majestic12 9
drupact 9
Lite 9
crawler 9
fluffy 9
Yahoo!Slurp/3.0 9
BeslistBot 9
NewsNow 9
Gooblebot 9
Yandex/1.01.001(compatible;Win16;H) 9
RealSpider 9
Yandex/1.01.001(compatible;Win16;I) 9
WWW-Mechanize 9
siteimprove 9
toplistbot 9
binlar 9
domaincrawler 9
Sysomos 9
EmeraldShield.com 9
RetrevoPageAnalyzer 9
//github.com/typhoeus/typhoeus 9
urlappendbot 9
//www.picsearch.com/bot.html) 9
MojeekBot\/ 9
MetaJobBot 9
Search 9
eknip 9
deadlinkchecker 9
webalta 9
Commons-HttpClient 9
backlinkcrawler 9
Mozilla/4.0(compatible;Zealbot1.0) 9
Megalodon 9
g00g1e.net 9
openindexspider 9
ec2linkfinder 9
DeuSu\/ 9
*Fetch* 9
MSNPTC 9
KomodiaBot 9
biglotron 9
webcompanycrawler 9
Cliqzbot\/ 9
wGet 9
gnamgnamspider 9
Twicelerwww.cuill.com/twiceler/robot.html 9
CapsuleChecker 9
WebEMailExtrac 9
panscient 9
europarchive.org 9
web-archive-net.com.bot 9
//tineye.com/crawler.html) 9
yandeximages 9
scribdbot 9
SlurpChina 9
Nusearch 9
//www.dotnetdotcom.org/,crawler@dotnetdotcom.org) 9
gslfbot 9
//research.microsoft.com/research/sv/msrbot/) 9
Trove 9
OfflineExplorer/1.5 9
findlink 9
SemrushBotSemrushBot-SA 9
UsineNouvelleCrawler 9
BorderManager* 9
WebduniaBot 9
*\ 9
buzzbot 9
yahoo-blogs 9
brainobot 9
edisterbot 9
bender 9
Mozilla/4.0 9
//feedback.redkolibri.com/ 9
Sonic 9
d3d9446802a44259755d38e6d163e820 9
b2w 9
ngbot 9
NTENTbot 9
Livelapbot 9
collection@infegy.com 9
//www.kototoi.org/zao/) 9
//www.searchtechnologies.com) 9
integromedb 9
webzip 9
IBM_Planetwide 9
wocbot 9
nerdybot 9
webmon 9
BDCbot 9
//fulltext.sblog.cz/screenshot/) 9
IOI 9
bibnum.bnf 9
GlutenFreeCrawler\/ 9
AddThis.comrobot 9
Mozilla/4.0(compatible;MSIE4.0;Windows9 9
IRLbot/3.0 9
CCMetadataScaper 9
MixrankBot 9
elisabot 9
contentcrawlerspider 9
y!j-asr 9
AcoonBot 9
acoonbot 9
bnf.fr_bot 9
Googlebot-Image/1.0 9
arabot 9
//www.semrush.com/bot/ 9
//www.WISEnutbot.com) 9
//webmeup-crawler.com/ 9
SeznamBot/3.2 9
OmniExplorer 9
sogoudevelopspider 9
simplepie 9
//help.naver.com/robots/) 9
findthatfile 9
tagoobot 9
HeinrichderMiragoRobot 9
VoilaBotBETA1.2 9
Java* 9
seokicks 9
google-proxy 9
cometrics-bot 9
* #*forallagents 9
spock 9
Psbot-Imagesearch 9
SemrushBot/0.92 9
page2rss 9
grub.org 9
summify 9
rabaz 8
istellabot/t.1 8
ezooms.bot 8
LemurWebCrawler 8
BecomeBot/3.0 8
Exabot-Images/1.0 8
iCjobs/3.2.3 8
Artabus 8
um-IC 8
WebReaperv9.7–www.webreaper.net 8
WebSnake 8
OfflineCommander 8
iCcrawler-iCjobsStellenangeboteJobs 8
ichiro/2.0 8
Owlin 8
YandexImages* 8
Buzzbot 8
SapphireWebCrawler/1.0 8
PeterBot 8
Mozilla/4.0(compatible;MSIE4.01;WindowsNT;MSSearch5.0Robot) 8
coc_coc 8
MicrosoftURLControl–5.01.4511 8
Arachnoidea 8
MissiguaLocator1.9 8
Mozilla/4.0(compatible;MSIE6.0;WindowsNT;MSSearch5.0Robot) 8
sproose 8
Zing-BottaBot/2.0 8
MSRBot 8
NGBot/4.5 8
adnbot 8
Twicelerwww.cuill.com/robots.html 8
ISCSystemsiRcSearch2.1 8
Java/1.6.0_13 8
Mozilla/4.0(compatible;MSIE4.01;WindowsNT;MSSearch4.0Robot) 8
GSiteCrawler 8
PingdomGIGRIB 8
support.voilabot@orange-ftgroup.com 8
ZipppBot 8
//www.majestic12.co.uk/projects/dsearch/mj12bot.php 8
dloader(SpeedySpider) 8
Bitaclebot/1.1 8
//help.goo.ne.jp/door/crawler.html) 8
CCBot/1.0 8
WebsiteQuester–www.esalesbiz.com/extra/ 8
//misc.yahoo.com.cn/help.html) 8
TutorialCrawler 8
boitho 8
SEMrushBot-Desktop 8
oodlebot/1.0 8
Leech 8
ImplisenseBot 8
webmeasurement-bot 8
ptd-crawler 8
//moz.com/researchtools/ose/dotbot 8
Pythonurllib 8
WebReaperv9.8–www.webreaper.net 8
Arachmo 8
//siteexplorer.info/Backlink-Checker-Spider/ 8
LinkWalker/2.0 8
zibber 8
Jetslide 8
PingdomGIGRIBv1.1 8
OfflineExplorerPro 8
RU_Bot 8
HMSE_Robot 8
magpie-crawler* 8
webmeup-crawler 8
WebReaperv9.1–www.otway.com/webreaper 8
AMZNKAssocBot 8
Snapbot/1.0 8
SurveyBot/2.3 8
Zend_Http_Client 8
WebsiteQuester–www.asona.org 8
//www.qwant.com/ 8
KINGSpider 8
MicrosoftURLControl–6.00.8169 8
* 8
*#matchallbots 8
WebReapervWebReaperv7.3–www,otway.com/webreaper 8
Infovell 8
Netvibes 8
Bitaclebot 8
HDnutchagent/Nutch-1.1(Think) 8
AutoSpider1.0 8
Googlebot#robotalquevadirigido 8
LoadTimeBot 8
//www.exalead.com/search/webmasterguide 8
sg-Orbiter 8
linkwalker 8
COMODOSpider/Nutch-1.2 8
mindUpBot 8
OSCE 8
Sosospider/2.0 8
FollowSiteBot 8
WebDataCentreBot 8
SixtrixCrawler 8
SurveyBot/2.3(WhoisSource) 8
HTTPWeazel 8
//urlm.nl/ 7
Exalead 7
Semager 7
StratagemsKumo 7
//moz.com/help/guides/moz-procedures/what-is-rogerbot 7
FeedBooster 7
yandex* 7
//napoveda.seznam.cz/en/seznamcz-web-search/crawling-control/ 7
//www.twenga.com/bot.html 7
peekyou 7
wikiwix-bot 7
Bad-Robot 7
ichiro/2.0(ichiro@nttr.co.jp) 7
Searcharoo 7
webcollage/1.77 7
naughtyrobot 7
Sogouweb 7
linkdexbot-mobile/2.1 7
Wget/1.9.1 7
Jobrapido 7
BigBrother 7
InktomiSlurp 7
Mozilla/4.0(compatible;B-l-i-t-z-B-O-T) 7
Bitacle* 7
snitch 7
netEstate 7
yodao 7
Java/1.6.0_07 7
TurnitinRobot 7
Exabot-Images 7
megaindex.com/crawler 7
mediapartners-google 7
NetSeer 7
LipperheySpider 7
HttpTool* 7
BecomeJPBot 7
FetchAPIRequest 7
FHscan 7
wink 7
Apple-PubSub 7
*#directedtoallrobots 7
Sproose 7
discovery 7
Java/1.5.0_06 7
CMSCrawler 7
SEOKicks 7
SEOKicks-Robot 7
//www.wise-guys.nl/webcrawler.php 7
legs 7
//majestic12.co.uk/bot.php?+) 7
zoominfo 7
bingbot #topreventbingbogfromcrawlingtoomuch 7
SearchmetericsBot 7
baidu* 7
businessdbbot 7
//yandex.com/support/webmaster/controlling-robot/robots-txt.xml 7
StackRambler/2.0 7
Pingdom.com 7
tweepz 7
Charlotte/1.0b 7
Owlinbot 7
ExaleadCloudView 7
Bilbo/2.3b-UNIX 7
abuse-report@terrykyleseoagency.com)" 7
bluemasterbot 7
ICCrawler 7
larbin_2.6.1 7
WebDownloader/5.7 7
Curl 7
Pcore-HTTP 7
opera 7
seostats 7
rapleaf 7
DataSpear 7
rogerBot 7
5 7
aihit 7
bspider 7
WebsiteWiki 7
/images/ 7
Java/1.5.0_05 7
ReadCube 7
NexaBot 7
yoname 7
wiederfreibot/1.0 7
OWR_Crawler 7
//help.yahoo.com/kb/SLN22600.html 7
YandexSomething/1.0 7
slurp@inktomi 7
YandexAccessibilityBot 7
YahooSeeker/CafeKelsa 7
fairshare.cc 7
*newsexpress* 7
Zing-BottaBot/1.0 7
Gigabot* 7
SEOprofiler 7
PricecloudBot 7
InfoPath.1 7
uw_cse_xwc 7
Veoozbot 7
Ichiro 7
LMQueueBot/0.2 7
LinksManager 7
scoutjet 7
allowfromall 7
iaskspider 7
BBot 7
yourtraces 7
Y!J-BRJ/YATScrawler 7
DoCoMoSpider 7
coccocbot-web 7
funnelback 7
ZyBorg/1.0 7
SiteSeekerCrawler/1.0 7
zyborg 7
MSNBot-NewsBlogs 7
findestars 7
PGBot 7
msnbot #topreventmsnbotfromcrawlingtoomuch 7
FAST-WebCrawler/3.6/FirstPage 7
360spider 7
WillyBot 7
OpenX 7
spinn3r 7
zermelo 7
bingbot* 7
crawl6.exabot.com 6
genieBot(wgao@genieknows.com) 6
HTTP/1.1zibber-v0.1(www.zibb.com/crawler/) 6
Slurp/2.0-KiteHourly 6
WebCopierv3.3.0 6
YandexBot/3.0-MirrorDetector 6
nexen 6
Snappybot 6
orderallow,deny 6
classroombot 6
Wada.vnVietnameseSearch/2.1 6
Mozilla/3.0(compatible;IndyLibrary) 6
webbandit 6
Java/1.5.0_10 6
//majestic12.co.uk/bot.php?) 6
XenuLinkSleuth1.2i 6
PSIBots 6
moget/2.1(moget@goo.ne.jp) 6
Webspider 6
www.websuche.de-spider 6
ApexooSpider1.0 6
voyager/2.0 6
SBIder/SBIder-0.8.2-dev 6
Cabot 6
SlySearch/1.0 6
//www.page-store.com) 6
YahooFeedSeeker 6
/sbconf/ 6
//search.yahoo.com/) 6
T-OnlineBrowser 6
page_prefetcher 6
/enc/ 6
ru_com_viewer 6
Aghaven 6
therarestparser 6
GOOGLEBOT 6
zibberv0.1 6
CiZilla 6
del.icio.us 6
YoudaoBot/1.0 6
Exabot-Images/3.0 6
aiHitBot/1.1 6
AttributorCorporationDMCABot 6
JavaBot 6
LaBot 6
OmniExplorer_Bot/1.09 6
//irl.cs.tamu.edu/crawler) 6
/storeadmin/ 6
QEAVis 6
whoisde.de 6
uberbot 6
SuperBot/4.4.0.60(WindowsXP) 6
fast-Webcrawler 6
percbotspider 6
compatible; 6
seolytics 6
JavaBot2.0 6
worio 6
//www.become.com/site_owners.html) 6
/www.innerprise.net/usp-spider.asp) 6
librabot 6
MagpieRSS 6
Scooter/2.0 6
Bork-edition 6
StackRambler/2.0+(MSIE+incompatible) 6
findlinks/1.1.3-beta9 6
Axonize-bot 6
RiceComputerArchitecture 6
Wada.vnVietnameseSearch 6
Caliperbot 6
msnbot/1.1 6
WebDownloader/6.9 6
IBMEVV 6
ASPSimply 6
AdIdxbot 6
Findexa 6
httpclient 6
//www.website-datenbank.de/) 6
Mozilla/4.0(compatible;MSIE5.05;WindowsNT4.0) 6
libcurl-agent 6
aiHitBot/1.0 6
cyberpatrol 6
Qwantify/2.3w 6
Webcollage 6
fatbot 6
OutfoxBot/0.5 6
fastmetawebcrawler 6
LexiBotWebImageCollector 6
YandexOntoDBAPI 6
ZapBot 6
LinkWalker/2.0-www.seventwentyfour.com 6
crawl13.exabot.com 6
sitecat 6
Win32 6
Googlebot-mobile 6
GoForIt.com 6
Cazoodle-Bot 6
sphsearch 6
whois.de 6
MouseBOT 6
libwww-perl/5.65 6
noxtrumbot/1.0 6
VeBot 6
Gasbot 6
wise-guys 6
AmazonCloudServices 6
Acoon-Robot 6
SeznamBot/1.0 6
python 6
Everest-Vulcan 6
denyfrom82.99.30. 6
nambu 6
speedyspider 6
Yandex/1.01.001(compatible;Win16;m) 6
OnTownsBot 6
TeleportProMIIxpc 6
LocalBot 6
SearchBot 6
R6_FeedFetcher(www.radian6.com/crawler) 6
Blexbot 6
NationalDirectory-SuperSpider 6
YandexSomething 6
RepoMonkeyBait&Tackle 6
slurp@inktomi.com 6
botmobi 6
IntegraTelecom 6
+BecomeBot/3.0 6
InfluenceBot 6
//www.turnitin.com/robot/crawlerinfo.html 6
searchme 6
paraisobot 6
Mozilla/4.0+compatible+ZyBorg/1.0+ 6
SensisWebCrawler(search_comments\\at\\sensis\\dot\\com\\dot\\au) 6
Wget/1.9+cvs-stable(RedHatmodified) 6
GWPImages 6
ViolaBot 6
bsalsa 6
OpenWebIndex/Nutch-1.6 6
Ichiro3.0 6
AdsBot-google 6
phpversion 6
PRTGCloudBot 6
VCIVCI 6
filterdb.iss.net/crawler 6
eventax 6
Websense 6
Fess 6
Piffany_Web_Cacher_v0.91 6
BobCrawl 6
HornySexSearch 6
ExDomain 6
aihitdata 6
larbin_2.6.1+(larbin2.6.1@unspecified.mail) 6
WEPSearch 6
ServageRobot 6
Mozilla/4.0(compatible;MSIE6.0;WindowsNT5.0) 6
Sensis+Web+Crawler+ 6
OmniExplorer_Bot/1.07 6
LinkAider 6
ActiveTouristBot 6
internetseer 6
Vocus 6
/svgallery/ 6
User-Agent 6
//;+bot@bot.bot) 6
ExaLeadCrawler 6
WebarooBot 6
Java/1.5.0_11 6
MaxPointBot 6
^Jakarta 6
SmabblerBot 6
YRSpider 6
AdsBot 6
setooz 6
DTSAgent 6
/template/ 6
//www.jaxcarpentry.com/sitemap.xml 6
EUDORA 6
Crowsnest 6
Plista 6
SeznamBot/1.1 6
WebDownloader/4.5 6
ThumbSniper 6
MJ12bot* 6
golabbot 6
JobCrawlerBot 6
fastEnterprisecrawler 6
findlinks/2.6 6
sogou+spider 6
SuperPages 6
YandexSomething/1. 6
GarlikCrawler 6
FunWebProductsf39-2350 6
ZupeeCrawler 6
Advista 6
perl 6
TencentTraveler 6
/www.spidersoft.com) 6
Mozilla/2.0(compatible;AskJeeves/Teoma) 6
kyluka 6
^Java 6
RootleCrawler 6
Wada.vn 6
Avant+Browser+ 6
ModifiedGMMBotRightHere 6
Slicehost 6
//discoveryengine.com/discobot.html) 6
eagle 6
majestic 6
shopwiki 6
WGet 6
ia_archive 6
tarspider 6
Servage 6
boitho.com 6
/secure/ 6
wegobot 6
IBM 6
complex_network_group 6
AITCSRobot 6
arks 6
genieBot+(wgao@genieknows.com) 6
Linkdex 6
FacebookExternalHit 6
CovarioIDS 6
/www.asona.org) 6
LinkStats 6
True_Robot/1.0turingos 6
sohuagent 6
RedCarpet 6
exooba 6
VWBot 6
urllib 6
centrumbot 6
genieBot+wgao@genieknows.com 6
OmniExplorer_Bot/1.10 6
MJ12bot/ 6
Accoona-AI-Agent 6
datacha0s/2.0 6
T-Rex 6
/go/ 6
DjangoTraineeBot 6
Sika 6
SmabblerBot/1.0 6
Cutbot 6
InnovantageBot 6
//help.yahoo.com/help/us/ysearch/slurp) 6
Mozilla/4.0(compatible;MSIE5.0;Windows95)VoilaBotBETA1.2 6
Mozilla/4.0(compatible;MSIE4.01;WindowsNT;MSSearch6.0Robot) 6
AideRSS2.0 6
robotgenius 6
MJ12bot/v1.2.0 6
Directcrawler 6
SlySearch/1.x 6
WebCopierv3.3.2 6
del.icio.us-thumbnails/1.0 6
OpenAcoon 6
[A-Z][a-z]{3,}[a-z]{4,}[a-z]{4,} 6
findlinks/1.1.1-a5 6
dcbot 6
cyberpatrolcrawler 6
yoogliFetchAgent/0.1 6
meds-online24.com 6
Mozilla/4.0(compatible;MSIE5.05;WindowsNT5.0) 6
REAP-Crawler 6
backlink-check.de 6
Mozilla/4.0(compatible;MSIE5.05;WindowsNT3.51) 6
//www.jobrapido.com) 5
MJ12bot/v1.0.7 5
DeepnetExplorer 5
AdMediabot 5
Mozilla/5.0 5
SBider 5
msnbot* 5
*msn* 5
Adidum 5
Tarantula 5
//www.dataprovider.com/spider/ 5
dubaiindex 5
spider_monkey 5
AportWorm 5
cfetch/1.0 5
Voila 5
baiduspider-video 5
Google-Sitemaps 5
Siteliner 5
DataproviderSiteExplorer 5
//openlinkprofiler.org/bot 5
WebVulnScan 5
CloudServerMarketSpider 5
Spanner 5
googlebot-video 5
DuckDuckBot/1.1 5
StanfordCompClub 5
MJ12bot/v1.0.8 5
Java/ 5
//www.xovibot.net/ 5
InternetSeer 5
EmailWolf1.00 5
MJ12bot/v1.0.5 5
istellabot-nutch/Nutch-1.10 5
Yankex 5
Nutraspace 5
jcrawler 5
Jooblebot 5
*Bing* 5
PathDefender 5
MJ12bot/v1.2.3 5
WordPress* 5
MJ12bot/v1.2.4 5
gooblogsearch 5
oBot/2.3.1 5
//www.linkdex.com/en-us/about/bots/ 5
MJ12bot/v1.0.6 5
BuzzSumo 5
Moreover 5
Mozilla/5.0(JobRapidoWebPump) 5
Proximic 5
Diffbot/0.1 5
Node/simplecrawler 5
OwlinBot 5
LoggerManager 5
OpenWebSpider 5
RedBot 5
WebWasher 5
OffByOne 5
ActiveAgent 5
orangebot 5
bixolabs 5
waybackarchive.org 5
turnitin 5
LinkAlarm 5
NING/1.0 5
alltheweb 5
portalU 5
Bitvorebot 5
WebCopier* 5
ConveraMultiMediaCrawler 5
StanfordCompSciClub 5
Majestic-12 5
baiduimagespider 5
netEstateFOAFcrawler 5
FeedBurner 5
192.comAgent 5
wakame 5
Linkdexbot 5
FlamingoSearch 5
lycos 5
Morfeus 5
WIJobRoboterSpider 5
WebVulnCrawl 5
FreeWebMonitoringSiteChecker/0.1 5
BizInformasjon 5
D115Crawler 5
Yet-Another-Spider 5
almaden 5
gazz 5
Slurp\ 5
InnyBot 5
Yahoo!DESlurp 5
URLspy 5
larbin_2.6.3 5
StanfordSpiderboys 5
BlekkoBot 5
msnbot/1.0 5
MindCrawler 5
iCorpusBot 5
scrapy 5
RB2B-bot 5
wikiwix-bot-3.0 5
FunnelBack 5
AhrefsBot/5.2 5
NaverBot-1.0 5
WebIndex 5
minibot(NaverRobot)/1.0 5
innosense 5
CityreviewRobot 5
//www.jobrapido.com)" 5
Sphider 5
baiduspider-image 5
newscan-online 5
msnbot-media/ 5
aspider 5
fido 5
Jyxobot/1 5
Googlebot 5
PercolateCrawler 5
Sensis.com.auWebCrawler 5
HouxouCrawler 5
SemrushBot* 5
likebot 4
jobo 4
Java/1.7.0_51 4
DuckDuckBot/1.0 4
Scooter-3.2 4
BuddhaBot 4
NetcraftWebServerSurvey 4
Y!J-BRJ/YATS 4
Arachnophilia 4
weborama-fetcher 4
AgentLinkSpammer 4
Scooter/1.0scooter@pa.dec.com 4
Jambot 4
echo! 4
Iron 4
Scooter/3.3 4
//URLFAN 4
Seobility 4
webreaper 4
simplecrawler 4
DemoBotDOT16b 4
Scooter/2.0G.R.A.B.X2.0 4
EBrowse1.4b 4
wapspider 4
Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+NT;+MS+Search+6.0+Robot) 4
warebay 4
seznam 4
Scooter2_Mercator_x-x.0 4
Mozilla/4.5(compatible;HTTrack3.0x;Windows98) 4
wbot 4
Marketwirebot 4
Onet.pl 4
EmailExtractor 4
Mozilla/2.0(compatible;AskJeeves) 4
URLmetrics 4
whowhere 4
verifybot 4
attentio 4
Freecrawl 4
webstripper 4
excite 4
WIRE/0. 4
EchO!/2.0 4
Psycheclone 4
solbot 4
W3C-checklink/ 4
/account/ 4
AVFetch1.0 4
FullWebBot0516B 4
Bloodhound 4
Scooter-3.2.JT 4
SynthesioCrawlerreleaseMonaLisa 4
research-spider 4
/* 4
SiteCheck-sitecrawlbySiteimprove.com 4
baiduspider-mobile 4
DataCha0s/2.0 4
Mozilla/5.0(compatible;Pogodak.co.yu/3.1) 4
combine 4
SogouPicAgent 4
wowrack 4
sitesnagger 4
teleportpro 4
AltaVistaV2.0Bcrawler@evreka.com 4
NetShelterContentScan 4
emBot 4
//www.openhose.org/bot.html) 4
MetagerBot 4
baiduspider-news 4
BrandProtect 4
Scooter_trk3-3.0.3 4
spiderbot 4
FDM3.x 4
WebZIP/3.65 4
Baiduspide* 4
Dulancebot 4
SiteSnagger* 4
ebiness 4
OkHttp 4
msnbot-Products 4
myfamilybot 4
marvin 4
baidumobaider 4
Scooter/1.1(custom) 4
DSurf15a81 4
MicrosoftURLControl�6.00.8169 4
Scooter/3.3.QA.pczukor 4
Mozilla/4.0(compatible;Y!J;forrobotstudy;keyoshid) 4
panoptaStudyBot 4
Scooter/ 4
Scooter-3.0.FS 4
Xenu�sLinkSleuth1.1c 4
Linkpad 4
webs 4
webcollage* 4
SpiderKU/0.9 4
ContentSmartz 4
FranklinLocator1.8 4
sygol 4
Synthesio 4
Mozilla/5.0(compatible;Najdi.si/3.1) 4
WebSauger* 4
Fairshare 4
fastcrawler 4
Seekbot/1.0 4
LincolnStateWebBrowser 4
BrightEdgeCrawler/1.0(crawler@brightedge.com) 4
FullWebBot0416B 4
radian6commentreader 4
Wise-Guys 4
Bitvore 4
TsolCrawler 4
twenga* 4
ESurf15a15 4
Butterfly/1.0 4
default.ida 4
waybackarchive 4
butterfly 4
Teleport* 4
gosospider 4
YaDirectBot 4
*1 4
Toweya 4
AWSCloudBased 4
ConveraMultiMediaCrawler/0.1 4
SpiderLing 4
hul-wax 4
WebnewsArianna 4
jikespider 4
inspectorwww 4
omgili 4
Scooter_bh0-3.0.3 4
ariadne 4
DSurf15a01 4
Getleft 4
WebDownloader/5.8 4
FamilyBot 4
Scooter-3.0.HD 4
DomainStatsBot 4
NetZip-Downloader 4
Tweetmeme 4
GomezAgent 4
ecollector 4
griffon 4
SEOsearch/ 4
perlcrawler 4
Ultraseek-www 4
anthill 4
VadixBot 4
openfind 4
WeblexBot 4
DBrowse1.4d 4
Willow 4
y!j-bsc/1.0 4
GlobalSpecLinkChecker 4
Scooter/3.3.vscooter 4
Boomtrain-Content-Bot* 4
BattleztarBazinga 4
MicrosoftURLControl�5.01.4511 4
searchbotadmin@google.com 4
BIGLOTRON(BETA2;GNU/Linux) 4
Scooter-3.2.DIL 4
SputnikBot/2.3 4
WebGather3.0 4
WinInetTest 4
NPBot/3 4
shopstylebot 4
YandexBot* 4
Watching/UnitCrawler 4
+spider@waybackarchive.org 4
Y!J/1.0 4
Gaisbot/3.0 4
FSurf15a01 4
Mozilla/2.0(compatible;NEWTActiveX;Win32) 4
Voinicsbot 4
Scooter/1.0 4
MSIE6.0 4
DBrowse1.4b 4
msnboot 4
seckicks 4
dotnetdotcom 4
Scooter-ARS-1.1-ih 4
WWW-Collector 4
Google-HTTP-Java-Client 4
Scooter-W3.1.2 4
QuickFinderCrawler 4
exabot. 4
DSurf15aVA 4
e-collector 4
YahooSlurp! 4
YahooSeeker/CafeKelsa-dev 4
Argus/1.1 4
msnbot\ 4
elfinbot 4
Scooter/3.3_SF 4
ScreamingFrogSEOSpider/3.3 4
Xenu�s 4
suke 4
GoogleAdSense 4
robofox 4
Scooter/2.0G.R.A.B.V1.1.0 4
Hackertarget.com 4
DinoPing 4
LjSEEK 4
core 4
Baiduspider+( 4
Sogouheadspider 4
Atomz 4
//www.pingdom.com) 4
EmailSmartz 4
mattie 4
Scooter-W3-1.0 4
dotBot 4
special_archiver 4
WebIndexer 4
waybackarchive.org/1.0 4
Omgili 4
spider72.yandex.ru 4
y!j 4
Site24x7 4
FlaxCrawler 4
wwwster 4
XenusLinkSleuth1.1c 4
Embedly 4
revivebot 4
Scooter-3.0.VNS 4
dumbBot 4
Scooter-3.3dev 4
Walhelloappie 4
Facebot/1.0 4
whitevectorcrawler 4
HiScan 4
ECCP/1.0(search@eniro.com) 4
wprocketbot 4
Tibot 4
Scooter-3.2.EX 4
DBLBot/1.0 4
//www.alexa.com/site/help/webmasters;crawler@alexa.com) 4
Scooter-3.0QI 4
LBot 4
WebFilter 4
favorstarbot 4
IstellaBot/1.23.15 4
curl/ 4
Mail.rubot 4
Scooter-ARS-1.1 4
searchprocess 4
ShopWiki/1.0 4
Deep-Crawl 4
+spider@spiderlytics.com 4
Whitevector+Crawler 4
MSWebServicesClientProtocol 4
webcrawl.net 4
radian6Feedfetcher 4
atSpider/1.0 4
ConveraCrawler/0.2 4
Mozilla/4.0(compatible;y!j;forrobotstudy;keyoshid) 4
* #Theserulesapplytoallcrawlers 4
SEOsearchCrawler/ 4
Firefly 4
AlexaBitlybot 4
Mozilla/4.0(compatible;Synapse) 4
Zeus2.6 4
TridentSpider 4
InternetSupervision 4
ContactBot/0.2 4
kraken-crawler/* 4
WebStripper/2.56 4
WebReaper* 4
BingBot/MSNBot 4
OfflineExplorer/1.3 4
ASpider/0.09 4
Scooter-3.2.BT 4
oneriot 4
Scooter-3.0.EU 4
esculapio 4
Voilabot 4
Alexa(IAArchiver) 4
DemoBotZ16b 4
MJ12bot/* 4
GuzzleHttp 4
microsoftbot 4
askjeeves 4
AVSearch-3.0(AltaVista/AVC) 4
NetCaptor 4
Jobo 4
AltaVistaIntranetV2.0AVSEVALsearch@freeit.com 4
pagefreezer 4
ConveraInternetSpiderV6.x 4
MetamojiCrawler 4
Kraken/0.1 4
Scooter-3.2.NIV 4
LinkedIn 4
ImageCollector 4
tbot-nutch 4
*bot 4
pjspider 4
packrat 4
TechnoratiBot/8.1 4
Aghaven/Nutch-1.2 4
templeton 4
*baiduspider.*$ 4
//www.google.com/feedfetcher.html;1subscribers;feed-id=14975829261283866692) 4
emBot-GalaBuzz 4
urlck 4
W3SiteSearchCrawler 4
Smetrics 4
Yahoo-MMCrawler* 4
NetcraftSurveyAgent/1.0 4
pegasus 4
polite-bot 4
LinksManager.com_bot 4
SeaMonkey 4
AltaVistaIntranetV2.0evreka.comcrawler@evreka.com 4
Scooter-3.2.SF0 4
mediafox 4
ZumBot* 4
bl.uk_lddc_bot 4
SoftlayerServer 4
cydralspider 4
scooter-venus-3.0.vns 4
Spbot 4
W3C_*Validator 4
Xenus 4
WebDownloader/4.9 4
PChomebot 4
spbot/4.0.9 4
msnbot-NewsBlogs/ 4
Gigabot/ 4
DuckDuck 4
msnbot-newsblogs 4
BoardPulse 4
MegaIndex.ru/* 4
NetLyzer 4
DSurf15a71 4
Echo! 4
WatchDog/3.0 4
Anonymous 4
Scooter-3.2.snippet 4
myweb 4
sato-crawler 4
FullWebBot2816B 4
DotBot* 4
AltaVistaIntranetV2.0CompaqAltavistaEvalsveand@altavista.net 4
BlueRobot 3
Semalt 3
confuzzledbot 3
msnbot-207-46-13-97.search.msn.com 3
Voltron 3
Alkaline 3
ca-crawler 3
KIT-Fireball 3
webquest 3
Raven 3
botify 3
NameIntelligence 3
Shopwiki 3
visionutils 3
SputnikImageBot 3
borg-bot 3
*MJ12bot* 3
CrawlDaddy 3
RedesScrapy 3
Zitebot 3
dataprovider.com 3
OneRiot 3
snooper 3
WorldBrewBot/2.1 3
InAGistURLResolver 3
pagesinventory 3
Valkyrielibwww-perl 3
crescent 3
LadenZeile 3
mmu-gsa-crawler 3
Webshag 3
OfflineExplorer/1.1 3
strucr 3
Ahrefs-Bot/1.0 3
teomaagent 3
Calculon 3
panopta.com 3
*"lines! 3
//www.bing.com/bingbot.htm) 3
Biglotron 3
Searchmetrics 3
escan 3
BackRub 3
qwant 3
quipu 3
website 3
arachnophilia 3
ko_yappo_robot 3
AlexaRobot 3
//support.paper.li/entries/20023257-what-is-paper-li) 3
scooter #Robotd'Astalavista 3
SemrushBot/1.1 3
webcollage/1.93 3
*MLBot* 3
PMAFind 3
tarantula 3
NaverRobot 3
Netsparker 3
Combine 3
VSynCrawler 3
*BLP_bbot* 3
Atomic_Email_Hunter/4.0 3
Libwhisker 3
qihoo 3
Aipbot 3
Asterias 3
Discoverybot 3
Daum 3
HTTrack\ 3
Monogosearch 3
GlutenFreeCrawler 3
EMailCollector 3
ltbot 3
arale 3
GrapeFX 3
Gootkitauto-rooterscanner 3
Musobot 3
jooblebot 3
WebAltaCrawler/1.3.25 3
Dragonfly 3
Nibbler 3
nzexplorer 3
//www.seokicks.de/robot.html 3
Lftp 3
WIJobRoboterSpiderVersion3 3
linkapediabot 3
patric 3
robi 3
webwalk 3
weblayers 3
Sitesucker 3
hambot 3
Needle 3
htmlgobble 3
NuSearchSpider 3
WebLinkValidator 3
EMailSiphon 3
Lanshanbot 3
titin 3
Niki-bot 3
Wprecon 3
*Birubot* 3
directhit 3
Seeker.lookseek.com 3
Suzuran 3
SiteBot* 3
boxseabot 3
verticrawl 3
CareerBot 3
CovarioCSE 3
WebCorp 3
msnbot-157-55-17-146.search.msn.com 3
Sougouspider 3
msnbot-65-52-104-29.search.msn.com 3
Y!J-BRZ/YATSHAcrawler 3
safesearch 3
grabber 3
*ISTHEFIRSTLINE------------------------------------------ 3
crawling@ubermetrics-technologies.com) 3
*PaperLiBot* 3
IntrafindBot 3
Woobot 3
zoominfobot 3
QIHOO 3
churl 3
deweb 3
*yacybot* 3
cusco 3
charlotte 3
Ecxi 3
Spammen 3
skymob 3
hbot 3
Yandex/* 3
irobot 3
Covario-IDS 3
Likse 3
//www.puritysearch.net/) 3
MetaURIAPI 3
Yandex/1.03.003(compatible;Win16;D) 3
Toata 3
opsh 3
openwebspider 3
,.;/\-) 3
Sitebeam 3
CCBot/2.0 3
trendictionbot0 3
ActiveCacheRequest 3
Re-Animator 3
Y!J-BRY/YATSHcrawler 3
PageScorer 3
Ahrefs-Bot 3
AppEngine 3
Bolt 3
TelegramBot 3
Ebingbong 3
yahoo!slurp 3
pageboy 3
Nettrack 3
*sindice-fetcher* 3
MorfeusFuckingScanner 3
Yeti/0.01 3
ChinaLocalBrowse2.6 3
Unknownrobot(identifiedby'bot*') 3
Funnelback 3
NEC-MeshExplorer 3
LinksCrawler 3
bbot 3
Sensis 3
msnbot/ 3
Quantcastbot/1.0 3
Litemage_walker 3
GravityStream 3
uMBot 3
Scanbot 3
Gromit 3
lifestyle_contributor 3
AhrefsBot* 3
youdao 3
msnbot-157-55-18-23.search.msn.com 3
URLMetriken 3
DomainTools 3
GoogleBot-Mobile 3
Humanlinks 3
LinkCheckbySiteimprove.com 3
EgotoBot 3
Drecombot 3
newsexpress 3
BetaBot 3
Flexumspider 3
*Flamingo_SearchEngine* 3
Collective 3
Rankbot 3
Twikle 3
Fimap 3
esther 3
willybot 3
sift 3
FreeWebMonitoringSiteChecker 3
duckduckgo 3
msnbot-157-55-18-24.search.msn.com 3
msnbot-65-52-110-34.search.msn.com 3
muninn 3
ShopAlikeBot 3
ccubee 3
xirq 3
Nederland.zoek 3
WISEnutbot 3
Masscan 3
PortalBSpider 3
jumpstation 3
CCbot 3
crawl31 3
Iblog 3
Zgrab 3
iajabot 3
msnbot-65-52-104-115.search.msn.com 3
twenga.com 3
Webclipping.com 3
Jbrofuzz 3
udmsearch 3
QueryCAT 3
bot*. 3
OfflineExplorer/1.8 3
SpamBayes 3
vebidoo 3
WebCopier\ 3
Mozilla/5.0(compatible;Pogodak.co.yu/3.1 3
Wget1.6 3
//www.facebook.com/externalhit_uatext.php) 3
libcurl 3
ia_archiverrr 3
meanpathbot/1.0 3
crawl31+ 3
peerindex 3
MyNutchSpider 3
ParaSite 3
GetProxi.es-bot 3
SnapSkout 3
gcreep 3
YahooFeedSeeker/2.0 3
Curious 3
Slurp* 3
DieBlindeKuh 3
*trendictionbot* 3
rogerbot/1.0 3
WallpapersHD 3
WEBDAVClient 3
rsstank 3
baiduspider/2.0 3
Hloader 3
CloudEndureScanner 3
TurnitinBot/2.0 3
JBot 3
BravoBrian 3
EmailSpider 3
KavamRingCrawler 3
PaperLiBot/2.1 3
internetVistamonitor 3
AboutWebsite 3
EventGuruBot 3
SpiderBot/1.0 3
HyperCrawl 3
ia_archiver(OS-Wayback) 3
Freeuploader 3
PulsepointXT3webscraper 3
AhrefsBo 3
nutchUL 3
ccubee/3.5 3
LinkTiger 3
rdfbot 3
webspider 3
Wonderbot 3
Dataprovider 3
iZSearch 3
Twengabot 3
Stan 3
Zermelo 3
EverbeeCrawler 3
gsa-crawler-MOS 3
Lmspider 3
WPScan 3
g2Crawler 3
ZeBot_www.ze.bz 3
SBSearch 3
XGET 3
masscan 3
Nessus 3
PeoplePal 3
Searchestate 3
LqwRobot 3
webkicks-Robot 3
SpyFu 3
IntershopWebAdapterAgent 3
Probethenet 3
golem 3
desertrealm 3
eZooms 3
YahooCacheSystem 3
araneo 3
tach_bw 3
YangaWorldSearchBotv1.1 3
coccocbot-image 3
Ltx71 3