Skip to content

Instantly share code, notes, and snippets.

@SamSaffron

SamSaffron/bots.csv

Created June 20, 2018 23:59
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save SamSaffron/8a646e9dbe5f8d7121988beaa75a522d to your computer and use it in GitHub Desktop.
Save SamSaffron/8a646e9dbe5f8d7121988beaa75a522d to your computer and use it in GitHub Desktop.
Download ZIP
Raw
bots.csv
Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/) 1124756
MauiBot (crawler.feedback+wc@gmail.com) 958011
MauiBot (crawler.feedback+dc@gmail.com) 810138
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 645846
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) 555638
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) 547675
Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com) 329547
Mozilla/5.0 (compatible; SemrushBot/2~bl; +http://www.semrush.com/bot.html) 228811
Treato-Bot/1.0 (TREATO LTD, www.treato.com, mailto: botadmin@treato.com) 223069
Googlebot-Image/1.0 133416
Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/) 70466
Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots) 62967
Pingdom.com_bot_version_1.4_(http://www.pingdom.com/) 55576
Slackbot 1.0 (+https://api.slack.com/robots) 48302
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) 47809
Mozilla/5.0 (compatible; Swiftbot/1.0; UID/58a5fef514cc8a66ed5c3e3e; +http://swiftype.com/swiftbot) 39119
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple. 38680
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of 38586
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-15@moz.com) 33934
TurnitinBot (https://turnitin.com/robot/crawlerinfo.html) 25713
Mozilla/5.0 (compatible; Swiftbot/1.0; UID/54b67f33c9f929733b000001; +http://swiftype.com/swiftbot) 24684
CCBot/2.0 (http://commoncrawl.org/faq/) 24296
Mozilla/5.0 (compatible; Cliqzbot/2.0; +http://cliqz.com/company/cliqzbot) 17751
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36 14552
Mozilla/5.0 (compatible; AlphaBot/3.2; +http://alphaseobot.com/bot.html) 13180
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-26@moz.com) 12888
Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/) 12833
Mozilla/5.0 (compatible; Swiftbot/1.0; UID/5b2a3222c9f929373665cc18; +http://swiftype.com/swiftbot) 11377
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-87@moz.com) 10776
Mozilla/5.0 (compatible; SputnikBot/2.3; +http://corp.sputnik.ru/webmaster) 9959
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-24@moz.com) 9940
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots) 9438
AppleNewsBot 9348
Twitterbot/1.0 9230
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-59@moz.com) 8670
Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/) 8321
Slack-ImgProxy (+https://api.slack.com/robots) 7766
robots 7730
SentiBot www.sentibot.eu (compatible with Googlebot) 7344
Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com) 6765
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-56@moz.com) 5687
Caliperbot/1.0 (+http://www.conductor.com/caliperbot) 5256
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-82@moz.com) 4500
Mozilla/5.0 (compatible; SearchmetricsBot; http://www.searchmetrics.com/en/searchmetrics-bot/) 4194
Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry) 4153
Mozilla/5.0 (compatible; AddSearchBot/1.0; +http://www.addsearch.com/bot/) 4031
Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; bot@linkfluence.com) 4008
Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html) 3686
ZoomBot (Linkbot 1.0 http://suite.seozoom.it/bot.html) 3587
Mozilla/5.0 (compatible; Scoopitbot/1.1; Googlebotlike; +https://www.scoop.it/bot.html) 3266
Googlebot (gocrawl v0.4) 3194
Mozilla/5.0 (compatible; Swiftbot/1.0; UID/54d8ffce14cc8ad5b1000005; +http://swiftype.com/swiftbot) 2916
Mozilla/5.0 (compatible; Pinterestbot/1.0; +http://www.pinterest.com/bot.html) 2852
Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; info@yooz.ir) 2721
rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com) 2709
Mozilla/5.0 (compatible; SemrushBot-BA; +http://www.semrush.com/bot.html) 2685
Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/) 2467
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-13@moz.com) 2464
Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot) 2453
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36 moatbot 2412
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-36@moz.com) 2372
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Img/2.0; +http://go.mail.ru/help/robots) 2143
Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots) 2112
Mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine) 2006
Mozilla/5.0 (compatible; AhrefsBot/5.2; News; +http://ahrefs.com/robot/) 1906
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-33@moz.com) 1867
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-46@moz.com) 1865
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-16@moz.com) 1731
Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0 1697
Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/) 1623
istellabot/t.1.13 1561
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-27@moz.com) 1516
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-75@moz.com) 1457
Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com) 1330
Mozilla/5.0 (compatible; SemrushBot/1.0~bm; +http://www.semrush.com/bot.html) 1282
Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/) 1208
Mozilla/5.0 (compatible; AhrefsSiteAudit/5.2; +http://ahrefs.com/robot/) 1057
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot T 988
Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li) 955
Googlebot/2.1 (+http://www.google.com/bot.html) 943
Mozilla/5.0 (compatible; Mappy/1.0; +http://mappydata.net/bot/) 921
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-08@moz.com) 907
SEMrushBot 843
Mozilla/5.0 (compatible; VoluumDSP-content-bot/2.0; +dsp-dev@codewise.com) 801
MoodleBot/1.0 797
Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com) 777
Mediatoolkitbot (complaints@mediatoolkit.com) 771
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-43@moz.com) 763
Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/) 758
Mozilla/5.0 (compatible; rss2tg bot; +http://komar.in/en/rss2tg_crawler) 733
RSSingBot (http://www.rssing.com) 682
Mozilla/5.0 (compatible; ExtLinksBot/1.5; +https://extlinks.com/Bot.html) 678
Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots) 666
GetIntent Crawler (http://getintent.com/bot.html) 653
Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+) 611
AdsBot-Google (+http://www.google.com/adsbot.html) 574
Clickagy Intelligence Bot v2 574
Netvibes (crawler/bot; http://www.netvibes.com) 571
Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B410 Safari/600.1.4 (Applebot/0.1; 565
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-55@moz.com) 561
Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html) 515
Mozilla/5.0 (compatible; Robot; +http://www.sitecrawl.de:81/bot.php) 472
ZoominfoBot (zoominfobot at zoominfo dot com) 468
Googlebot/Nutch-1.7 454
rbot http util 0.9.15 (master branch, revision e358601 [fixes registry for 2.3.0, bug caused by wrong #23get_impl], 1 file changed) (http://ruby-rbot.or 433
Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot) 425
Mozilla/5.0 (Windows NT 10.0; ZumIBot; WOW64;Trident/7.0;rv:11.0) Chrome/50.0.2661.94 (KHTML, like Gecko) 416
Mozilla/5.0 (compatible; Feedspotbot/1.0; +http://www.feedspot.com/fs/bot) 402
FeedlyBot/1.0 (http://feedly.com) 388
Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html) 358
Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.133 Mobile Safari/535.19 moatbot 339
Mozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback) 331
Mozilla/5.0 (compatible; Dow Jones Searchbot) 330
Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 6.0 Robot) 326
Mozilla/5.0 (compatible; BomboraBot/1.0; +http://www.bombora.com/bot) 325
TelegramBot (like TwitterBot) 310
Mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine) 309
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0; dbot) 293
Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8.1.11)Gecko/20071127 Firefox/2.0.0.11 BugBountyBot 256
nanami.fr RSS-aggregator bot(68) 253
nanami.fr RSS-aggregator bot(101) 252
nanami.fr RSS-aggregator bot(78) 247
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-70@moz.com) 238
Mozilla/5.0 (compatible; coccocbot-fast/1.0; +http://help.coccoc.com/searchengine) 236
Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot) 225
Mozilla/5.0 (compatible; ScooperBot/3.0; +http://www.carma.com) 225
Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot) 221
Mozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots) 220
Mozilla/5.0 (compatible; AddSearchBot/0.9; +http://www.addsearch.com/bot; info@addsearch.com) 210
Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot ) 206
Swiftbot TEST 205
Mozilla/5.0 (compatible; Jooblebot/2.0; Windows NT 6.1; WOW64; +http://jooble.org/jooble-bot) AppleWebKit/537.36 (KHTML, like Gecko) Safari/537.36 198
V-BOT/0.001 (+mailto://bot@voyagerx.com) 196
Mozilla/5.0 (compatible; archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html) 187
Mozilla/5.0 (compatible; Swiftbot/1.0; UID/59de6e91eaa0b268b71483e6; +http://swiftype.com/swiftbot) 177
Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:129224754 173
Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:602837654 173
AwarioRssBot/1.0 (+https://awario.com/bots.html; bots@awario.com) 171
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=813647) 162
Flamingo_SearchEngine (+http://www.flamingosearch.com/bot) 158
Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html) 158
Dalvik/2.1.0 (Linux; U; Android 6.0; CUBOT MAX Build/MRA58K) 154
WordPress/4.9.6; https://www.bluerobotics.com 150
soleobot (+http://www.soleo.com) 149
Mozilla/5.0 (compatible; changedetection/7.0; +http://www.changedetection.com/bot.html) 144
Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot) 139
Mozilla/5.0 (Linux; Android 6.0; CUBOT DINOSAUR Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 134
Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0 (IndeedBot 1.1) 134
Mozilla/5.0 (Linux; Android 6.0; CUBOT MAX Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 133
Mozilla/5.0 (compatible; FemtosearchBot/1.0; http://femtosearch.com) 119
Skopaibot 114
Mozilla/5.0 (compatible; SeznamBot/3.2-test1-1; +http://napoveda.seznam.cz/en/seznambot-intro/) 113
Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler) 111
Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://napoveda.seznam.cz/en/seznambot-intro/) 109
Mozilla/5.0 (compatible; Googlebot/2.1; http://robotto.org/robotto-bot/) 90
rogerbot/1.1 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+vanguard@moz.com) 89
Gigabot 88
Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/) 84
Mozilla/5.0 (compatible;contxbot/1.0) 82
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=814193) 80
wp.com feedbot/1.0 (+https://wp.com) 80
WordPress/4.9.6; https://bluerobotics.com 78
HybridBot (hybrid.ru/about. If our bot caused problems please contact us. Contact email: abuse@hybrid.ru) 77
techfetch-bot/Nutch-1.14 72
Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; +http://www.leiki.com) 71
Mozilla/5.0 (compatible; musobot/1.0; info@muso.com; +http://www.muso.com) 67
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)/1.0 (LinkedInBot; https://www.linkedin.com/; omni-c 65
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.10) Gecko/20050716 Thunderbird/1.0.6 - WebCrawler http://cognitiveseo.com/bot.html 65
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=864188) 65
Java 1.7 Apache HttpClient (Linux x86_64) / GnowitNewsbot / Contact information at http://www.gnowit.com 64
Mozilla/5.0 (compatible; Swiftbot/1.0; UID/4fa97404dba8884d1b00003e; +http://swiftype.com/swiftbot) 60
Mediumbot-MetaTagFetcher/0.3 (+https://medium.com/) 59
Mozilla/5.0 (compatible; BehloolBot/beta; +http://www.webeaver.com/bot) 58
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-84@moz.com) 57
Mozilla/5.0 (Windows NT 10.0; ZumBot/1.0; http://help.zum.com/;WOW64;Trident/7.0;rv:11.0) Chrome/50.0.2661.94 (KHTML, like Gecko) 55
Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+) 55
Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots) 55
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0 / GnowitNewsbot / Contact information at http://www.gnowit.com 54
Mozilla/5.0 (compatible; YandexVideoParser/1.0; +http://yandex.com/bots) 53
WordPress/4.9.6; https://crypto-tradingbot.com 51
calculon spider/0.1.0 mailto:abuse@yieldbot.com 51
ArchiveTeam ArchiveBot/20171212.01 (wpull 2.0.3) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.9 49
Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/) 49
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=864195) 48
Googlebot-Video/1.0 47
Mozilla/5.0 (Linux; Android 8.0.0; CUBOT_X18_Plus Build/O00623) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 47
LivelapBot/0.2 (http://site.livelap.com/crawler) 44
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/4.3 +http://www.linkedin.com) 42
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com) 41
Mozilla/5.0 (compatible; Fluxiobot/0.1) 40
Mozilla/5.0 (Windows NT 10.0; ZumBot/1.0; WOW64;Trident/7.0;rv:11.0) Chrome/50.0.2661.94 (KHTML, like Gecko) 36
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=864190) 35
FreeWebMonitoring SiteChecker/0.2 (+https://www.freewebmonitoring.com/bot.html) 34
Googlebot 34
Mozilla/5.0 (compatible; SEOkicks-Robot; +https://www.seokicks.de/robot.html) 34
DomainStatsBot/1.0 (http://domainstats.io/our-bot) 33
adbeat_bot 32
Mozilla/5.0 (compatible; AndersPinkBot/1.0; +http://anderspink.com/bot.html) 31
Mozilla/5.0 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot) 31
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-64@moz.com) 31
Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/ 30
Patreon HTTP Robot 30
bitlybot/3.0 (+http://bit.ly/) 30
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=864194) 30
Ocarinabot 29
uipbot/1.0 (uipbot@semasio.net) 29
Mozilla/5.0 (Windows; U; WindowsNT 5.1; en-US; rv1.8.1.6;compatible; Googlebot/2.1; +http://www.google.com/bot.html) Gecko/20070725 Firefox/2.0.0.6 28
Mozilla/5.0 (Linux; Android 7.0; CUBOT X18 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 27
bidswitchbot/1.0 27
Mozilla/5.0 (Linux; Android 6.0; CUBOT_NOTE_S Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 26
Mozilla/5.0 (Linux; Android 6.0; CUBOT CHEETAH 2 Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 25
Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv 11.0) like Gecko (compatible; Zombiebot/2.1; +http://www.zombiedomain.net/robot/) 25
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=865357) 25
Mozilla/5.0 (Linux; Android 6.0; CUBOT_MANITO Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 24
Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv 11.0) like Gecko (compatible; Zombiebot/2.1; +http://www.zombiedomain.net/robot/) 23
RSSMicro.com RSS/Atom Feed Robot 23
Mozilla/5.0 (compatible; ZuperlistBot/1.0) 22
yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_111; Europe/en) http://yacy.net/bot.html 22
yacybot (/global; amd64 Linux 4.17.2-1-ARCH; java 1.8.0_172; America/en) http://yacy.net/bot.html 21
Mozilla/5.0 (Linux; Android 5.1; CUBOT_NOTE_S Build/LMY47I) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 20
Mozilla/5.0 (compatible; TinEye-bot/1.31; +http://www.tineye.com/crawler.html) 20
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=864191) 20
Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9 19
SafeDNSBot (https://www.safedns.com/searchbot) 19
Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://napoveda.seznam.cz/en/seznambot-intro/) 18
Mozilla/5.0 (compatible; NTENTbot; +http://www.ntent.com/ntentbot) 17
WordPress/4.9.6; https://crypto-tradingbot.com EWWW/421.0 17
yacybot (/global; amd64 FreeBSD 11.1-RELEASE-p10; java 1.8.0_162; Europe/en) http://yacy.net/bot.html 17
yacybot (/global; amd64 Linux 4.15.15-1-ARCH; java 1.8.0_171; Etc/en) http://yacy.net/bot.html 17
Mozilla/5.0 (Linux; Android 7.0; CUBOT CHEETAH 2 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36 16
Mozilla/5.0 (Linux; Android 7.0; CUBOT KING KONG Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 16
Mozilla/5.0 (Linux; Android 7.0; CUBOT MAGIC Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 16
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots) 16
Mozilla/5.0 (compatible; SputnikImageBot/2.3) 16
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=279589) 16
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=789505) 16
g2reader-bot/1.0 (+http://www.g2reader.com/; 2 subscribers; feed-id=588716) 16
yacybot (/global; amd64 Linux 4.16.11-1-ARCH; java 1.8.0_172; America/en) http://yacy.net/bot.html 16
Lynx/2.8 (;http://seebot.org) 15
Mozilla/5.0 (Linux; Android 6.0; CUBOT DINOSAUR Build/MRA58K; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/44.0.2403.119 Mobile Safar 15
Mozilla/5.0 (Linux; Android 6.0; VR BOT 552 PLUS Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.81 Mobile Safari/537.36 15
Mozilla/5.0 (Linux; Android 7.0; M bot 60 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 15
Mozilla/5.0 (compatible; Googlebot/2.1 +http://www.googlebot.com/bot.html) 15
Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot) 15
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=777516) 15
g2reader-bot/1.0 (+http://www.g2reader.com/; 2 subscribers; feed-id=588717) 15
Mozilla/5.0 (compatible; AhrefsBot/5.2; SA; +http://ahrefs.com/robot/) 14
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails) 14
Mozilla/5.0 (compatible;FindITAnswersbot/1.0;+http://search.it-influentials.com/bot.htm) 14
OutclicksBot/4 +https://www.outclicks.net/agent/80sPGE1AQJjSvR0DE9lPx0bWjzeUQg1h9QMnIslSDv3 14
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=705957) 14
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=864208) 14
g2reader-bot/1.0 (+http://www.g2reader.com/; 2 subscribers; feed-id=351218) 14
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com) 13
Mozilla/5.0 (compatible; seoBOT/0.2) 13
g2reader-bot/1.0 (+http://www.g2reader.com/; 2 subscribers; feed-id=588719) 13
Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; http://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0 12
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot 12
Mozilla/5.0 (compatible; Eyeotabot/1.0; +http://www.eyeota.com) 12
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-02@moz.com) 12
rrainn-Bot-Test 12
Mozilla/5.0 (Linux; Android 7.1.2; M bot Tab 103 Build/NHG47K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.81 Safari/537.36 11
Mozilla/5.0 (compatible; Pvblcbot/1.0; +http://www.pvblc.com/bot) 11
Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html) 11
Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/) 11
msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) 11
yacybot (/global; x86 Windows 7 6.1; java 1.8.0_171; Europe/fr) http://yacy.net/bot.html 11
AddSearchBot 10
Mozilla/5.0 (Linux; Android 7.0; CUBOT CHEETAH 2 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.109 Mobile Safari/537.36 10
Mozilla/5.0 (Linux; Android 7.0; ID bot 53+ Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.91 Mobile Safari/537.36 10
Mozilla/5.0 (Linux; Android 7.0; ID bot 56 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 10
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.63 Safari/537.36 (compatible; aylienbot/0.2; +http://www.aylie 10
OutclicksBot/4 +https://www.outclicks.net/agent/22fjsSYqDUB9YIk6vR8Jkkca5nlBNCue5ICL1tdAo4X0ykDuwLQlNmSULlnsj0HuH 10
PK bot: passkart.com 10
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=864192) 10
Deskyobot/1.0 (+https://www.deskyo.com/bot) 9
IABot: Checking if link from Wikipedia is broken and needs removal - See https://meta.wikimedia.org/wiki/InternetArchiveBot/FAQ_for_sysadmins 9
Mozilla/5.0 (Linux; Android 6.0; CUBOT ECHO Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 9
Mozilla/5.0 (Linux; Android 7.0; CUBOT NOTE Plus Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 9
Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36 _zbot 9
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots) 9
Mozilla/5.0 (compatible; epicbot; +http://www.epictions.com/epicbot) 9
Superfeedr bot/2.0 http://superfeedr.com - Checking HTML reverse link. 9
awesome_bot 9
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=864189) 9
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=864209) 9
Mozilla/5.0 (Linux; Android 7.0; CUBOT CHEETAH 2 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.68 Mobile Safari/537.36 8
Mozilla/5.0 (Linux; Android 7.0; M bot 51 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 8
Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0 8
Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot) 8
Mozilla/5.0 (compatible; Swiftbot/1.0; UID/5a83208714cc8a07b299b4ae; +http://swiftype.com/swiftbot) 8
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01i.yandex.ru) 8
PiplBot (+http://www.pipl.com/bot/) 8
WordPress/4.9.6; https://ftp.bluerobotics.com 8
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) Facebot 8
Mozilla/5.0 (Linux; Android 6.0; CUBOT CHEETAH 2 Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.68 Mobile Safari/537.36 7
Mozilla/5.0 (Linux; Android 6.0; CUBOT DINOSAUR Build/MRA58K; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/67.0.3396.87 Mobile Safari 7
Mozilla/5.0 (Linux; Android 6.0; CUBOT_NOTE_S Build/MRA58K; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/67.0.3396.87 Mobile Safari/5 7
Mozilla/5.0 (Linux; Android 6.0; VR BOT 552 PLUS Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 7
Mozilla/5.0 (Linux; Android 7.0; CUBOT H3 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.158 Mobile Safari/537.36 7
Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 +http://archive.org/details/archive.org_bot) 7
datagnionbot (+http://www.datagnion.com/bot.html) 7
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=864207) 7
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=866154) 7
mediawords bot (http://cyber.law.harvard.edu) 7
yacybot (/global; amd64 Linux 4.13.0-43-generic; java 1.8.0_171; Europe/en) http://yacy.net/bot.html 7
yacybot (/global; amd64 Linux 4.4.0-128-generic; java 1.8.0_171; Europe/de) http://yacy.net/bot.html 7
ETI SEO bot - www.seo.eti.pw 6
GoogleBot/2.1 6
Googlebot/2.1; +http://www.google.com/bot.html) 6
Mozilla/5.0 (Linux; Android 6.0; CUBOT MAX Build/MRA58K; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/67.0.3396.87 Mobile Safari/537. 6
Mozilla/5.0 (Linux; Android 7.0; CUBOT R9 Build/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/67.0.3396.87 Mobile Safari/537.3 6
Mozilla/5.0 (compatible; Cliqzbot/1.0 +http://cliqz.com/company/cliqzbot) 6
Mozilla/5.0 (compatible; Fess/11.2; +http://fess.codelibs.org/bot.html) 6
Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) 6
Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots) 6
Mozilla/5.0 (compatible; Plukkie/1.6; http://www.botje.com/plukkie.htm) 6
Mozilla/5.0 (compatible; Semanticbot/1.0; +http://sempi.tech/bot.html) 6
Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/) 6
Mozilla/5.0 (compatible; Windows NT 6.1?; ZumBot/1.0; http://help.zum.com/inquiry) 6
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 (Refindbot/1.0) 6
RobotsTxtParser-VIPnytt/2.0 (+https://github.com/VIPnytt/RobotsTxtParser/blob/master/README.md) 6
SerendeputyBot/0.8.6 (http://serendeputy.com/about/serendeputy-bot) 6
SuperPagesUrlVerifyBot/1.0 6
WordChampBot 6
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-50@moz.com) 6
yacybot (/global; amd64 Linux 4.15.0-13-generic; java 1.8.0_171; Europe/de) http://yacy.net/bot.html 6
yacybot (/global; amd64 Linux 4.15.0-15-generic; java 10.0.1; Europe/de) http://yacy.net/bot.html 6
yacybot (/global; amd64 Windows 10 10.0; java 1.8.0_171; Europe/en) http://yacy.net/bot.html 6
Googlebot/2.1 (+http://www.googlebot.com/bot.html) 5
Livechat OpenGraph Robot 5
Mozilla/5.0 (Linux; Android 6.0; CUBOT_MANITO Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.83 Mobile Safari/537.36 5
Mozilla/5.0 (Linux; Android 6.0; M BOT 551 Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.81 Mobile Safari/537.36 5
Mozilla/5.0 (Linux; Android 7.0; M bot 54 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.111 Mobile Safari/537.36 5
Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html; Linux; Android 4; iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KH 5
Mozilla/5.0 (compatible; Veoozbot/1.0; +http://www.veooz.com/veoozbot.html) 5
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Accept-Language: en, en-us;q=0.8 5
Mozilla/5.0 (compatible; vebidoobot/1.0; +https://blog.vebidoo.de/vebidoobot/) 5
Mozilla/5.0 (compatible; woobot/2.0; +https://www.woorank.com/bot) 5
BrandONbot (http://brandonmedia.net) 4
Fyrebot/1.0 4
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/4.1 +http://www.linkedin.com) 4
Mozilla/5.0 (Linux; Android 5.1; CUBOT_NOTE_S Build/LMY47I) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/39.0.0.0 Mobile Safari/537.36 4
Mozilla/5.0 (Linux; U; Android 7.0; CUBOT MAGIC Build/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/67.0.3396.87 Mobile Safari 4
Mozilla/5.0 (compatible; BotPants/1.0; Linux; +info@copypants.com) KHTML/3.5.5 (like Gecko) 4
Mozilla/5.0 (compatible; Linux x86_64; Meta_Bot/1.1) 4
Mozilla/5.0 (compatible; SputnikImageBot/2.3; +http://corp.sputnik.ru/webmaster) 4
Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com) 4
PubMatic Crawler Bot 4
ShowyouBot (http://showyou.com/crawler) 4
Testomatobot/1.0 (Linux x86_64; +https://www.testomato.com/testomatobot) minicrawler/5.1.0 4
WeSEE:Ads/PictureBot (http://www.wesee.com/bot/) 4
msnbot/2.0b (+http://search.msn.com/msnbot.htm) 4
rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) 4
tigerbot 4
yacybot (-global; arm Linux 4.14.14-sunxi; java 9-Debian; Etc/en) http://yacy.net/bot.html 4
yacybot (/global; amd64 Linux 4.17.1-gentoo; java 1.8.0_171; Europe/de) http://yacy.net/bot.html 4
yacybot (/global; x86 Windows 10 10.0; java 1.8.0_171; Europe/fr) http://yacy.net/bot.html 4
GumGum Bot (http://gumgum.com; support@gumgum.com) 3
Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com) 3
Mozilla/5.0 (Linux; Android 6.0.1; M BOT TAB 100 Build/MOB30R) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36 3
Mozilla/5.0 (Linux; Android 6.0; CUBOT_NOTE_S Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.68 Mobile Safari/537.36 3
Mozilla/5.0 (Linux; Android 6.0; CUBOT_NOTE_S Build/MRA58K; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/63.0.3239.111 Mobile Safari/ 3
Mozilla/5.0 (Linux; Android 7.0; CUBOT CHEETAH 2 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.126 Mobile Safari/537.36 3
Mozilla/5.0 (Linux; Android 7.0; CUBOT NOTE Plus Build/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/67.0.3396.87 Mobile Safar 3
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.79 Safari/537.36 KoboTest 3
Mozilla/5.0 (compatible; DomainSigmaCrawler/0.1; +http://domainsigma.com/robot) 3
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html ) 3
Mozilla/5.0 (compatible; MarketwireBot/1.0; +http://www.marketwire.com) 3
Mozilla/5.0 (compatible; Parsijoobot; +http://www.parsijoo.ir/bot.html; crawler@parsijoo.ir) 3
Mozilla/5.0 (compatible; Webliobot/0.1; +http://www.weblio.jp/info/crawler.jsp) 3
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon1.yandex.ru) 3
Mozilla/5.0 (compatible; YandexNews/4.0; +http://yandex.com/bots) 3
Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm) 3
Mozilla/5.0 (compatible; wc-test-dev-bot/1.0; +http://wc-test.dev.area) 3
bitlybot 3
g2reader-bot/1.0 (+http://www.g2reader.com/; 1 subscribers; feed-id=536663) 3
wp.com feedbot/1.0 (+https://wp.com;subscribers=1;feed-id=32818961) 3
yacybot (/global; amd64 Linux 4.13.0-45-generic; java 1.8.0_171; Europe/pl) http://yacy.net/bot.html 3
yacybot (/global; x86 Windows 7 6.1; java 1.8.0_171; Europe/en) http://yacy.net/bot.html 3
CrunchBot/1.0 (+http://www.leadcrunch.com/crunchbot) 2
Dalvik/2.1.0 (Linux; U; Android 7.0; CUBOT NOTE Plus Build/NRD90M) 2
Fuze_Bot/1.0 2
Gigbot/1.0; (+http://gigrocket.co/about/spiderfaq) 2
Google Bot 2
GoogleBot 2
Googlebot/1.0 (googlebot@googlebot.com http://googlebot.com/) 2
Googlebot/2.1 (http://www.googlebot.com/bot.html) 2
Googlebot/2.X (http://www.googlebot.com/bot.html) 2
Googlebot/Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:28.0) Gecko/20100101 Firefox/28.0 2
LABLEBot 2
Magus Bot 1.0 2
Mozilla/5.0 (Linux; Android 5.1; CUBOT_NOTE_S Build/LMY47I) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/39.0.0.0 Mobile Safari/537.36 [F 2
Mozilla/5.0 (Linux; Android 6.0; CUBOT CHEETAH 2 Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.109 Mobile Safari/537.36 2
Mozilla/5.0 (Linux; Android 6.0; CUBOT DINOSAUR Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.126 Mobile Safari/537.36 2
Mozilla/5.0 (Linux; Android 6.0; CUBOT MAX Build/MRA58K; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/64.0.3282.137 Mobile Safari/537 2
Mozilla/5.0 (Linux; Android 6.0; CUBOT_MANITO Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.111 Mobile Safari/537.36 2
Mozilla/5.0 (Linux; Android 7.0; CUBOT H3 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 2
Mozilla/5.0 (Linux; Android 7.0; M bot 60 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36 2
Mozilla/5.0 (Linux; Android 8.0.0; CUBOT_X18_Plus Build/O00623) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3457.4 Mobile Safari/537.36 2
Mozilla/5.0 (Linux; Android 8.0.0; HABOTHAT Build/O00623) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3457.4 Mobile Safari/537.36 2
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; http://www.apple. 2
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.79 Safari/537.36 FullStoryBot/1.0 (+https://ww 2
Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US) AppleWebKit/533.3 (KHTML, like Gecko) cXensebot/1.1a; http://www.cxense.com/bot.html/0.1 Safari/533 2
Mozilla/5.0 (Windows NT 10.0; ZumBot/1.0; http://help.zum.com/;WOW64;Trident/7.0;rv:11.0) Chrome/50.0.2661.94 (KHTML, like Gecko) 2
Mozilla/5.0 (Windows NT 6.1; rv:34.0) Gecko/20100101 Firefox/34.0; Dragonbot; http://www.dragonmetrics.com) 2
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36Aribot 2
Mozilla/5.0 (compatible; AddSearchBot; +http://www.addsearch.com/bot/) 2
Mozilla/5.0 (compatible; Girobot; +http://192.168.0.116/se/bot.php) 2
Mozilla/5.0 (compatible; Monsidobot/2.2; +http://monsido.com/bot.html; info@monsido.com) 2
Mozilla/5.0 (compatible; Primalbot; +https://www.primal.com;) 2
Mozilla/5.0 (compatible; ResearchBot/0.7) 2
Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/) 2
Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots) 2
Mozilla/5.0 (compatible; tracemyfile/1.0; +bot@tracemyfile.com) 2
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) (compatible; Coveobot/2.0;+http://www.coveo.com/bot.html) 2
Neticle Crawler v1.0 ( http://bot.neticle.hu/ ) 2
SeobilityBot (SEO-Check; http://bit.ly/1dJuuzs) 2
Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:606385210 2
Terminus URL Monitoring Bot (+https://www.terminusapp.com) 2
Twitterbot/1.3 2
Who.is Bot 2
WordPress/4.9.6; https://www.dev.bluerobotics.com 2
repology-linkchecker/0 (+https://repology.org/bots) 2
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-07@moz.com) 2
wp.com feedbot/1.0 (+https://wp.com;subscribers=1;feed-id=11112917) 2
yacybot (/global; amd64 Linux 4.9.0-6-amd64; java 1.8.0_141; Etc/en) http://yacy.net/bot.html 2
yacybot (/global; arm Linux 4.14.33-raspberrypi+; java 1.8.0_162; Europe/de) http://yacy.net/bot.html 2
yacybot (freeworld/global; amd64 Linux 4.4.0-128-generic; java 1.8.0_171; Etc/en) http://yacy.net/bot.html 2
@LinkArchiver twitter bot 1
ArchiveTeam ArchiveBot/20170106.02 (wpull 2.0.2) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.9 1
Faws.com bot 1
Google-Adwords-Instant (+http://www.google.com/adsbot.html) 1
ImplisenseBot 1.1 1
Mozilla/5.0 (Compatible; Supybot 0.83.4.1+limnoria installed on 2015-12-15T21-16-42) 1
Mozilla/5.0 (Linux; Android 6.0; CUBOT DINOSAUR Build/MRA58K; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/61.0.3163.98 Mobile Safari 1
Mozilla/5.0 (Linux; Android 6.0; CUBOT_MANITO Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.158 Mobile Safari/537.36 1
Mozilla/5.0 (Linux; Android 7.0; CUBOT CHEETAH 2 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.98 Mobile Safari/537.36 1
Mozilla/5.0 (Linux; Android 7.0; CUBOT MAGIC Build/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/66.0.3359.158 Mobile Safari/5 1
Mozilla/5.0 (Linux; Android 7.0; CUBOT R9 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36 1
Mozilla/5.0 (Linux; Android 7.1.1; M_bot_tab_1150 Build/N9F27F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36 1
Mozilla/5.0 (Linux; Android 7.1.2; M bot Tab 103 Build/NHG47K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.158 Safari/537.36 1
Mozilla/5.0 (Linux; Android 7.1.2; M bot Tab 103 Build/NHG47K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36 1
Mozilla/5.0 (Linux; Android 7.1.2; M_bot_tab_71 Build/NHG47K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.158 Safari/537.36 1
Mozilla/5.0 (Linux; U; Android 4.2.2; es-es; S222 Build/CUBOT) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30 GSA/6.5.35.16.a 1
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.117 Safari/537.36 FullStoryBot/1.0 (+https://www.ful 1
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534 (KHTML, like Gecko) MsnBot-Media /1.0b 1
Mozilla/5.0 (Windows NT x.y; rv:10.0; +http://tcptools.podofthing.com/bot.htm) Gecko/20100101 Firefox/10.0 1
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) JobBot/5.0 (compatible; http://www.jobseeker.com.au/bot.html) Safari/538.1 1
Mozilla/5.0 (compatible& Googlebot/2.1& +http://www.google.com/bot.html) 1
Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~javaruntimes-hrd) 1
Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~nowwhocan-hrd) 1
Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~proxy5070-hrd) 1
Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~proxy6003-hrd) 1
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html); yaka 1
Mozilla/5.0 (compatible; Googlebot/2.1;+http://www.google.com/bot.html) 1
Mozilla/5.0 (compatible; LinkisBot/1.0; bot@linkis.com) 1
Mozilla/5.0 (compatible; LinkisBot/1.0; bot@linkis.com) (iPhone; CPU iPhone OS 8_4_1 like Mac OS X) Mobile/12H321 1
Mozilla/5.0 (compatible; Please Name Your robot; +http://localhost/yioop/src/bot.php) 1
Mozilla/5.0 (compatible; Prezibot/1.0; +https://prezi.com) 1
Mozilla/5.0 (compatible; Swiftbot/1.0; UID/5b2a3222c9f929373665cc18; +http://swiftype.com/swiftbot) (2.8.3, ruby 2.4.4 (2018-03-28)) 1
Mozilla/5.0 (compatible; Swiftbot/1.0; UID/; +http://swiftype.com/swiftbot) (2.8.3, ruby 2.4.4 (2018-03-28)) 1
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppEngine-Google; (+http://code.google.com/appengine; appid: s~rrdsoa) 1
Mozilla/5.0 (compatible; YandexDirectDyn/1.0; +http://yandex.com/bots 1
Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots) AppEngine-Google; (+http://code.google.com/appengine; appid: s~proxycgi7-hrd) 1
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots TEST) 1
Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots mtmon01e.yandex.ru) 1
Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots) 1
Mozilla/5.0 (compatible; adidxbot/2.0; http://www.bing.com/bingbot.htm) 1
Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/) 1
NinjaBot/2.0 (+http://www.imninjas.com) 1
OnPageBot (compatible; Googlebot 2.1; +https://bot.onpage.org/) 1
OutclicksBot/4 +https://www.outclicks.net/agent/3ljq0Kzo0ipN44TvZ1SZSvrCgxEmm62dfy0DpKLh2qR 1
RankurBot/3.3 (+http://rankur.com) 1
SBL-BOT (http://sbl.net) 1
SE Ranking Gentle bot 1
SiteCheckerBot/1.0 (+http://sitechecker.pro) 1
Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:77917838 1
WeSEE_Bot:we_help_monitize_your_site (http://www.wesee.com/bot/) 1
WordPress/4.9.6; https://www.businessinchina.com/robot2 1
Wotbox/2.01 (+http://www.wotbox.com/bot/) 1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment