Skip to content

Instantly share code, notes, and snippets.

@acdha
Forked from edsu/bots.py
Created May 2, 2012 18:52
Show Gist options
  • Save acdha/2579174 to your computer and use it in GitHub Desktop.
Save acdha/2579174 to your computer and use it in GitHub Desktop.
Report the percentage of your site traffic generated by robots
'Mozilla/5.0 (compatible; BMC/1.0 (Y!J-AGENT))'
(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.talktalk.co.uk/products/virus-alerts/)
(TalkTalk Virus Alerts Scanning Engine)
*/Nutch-0.9
*/Nutch-1.0
*/Nutch-1.0 (*; *; *)
*/Nutch-1.2
+http://hilfe.acont.de/bot.html ACONTBOT
-
- (larbin2.6.3@unspecified.mail)
08_800w_web_an (larbin2.6.3@unspecified.mail)
1.2 beast/Nutch-1.0 (agentspider; bt@mail.com)
123peoplebot/1.0
163tester (tester@163.com)
2000_sites/Nutch-1.0 (Small_crawldb)
20090608_web_aq (larbin2.6.3@unspecified.mail)
222.243.160.126/Nutch-1.0
312/Nutch-1.0
50.nu V0.01 beta ( http://www.50.nu/bot.html )
50.nu/0.01 ( +http://50.nu/bot.html )
8dorm/Nutch-1.0 (8dorm; http://www.8dorm.com; webmaster@8dorm.com)
: Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1
=Mozilla/4.0
???/Nutch-1.0 (MyTest; 192.168.143.171; lai)
A1 Website Download/2.0.1 (+http://www.microsystools.com/products/website-download/) miggibot
AAfter.com Crawler/AAfter-1.0 (This bot is very focused, well-behaved, and wants to do good to internet community. For any questions, please call collect +1 214-714-2224. Team AAfter, Dallas, TX, USA; http://aafter.com/crawler.htm; crawler at aafter.com)
AAfter.com Crawler/Nutch-1.0 (This bot is very focused, well-behaved, and wants to do good to internet community. For any questions, please call collect +1 214-714-2224. Team AAfter, Dallas, TX, USA; http://aafter.com/crawler.htm; crawler at aafter.com)
Abortion.sg/Nutch-1.1 (www.Abortion.sg; crawler@Abortion.sg)
Aboundex/0.2 (http://www.aboundex.com/crawler/)
Abrave Spider v5.3 Robot 5 (http://robot.abrave.com)
ACME Corporation/Nutch-1.0 (ACME Spider; http://www.acme.com; test123@spam.la)
Acoon v4.1.0 (www.acoon.de)
Acoon v4.10.1 (www.acoon.de)
Acoon v4.10.3 (www.acoon.de)
Acoon v4.10.4 (www.acoon.de)
Acoon v4.10.5 (www.acoon.de)
Acoon v4.9.3 (www.acoon.de)
Acoon v4.9.5 (www.acoon.de)
AcoonBot/4.10.5 (+http://www.acoon.de)
acquia-crawler
acquia-crawler (detected bad behaviour? please tell us at it@acquia.com)
Acushare_Spider
AddThis.com robot tech.support@clearspring.com
adidxbot/1.1 (+http://search.msn.com/msnbot.htm)
AdnormCrawler www.adnorm.com/crawler
AdultShop.Sg/Nutch-1.1 (www.AdultShop.Sg; crawler@AdultShop.Sg)
AdultsVisit.us/Nutch-1.0 (www.AdultsVisit.us; crawler@AdultsVisit.us)
AdultsVisitUs/Nutch-1.1 (www.AdultsVisitUs.com; crawler@AdultsVisitUs.com)
agbot/1.0 (AgHaven.com search engine crawler; http://search.aghaven.com; bot@aghaven.com)
Agent
agentname/Nutch-1.0-dev
Aghaven/Nutch-1.2
Aghaven/Nutch-1.2 (www.aghaven.com)
Ahacc.Ru bot
AIR1/Nutch-0.9
AISearchBot (Email: aisearchbot@gmail.com; If your web site doesn't want to be crawled, please send us a email.)
akhomraven/Nutch-1.2 ( using Nutch 1.2; http://www.creativewires.com/; akhomraven@gmail.com)
AlexionResearchBot/Nutch-1.3
alhiqma 0.1 (Al-Hiqma Engine; http://labs.unrealasia.net/engine/; admin@unrealasia.net)
all_web3 (larbin2.6.3@unspecified.mail)
al_org_net_viewer (larbin2.6.3@unspecified.mail)
al_org_viewer (larbin2.6.3@unspecified.mail)
al_viewer (larbin2.6.3@unspecified.mail)
amibot - http://www.amidalla.de - tech@amidalla.com libwww-perl/5.831
andrologia medicina/Nutch-1.0 (andrologia medica; http://www.studiodiandrologia.it; profscandurra@alice.it)
Anemone/0.6.1
animalstxt.org/bot (http://animalstxt.org/)
AnnunciPazzi/Wauuu 0.1 (AnnunciPazzi; http://www.annuncipazzi.it/; admin@annuncipazzi.it)
Ant/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com)
Antbot/1.0
AntBot/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com)
Apache-HttpClient/4.1.1 (java 1.5)
Apache-HttpClient/4.1.2 (java 1.5)
AppEngine-Google; (+http://code.google.com/appengine; appid: thedomaintips)
Apple iPhone OS v2.2 CoreMedia v1.0.0.5G77
AppleWebKit/531.21.10
Applied-Technologies-Inc-Spider/Nutch-1.4
aquarelabot (no-reply@noemail.com)
Ara.com.tr/Nutch-1.2
AramaM/Nutch-2.0-dev
araneo
Argh!/Nutch-1.2
arianna.libero.it
Arquivo-web-crawler (compatible; heritrix/1.12.1 +http://arquivo-web.fccn.pt)
Arquivo-web-crawler (compatible; heritrix/1.14.3 +http://arquivo.pt)
Arquivo-web-crawler (compatible; heritrix/1.14.3 +http://arquivo.pt/faq-crawling)
asearcher/Nutch-1.2 (asearcher; www.asearcher.com; asearcher@yeah.net)
asearcher/Nutch-1.2 (asearcher; www.asearcher.com; asearcher@yearch.net)
ASER-ORNL/Nutch-1.0
Aspire-Tech Nutch/Nutch-1.1-dev (Aspire-Tech Nutch Robot Agent)
ASPseek/1.4
Atomic_Email_Hunter/4.0
Atomz/1.0
Atrax Solutions atraxbot/0.3; http://www.atraxsolutions.com/atraxbot
atraxbot/0.2
Attentio/Nutch-0.9-dev (Attentio's beta blog crawler; www.attentio.com; info@attentio.com)
avishayResearchSpider/Nutch-1.2
axtar.az/axtar-1 (axtar crawler; http://axtar.az; admin@axtar.az)
A_TEST_CRALWER_BY_ISSEL/Nutch-1.4 (http://issel.ee.auth.gr; kvavliak at issel dot ee dot auth dot gr)
B656_testing/Nutch-1.0 (Nutch spiderman)
BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)
BacklinkCrawler (http://www.backlinktest.com/crawler.html)
BaiDuSpider
Baiduspider+(+http://help.baidu.jp/system/05.html)
Baiduspider+(+http://www.baidu.com/search/spider.htm)
Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
Baiduspider+(+http://www.baidu.jp/spider/)
Balihoo/Nutch-1.0-dev (Crawler for Balihoo.com search engine - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com)
baypup/1.1 (Baypup; http://www.baypup.com/; jason@baypup.com)
baypup/1.1 (Baypup; http://www.baypup.com; jason@baypup.com)
baypup/colbert (Baypup; http://sf.baypup.com/webmasters; jason@baypup.com)
baypup/realtime (Baypup Realtime; http://sf.baypup.com/webmasters; jason@baypup.com)
BDFetch
beast/Nutch-0.9 (agentspider; beast@mail.com)
beast/Nutch-1.0 (agentspider; beast@mail.com)
Bee
belief,*/Nutch-1.2 (belief; belief; belief)
belief/belief
berlin-fu-cow (+http://hpsg.fu-berlin.de/cow)
betaBot
BigTMBot/1.0
Biible/Nutch-1.2 (Biible; http://www.biible.info ; raymond.balmes@gmail.com)
BilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com)
bim-zone (+http://www.bim-zone.com)
binlar_2.6.3 (binlar2.6.3@unspecified.mail)
binlar_2.6.3 (larbin2.6.3@unspecified.mail)
binlar_2.6.3 (phanendra_kalapala@McAfee.com)
binlar_2.6.3 (test@mgmt.mic)
bitlybot
BizwikiBot/0.3 (http://www.bizwiki.com/bizwikibot.htm)
BlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent
BlackBerry9630/5.0.0.732 Profile/MIDP-2.1 Configuration/CLDC-1.1 VendorID/102 ips-agent
blackcrawl/Nutch-0.9 (crawl for fun; www.zipfelchappe.com; zipfelchappe@localhost)
bldbCrawler/Nutch-1.4
BlevintronBot version 2012-04-02 contact http://en.wikipedia.org/wiki/User_talk:Blevintron
BlevintronBot version 2012-04-03 contact http://en.wikipedia.org/wiki/User_talk:Blevintron
BlevintronBot version 2012-04-04 contact http://en.wikipedia.org/wiki/User_talk:Blevintron
BlevintronBot version 2012-04-09 contact http://en.wikipedia.org/wiki/User_talk:Blevintron
BlevintronBot version 2012-05-01 contact http://en.wikipedia.org/wiki/User_talk:Blevintron
Blips Bot (www.dailyradar.com)
BlogPulseLive (support@blogpulse.com)
bloo/Nutch-1.0 (bloo; wiki.bloo.com; bloo at bloo dot com)
BLT/1.0.3
BLT/1.0.4 (http://www.bltformac.com/)
bobcrawler (contact bobcrawler@gmail.com with questions or concerns)
bobcrawler/1.0 (respects robots.txt including crawl-delay, send concerns to bobcrawler@gmail.com)
boitho.com-dc/0.86 ( http://www.boitho.com/dcbot.html )
bookagent/Nutch-1.0
boosker/Nutch-1.2
bot (Java Hotspot(TM) Server 64-Bit VM; +http://www.daniel-nowak-ug.de) Resource Description Engine
bot/1.0 (bot; http://; bot@bot.bot)
BotBot
BotLaju v2.0 (http://laju.my/)
BrokenLinkBot version 2012-03-31 contact http://en.wikipedia.org/wiki/User_talk:Blevintron
BrokenLinkBot version 20120319.102558 contact http://en.wikipedia.org/wiki/User_talk:Blevintron
BrokenLinkBot version 20120320.224751 contact http://en.wikipedia.org/wiki/User_talk:Blevintron
Browsershots
bss (meng1986626@163.com)
Btsearch/1.1 (Btsearch; http://www.baotongsoft.com/search.html; Btsearch-agent@baotongsoft.com)
Btsearch/1.1 (Btsearch; http://www.test.com/test.html; test@test.com)
Bubble Spider/Nutch-1.0 (Bubble Bot; http://f-desk/hwa; frank@verogem.com)
buscador-age (Enterprise; T2-C9QZ24BXJWWK2; gep@ti.eptisa.com)
businessdbbot/v1.2 (http://www.businessdb.com/bot.php)
Cabot/Nutch-1.0 (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; help at amfibi dot com)
Cabot/Nutch-1.0-dev (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; help at amfibi dot com)
Cabot/Nutch-1.2 (Amfibi's webcrawler robot; http://www.amfibi.com/cabot; cabot@amfibi.com)
CamontSpider/1.0 +http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html
CatchBot/1.0; +http://www.catchbot.com
CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot.php; cbot@cazoodle.com)
cc/Nutch-1.0
CCBot/1.0 (+http://www.commoncrawl.org/bot.html)
cdacp/Nutch-0.9 (IIT Kharagpur; http://www.cdac.in; asoni@cdac.in)
cds.robot/Nutch-1.2
CE-Preload
CEIT/Nutch-1.1
Cercador AOC / Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)/Nutch-1.0
CervantesRIBot
Charlotte
Charlotte/0.05
check.rdfa/0.0.1 (+http://check.rdfa.info/) RDF-RDFa-Parser/1.09_09 RDF-RDFa-Linter/0.04 RDF-RDFa-Generator/0.04 libwww-perl/5.831
Chemical Search Engine/Nutch-1.2 (ChemSE crawler, Build a chemical sourcing database; '; info123@chemse.com)
Chen Li/Nutch-1.0 (Nutch spiderman; http://chenli.com.cn; chenlibiti@163.com)
chklinks-robot/3.09 (using libwww-perl/5.837)
cierzo-development/Nutch-1.1-dev
cierzo/Nutch-0.9
cinvestav/Nutch-1.1
cinvestav/Nutch-1.4
citeseerxbot (compatible; citeseerxbot; +http://citeseer.ist.psu.edu/)
Cityreview Robot (+http://www.cityreview.org/crawler/)
cizilla.com/Cizilla-1.0 (CIZILLA - Complete Internet Search Engine; http://www.cizilla.com; crawl@cizilla.com)
CjxSearch/Nutch-1.4
ck's crawler/Nutch-1.2
CKC Spider/Nutch-1.4
CLARAbot
CligooRobot/1.0
clj-httpc
cloud-valley/bigdata-1.0 (http://www.cloud-valley.com/; news@mobier.mobi)
CloudACL/Nutch-1.4
CloudCrawler606/Nutch-1.2
cn_1 (larbin2.6.3@unspecified.mail)
cn_1_viewer (larbin2.6.3@unspecified.mail)
cn_ab_viewer (larbin2.6.3@unspecified.mail)
cn_org_viewer (larbin2.6.3@unspecified.mail)
cn_org_viewer2 (larbin2.6.3@unspecified.mail)
coccoc
Code Search Crawler/Nutch-1.2 (Code Search Crawler; www.iai.uni-bonn.de)
Codeton Software RSS Bot/1.0
colkbrowser/0.7; Linux OS
colorgo agent/1.0
colorgo.com/Nutch-1.2
Combine/3 http://combine.it.lth.se/
Comodo SSL Checker
Comodo-Certificates-Spider
COMODOspider/Nutch-1.0
COMODOSpider/Nutch-1.2
compatible; Mozilla 4.0; MSIE 5.5; (SqwidgeBot b1 - http://www.sqwidge.com/bot/)
Content Crawler
Contextured Site Crawler/Nutch-1.0 (http://contextured.com/features/crawler; crawler@contextured.com)
copyright sheriff (+http://www.copyrightsheriff.com/)
core-I-Bot/1.0 (+http://www.core.at/core-i-bot/)
coruscan/Nutch-1.4
Covario IDS/Nutch-1.2
Covario-IDS/1.0 (Covario; http://www.covario.com/ids; support at covario dot com)
Covario-IDS/1.0 (Covario; http://www.covario.com; support at covario dot com)
CovarioIDS/1.1 (http://www.covario.com/ids; support at covario dot com)
CowCrawler/CowCrawler-dev (+http://beta.cow.com)
CPSE Crawler(http://www.clusterpoint.com)
crabbyBot/1.0
Crawl Test/Nutch-1.2
Crawl/0.01 libcrawl/0.3
crawler
Crawler 0.1
Crawler/Nutch-1.2
crawler/Nutch-1.3
Crawler0.1
crawler01.DiverseBooks.com/Nutch-1.0 (Web crawler aimed at collecting information about Books. Mostly SF for now; http://diversebooks.com/crawler; info at diversebooks tot kom)
crawler4j (http://code.google.com/p/crawler4j/)
CrawlerProcess (http://www.PowerMapper.com) /3.3.300.0
CrawlerTest/Nutch-1.0-dev
crawly_2.6.3 (crawly@commandcom.com)
CRW1/Nutch-1.2
crwal_zkzy (chenchaoyin@software.ict.ac.cn)
CSimpleSpider Robot
CSVNutch/Nutch-1.3 (Academic test crawl; pshopcrawl at gmail)
curl/7.15.5 (i686-suse-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8d zlib/1.2.3 libidn/0.6.8
curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
curl/7.19.0 (x86_64-suse-linux-gnu) libcurl/7.19.0 OpenSSL/0.9.8h zlib/1.2.5 libidn/1.10
curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
curl/7.21.0 (x86_64-apple-darwin10.4.0) libcurl/7.21.0 OpenSSL/0.9.8l zlib/1.2.3 libidn/1.19
curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
CyberSpyder Link Test/2.2.1
CydralSpider/1.9 (Cydral Web Image Search; http://www.cydral.com)
CydralSpider/3.2 (Cydral Image Search; http://www.cydral.com)
CydralSpider/3.2.6 (Cydral Image Search; http://www.cydral.com)
DAgent
daodao (larbin@unspecified.mail)
DatenschutzRobot/0.01
Dawwer/1.0
Deafindex.se/Nutch-1.3
DealGates Bot/1.1 by Luc Michalski (http://spider.dealgates.com/bot.html)
del.icio.us-thumbnails/1.0 Mozilla/5.0 (compatible; Konqueror/3.4; FreeBSD) KHTML/3.4.2 (like Gecko)
DelvuBot/1.0
Deneme/Nutch-0.9
Denodo IECrawler/4.5
deva/deva-1.0 (Deva Fetcher; devarajaswami at yahoo dot com)
Digital Dialogue Media Pty Ltd Perth Australia /Nutch-1.2 (Nutch spiderman; www.amnet.net.au ; bingatron@digitaldialogue.com.au )
DigitalPebble/Nutch-1.2 (DigitalPebble's robot; http://www.digitalpebble.com; robot@digitalpebble.com)
DiscoverED by ccLearn/Nutch-0.9
Dit/1 (Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; YPC 3.0.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727); http://www.dit.unitn.it; bot@dit.unitn.it)
dlab spider/Nutch-1.3
dmzg/Nutch-1.0
DNSTallyBot/0.2
DoCoMo/1.0/N505i/c20/TB/W20H10 (compatible; RFCrawler-Mobile/1.0; +http://www.rfms.jp/crawler.html)
DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; +http://mgw.hatena.ne.jp/help)
DoCoMo/2.0 i-robot(c10;TC)
DoCoMo/2.0 N902iS(c100;TB;W24H12)(compatible; moba-crawler; http://crawler.dena.jp/)
DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
DoCoMo/2.0 P05A(c100;TB;W24H15) (compatible; BaiduMobaider/1.0; +http://www.baidu.jp/spider/)
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
DogSniffer
Domnutch-Bot/Nutch-1.0 (Domnutch; http://www.Nutch.de/)
Doot/1.0/Nutch-1.2
doraemon (wangzheng088@gmail.com)
DownloadIpTxtFile
drone
droreSpider (gaowenbinmarr@gmail.com)
DrRajendra/Nutch-0.9 (Senior Researcher; http://cse.iitkgp.ac.in/~rpr; rajendra@cse.iitkgp.ernet.in)
dtSearchSpider
DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
ealbum/Nutch-1.0 (ealbum crawler; www.ealbum.com; steley2000 at yahoo dot com)
easyDigGet/1.0
ebooksearchtool
ebsearch (larbin@unspecified.mail)
EC2LinkFinder
ecxi/Nutch-1.0 (esCERT-UPC-ecxi; http://escert.upc.edu/; admin escert edu)
EdisterBot (http://www.edister.com/bot.html)
edu.pjwstk.focusedcrawler.training-data-gatherer
eFills/Nutch-0.9
Elblindo the Blind Bot (Version: 1.06, powered by www.elblindo.com +http://www.elblindo.com/pfadzurbotseite/bot.html)
Element/Nutch-2.0-dev
emBot-GalaBuzz/Nutch-1.0 (http://emining.jp/; em@galabuzz.jp)
emefgebot/BETA (+http://emefge.de/bot.html)
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)
EoLarbin_2.6.3.1 (ns6934@ovh.net)
EPiServer Link Checker
ePochta_Extractor/4.0
Eric Osgood/Nutch-1.0 (Nutch spiderman; http://www.calpoly.edu/~eosgood ; MyEmail)
es_com_viewer (larbin2.6.3@unspecified.mail)
es_org_viewer (larbin2.6.3@unspecified.mail)
es_org_web (larbin2.6.3@unspecified.mail)
Eurobot/1.1 (http://eurobot.ayell.eu)
Eurobot/Nutch-1.0-dev (1.0)
Eurosoft-Bot (Version: 1.06, powered by www.eurosoftware.de +http://www.eurosoftware.de/zeige/bot.html)
eventax/1.0 (http://www.eventax.de)
EventMachine HttpClient
EventusCrawler/0.1 (Collects event information)
ExactSeek Crawler (http://www.exactseek.com/)/Nutch-1.4
ExactSeek Crawler (nutch 1.4)/Nutch-1.4
example/Nutch-0.9 (listallabout; http://www.listallabout.com; contact@listallabout.com)
Facebook Labs/Nutch-1.0 (Test crawler for facebook.; http://www.facebook.com/kma; kma at facebook dot com)
Facebook share follower
facebookexternalhit/1.0 (http://www.facebook.com/externalhit_uatext.php)
facebookscraper/facebookscraper/1.0
fairshare2.0 test crawler/fairshare2.0 (Check us out at http://fairshare.cc )
fangdudu_1.0
FAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg)
fc2bots (http://fc2.com/)
FeedChecker/0.01
fetch/Nutch-1.0 (TCGfetch; http://fetch.thecyberguardian.com; TCGEmail)
fetch/Nutch-1.2 (TCGfetch; http://fetch.thecyberguardian.com; TCGEmail)
Filter/Nutch-1.2
findfiles.net/0.96 (Robot;test_robot@gmx-topmail.de)
findfiles.net/0.97 (Robot;test_robot@gmx-topmail.de)
findfiles.net/0.98 (Robot;test_robot@gmx-topmail.de)
findlinks/1.1.5-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta3 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0.4 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.1.3 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.1.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
FINLY
Firefox
Firefox/3.0.10 es-ES U Ubuntu jaunty 9.04
firesignbranding/mokum (A friendly experimental http fetcher.; www.firesignbranding.com; info@firesignbranding.com)
firtex2 agent/1.0
Flaptor RT/Nutch-1.0 (Flaptor test; http://www.flaptor.com; rodriguez arroba gmail punto com)
Flaptor/Nutch-1.0 (Flaptor crawling; http://www.flaptor.com; info@flaptor.com)
flatlandbot/baypup (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; jason@flatlandindustries.com)
Flexum/2.0
FlightDeckReportsBot/2.0 (http://www.flightdeckreports.com/pages/bot)
Focal/Focal-1.0-dev (Syllabs focused web crawler; http://www.syllabs.com/crawler.html; crawler@syllabs.com)
FOIIABot/1.1 (http://foiia.com/bot/; info at foiia.com)
foxman/Nutch-1.2 (Nutch spiderman; http://www.163.com/; MyEmail)
Free Link Checker - http://www.Control-F.com - based on W3C-checklink/4.3 [4.42] libwww-perl/5.801
FreeNutch/Nutch-1.2
FrequencyAnalyzer/0.2
frobee.com Link Checker/0.7
fr_org_viewer (larbin2.6.3@unspecified.mail)
Fumae2Oo/Nutch-1.4
FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php)
g10_132_yy_x11/110510 (Mesin Pencari bahasa Indonesia; http://www.nguber.com; websearch at nguber dot com)
Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
GarlikCrawler/1.1 (http://garlik.com/)
GarlikCrawler/1.1 (http://garlik.com/, crawler@garik.com)
GarlikCrawler/1.1 (http://garlik.com/, crawler@garlik.com)
gaurav/Nutch-0.9
GematchCrawler/1.8 (http://www.gematch.com/crawler.html)
GematchCrawler/2 (http://www.gematch.com/crawler.html)
Generic
geobucket/Nutch-1.4
GeoHasher/Nutch-1.0 (GeoHasher Web Search Engine; geohasher.gotdns.org; geo_hasher at yahoo * com)
gettor/0.1
GFJ-Hosting.de_PerformanceCrawler
gh-index-bot/Nutch-1.0 (GH Web Search.; lucene.apache.org; gh_email at someplace dot com)
Gigabot/3.0 (http://www.gigablast.com/spider.html)
Gimmie60/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.11)
GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)
Gizmick/1.0 (A non-profit search engine created for the benefit of charity.; http://www.i-s-a-r-a.org/search/faq/ (take out the
GMI sentiment crawler/Nutch-1.0 (GMI sentiment crawler; http://GMI.googlepages.com ; MyEmail)
Gnam Gnam Spider
Go http package
GOFORITBOT ( http://www.goforit.com/about/ )
gold crawler
Goldfire Server
gonzo/1[P] (+http://www.suchen.de/faq.html)
gonzo1[D] +http://www.suchen.de/faq.html
gonzo1[P] +http://www.suchen.de/faq.html
gooblogsearch/2.0 (http://help.goo.ne.jp/contact/)
Google/4.0
google/Nutch-1.1
googlebot
Googlebot (larbin2.6.3@unspecified.mail)
googlebot (search@socbay.com)
Googlebot-Image/1.0
Googlebot-Image/1.0 (larbin@unspecified.mail)
Googlebot-Trojan/0.1.1/Nutch-1.1-dev (Trojan Scanner v.0.7; http://www.google.com/bot.html)
Googlebot/1.0 (Googlebot; http://www.google.com/; googlebot@google.com)
Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Googlebot/2.1+(+http://www.google.com/bot.html)
Googlebot/2.1/Nutch-1.1
Googlebot/Googlebot-1.0
Googlebot/Nutch-1.0
Googlebot/Nutch-1.0 (Prototype; http://en.wikipedia.org/wiki/Web_crawler; donotreply at prototype dot com)
Googlebot/Nutch-1.4
googleru_orgnet_viewer (larbin2.6.3@unspecified.mail)
google_web (larbin2.6.3@unspecified.mail)
goole.crawler/Nutch-1.0
Gperp (Gperp@163.com)
Gperp (UserAgent)
GreenIvory/Nutch-0.9 (GreenIvory-BlueCrane; http://greenivory.fr; v.taesch@greenivory.com)
Grewatsch/Nutch-1.2 (Hochschule Lausitz(; http://www.HS-Lausitz.de ; Morris.Grewatsch@HS-Lausitz.de)
gsa-bizkaia (Enterprise; T2-B84TKEFXKWWBK; gsa@bizkaia.net)
gsa-crawler (Enterprise; C5-QP8LD5K6A2JJA; me@mycompany.com)
gsa-crawler (Enterprise; GIX-04247; me@mycompany.com)
gsa-crawler (Enterprise; M2-ANLVCZEKDC2AA; admin@jadu.net)
gsa-crawler (Enterprise; M2-B3X6EDEBUC6JB; serveradm@dickinson.edu)
gsa-crawler (Enterprise; M2-B49PXEELPJSAA; webmaster@bethel.edu)
gsa-crawler (Enterprise; M2-FLTWN9EKUC6JS; uriah.liggett@etegent.com)
gsa-crawler (Enterprise; M2-FZL5CXEKDA2AA; google_mini@tribalddb.de)
gsa-crawler (Enterprise; M2-JGHL8PEKVA2JS; webmaster@bethel.edu)
gsa-crawler (Enterprise; M2-JS5323WKVA6JA; dana.champine@slcc.edu)
gsa-crawler (Enterprise; M2-KKXEPREKUC2AT; biggersj@hillsboroughcounty.org)
gsa-crawler (Enterprise; M2-KQMM4KAKDA6JS; sesame@u.washington.edu)
gsa-crawler (Enterprise; M2-KWZGA5SKCA2AB; googlemini@bvt.org.uk,helpdesk@bvt.org.uk)
gsa-crawler (Enterprise; M2-PA8WZ3SKDA2AS; dana.champine@slcc.edu)
gsa-crawler (Enterprise; MID-00215; jreeve@weber.k12.ut.us)
gsa-crawler (Enterprise; S5-DPJCVFSVSEWAS; itsunixadmins@vanderbilt.edu)
gsa-crawler (Enterprise; S5-F5KK3P63Y6NAS; datacenter2@digitalinsight.com,noc@digitalinsight.com,kenneth.lemoine@digitalinsight.com)
gsa-crawler (Enterprise; S5-GE8RVQN3J4NJS; nobody@google.com)
gsa-crawler (Enterprise; S5-HX5HMEAVAESJS; marcos@justdigital.com.br,marco@justdigital.com.br,aseri@justdigital.com.br)
gsa-crawler (Enterprise; S5-HX5HMEAVAESJS; marcos@justdigital.com.br,mteixeira@justdigital.com.br,aseri@justdigital.com.br)
gsa-crawler (Enterprise; S5-HYU2NGRTV6JJA; wwwadm@ua.es)
gsa-crawler (Enterprise; S5-LRNEWUP3EUJAT; nobody@google.com)
gsa-crawler (Enterprise; S5-Q7C5KR6TY8JAS; datacenter2@digitalinsight.com,noc@digitalinsight.com,kenneth.lemoine@digitalinsight.com)
gsa-crawler (Enterprise; S5-RLAPLGRTU6NAT; networkadmin@figleaf.com)
gsa-crawler (Enterprise; T1-EW7TFYE5SGSAS; allan.s.stilwell@nasa.gov)
gsa-crawler (Enterprise; T2-A6WTP66PZJWAS; webmaster@uva.es)
gsa-crawler (Enterprise; T2-CBUNHYW9ENWAB; webservices@calstate.edu)
gsa-crawler (Enterprise; T2-CVDXLNS9ENWAB; julien.marchet@sword-group.com)
gsa-crawler (Enterprise; T2-K6FCMUW9WQWJT; webservices@calstate.edu)
gsa-crawler (Enterprise; T2-KX74XYNZFQSAS; gsa@searchtechnologies.com)
gsa-crawler (Enterprise; T2-LXRKXYNZENSAA; gsa4.admin@searchtechnologies.com)
gsa-crawler (Enterprise; T3-A3BDGUFT8YSGH; rgerry@fsc.follett.com)
gsa-crawler (Enterprise; T3-GNGZGLT73YWB2; search@byu.edu)
gsa-crawler (Enterprise; T3-H5CRHE772YWKK; tech@typhon.com)
gsa-crawler (Enterprise; T3-L39FLJP3RYWGY; dlfr-france.fr-system@atos.net)
gsa-crawler (Enterprise; T3-LLNR7LKTRYSQY; dlfr-france.fr-system@atos.net)
gsa-crawler (Enterprise; T3-MRHJJGX73YWBJ; estauthamer@emidconsult.com)
gsa-crawler (Enterprise; T3-Q3A8GJTXKYWBJ; tech@typhon.com)
gsa-crawler-ARD (Enterprise; S5-C6H8MTSZNSJJB; webmaster@ard.de)
gsa-crawler-ARD (Enterprise; T1-NZSVYYSPYJSJS; Technik-Support@ard.de)
gsa-crawler-ARD (Enterprise; T1-NZSVYYSPYJSJS; webmaster@ard.de)
gsa-crawler1 (Enterprise; S5-DPJCVFSVSEWAS; itsunixadmins@vanderbilt.edu)
gsa-crawler2 (Enterprise; S5-DWXMDHSVBGSAS; itsunixadmins@vanderbilt.edu)
GSLFbot
GurujiImageBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)
gva/europe (Geneva Web Explorer.; http://www.genevasearch.com/; info@genevasearch.com)
gyplayer/1.0
HablaCentro/1.0 (HablaCentro Crawler; http://hablacentro.com/; webmaster@hablacentro.com)
HaDoot/1.0/Nutch-1.2 (Hadoot Open Source Search Engine; http://www.hadoot.com/)
hao123_viewer (larbin2.6.3@unspecified.mail)
Harmonia/Nutch-1.2
HarvestMan 1.4
Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)
HatenaScreenshot/1.0 (checker)
HAZY.SPIDER/Nutch-1.4
hclsreport-crawler (+http://hclsreport.com/
HD nutch agent/1.0
HD nutch agent/1.2
HD nutch agent/Nutch-1.2
HD search agent/1.0
HD search agent/1.1
HeartRails Robot/0.1 (http://www.heartrails.com)
hitcrawler_0.1 (ringken@gmail.com)
hitspider (larbin2.6.3@unspecified.mail)
hoge (co2h2onacl@gmail.com)
hogna v0.1
holmes/3.12 (OnetSzukaj/5.0; +http://szukaj.onet.pl)
holmes/3.12.4 (http://morfeo.centrum.cz/bot)
holmes/3.9
HolmesBot (http://holmes.ge)
HopperBot/1.1 (compatible; heritrix/3.1.0 +http://www.hopper.com)
HostHarvest/0.4.28 (http://www.raskgaard.dk/~cara/hostharvest.html)
HounderCrawl/Nutch-0.9 (Hounder Search Bot; http://www.flaptor.com/)
http://ask.24x.info/
http://domino.research.ibm.com/comm/research_projects.nsf/pages/sai-crawler.callingcard.html
http://SiteIntel.net Bot
http://www.amagit.com/
http://www.baidu.com/Nutch-1.2
http://www.sidusoft.cn/Nutch-1.0 (http://www.sidusoft.cn)
http://www.sidusoft.com.cn/Nutch-1.0 (http://www.sidusoft.com.cn)
http://www.sina.com.cn/Nutch-1.0 (http://www.sina.com.cn)
http://www.stero.pl/splywy-kajakowe | kajaki@onet.pl | tel. 601 41 88 60
http://www.uni-koblenz.de/~flocke/robot-info.txt
https_page_test (larbin2.6.3@unspecified.mail)
https_url_test (larbin2.6.3@unspecified.mail)
HTTrack Website Copier/3.0x (offline browser; web mirror utility)
HuaweiSymantec spider (compatible; MSIE 8.0; wangdanfeng@huaweisymantec.com)
Huaweisymantecspider (compatible; MSIE 8.0; DSE-support@huaweisymantec.com)
Huaweisymantecspider (compatible; MSIE 8.0; wangdanfeng@huaweisymantec.com)
HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/cn/IRL/spider)
HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/en/IRL/spider)
HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.talktalk.co.uk/products/virus-alerts/)
I am a friendly (BETA) R.O.B.O.T. I am crawling the web. If I do not respect your robots.txt, e-mail me. Soon I will have a website. More info at ggggggg73@hotmail.com
ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)
ia_archiver(OS-Wayback)
ia_archiver-web.archive.org
iBot
ibuena (robot@ibuena.net)
ICC-Crawler (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
ICC-Crawler(Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
ICC-Crawler(Mozilla-compatible; icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp; http://kc.nict.go.jp/project1/crawl.html)
ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.csk.com/sys-nishi/)
Iceweasel2.0.0.16/Nutch-1.0 (Webbrowser; http://iceweasel.com; info@iceweasel.com)
Iceweasel2.0.0.16/Nutch-1.1 (Webbrowser; http://iceweasel.com; info@iceweasel.com)
ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)
ichiro/3.0 (http://help.goo.ne.jp/help/article/1142)
ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)
ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)
ichiro/6.0 (http://help.goo.ne.jp/door/crawler.html)
ict/Nutch-1.1 (ict; http://idoc.ict.ac.cn)
iDEAL/Nutch-1.2
IE-9
IE/6.0 (compatible; heritrix/1.14.4 +http://202.206.249.110)
IE/8.0 (compatible; heritrix/1.14.3+http://179.39.11.19/)
IE/Nutch-1.0
IE6_IE7_IE8 (larbin2.6.3@unspecified.mail)
IIT Bombay CFILT NLP Bot/Nutch-1.1 (IITB CFILT Crawler)
IITB-CFILT-Bot/Nutch-1.1 (This is the crawler of IIT Bombay, India. The data will be used for research purposes.; http://www.cfilt.iitb.ac.in/; pb@cse.iitb.ac.in)
ikubinfo/Nutch-1.2 (research)
ilocal company search
IlseBot/1.1
imrd-1.2 (walker@imrd.com)
IMRD/1.0 (robot@imrd.com)
imrdbot1.0 (imrdbot@imrd.com)
INDEXACIONEDUCAREX/Nutch-1.0
InetURL:/1.0
Infoaxe./Nutch-0.9
Infoaxe./Nutch-1.0
informerabot/Nutch-1.2 (test crawler; http://www.informera.de; info at informera dot de)
innocrawl/Nutch-1.4
innoventage/Nutch-1.0 (poc; www.google.com; proof of concept)
innoventage/Nutch-1.0 (poc; www.innoventage.com; proof of concept)
inonsh/Nutch-1.1
inonsh/Nutch-1.1 (Web Crawler; www.inonsh.com; webmaster@inonsh.com)
inonsh/Nutch-1.2
Internet Explorer
Internet Explorer/8.0 (compatible; heritrix/1.10.1+http://192.168.0.1)
intumit/Nutch-1.1
intumit/Nutch-1.2
intumit/Nutch-1.2 (intumit)
intumit/Nutch-1.3
intumit/Nutch-1.4
ipec/Nutch-1.2 (ipec; ipec.cn; info@ipec.cn)
IPv6'n/0.1; +http://draft.scyphus.co.jp/ipv6.html
ir1.0 (jessievop@hotmail.com)
IS Alpha/Nutch-1.0
IS Alpha/Nutch-1.1
IS Alpha/zcspider-0.1
IS Alpha/zcspider-0.1 (Wisshow Network Technology LTD.CO)
Isara/Isara-1.0 (A non-profit search engine benefiting charity.; http://www.isara.org; webmaster@isara.org)
Isara/Isara-1.0 (A non-profit search engine for the benefit of charity.; http://www.isara.org; search@isara.org)
Isara/Nutch-1.2 (A non-profit search engine for the benefit of charity.; http://www.isara.org; community@isara.org)
Itah/1.1 (Itah; http://www.itah.com; support@itah.com)
itim
iVia/5.0 Adders (http://infomine.ucr.edu:80/iVia/useragents.shtml)
Jabse.com Crawler v.1.0 www.jabse.com/crawler.php//imagecrawler
jadu-crawler (Enterprise; M2-ANLVCZEKDC2AA; admin@jadu.net)
Jakarta Commons-HttpClient/3.0.1
Jakarta Commons-HttpClient/3.1
James Prabhu/Nutch-1.0 (Nutch spiderman; http://www.cosmoage.com ; MyEmail)
Jasmine/Nutch-1.2 (searching for The One; www.eCartesia.com; Robin.Hood@SherwoodForest.com)
Java/1.5.0
Java/1.5.0_01
Java/1.5.0_13
Java/1.5.0_14
Java/1.5.0_15
Java/1.5.0_16
Java/1.5.0_17
Java/1.5.0_18
Java/1.5.0_19
Java/1.6.0
Java/1.6.0_01
Java/1.6.0_03
Java/1.6.0_06
Java/1.6.0_07
Java/1.6.0_10
Java/1.6.0_12
Java/1.6.0_13
Java/1.6.0_15
Java/1.6.0_16
Java/1.6.0_17
Java/1.6.0_18
Java/1.6.0_19
Java/1.6.0_20
Java/1.6.0_21
Java/1.6.0_22
Java/1.6.0_24
Java/1.6.0_26
Java/1.6.0_27
Java/1.7.0_02
JavaCrawler/1.1
Jawler Beta - arman(dot)vartan(at)itwm(dot)fhg(dot).com
Jawler Beta - arman(dot)vartan(at)itwm(dot)fhg(dot)de
jcrawler4/Nutch-1.3 (bot)
jcrobot
Jennifer/Nutch-0.9 (Jennifer; Jennifer; Jennifer)
Jet/Nutch-1.4
jetbot(+http://premium.jetrun.jp/help/robot.html)
JikeSpider Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html)
JoBo/1.3 (http://www.matuschek.net/jobo.html)
Joy CMU/Nutch-1.2 (Data collection for research ; http://mlt.sv.cmu.edu/joy; joy@cs.cmu.edu)
jrobot (jazy333@gmail.com)
Just another academic search engine/Nutch-1.2 (search; fcprogs (gmail))
JUST-CRAWLER(+http://www.justsystems.com/jp/tech/crawler/)
Jyxobot/1
k2spider
kacst/Nutch-1.4
KAIST AITrc Crawler( crawler . crawling .
kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler)
kapooha/Nutch-1.4 (crawler; http://www.kapooha.com; aeroganic@gmail.com)
KAS/Nutch-1.2 (Nutch ARGUS; MyEmail)
Kavande Crawler 1.0/Nutch-1.4 ( Iranian National Web Crawler ; kh3rad@gmail.com)
Kavande Crawler 1.0/Nutch-1.4 (Iranian National Web Crawler)
Kavande Crawler 1.0/Nutch-1.4 (Kavande)
Kavande Crawler 1.0/Nutch-1.4-dev (Iranian National Web Crawler)
Keybot Translation-Search-Machine
KeywordSearchTool.co/Nutch-1.4 (http://KeywordSearchTool.co/robot)
kindsight/Nutch-1.0 (kscrawler; www.projectrialto.com; crawler@projectrialto.com)
Kiodia Spider/Nutch-1.4
KiwiStatus (NZS.com)/0.2 (NZS.com KiwiStatus Spider, Local Search New Zealand; http://www.nzs.com; bot-at-nzs dot com)
KiwiStatus/0.3 (NZS.com New Zealand Search; http://www.nzs.com/kiwi-status/)
KoopitBot/0.1 (http://www.koopit.com/)
KS Crawler/Nutch-1.0 (http://www.kindsight.net/kscrawler; crawler@kindsight.net)
KS Spider/Nutch-0.9
KSCrawler/Nutch-1.0 (http://www.kindsight.net/en/kscrawler; crawler@kindsight.net)
Kyoto-Crawler/2.0 (Mozilla-compatible; kyoto-crawler-contact(at)nlp(dot)kuee(dot)kyoto-u(dot)ac(dot)jp; http://nlp.ist.i.kyoto-u.ac.jp/)
L.webis/0.19
L.webis/0.32 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
L.webis/0.35 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
L.webis/0.40 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
L.webis/0.42 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
L.webis/0.44 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
L.webis/0.47 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
L.webis/0.50 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
L.webis/0.53 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
L.webis/0.78 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
L.webis/0.85 (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
L.webis/posttool (http://webalgo.iit.cnr.it/index.php?pg=lwebis)
larbin (larbin@unspecified.mail)
larbin (xieys19860101@163.com)
larbin-m (larbin@magellan.cc)
larbin_2.6.3
larbin_2.6.3 (airborne08@gmail.com)
larbin_2.6.3 (changzhiwin@163.com)
larbin_2.6.3 (chowtin@qq.com)
larbin_2.6.3 (cswchli@polyu.edu.hk)
larbin_2.6.3 (fuxiang90@gamil.com)
larbin_2.6.3 (gordon0640@gmail.com)
larbin_2.6.3 (h0451bct@rz.hu-berlin.de)
larbin_2.6.3 (hhgg20000@21cn.com)
larbin_2.6.3 (hustguoguo@qq.com)
larbin_2.6.3 (hwq_super@126.com)
larbin_2.6.3 (kevinew.1221@gmail.com)
larbin_2.6.3 (laomadadupi@gmail.com)
larbin_2.6.3 (larbin2.6.3@unspecified.mail)
larbin_2.6.3 (larbin263@126.com)
larbin_2.6.3 (larbincrawler@gmail.com)
larbin_2.6.3 (myemail@address.co.uk)
larbin_2.6.3 (pcwang@pchome.com.tw)
larbin_2.6.3 (talex@astaro.com)
larbin_2.6.3 (tmdss@163.com)
larbin_2.6.3 (wangqun09@mails.gucas.ac.cn)
larbin_2.6.3 (whoismickey@gmail.com)
larbin_2.6.3 (ymyang2010@hotmail.com)
larbin_2.6.4 (larbin2.6.4@unspecified.mail)
LargeSmall Crawler (LargeSmall; http://onespot.com; info@onespot.com)
larsrc (nobody@dreamland.com)
lb-spider/Mozilla/5.0 Gecko/20100101 Firefox/10.0.2 (lb-spider; http://www.linkbutler.de/spider; spider@linkbutler.de)
Leo Chow/Nutch-0.9 (Nutch spiderman; http://www-u1.icil.net; MyEmail)
lessonupgrades/Nutch-1.2
LexxeBot/1.0 (lexxebot@lexxe.com)
lhsoso.com/1.0
libcurl-agent/1.0
libiViaCore/5.0 generate_infomine_category_classifiers (http://changesiv.ucr.edu/useragents.html)
libiViaCore/5.0 generate_infomine_category_classifiers (http://irsdev.intute.ac.uk:8999/useragents.html)
librabot/1.0 (+http://search.msn.com/msnbot.htm)
librabot/1.0 (compatible; MSIE 6.0; Windows NT 5.0)
librabot/2.0 (+http://academic.research.microsoft.com/)
librabot/2.0 (+http://search.msn.com/msnbot.htm)
Libri amici, libri magistri
libwww-perl/5.805
libwww-perl/5.831
libwww-perl/5.834
LICrawler licrawler@gmail.com
Lijit Crawler (+http://www.lijit.com/robot/crawler)
Lijit Crawler/Nutch-0.9 (Reports crawler; http://www.lijit.com/robot/crawler; info(a)lijit(d)com)
LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/robot/crawler; info(a)lijit(d)com)
LinguaBot/v0.001-dev (MultiLinual Sarch Engine v0.001; LinguaSeek; admin at linguaseek dot com)
Linguee Bot (bot@linguee.com)
Linguee Bot (http://www.linguee.com/bot; bot@linguee.com)
LinkAider (http://linkaider.com/crawler/)
LinkAlarm/3.0
LinkChecker/3.4 (+http://linkchecker.sourceforge.net/)
LinkChecker/4.8 (+http://linkchecker.sourceforge.net/)
LinkChecker/4.9 (+http://linkchecker.s/urceforge.net/)
LinkChecker/4.9 (+http://linkchecker.sourceforge.net/)
LinkChecker/5.1 (+http://linkchecker.sourceforge.net/)
LinkChecker/5.2 (+http://linkchecker.sourceforge.net/)
LinkChecker/5.3 (+http://linkchecker.sourceforge.net/)
LinkChecker/6.2 (+http://linkchecker.sourceforge.net/)
LinkChecker/6.3 (+http://linkchecker.sourceforge.net/)
LinkChecker/6.4 (+http://linkchecker.sourceforge.net/)
LinkChecker/6.5 (+http://linkchecker.sourceforge.net/)
LinkChecker/6.6 (+http://linkchecker.sourceforge.net/)
LinkChecker/6.7 (+http://linkchecker.sourceforge.net/)
LinkChecker/6.8 (+http://linkchecker.sourceforge.net/)
LinkChecker/6.9 (+http://linkchecker.sourceforge.net/)
LinkChecker/7.0 (+http://linkchecker.sourceforge.net/)
LinkChecker/7.1 (+http://linkchecker.sourceforge.net/)
LinkChecker/7.2 (+http://linkchecker.sourceforge.net/)
LinkChecker/7.3 (+http://linkchecker.sourceforge.net/)
LinkChecker/7.4 (+http://linkchecker.sourceforge.net/)
LinkChecker/7.5 (+http://linkchecker.sourceforge.net/)
LinkStar Research Crawler (http://linkstar.com/)
LinkWalker/2.0
linky
Lisboa/Nutch-1.2
LitlrBot (http://litlr.me/bot.html)
livedoor ScreenShot/0.10
li_viewer (larbin2.6.3@unspecified.mail)
lmspider/Nutch-0.9-dev (For research purposes.; www.nuance.com; lmspider@nuance.com)
lmspider/Nutch-1.2 (For research purposes.; www.nuance.com; tegic-SMS atdomain nuance dot com)
lobot/0.3 (http://www.lofoso.com/robot.html)
local/Nutch-1.2
LOCIndexer/Nutch-1.0
LookBot
loosos_bot/Loos-1.2 (loosos_crawler_bot; loosos.com; bot@loosos.com)
loosos_bot/loosos v1.0 (loosos.com crawler; loosos.com; loosos@126.com)
LucidMedia ClickSense/4.0 (support@lucidmedia.com; http://www.lucidmedia.com/)
Lycos_Spider_(modspider)
LynnBot/1
Lynx/2.8.4rel.1 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.6c ()
Lynx/Lynx(2.0)
Lynx/Lynx(2.2) (Lynx)
Lynx/Lynx(8899)
Lynx/Nutch-1.4
m0cca.com/Nutch-1.1 (www.m0cca.com; crawler@m0cca.com)
M42bot/Nutch-1.0 (Prototype; http://en.wikipedia.org/wiki/Web_crawler; donotreply at prototype dot com)
magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
magpie-crawler/1.1 (U; Linux i386; en-GB; +http://www.brandwatch.net)
mahonie, neofonie search:robot (http://spider.neofonie.de)/Nutch-1.4 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to spider@neofonie.de; http://spider.neofonie.de)
mahonie, neofonie search:robot/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to spider@neofonie.de; http://spider.neofonie.de; spider@neofonie.de)
Mail.Ru/1.0
Mail.RU/2.0
male.com.sg/Nutch-1.1 (http://www.male.com.sg; crawler@male.com.sg)
male.com.sg/Nutch-1.1 (www.male.com.sg; crawler@male.com.sg)
MANTIS/Nutch-1.4
Mediapartners-Google
Medio HttpUrlSearchProvider(medio.com)
MegaSpider
MELANI/Nutch-0.9 (Nutch spiderman; http://www.melani.admin.ch; MyEmail)
memkompreneble
mercury/Nutch-1.2
merocrawl/Nutch-1.4 (merobase crawler; www.merobase.com; info@merobase.com)
mesiuma/Nutch-1.0
MetaGeneratorCrawler/1.1 (www.metagenerator.info)
Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner.de/)
Metaspinner/1.0 (Metaspinner Search Engine; http://www.meta-spinner.de/; support@meta-spinner.de)
MetaTagRobot/1.9 (http://www.widexl.com/remote/search-engines/metatag-analyzer.html)
MetaTagRobot/2.1 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html)
Metz/Nutch-1.2
Microsoft Bing Mobile SocialStreams Bot
Midas/Nutch-1.0 (Nutch Spider; www.unal.edu.co; yamile.jaime@gmail.com)
MireoBot
mixi-mobile-converter/1.0 (http://mixi.jp/)
miyatai/Nutch-1.4
MLBot (www.metadatalabs.com/mlbot)
mmonitor/Nutch-1.0 (MarkMonitor Robots; http://www.markmonitor.com; tlin@markmonitor.com)
mnoGoSearch (http://www.truthsearch.us/)
MnoGoSearch/3.3.8
MnoGoSearch/3.3.9
Mobier/Mobier-1.0 (http://mobier.mobi/; abc@mobier.mobi)
Mobier/Mobier-1.0 (http://mobier.mobi/; service@mobier.mobi)
Modiphibot/0.9 (Rev.Nagayama1; http://www.modiphi.com/)
Mojolicious (Perl)
MOMspider/1.00 libwww-perl/0.40
Monkeyworx Spider/Nutch-1.2 (http://example.com; noreply@example.com)
mootlaw/Nutch-1.2 (MootLaw Searc Bot; mootlaw.com; eric at moot law dot com)
Motoricerca-Robots.txt-Checker/1.0 (http://tool.motoricerca.info/robots-checker.phtml)
Mozilla crawl/5.0 (compatible; fairshare.cc +http://fairshare.cc)
mozilla firefox
Mozilla incompatible Wget (http://www.artiesoft.com/lexxbot.php mailto:info-429artiesoft.com)
Mozilla/4.0 (+http://www.up.ru)
Mozilla/4.0 (compatible: FDSE robot)
Mozilla/4.0 (compatible; Coosto linkresolver 1.0; http://www.coosto.nl/)
Mozilla/4.0 (compatible; Vagabondo/4.0; http://webagent.wise-guys.nl/)
Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
Mozilla/4.0 (compatible; ApocalXExplorerBot/0.9; +http://explorer.apocalx.com/bot.php)
Mozilla/4.0 (compatible; http://search.thunderstone.com/texis/websearch/about.html)
Mozilla/4.0 (compatible; Kwaclebot/0.8; http://www.kwacle.com/kbot.html)
Mozilla/4.0 (compatible; MSIE 4.01; Windows 95; http://www.freefind.com/spider.html )
Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19.123.2.733) OrangeBot-Mobile 2008.0 (mobilesearch.support@orange-ftgroup.com)
Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 5.0 Robot)
Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 6.0 Robot)
Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Instituto Cervantes
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; FeedFinder-2.0; http://bloggz.se/crawler)
Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)
mozilla/4.0 (compatible; msie 6.0; windows 2000)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) (larbin2.6.3@unspecified.mail)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) (larbin@unspecified.mail)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Embedded Web Browser from: http://bsalsa.com/; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322; +http://www.archive.org)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 2.0.50727; CIBA; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 2.0.50727; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Embedded Web Browser from: http://bsalsa.com/; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts; GTB5; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 2.0.50727; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; InfoPath.1; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB5; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 1.1.4322; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB6.4; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 2.0.50727; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 2.0.50727; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; TuneUp HTML Client Embedded Web Browser from: http://bsalsa.com/; .NET CLR 1.1.4322; InfoPath.2; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 1.7; EmbeddedWB 14.52 from: http://www.bsalsa.com/ EmbeddedWB 14.52; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; TencentTraveler 4.0; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 2.0.50727; CIBA; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; FDM; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; FunWebProducts; GTB6; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 1.0.3705; Media Center PC 4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 1.1.4322; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; AskTB5.3)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; FunWebProducts; MindQuizSearchToolbar 1.1; GTB6.5; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 1.0.3705; Media Center PC 4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 1.1.4322; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; AskTB5.3)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6.4; FunWebProducts; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 1.0.3705; Media Center PC 4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 1.1.4322; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; AskTB5.3)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; livebot-searchsense/0.1; +http://search.msn.com/msnbot.htm)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; TencentTraveler 4.0; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; IE7Pro)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Webaroo/1.3.940; MSDigitalLocker; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Wysigot 6.01; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; FDM; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; MS-RTC LM 8; InfoPath.2; OfficeLiveConnector.1.3; OfficeLivePatch.1.3; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; WOW64; User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://bsalsa.com) ; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; +http://www.europarchive.org)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30; IE7Pro; .NET CLR 3.0.04506.648; .NET CLR 1.1.4322; .NET CLR 3.5.21022)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://bsalsa.com) ; FDM; .NET CLR 2.0.50727)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://bsalsa.com) ; Tablet PC 1.7; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR 2.0.50727; IE7Pro; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; InfoPath.2; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.jsp)
Mozilla/4.0 (larbin2.6.3@unspecified.mail)
Mozilla/4.0 (Toread-Crawler/1.1; +http://news.toread.cc/crawler.php)
Mozilla/4.0 http://referencement.abaabaa.com/add_url.php : Easy referencement (version 1.0)
Mozilla/4.0 http://referencement.abaabaa.com/add_url_GRATUIT.php : Easy referencement (version 1.0)
Mozilla/4.0(compatible;MSIE7.0) (larbin2.6.3@unspecified.mail)
Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html )
Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html ; agentname at setooz dot_com )
Mozilla/5.0 ((compatible;+Baiduspider/2.0;++http://www.baidu.com/search/spider.html))
Mozilla/5.0 (compatable; FNRBot/1.0; http://www.fairwaysandroughs.com/bot.php)
Mozilla/5.0 (compatible; heritrix/3.1.0 +http://mozilla.org)
Mozilla/5.0 (compatible; +http://pk.com)
Mozilla/5.0 (compatible; +http://www.evri.com/evrinid)
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620
Mozilla/5.0 (compatible; 80bot/0.71; http://www.80legs.com/spider.html;) Gecko/2008032620
Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html)
Mozilla/5.0 (compatible; AcoonBot/4.10.6; +http://www.acoon.de/robot.asp)
Mozilla/5.0 (compatible; AcoonBot/4.10.7; +http://www.acoon.de/robot.asp)
Mozilla/5.0 (compatible; AhrefsBot/1.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/2.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/3.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; aiHitBot/1.0-DS; +http://www.aihit.com/)
Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/)
Mozilla/5.0 (compatible; aiHitBot/1.1; +http://www.aihit.com/)
Mozilla/5.0 (compatible; akula/k311; +http://k311.fd.cvut.cz/)
Mozilla/5.0 (compatible; AntBot/1.0; +http://www.ant.com/)
Mozilla/5.0 (compatible; ApptusBot/1.0; +http://www.apptus.com)
Mozilla/5.0 (compatible; archive.org_bot +http://pandora.nla.gov.au/crawl.html)
Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; archive.org_bot +http://www.bne.es/es/LaBNE/PreservacionDominioES/AvisoWebmasters/index.html)
Mozilla/5.0 (compatible; archive.org_bot/1.14.1 +http://www.loc.gov/minerva/crawl.html)
Mozilla/5.0 (compatible; archive.org_bot/1.15.4 +http://www.loc.gov/minerva/crawl.html)
Mozilla/5.0 (compatible; archive.org_bot/1.5.0 +http://www.bne.es/es/LaBNE/PreservacionDominioES/AvisoWebmasters/index.html)
Mozilla/5.0 (compatible; archive.org_bot/1.5.0 +http://www.loc.gov/minerva/crawl.html)
Mozilla/5.0 (compatible; archive.org_bot/1.5.4 +http://www.loc.gov/minerva/crawl.html)
Mozilla/5.0 (compatible; archive.org_bot/1.8.0 +http://www.loc.gov/minerva/crawl.html)
Mozilla/5.0 (compatible; archive.org_bot/3.1.0 +http://www.archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; archive.org_bot/3.1.1 +http://pandora.nla.gov.au/crawl.html)
Mozilla/5.0 (compatible; archive.org_bot/3.1.1 +http://www.bne.es/es/LaBNE/PreservacionDominioES/AvisoWebmasters/index.html)
Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.archive.org)
Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.bne.es/es/LaBNE/PreservacionDominioES/AvisoWebmasters/index.html)
Mozilla/5.0 (compatible; archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html)
Mozilla/5.0 (compatible; archive_crawler/3.0.0-SNAPSHOT-20091205.013431 +http://www.archive.org/details/archive_crawler)
Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://www.ask.com/about/help/webmasters)
Mozilla/5.0 (compatible; Atomic5 Search; +http://atomic5.com/)
Mozilla/5.0 (compatible; attributor/1.13.2 +http://www.attributor.com)
Mozilla/5.0 (compatible; Ayna +http://www.ayna.com)
Mozilla/5.0 (compatible; ayna-crawler +http://www.ayna.com)
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Mozilla/5.0 (compatible; BeatrixxX/1.14.4 +http://beerwurstonkelz.org)
Mozilla/5.0 (compatible; Bender; http://benderthewebrobot.tumblr.com)
Mozilla/5.0 (compatible; Bender; http://sites.google.com/site/bendercrawler)
Mozilla/5.0 (compatible; bingbot-media/2.0 +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; Birubot/1.0) Gecko/2009032608 Firefox/3.0.8
Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; crawler@bixolabs.com)
Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; crawler@mail.bixolabs.com)
Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)
Mozilla/5.0 (compatible; BlinkaCrawler/1.0; +http://www.blinka.jp/crawler/)
Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/)
Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/bnf.html)
Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)
Mozilla/5.0 (compatible; bush-crawler +http://www.arkevista.com)
Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) Gecko/2009032608 Firefox/3.0.8
Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8
Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)
Mozilla/5.0 (compatible; cdlwas_bot; heritrix/1.14.1 +http://webarchives.cdlib.org/p/webmasters)
Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/)
Mozilla/5.0 (compatible; CligooRobot/2.0; +http://www.cligoo.de/wk/technik.php)
Mozilla/5.0 (compatible; crawler/3.0.0 +http://www.notconfigured.com/)
Mozilla/5.0 (compatible; crawler/3.1.0-SNAPSHOT-20110914.213704 +http://www.notconfigured.com/)
Mozilla/5.0 (compatible; cuil.Com/1.14.3 +http://www.cuil.com)
Mozilla/5.0 (compatible; Cyberz Communication Agent; +http://www.cyberz.co.jp/)
Mozilla/5.0 (compatible; DCPbot/1.0; +http://domains.checkparams.com/)
Mozilla/5.0 (compatible; DCPbot/1.1; +http://domains.checkparams.com/)
Mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html
Mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
Mozilla/5.0 (compatible; discobot/2.0; +http://discoveryengine.com/discobot.html)
Mozilla/5.0 (compatible; Dolphin/1.0; +http://tele-house.ru/crawler.html)
Mozilla/5.0 (compatible; Dolphin/1.0; +http://telehouse.ru/crawler.html)
Mozilla/5.0 (compatible; dombot +http://domlog.org)
Mozilla/5.0 (compatible; dooplr/2.3.5 +http://www.yahoo.com)
Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)
Mozilla/5.0 (compatible; dotSemantic/1.0; +http://www.dotsemantic.org)
Mozilla/5.0 (compatible; Dow Jones Searchbot)
Mozilla/5.0 (compatible; EllerdaleBot/ 1.0; +http://www.ellerdale.com/crawler.html)
Mozilla/5.0 (compatible; en-US; ReverseGet/1.0; http://reverseget.com/; robot@reverseget.com)
Mozilla/5.0 (compatible; Estbot/1.0; http://www.estsoft.com/)
Mozilla/5.0 (compatible; EventGuruBot/1.0; +http://www.eventguru.com/spider.html)
Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot)
Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)
Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)
Mozilla/5.0 (compatible; Falconsbot; +http://iws.seu.edu.cn/services/falcons/contact_us.jsp)
Mozilla/5.0 (compatible; Falconsbot; +http://ws.nju.edu.cn/falcons/)
Mozilla/5.0 (compatible; Feedtrace-bot/0.2; bot@feedtrace.com)
Mozilla/5.0 (compatible; FlipboardBrowserProxy/1.1; +http://flipboard.com/browserproxy)
Mozilla/5.0 (compatible; FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
Mozilla/5.0 (compatible; FMSH; http://ticmigrations.fr/)
Mozilla/5.0 (compatible; foobot/0.0.1 +http://foofind.com)
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html)
Mozilla/5.0 (compatible; Hailoobot/1.2; +http://www.hailoo.com/spider.html)
Mozilla/5.0 (compatible; HatoPoppo/1.0b; http://potaru.com/HatoPoppo.html)/Nutch-1.2
Mozilla/5.0 (compatible; heritrix/ 1.14.3+http://202.200.48.59)
Mozilla/5.0 (compatible; heritrix/${pom.version} +http://crawler.archive.org)
Mozilla/5.0 (compatible; heritrix/${pom.version} +http://seekda.com)
Mozilla/5.0 (compatible; heritrix/${pom.version} +http://webservices.seekda.com)
Mozilla/5.0 (compatible; heritrix/${pom.version} +http://www.adsafemedia.com)
Mozilla/5.0 (compatible; heritrix/1.10.0+http://bbs.ustc.edu.cn/)
Mozilla/5.0 (compatible; heritrix/1.10.1 +http://www2.zzu.edu.cn/xxgl)
Mozilla/5.0 (compatible; heritrix/1.10.2 +http://netarkivet.dk/website/info.html; http://eia.libis.lt)
Mozilla/5.0 (compatible; heritrix/1.12.0 +http://www.webcorp.org.uk/)
Mozilla/5.0 (compatible; heritrix/1.12.1 +http://archive.aueb.gr)
Mozilla/5.0 (compatible; heritrix/1.12.1 +http://curiousgeorge.info)
Mozilla/5.0 (compatible; heritrix/1.12.1 +http://eia.libis.lt)
Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.webarchiv.cz)
Mozilla/5.0 (compatible; heritrix/1.12.1b +http://netarkivet.dk/website/info.html)
Mozilla/5.0 (compatible; heritrix/1.14.0 +http://10.0.11.109)
Mozilla/5.0 (compatible; heritrix/1.14.0 +http://lib.csu.edu.cn/pub/zndxtsg/index.htm)
Mozilla/5.0 (compatible; heritrix/1.14.0 +http://webarchive.nlc.gov.cn)
Mozilla/5.0 (compatible; heritrix/1.14.0 +http://www.banq.qc.ca/archivageweb/carte.html)
Mozilla/5.0 (compatible; heritrix/1.14.0 +http://www.cienciapt.net)
Mozilla/5.0 (compatible; heritrix/1.14.0+http://www.nationallibrary.fi/)
Mozilla/5.0 (compatible; heritrix/1.14.1 +http://www.cdlib.org/)
Mozilla/5.0 (compatible; heritrix/1.14.1 +http://www.enxnc.com.cn)
Mozilla/5.0 (compatible; heritrix/1.14.1 +http://www.nlc.gov.cn/)
Mozilla/5.0 (compatible; heritrix/1.14.1 +http://www.padi.cat)
Mozilla/5.0 (compatible; heritrix/1.14.1 +http://www.webarchive.org.uk/)
Mozilla/5.0 (compatible; heritrix/1.14.1 +http://www.widernet.org)
Mozilla/5.0 (compatible; heritrix/1.14.1+http://192.168.0.1)
Mozilla/5.0 (compatible; heritrix/1.14.2 +http://www.novinky.cz)
Mozilla/5.0 (compatible; heritrix/1.14.2 yptrino +http://www.buddybuzz.net/yptrino)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://127.0.0.1)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://192.168.0.1)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://192.168.0.91)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://202.200.49.123/)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://archive.org)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://badcheese.com)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://dtn.net)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://ece.ut.ac.ir)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://europarchive.org)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://mytest.com)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://nipg.inf.elte.hu)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://web.archive.org)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.accelobot.com)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.cs.ucl.ac.uk/staff/C.Raiciu/crawl.html)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.google.com)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.it.kmitl.ac.th)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.webarchiv.cz)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.xbmu.edu.cn/)
Mozilla/5.0 (compatible; heritrix/1.14.3+http://202.200.59.7)
Mozilla/5.0 (compatible; heritrix/1.14.3.r6601 +http://www.buddybuzz.net/yptrino)
Mozilla/5.0 (compatible; heritrix/1.14.3.r6601 +http://www.yp.com)
Mozilla/5.0 (compatible; heritrix/1.14.3.r6601+http://www.buddybuzz.net/yptrino)
Mozilla/5.0 (compatible; heritrix/1.14.3.r6601+http://www.yellowpages.com/about/legal/crawl)
Mozilla/5.0 (compatible; heritrix/1.14.3.r6601+http://www.yp.com)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://192.168.1.11)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://221.204.244.37)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://ce.yazduni.ac.ir)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://censoweb.speed.dcc.ufmg.br)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://graphseeker.com)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://hpsg.fu-berlin.de)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://iran.ir/)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://it.usyd.edu.au/~smerity/schwa/crawler/)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://lib.ccut.edu.cn/)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://susanszky.tatk.elte.hu)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://swr.de)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.attrakt.com)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.cacak-dis.rs)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.ccert.edu.cn/)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.dlsu.edi.ph)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.findthatfile.com)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.google.com)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.iran.ir/)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.kb.se)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.speed.dcc.ufmg.br)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.test.test)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.youscribe.com)
Mozilla/5.0 (compatible; heritrix/1.14.4+http://today.hit.edu.cn/)
Mozilla/5.0 (compatible; heritrix/1.15.1-200807172326 +http://www.accelobot.com)
Mozilla/5.0 (compatible; heritrix/1.15.4-200909230304 +http://pirun.ku.ac.th/~g5165280)
Mozilla/5.0 (compatible; heritrix/1.15.5-201006012335 +http://yandex.ru)
Mozilla/5.0 (compatible; heritrix/1.5 +http://www.metacarta.com)
Mozilla/5.0 (compatible; heritrix/1.6.0 +http://www.offis.de)
Mozilla/5.0 (compatible; heritrix/1.6.0 +http://www.schluetersche.de)
Mozilla/5.0 (compatible; heritrix/1.6.0-OFFIS +http://www.test.test)
Mozilla/5.0 (compatible; heritrix/1.8.0 +http://www.bl.uk/)
Mozilla/5.0 (compatible; heritrix/1.8.0 +http://www.cs.vt.edu)
Mozilla/5.0 (compatible; heritrix/1.8.0 +http://yourwebsite.domain.com/)
Mozilla/5.0 (compatible; heritrix/2.0.0 +http://www.aol.com)
Mozilla/5.0 (compatible; heritrix/2.0.0 +http://www.google.com)
Mozilla/5.0 (compatible; heritrix/2.0.1 +http://tibesti.research.att.com/research-crawler.html)
Mozilla/5.0 (compatible; heritrix/2.0.2 +http://getprotected.asu.edu/files/wass.pdf)
Mozilla/5.0 (compatible; heritrix/2.0.2 +http://localhost)
Mozilla/5.0 (compatible; heritrix/2.0.2 +http://seekda.com)
Mozilla/5.0 (compatible; heritrix/2.0.2 +http://www.loc.gov/webcapture/)
Mozilla/5.0 (compatible; heritrix/2.0.2 +http://www.yell.com)
Mozilla/5.0 (compatible; heritrix/3.0.0 +http://dataclip.com/crawler.html)
Mozilla/5.0 (compatible; heritrix/3.0.0 +http://vefsofnun.bok.hi.is)
Mozilla/5.0 (compatible; heritrix/3.0.0 +http://webarchive.loc.gov)
Mozilla/5.0 (compatible; heritrix/3.0.0 +http://www.bl.uk/aboutus/stratpolprog/digi/domresproj/index.html)
Mozilla/5.0 (compatible; heritrix/3.0.0 +http://www.dataclip.com)
Mozilla/5.0 (compatible; heritrix/3.0.0 +http://www.webarchive.org.uk/)
Mozilla/5.0 (compatible; heritrix/3.0.0-beta +http://webarchive.loc.gov)
Mozilla/5.0 (compatible; heritrix/3.0.0-beta +http://webarchives.loc.gov)
Mozilla/5.0 (compatible; heritrix/3.0.1-SNAPSHOT-20110209.020021 +http://newsconsole.com)
Mozilla/5.0 (compatible; heritrix/3.0.1-SNAPSHOT-20110328.124355 +http://www.neoscio.de)
Mozilla/5.0 (compatible; heritrix/3.1.0 +http://mith.umd.edu)
Mozilla/5.0 (compatible; heritrix/3.1.0 +http://sid.uncu.edu.ar/sid/cdh)
Mozilla/5.0 (compatible; heritrix/3.1.0 +http://www.library.unt.edu/digitalprojects/web-archiving)
Mozilla/5.0 (compatible; heritrix/3.1.0 +http://www.northgrid.co.jp)
Mozilla/5.0 (compatible; heritrix/3.1.0-SNAPSHOT-20110614.230024 +http://www.sina.com.cn)
Mozilla/5.0 (compatible; heritrix/3.1.0-SNAPSHOT-20110920.235706 +http://www.archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; heritrix/3.1.0-SNAPSHOT-20111013.155210 +http://archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20100429.232622 +http://crawler.archive.org)
Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20100915.040035 +http://www.archive.bibalex.org)
Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20100928.230034 +http://www.dataclip.com)
Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20110411.223526 +http://efca.pl/crawler)
Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 +http://www.archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120320.221924 +http://archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; heritrix/UNKNOWN +http://www.adsafemedia.com)
Mozilla/5.0 (compatible; heritrix/UNKNOWN +http://www.google.com)
Mozilla/5.0 (compatible; HomeTags/1.0; +http://www.hometags.nl/bot)
Mozilla/5.0 (compatible; http://summarity.com/)
Mozilla/5.0 (compatible; http; +http://www.idrc.ca/cp/ev-136691-201-1-DO_TOPIC.html; tangyang9@gmail.com)
Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/
Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/)
mozilla/5.0 (compatible; INA dlweb; +http://www.ticm.msh-paris.fr/spip.php?article6)
Mozilla/5.0 (compatible; IntelCSbot/0.2.1beta)
Mozilla/5.0 (compatible; IstellaBot/1.01.18 +http://www.tiscali.it/)
Mozilla/5.0 (compatible; janforman.com/2.4; +http://janforman.com/)
Mozilla/5.0 (compatible; KaloogaBot; http://kalooga.com/crawler)
Mozilla/5.0 (compatible; KaloogaBot; http://www.kalooga.com/info.html?page=crawler)
Mozilla/5.0 (compatible; KLSH-PageGet/1.03; http://www.kondo-net.gr.jp/klsh/)
Mozilla/5.0 (compatible; KoDe/0.2; +http://www.knoova.com/)
Mozilla/5.0 (compatible; kulturarw3 +http://www.kb.se/soka/internet/sv-webbsidor/)
Mozilla/5.0 (compatible; lemurwebcrawler admin@lemurproject.org; +http://boston.lti.cs.cmu.edu/crawler_12/)
Mozilla/5.0 (compatible; linkfluence/0.9; http://linkfluence.net/)
Mozilla/5.0 (compatible; linkfluence; http://linkfluence.net/)
Mozilla/5.0 (compatible; loc-crawler/0.11.0 +http://archive.org)
Mozilla/5.0 (compatible; loc-crawler/0.11.0 +http://loc.gov)
Mozilla/5.0 (compatible; loc-crawler/3.0.0 +http://webarchive.loc.gov)
Mozilla/5.0 (compatible; loc-crawler/3.0.1-SNAPSHOT-20110412.010027 +http://webarchive.loc.gov)
Mozilla/5.0 (compatible; loku; +http://loku.com/; development@loku.com)
Mozilla/5.0 (compatible; ltbot/1.3; bot@language-tools.com)
Mozilla/5.0 (compatible; LucidWorks/; +; crawler at example dot com)
Mozilla/5.0 (compatible; LuminateBot/1.0; +http://www.luminate.com/bot/)
Mozilla/5.0 (compatible; MendeleyBot +http://mendeley.com))
Mozilla/5.0 (compatible; MetamojiCrawler/1.0; +http://www.metamoji.com/jp/crawler.html
Mozilla/5.0 (compatible; METASpider; +http://meta.ua/spider)
Mozilla/5.0 (compatible; Microsoft MSIE 7.0 +http://www.europarchive.org)
Mozilla/5.0 (compatible; MJ12bot/v1.2.1; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.2.4; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.2.5; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.3.0; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.3.1; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.3.2; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+),gzip(gfe),gzip(gfe),gzip(gfe) AppEngine-Google; (+http://code.google.com/appengine; appid: pr0xy--me)
Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html#relaunch)
Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html)
Mozilla/5.0 (compatible; Mozilla/5.0 +http://www.up.ru)
Mozilla/5.0 (compatible; Mp3Bot/0.4; +http://mp3realm.org/mp3bot/)
Mozilla/5.0 (compatible; Mp3Bot/0.6; +http://mp3realm.org/mp3bot/)
Mozilla/5.0 (compatible; Mp3Bot/0.7; +http://mp3realm.org/mp3bot/)
Mozilla/5.0 (compatible; MSIE 6.0 Win 5.1 +http://www.europarchive.org)
Mozilla/5.0 (compatible; MSIE 7.0 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; MSIE 7.0 +http://internetmemory.org/en/); crawling on behalf of The National Archives UK Government Web Archive: http://www.nationalarchives.gov.uk/webarchive/; email: webarchive@nationalarchives.gsi.gov.uk
Mozilla/5.0 (compatible; MSIE 7.0 +http://www.europarchive.org)
Mozilla/5.0 (compatible; MSIE 7.0 Win32 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; MSIE 7.0 Win32 +http://www.europarchive.org/)
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/2.0
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/3.0
Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)
Mozilla/5.0 (compatible; NerdByNature.Bot; http://www.nerdbynature.net/bot)
Mozilla/5.0 (compatible; Netseer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com)
Mozilla/5.0 (compatible; Nigma.ru/3.0; crawler@nigma.ru)
Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)
Mozilla/5.0 (compatible; oBot/2.3.1; +http://www-935.ibm.com/services/us/index.wss/detail/iss/a1029077?cntxt=a1027244)
Mozilla/5.0 (compatible; OGSearchSpider 3508 tom@lemurconsulting.com)
Mozilla/5.0 (compatible; OntoPort/; +; admin@ontoport.de)
Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/)
Mozilla/5.0 (compatible; Plukkie/1.1; http://www.botje.com/plukkie.htm)
Mozilla/5.0 (compatible; Plukkie/1.2; http://www.botje.com/plukkie.htm)
Mozilla/5.0 (compatible; Plukkie/1.3; http://www.botje.com/plukkie.htm)
Mozilla/5.0 (compatible; Plukkie/1.4; http://www.botje.com/plukkie.htm)
Mozilla/5.0 (compatible; Popular-Bookmark-Bot/1.0; +http://www.popular-bookmark.com/)
Mozilla/5.0 (compatible; Presansbot/1.0 +http://www.presans.com)
Mozilla/5.0 (compatible; proximic; +http://www.proximic.com)
Mozilla/5.0 (compatible; ptd-crawler; +http://bixolabs.com/crawler/ptd/; crawler@bixolabs.com)
Mozilla/5.0 (compatible; Purebot/1.1; +http://www.puritysearch.net/)
Mozilla/5.0 (compatible; q1; +http://www.qleeq.com; info@qleeq.com)
Mozilla/5.0 (compatible; Robo/1.0b; http://potaru.com/robo.html)/Nutch-1.2
Mozilla/5.0 (compatible; RTGI; http://rtgi.fr/)
Mozilla/5.0 (compatible; RTSbot/1.0; +http://adamcaudill.com/rtsbot/)
Mozilla/5.0 (compatible; Scarlett/ 1.0; +http://www.ellerdale.com/crawler.html)
Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)
Mozilla/5.0 (compatible; ScribdReader/1.0; +http://www.float.com)
Mozilla/5.0 (compatible; ScribdReader/1.0; +http://www.scribd.com/reader.html)
Mozilla/5.0 (compatible; Search17Bot/1.1; http://www.search17.com/bot.php)
Mozilla/5.0 (compatible; Semager/1.4; http://www.semager.de/blog/semager-bots/)
Mozilla/5.0 (compatible; Semager/1.4c; +http://www.semager.de/blog/semager-bots/)
Mozilla/5.0 (compatible; SEODat/0.1 http://crawler.seodat.com)
Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)
Mozilla/5.0 (compatible; sgbot v0.01a, sgcrawlerbot@gmail.com)
Mozilla/5.0 (compatible; sindice-fetcher/0.1.0 +http://sindice.com/developers/bot)
Mozilla/5.0 (compatible; sindice-site-manager/0.1.0 +http://sindice.com/developers/bot)
Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)
Mozilla/5.0 (compatible; SiteBot/0.1; +http://www.sitebot.org/robot/)
Mozilla/5.0 (compatible; SkreemRBot +http://skreemr.com)
Mozilla/5.0 (compatible; socketcrawler; http://nlp.fi.muni.cz/projects/biwec/)
Mozilla/5.0 (compatible; spbot/1.0; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/1.1; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/1.2; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/1.5; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/2.0.1; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/2.0.2; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/2.0.3; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/2.0.4; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ )
Mozilla/5.0 (compatible; spbot/2.1; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; spbot/3.0; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; special_archiver/1.5.0 +http://www.bne.es/es/LaBNE/PreservacionDominioES/AvisoWebmasters/index.html)
Mozilla/5.0 (compatible; special_archiver/1.5.0 +http://www.loc.gov/minerva/crawl.html)
Mozilla/5.0 (compatible; special_archiver/1.5.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)
Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); http://nlp.fi.muni.cz/projects/biwec/)
Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/)
Mozilla/5.0 (compatible; suggybot v0.01a, sgcrawlerbot@gmail.com)
Mozilla/5.0 (compatible; Swarm; +swarm@webscaled.com)
Mozilla/5.0 (compatible; Tagoobot/3.0; +http://www.tagoo.ru)
Mozilla/5.0 (compatible; TopBlogsInfo/2.0; +topblogsinfo@gmail.com)
Mozilla/5.0 (compatible; U; Linux i686 (x86_64); de-DE; <a href=http://wikimpress.org/>Wikimpress</a>) Wikimpress/1.0
Mozilla/5.0 (compatible; Urlfilebot/2.2; +http://urlfile.com/bot.html)
Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html)
Mozilla/5.0 (compatible; Voluniabot/1.0; spider@volunia.com)
Mozilla/5.0 (compatible; WASALive Bot ; http://blog.wasalive.com/wasalive-bots/)
Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/)
Mozilla/5.0 (compatible; Wazzup1.0.1028; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.1446; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.1630; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.1830; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.2280; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.2496; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.2926; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.3265; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.3635; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.4087; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.4172; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.5321; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.6644; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.6811; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.6957; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.7005; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.7225; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.8567; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.9277; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.9813; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; Wazzup1.0.9989; http://75.126.176.112/Wazzup)
Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)
Mozilla/5.0 (compatible; Web CEO Online robot)
Mozilla/5.0 (compatible; WebCensusProjectBot/0.03 +http://www.webcensusproject.org/)
Mozilla/5.0 (compatible; WebmasterCoffee/0.7; +http://webmastercoffee.com/about)
Mozilla/5.0 (compatible; woriobot +http://worio.com)
Mozilla/5.0 (compatible; XmarksFetch/1.0; +http://www.xmarks.com/about/crawler; info@xmarks.com)
Mozilla/5.0 (compatible; xrss; robot; http://www.xrss.eu/robot; version 2.0)
Mozilla/5.0 (compatible; Yahoo! Slurp Asia; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
Mozilla/5.0 (compatible; Yahoo! Slurp/si-emb; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; R; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; Yeti/1.0; NHN Corp.; http://help.naver.com/robots/)
Mozilla/5.0 (compatible; YioopBot +http://www.yioop.com/bot.php)
Mozilla/5.0 (compatible; YioopBot +http://yioop.com/bot.php)
Mozilla/5.0 (compatible; YioopBot +http://www.yioop.com/bot.html)
Mozilla/5.0 (compatible; YioopBot; +http://www.yioop.com/bot.php)
Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
Mozilla/5.0 (compatible;+LolongBot/1.0; +http://www.yisou.us/help/webmaster/)
Mozilla/5.0 (compatible;archive.org_bot/1.7.0; Archive-It; +http://archive-it.org/files/site-owners.html)
Mozilla/5.0 (compatible;archive.org_bot/1.7.0; Archive-It; +http://www.archive-it.org)
Mozilla/5.0 (compatible;archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html) Firefox
Mozilla/5.0 (compatible;archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html) Firefox/0.0
Mozilla/5.0 (compatible;picmole/1.0 +http://www.picmole.com)
Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;)
Mozilla/5.0 (http://textdigger.com) Gecko/20061208 Firefox/2.0.0.1
Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) (compatible; Yeti-Mobile/0.1; +http://help.naver.com/robots/)
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0_1 like Mac OS X; fr-fr) OrangeBot-Mobile AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8A306 Safari/(mobilesearch.support@orange-ftgroup.com)
mozilla/5.0 (larbin2.6.3@unspecified.mail)
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; ko; rv:1.9.1b2) Gecko/20081201 Firefox/3.1b2/Nutch-0.9 (It's a Mozilla based browser; http://www.mozilla.org/products/firefox/; no@mail.com)
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardBrowserProxy/0.0.5; +http://flipboard.com/browserproxy)
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; +http://flipboard.com/browserproxy)
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.8; flipboard.com/crawler rv:0.0.5) Gecko Firefox
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us; +http://stanford.edu) AppleWebKit/533.21.1 (KHTML, like Gecko) Version/5.0.5 Safari/533.21.1
Mozilla/5.0 (MetaCarta Labs GeoWebCrawler; +http://labs.metacarta.com/geowebcrawler)
Mozilla/5.0 (someone@somewhere.any)
Mozilla/5.0 (Thriceler-0.1 http://www.kuill.com/robots/spider.html)
Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0a2 +http://mozilla.org)
Mozilla/5.0 (Windows; N; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)/Nutch-1.0 (Crawler; lucene.apache.org/nutch/; a@b.net)
Mozilla/5.0 (Windows; N; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)/Nutch-1.1 (Crawler; lucene.apache.org/nutch/; a@b.net)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6/Nutch-1.1 (browser; http://www.g.cn; google@gmail.com)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0/1.0 (bot; http://)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.12) Gecko/2009070611 Firefox/3.0.12 (.NET CLR 3.5.30729) AutoPager/0.5.2.2 (http://www.teesoft.info/)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 AutoPager/0.5.2.2 (http://www.teesoft.info/)
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1 (http://www.voila.com/)
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.2; trendiction media ssppiiddeerr; http://www.trendiction.com/bot/; please let us know of any problems; ssppiiddeerr at trendiction.com) Gecko/20071127 Firefox/2.0.0.11
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1 GTB5 (.NET CLR 3.5.30729) AutoPager/0.5.2.2 (http://www.teesoft.info/)
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1) Gecko/20090624 Firefox/3.5 AutoPager/0.5.2.2 (http://www.teesoft.info/)
Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.112 Safari/535.1/Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.112 Safari/535.1 (Mozilla; Mozilla; mozilla@mozilla.org)
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.12) Gecko/2009070811 Ubuntu/9.04 (jaunty) Firefox/3.0.12/Nutch-1.0-dev (imcs; http://imcs.ro; admin@imcs.ro)
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19
Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:vertical-crawl-support@yahoo-inc.com)
Mozilla/5.0 (Yahoo-Test/4.0; mailto:vertical-crawl-support@yahoo-inc.com)
Mozilla/5.0 GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)
Mozilla/5.0 [en] (X11; U; Linux 2.2.15 i686 +http://www.yahoo.com/index.html)
Mozilla/5.0(compatible;MSIE/6.0+http://www.myspace.com)
Mozilla/5.0+(compatible;+CukBot;++like+Googlebot/2.1;+Not+a+dirty+rotten+spammer;+++http://www.companiesintheuk.co.uk/bot.html)
Mozilla/5.0+(compatible;+CukBot;+Not+a+spammer;+++http://www.companiesintheuk.co.uk/bot.html)
Mozilla/5.0-Firefox/3.0.15 (Firefox@mozilla.com)
Mozilla/5.0/heritrix/3.1.0 (compatible;; http://vancouver.cs.washington.edu:8080/)
Mozilla/5.0/Nutch-1.0-dev ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html ; agentname at setooz dot_com )
Mozilla/6.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
MozillaTest/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
Mozzila/5.0 (compatible; YamanaLab-Robot/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)
MPICrawler/0.1 +http://kermit.news.cs.nyu.edu/crawler.html
MR Crawler/Nutch-1.3
msbot (http://www.malaysiansearch.com/faq.php)
MSIE 5.0
MSIE 7.0/Nutch-1.0-dev
MSIndianWebcrawl
MSMOBOT/1.1 (+http://search.msn.com/msnbot.htm)
msnbot
msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
msnbot-NewsBlogs/1.1 (+http://search.msn.com/msnbot.htm)
msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)
msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
msnbot/1.1 (+http://search.msn.com/msnbot.htm)
msnbot/2.0b (+http://search.msn.com/msnbot.htm)
msnbot/2.0b (+http://search.msn.com/msnbot.htm).
msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
MSNPTC/1.0 (compatible; MSIE 6.0; Windows NT 5.2; MyIE2; .NET CLR 1.1.4322; .NET CLR 1.0.3705)
MSR-ISRCCrawler
MSRBOT
MSRBOT (http://research.microsoft.com/research/sv/msrbot/
MSU Libraries Search Engine/Nutch-0.9 (MSU Libraries Search Engine Description; none; webmaster at library dot msstate dot edu)
MT-Soft (http://www.mt-soft.com.ar)
multicrawler (+http://sw.deri.org/2006/04/multicrawler/robots.html)
MULTIPRISE/Nutch-1.0 (Robot d'indexation; http://www.multiprise.biz; admin@multiprise.fr)
my nutch agent/1.5
My Nutch Spider/Nutch-1.2
My Nutch Spider/Nutch-1.3
My Nutch Spider/Nutch-1.4
My Nutch Spider/Nutch-1.5-SNAPSHOT
my nutch/Nutch-1.4
My Spider/Nutch-1.4
my-robot/0.1
my-robot/0.2
mybot/Nutch-1.0 (mybot; http://mybot.com; mybot@mybot.com)
myCrawl/Nutch-1.3
myfirsttest/Nutch-1.1 (myfirsttest; myfirsttest.com; test@test.com)
MyNutchTest/Nutch-1.4-dev (Testing some algorithms)
MySearch/feiseek-0.09
MySearch/Nutch-1.0
MySpider/1.0
MyThesisCrawler/Nutch-1.3
myWebCrawler (Sharpsword@esearch.com)
N/A (farmer.tw@gmail.com)
naba/Nutch-1.3
naba/Nutch-1.4
NameOfAgent (CMS Spider)
NameSpider/1.0
NameSpier/1.0
nametest/test-version (http://google.com/robots.html)
naturaxalli/Nutch-1.2 (naturaxalli) (A bot for our semantic search engine, look at http://naturaxalli.com/sse; http://naturaxalli.com/sse; info at naturallaxi.com)
Naveen/Nutch-1.0 (Naveen; http://www.google.org; root@me)
netEstate FOAF crawler (+http://www.foaf-search.net/)
netEstate NE Crawler
netEstate NE Crawler (+http://www.sengine.info/)
NetinfoBot/1.1 (http://netinfo.bg/netinfobot.html)
NetSprint -- 3.0
NetSrcherP/0.2.3
NetStudyBot1.0
NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx)
NG/2.0
NgapaBot/Nutch-1.3 (NgapaBot is crawler for Indonesia Search Engine; http://www.ngapa.com; admin at ngapa dot com)
Nica Spider/Nutch-1.1
NicheBot (http://nb.bretlester.com/)
NIF/Nutch-1.0-dev (Neuroscience Information Framework Crawl; http://neuinfo.org/; support at neuinfo dot org)
nightsuns.nutch/Nutch-1.0
NISE/NISE-1.0 (NISE)
NISE/Nutch-2.0-dev
Nokia (larbin2.6.3@unspecified.mail)
Nokia3650/1.0 SymbianOS/6.1 Series60/1.2 Profile/MIDP-1.0 Configuration/CLDC-1.0/ (compatible; YodaoBot-Mobile/1.0; http://www.youdao.com/help/webmaster/spider/; )
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
NokiaN70/. FASTMobileCrawl/6.6 Profile/MIDP-2.0 Configuration/CLDC-1.1
nolosear/Nutch-1.2 (Nolosearch.com is searching the web for law related material to index on its server. This is built by a law student who is tired of not finding the law.; http://nolosearch.com/node6; eric at law brief case dot com)
nolosearch/nolo2.0 (law student search engine; nolosearch.com; eric@nolosearch.com)
NoNSTOPRobot/Nutch-1.3
noopsis Spider/Nutch-1.1 (noopsis crawler)
NoteworthyBot/0.1
nrsbot/5.0(loopip.com/robot.html)
nrsbot/6.0(loopip.com/robot.html)
NSE/Nutch-1.2
NSE/Nutch-1.3
NSE/Nutch-1.3 (saeed@mci.ir)
Nuhk/2.4 (+http://www.neti.ee/cgi-bin/abi/otsing.html)
Nutch 1.0/Nutch-1.0
Nutch 1.2/Nutch-1.2 (Facet Engine Nutch Crawler; spider@facetengine.com)
Nutch Crawler/Nutch-1.1
Nutch crawler/Nutch-1.2
Nutch Experimental Crawler/Nutch-1.4
nutch search engine 1.2/Nutch-1.2 (Nutch Search Engine)
Nutch Spider 1.1-dev/Nutch-1.0
Nutch Spider Evaluation/Nutch-1.4
Nutch Spider/Nutch-1.3
Nutch test crawler/Nutch-1.4-dev (Crawler testing; crawler/a/t/luminouslabs.com)
nutch test/Nutch-1.0 (nutch test)
Nutch, Leistungsschutz-Debatte Crawler/Nutch-1.4
Nutch-1.0/Nutch-1.0 (Nutch-1.0; http://lucene.apache.org/nutch/; admin@lucene.apache.org)
nutch-1.1/Nutch-1.1
nutch-1.2/Nutch-1.2
Nutch-1.2/Nutch-1.2 (http://lucene.apache.org/nutch/; admin@lucene.apache.org)
nutch-1.3/Nutch-1.3
nutch-1.4/Nutch-1.4
Nutch-1/Nutch-1.4
nutch-buibui/0.0.1/Nutch-1.0
nutch-crawl/Nutch-1.0-dev (imcs; http://imcs.ro; admin@imcs.ro)
nutch-crawler/Nutch-1.2
nutch-solr-integration-test/Nutch-1.1 (FreeCode Norway Web Crawler using Nutch 1.0; http://www.freecode.no/; mrskyhorse@163.com)
nutch-solr-integration-test/Nutch-1.2 (ADR Software Web Crawler using Nutch 1.2; http://www.adrsoftware.gr/; info@adrsoftware.gr)
nutch-solr-integration-test/Nutch-1.2 (FreeCode Norway Web Crawler using Nutch 1.0; http://www.bupt.edu.cn/; guorui15@sohu.com)
nutch-solr-integration-test/Nutch-1.2 (FreeCode Norway Web Crawler using Nutch 1.0; http://www.freecode.no/; reza.keshmir.90@gmail.com)
nutch-solr-integration-test/Nutch-1.2 (MoonValley Web Crawler using Nutch 1.2; http://www.moonvalley.com/; cwoolum@moonvalley.com)
nutch-solr-integration-test/Nutch-1.2 (Tagvillage Web Crawler using Nutch 1.0; http://www.tagvillage.com/; admin@tagvillage.com)
nutch-solr-integration/1.0
nutch-solr-integration/1.1
nutch-solr-integration/Nutch-1.0
nutch-solr-integration/Nutch-1.2
nutch-solr-integration/Nutch-2.0-dev
nutch-solr-integration/Nutch-2.0-dev (aeq123dec; www.aequor.com; gopal.dega@aequor.com)
nutch.biz/Nutch-1.0 (nutch.biz; crawler@nutch.biz)
nutch.us/Nutch-1.0 (nutch.us; crawler@nutch.us)
nutch.us/Nutch-1.0 (www.nutch.us; crawler@nutch.us)
Nutch/1.2/Nutch-1.2
Nutch/Nutch-0.8.1 (Nutch; http://www.nutch.es; email)
Nutch/Nutch-0.8.1 (Nutch; Nutch; Nutch)
Nutch/Nutch-0.9 (Nutch Search Engineer; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
Nutch/Nutch-1.0
Nutch/Nutch-1.0 (academic purpose; cats.kaist.ac.kr; smiler82@naver.com)
Nutch/Nutch-1.0 (academic purpose; cats2.kaist.ac.kr; smiler82@naver.com)
nutch/Nutch-1.0 (MyTest; 172.18.5.30; crawler@crawlzilla)
nutch/Nutch-1.0 (MyTest; set.your.ip.or.domain; your@email)
nutch/Nutch-1.0 (none; http://www.163.com; some@some.com)
Nutch/Nutch-1.0 (Nutch Search Engineer; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
nutch/Nutch-1.0 (nutch)
nutch/Nutch-1.0 (nutch; http://www.hp.com; nutch@apache.org)
Nutch/Nutch-1.0-dev (A Nutch-based crawler.; http://lucene.apache.org/nutch/bot.html; nutch-agent AT lucene.apache.org)
Nutch/Nutch-1.1 (Nutch Search Engineer; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
Nutch/Nutch-1.1-dev
Nutch/Nutch-1.2
nutch/Nutch-1.2 (MyTest; set.your.ip.or.domain; your@email)
Nutch/Nutch-1.2 (Nutch Search Engineer; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
Nutch/Nutch-1.3
Nutch/Nutch-1.4 (A semantic crawler for a PhD thesis; http://issel.ee.auth.gr/doku.php/research; kvavliak at issel dot ee dot auth dot gr)
Nutch/Nutch-1.4 (A semantic crawler for a PhD thesis; http://issel.ee.auth.gr; kvavliak at issel dot ee dot auth dot gr)
nutch/Nutch-1.5-SNAPSHOT (Viterbi Web Crawler using Nutch 1.0)
nutch1.1/Nutch-1.0
nutch1/huntsman1.0 (huntsman@mailguard.com.au)
NutchBot/Nutch-1.2 (Nutch Tester)
NUTCHCRAWLER/Nutch-1.0
NUTCHCRAWLER/Nutch-1.2
NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)
NutchCVS/0.7 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
NutchCVS/Nutch-1.0 (http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
NutchRobot/Nutch-1.3
NutchRobot/Nutch-2.0-dev
NutchTest/Nutch-1.0
NutchTestJb UB Heidelberg, Germany/Nutch-0.9 (b-a-r-t-h-at-u-b-punkt-u-n-i---h-e-i-d-e-l-b-e-r-g-punkt-d-e)
NutchTestRobot/Nutch-1.2
nutchwax.com/Nutch-1.0 (nutchwax.com; crawler@nutchwax.com)
nutchy/Nutch-1.0
Nutch_Crawler/Nutch-1.3
nutch_MR/Nutch-1.0 (nutch_MR; http://www.hp.com; nutch@apache.org)
nutch_solr/Nutch-1.2
nutch_test/Nutch-0.9 (nutch_test; http://www.8dorm.com; webmaster@8dorm.com)
Nutraspace/Nutch-1.2 (www.nutraspace.com)
nu_tch-princeton/Nu_tch-1.0-dev (princeton crawler for cass project; http://www.cs.princeton.edu/cass/; zhewang a_t cs ddot princeton dot edu)
nWormFeedFinder (http://www.nworm.com)
nyobot/1.1 (Noyb.com search engine crawler; http://www.noyb.com; bot@noyb.com)
NYU CS Nutch 1.0/Nutch-0.9 (NYU CS Nutch 1.0; http://lucene.apache.org/nutch/; webmaster at cs dot nyu dot edu)
OAPS-Document-Fetcher
oBot
Obvius external linkcheck/1.3
oco/Nutch-1.3 (oco crawler; oco.com; jgrey@oco.com)
OfelioBot
oh,fuck (larbin2.6.3@unspecified.mail)
OKArabia search engine/Nutch-1.4
OMGCrawler 1.0
OmniWeb
online-intel/Nutch-1.0
OOZBOT/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
OpenAcoon v4.1.0 (www.openacoon.de)
OpenAcoon v4.10.5 (www.openacoon.de)
OpenCalaisSemanticProxy
OpenIntelligenceData/1.1 (+http://www.worldwideweb-x.com/openData.html)
Openstat/0.1
OpenWebSpider (link collector; http://links.port30.se/cia.html) v (http://www.openwebspider.org/)
OpenWebSpider v0.1.2.B (http://www.openwebspider.org/)
OpenWebSpider v0.1.4 (http://www.openwebspider.org/)
Opera/6.05 (Windows 2000; U)
Opera/9.00 (Windows NT 5.1; U; zh-cn)
Opera/9.26 (Windows NT 5.1; U; zh-cn)
Opera/9.63 (Windows NT 5.1; U; en) Presto/2.1.1
Opera/9.64 (Windows NT 5.1; U; en) Presto/2.1.1
Opera/9.80
Opera/9.80 (Macintosh; Intel Mac OS X; U; en)
Opera/9.80 (Windows NT 5.1; U; en) Presto/2.5.22 Version/10.50
Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.2.15 Version/10.10
Opera/9.80 (Windows NT 6.1; U; cs) Presto/2.10.229 Version/11.61
Opera/9.80 (Windows NT 6.1; U; en-US) Presto/2.7.62 Version/11.01
opera_a (lostall@gmail.com)
oracle-cag-http-agent/Nutch-1.0 (Crawler to check site html; http://www.oracle.com; arist at oracle dot com)
OrangeCrawler/Nutch-1.0 (ldorange.crawler@orange-ftgroup.com)
OrbitSpiderBot/1
orgbybot/OrgbyBot v1.2 (Spidering for Orgby; http://www.orgby.com/ ; orgby.com search engine)
Orgbybot/OrgbyBot v1.2 (Spidering the net for Orgby; http://www.orgby.com/ ; Orgby.com Search Engine)
Orgbybot/OrgbyBot v1.3 (Spider; http://orgby.com/bot/ ; Orgby.com Search Engine)
ORISBot
ossse.com spider/Nutch-1.2 (ossse.com spider; http://ossse.com; ossse email address)
PadosAttilaCrawler/Nutch-1.0 (Ozi,PolandWiz,AustriaWiz,WiennaWiz crawlers, Attila Pados, attila.pados@gmail.com; www.ozi.hu, www.polandwiz.com,www.wiennawiz.com,www.austriawiz.com; attila dot pados at gmail dot com)
PagePeeker.com
PagePeeker.com (info: http://pagepeeker.com/robots)
page_test (larbin2.6.3@unspecified.mail)
page_verifier (http://www.securecomputing.com/goto/pv)
page_viewer (larbin2.6.3@unspecified.mail)
Paleoweb.org Friendly Bot ;-)/1
panelbot
Pangalactic Gargleblaster/Nutch-1.2 (Very intoxicating; http://circuitbomb.com; DigitalMail)
panscient.com
PArchiveCrawler/1.0 (PArchiveCrawler)
parsa/Nutch-1.1 (parsa; http://parsa.epfl.ch; stavros.volos@epfl.ch)
pavuk/0.9.35 linux-gnu
Pawler Beta - pawler(dot)net
Paxbot/1.0 (http://www.pax.com/paxbot.html)
PEAR HTTP_Request class ( http://pear.php.net/ )
Pentafocus.com pentafocusbot/1.0 (+http://www.pentafocus.com/pentafocusbot)
PenTest.sg/Nutch-1.1 (www.PenTest.sg; crawler@PenTest.sg)
percbotspider
percbotspider <ops@percolate.com>
Peter Wang/Nutch-0.9 (Nutch spiderman; http://peterpuwang.googlepages.com ; MyEmail)
Peter Wang/Nutch-1.0 (Nutch spiderman; http://peterpuwang.googlepages.com ; MyEmail)
Peter Wang/Nutch-1.0 (Nutch spiderman; local host ; saharelkazaz@hotmail.com)
Peter Wang/Nutch-1.2 (Nutch spiderman; http://peterpuwang.googlepages.com ; MyEmail)
pflab (cboc-test@lab.ntt.co.jp)
pflab (co2h2onacl@gmail.com)
Phex 2.02.8.2.92 Java/1.6.0_10
Phex 2.02.8.2.92 Java/1.6.0_26
PHP version tracker (http://www.nexen.net/phpversion/bot.php)
PHP/5.2.17p1
PHPCrawl
PicklBot 1.0
Pills.com.sg/Nutch-1.1 (Pills.com.sg; crawler@Pills.com.sg)
pinakilabs/Nutch-1.0
Pingthesemanticweb-robotvalidator/3.0
Pixray-Seeker/1.1 (Pixray-Seeker; http://www.pixray.com/pixraybot; crawler@pixray.com)
PizzaFace Crawler/Nutch-1.2
Pockey-GetHTML/4.11.6 (Win32; GUI; ix86)
Pockey-GetHTML/5.1.1 (Win32; GUI; ix86)
Ponyfish
PornoBug Crawler 0.8 * www.pornobug.com
PortalNG-Crawler/Nutch-1.4 (Crawler des Portals der Uni Ulm.; http://portal.uni-ulm.de; christian.aschoff at uni-ulm.de)
PostPost/1.0 (+http://postpo.st/crawlers)
PostPost/1.0 (+http://postpost.com/crawlers)
PR Crawler/Nutch-1.0 ( ; crawler@projectrialto.com)
PR Crawler/Nutch-1.0 (data mining develpment project; crawler@projectrialto.com)
PRBot/0.1 (+http://www.paulridgway.co.uk/prbot)
PRCrawler/Nutch-0.9 (data mining development project; crawler@projectrialto.com)
Primo Web Spider/Nutch-1.4
PrivacyFinder/1.1
private test crawler curl-based
Proto_Crawler/Nutch-0.9 (Prototype of a Crawler used in order to test Nutch's limits and performances.; --; lorack@gmail.com)
PRUEBA/Nutch-1.0
psbot/0.1 (+http://www.picsearch.com/bot.html)
PShop-Nutch/Nutch-1.1 (Academic test crawl; livne at umich)
PShop-Nutch/Nutch-1.1 (Academic test crawl; pshopcrawl at gmail)
PSS-Bot (+http://www.dcs.shef.ac.uk/~aca07pdr/pss/)
pt_viewer (larbin2.6.3@unspecified.mail)
pt_viewer_org (larbin2.6.3@unspecified.mail)
pt_viewer_page (larbin2.6.3@unspecified.mail)
Python-urllib/1.16
Python-urllib/1.17
Python-urllib/1.17 AppEngine-Google; (+http://code.google.com/appengine; appid lemmatica)
Python-urllib/2.1
Python-urllib/2.4
Python-urllib/2.5
Python-urllib/2.6
Python-urllib/2.7
QCF Testname/Nutch-0.9 (QCF Test description; QCF Test url; QCF Test email )
Qirina Hurdler v. 1.05 10.11.01 (+http://www.qirina.com/hurdler.html)
QleeQ1/Nutch-1.4
QrobotW
Qryos
Qseero v1.0.0
quickobot/quickobot-1 (Quicko Labs; http://quicko.co; robot at quicko dot co)
R6_CommentReader(www.radian6.com/crawler)
R6_FeedFetcher(www.radian6.com/crawler)
R6_Primer(www.radian6.com/crawler)
RADaR-Bot/Nutch-1.4 (http://radar-bot.com/)
radian6_default_(www.radian6.com/crawler)
random house/Nutch-1.0 (Lucene Random House Crawler; http://www.randomhouse.com/; somewhere@neverexisted.com)
random house/Nutch-1.0 (Lucene Random House Crawler; http://www.randomhouse.com/; unixadmin@randomhouse.com)
RankurBot/Rankur2.1 (http://rankur.com; info at rankur dot com)
Raymond Balm s/Biible 1.0 (Biible; http://www.biible.info ; raymond.balmes@gmail.com)
Raymond Balm s/Bible-Ref: (spiderman; http://www.balmes.com ; raymond.balmes@gmail.com)
Raymond Balm s/Biible 1.0 (Biible; http://www.bibleref.com ; raymond.balmes@gmail.com)
Raymond Balm s/Biible 1.0 (Biible; http://www.biible.info ; raymond.balmes@gmail.com)
Raymond Balm s/Biible 1.0 (spiderman; http://www.bibleref.com ; raymond.balmes@gmail.com)
Raymond Balm s/Nutch-1.0 (spiderman; http://www.balmes.com ; raymond.balmes@gmail.com)
Raymond Balm s/Nutch-1.0 (spiderman; http://www.bibleref.com ; raymond.balmes@gmail.com)
RDFaS-Bot/1.0 (+http://www.rdfas.com/bot)
rdfbot/1.0 ( Indian Language Web Search Engine ; Rediff.com ; rdfbotsupport AT rediffmailpro DOT com )
rdfbot/1.0 (Indian Language Web Search Engine; Rediff.com; rdfbotsupport AT rediffmailpro DOT com)
rdfbot/1.0 (rdfbotsupport AT rediffmailpro DOT com)
research spider[AT]mikelab.net
research-scan-bot/Nutch-1.0
rganalytics
RhubarbPIe/Nutch-1.3
RiceComputerArchitecture Crawler/Nutch-1.0
rincewind/rincewind-0.1 (http://labs.ischroedi.de)
RISSBot/dev (http://www.tkolb.de/rissbot.txt)
RiverglassScanner
rmatie.com spider/Nutch-1.4
rmatie.com/Nutch-1.4
RoadRunner/Nutch-1.0 (webmaster@fieldtech.org)
robis-webmasterportal-Bot (Version: 1.06, powered by www.robis-webmasterportal.de +http://www.robis-webmasterportal.de/bot.html)
roboo/Nutch-1.0 (roboo; http://wap.roboo.com; winter.pi@roboo.com)
roboobot/1.0 (roboo; http://wap.roboo.com; winter.pi@roboo.com)
roboobot/Nutch-1.0 (roboobot; http://wap.roboo.com; winter.pi@roboo.com)
robotgenius (http://robotgenius.net)
Robots Bot
Ronbot/Nutch-1.2
Ronzoobot/1.4 (http://www.ronzoo.com/about.php)
Ronzoobot/1.4 (http://www.ronzoo.com/about/)
Ronzoobot/1.5 (http://www.ronzoo.com/about/)
RSpider/1.0
ruijie (larbin@unspecified.mail)
Ruky-Roboter (Version: 1.06, powered by www.ruky.de +http://www.ruky.de/zeige/bot.html)
ru_org_net_viewer (larbin2.6.3@unspecified.mail)
ru_org_viewer (larbin2.6.3@unspecified.mail)
ru_prgnet_viewer (larbin2.6.3@unspecified.mail)
rwack/Nutch-1.0 (learning about distributed computing; http://rdsd.com; jhires at gmail dot com)
rwack/Nutch-1.0 (learning about distributed computing; http://rwack.com; jhires at gmail dot com)
S2-Crawler/1.1 (your search engine; http://www.search2.net/; info at search2 dot net)
S2Bot/1.0 (http://search2.net)
S2Bot/1.0 (http://search2.net; bot[at]search2[dot]net)
S2Robot
sadsa/Nutch-1.4
SaladSpoon/ShopSalad 1.0 (Search Engine crawler for ShopSalad.com; http://shopsalad.com/en/partners.html; crawler AT shopsalad.com)
SaladSpoon/ShopSalad 1.0 (Search Engine crawler for ShopSalad.com; http://shopsalad.com/en/partners.html; crawler@shopsalad.com)
SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
SAP-KM/WebRepository 1.2
SBIder/Nutch-1.0-dev (http://www.sitesell.com/sbider.html)
SBIder/SBIder-1.0 (http://www.sitesell.com/sbider.html)
SBIRobobot/0.9 (Rev.Nagayama1; http://news.sbibusiness.com/)
Schloerbot/Nutch-1.0 (Schloer consulting bot; http://schloerconsulting.com/schloerbot)
ScholarScope/Nutch-1.2 (Scholar Research Engine)
ScholarScope/Nutch-1.5 (Scholar Research Engine)
SciCrawl/Nutch-1.1
ScooperBot www.customscoop.com
Scooter/3.3
Screaming Frog SEO Spider/1,30
Screaming Frog SEO Spider/1,40
Screaming Frog SEO Spider/1,60
Screaming Frog SEO Spider/1,70
Screaming Frog SEO Spider/1,80
Screaming Frog SEO Spider/1.30
Screaming Frog SEO Spider/1.40
Screaming Frog SEO Spider/1.80
Screaming Frog SEO Spider/1.90
SD Crawler/Nutch-0.9 (automated Crawler)
search.ch V1.4.2 (spiderman@search.ch; http://www.search.ch)
search.KumKie.com
SearchBot
SearchEngine/Nutch-1.4
SearchEngineVerificationCrawler/Nutch-1.0 (The purpose of this crawling is to collect web pages for verifying search engines.; http://www.yama.info.waseda.ac.jp/~takuya/en/aboutSRVC.html; srvc at yama dot info dot waseda dot ac dot jp)
searcher/Nutch-1.1
Searchmee! Spider v0.98a
Seegul/Nutch-1.4
SeekBot/1.0 RobotsTxtFetcher/1.3
Seengine/1.0/Nutch-1.3-dev
Seengine/1.0/Nutch-2.0-dev
Semager/1.3 http://www.semager.de/blog/semager-bots/
Semantifire/0.20 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html ; agentname at setooz dot_com )
SemrushBot/0.9
SemrushBot/0.91
SemrushBot/0.92
SemrushBot/Nutch-1.5-SNAPSHOT
SemtoBot/0.1
SentinelCrawler 0.2/Nutch-1.0-dev
SeoBot
SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm)
service-public-indexeur
Setooz/Nutch-1.0 (http://www.setooz.com)
setoozbot/0.30 ( compatible; SETOOZBOT/0.30 ; setooz.com ; bot AT setooz DOT com )
SETOOZBOT/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html )
SexSearchBot (+http://gou-sex.com/)
SexShop.Sg/Nutch-1.1 (www.SexShop.Sg; crawler@SexShop.Sg)
SexShop.sg/Nutch-1.1 (www.SexShop.sg; crawler@www.sexshop.sg)
SexyBoom/Nutch-1.2 (SexyBoom; http://www.sexyboom.it/; service@sexyboom.it)
SexyBoom/Wauuu 0.1 (AnnunciPazzi; http://www.sexyboom.it/; service@sexyboom.it)
SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/)
SeznamBot/2.0 (+http://fulltext.seznam.cz/)
SeznamBot/3.0 (+http://fulltext.sblog.cz/)
SeznamBot/3.0-alpha (+http://fulltext.sblog.cz/)
SeznamBot/3.0-beta (+http://fulltext.sblog.cz/)
SeznamBot/3.0-test (+http://fulltext.sblog.cz/)
sGroup crawler 1/Nutch-1.3
Shared Content Fetcher/Nutch-0.9 (http://www.sharethis.com; gordon@sharethis.com)
SHC/Nutch-1.0 (SemanticHacker Crawler; http://www.semantichacker.com/crawler-info; abuse@semantichacker.com)
SheenBot/SheenBot-1.0.1 (Sheen web crawler)
SheenBot/SheenBot-1.0.2 (Sheen web crawler)
SheenBot/SheenBot-1.0.3 (Sheen web crawler)
SheenBot/SheenBot-1.0.4 (Sheen web crawler)
Shelob (shelob@gmx.net)
ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)
sihuifeng/Nutch-1.2
silversurfer.cc
SimilarPages/Nutch-1.0-dev (SimilarPages Nutch Crawler; http://www.similarpages.com; info@similarpages.com)
sindice.net sitemap fetcher
SindiceBot (heritrix/2.0.2 +http://sindice.com/developers/bot)
Sirketcebot/v1.0 (http://www.sirketce.com/bot.php)
SiteGuardBot (support@@siteguard.com)
SiteGuardBot (support@siteguard.com)
SiteSucker/1.6.9
SiteSucker/2.2.2
SiteSucker/2.2.2 CFNetwork/454.11.5 Darwin/10.6.0 (i386) (iMac7%2C1)
SiteSucker/2.2.3
SiteSucker/2.2.4
SiteSucker/2.3
SiteSucker/2.3.1
SiteSucker/2.3.2
SiteSucker/2.3.3
SiteSucker/2.3.4
SiteSucker/2.3.4 CFNetwork/454.11.12 Darwin/10.7.0 (i386) (MacBookPro5%2C5)
SiteSucker/2.3.5
SiteSucker/2.3.6
SiteSucker2.2.3 CFNetwork/438.12 Darwin/9.7.0 (i386) (MacBookPro5%2C1)
sixtoCrawler/Nutch-0.9 (sixtoCrawler; www.sixto.org; info.sequic@gmail.com)
SkimBot/1.0 (www.skimlinks.com <dev@skimlinks.com>)
SkimBot/1.0 (www.skimlinks.com)
SkyGrid/1.0 (+http://skygrid.com/partners)
SlimBot (SlimBot; http://www.slimbot.com; info@slimbot.com)
SlugBug Spider/0.1 beta (SlugBug.com search engine; http://www.slugbug.com)
SmallBlue/Synergy-2.0; Nutch-1.0-rc1
Smarte Bot v1.0
smartview (larbin2.6.3@unspecified.mail)
SMXEchoCrawl/Nutch-0.9 (Social Metrix Bot; http://www.socialmetrix.com/; bot@socialmetrix.com)
Snapbot/1.0 (Snap Shots, +http://www.snap.com)
SnapOn Long-Crawl/Nutch-0.9 (CCPV Crawler; http://thecrawlerproject.com; aherres@gmail.com)
Snoopy v1.2.4
SNSpider_0.0.1 (snspider@salatti.net)
socbot (search@socbay.com)
SocSciBot
SoftNet Nutch Spider/Nutch-1.4
Sogou Video /3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98)
soni/Nutch-0.9 (RnD; www.cdac.in; asoni@cdac.in)
Soonxun/1.0 (web spider; )
Sosospider+(+http://help.soso.com/webspider.htm)
sou300-spider/1.0 (www.sou300.com)
sou300-spider/Nutch-1.0
sou300-spider/sou300 (www.sou300.com)
sou300/1.0 (www.sou300.com)
sou300/1.1 (www.sou300.com)
Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
Sphider
sphsearch.org/Nutch-1.0 (sphsearch.org; crawler@sphsearch.org)
Spider
Spider (Java EE; Windows NT; +http://www.daniel-nowak-ug.de) Resource Description Engine
Spider/5.0
Spider/Nutch-1.0
spider/Nutch-1.1 (Nutch; http://lucene.apache.org/nutch/; nutch-agent@lucene.apache.org)
spider/Nutch-1.3 (spider; http://www.xxx.com)
spiderbot119
SpiderMan/1.0
Spiprabot
SpokeSpider/1.0 (http://support.spoke.com/webspider/) Mozilla/5.0 (not really)
spotinfluence/Nutch-1.2 (Spot Influence crawler; http://spotinfluence.com; hello at spotinfluence dot com)
spotinfluence/Nutch-1.4 (Spot Influence crawler; http://spotinfluence.com; hello at spotinfluence dot com)
Squid Prefetch 1.0
ss (meng1985636@163.com)
SS/Nutch-1.5 (SS)
ssearch_bot (sSearch Crawler; http://www.semantissimo.de)
ssearch_bot/Nutch-1.0 (SSearch Crawler; http://www.semantissimo.de)
SSSE/0.08
StackRambler/2.0 (MSIE incompatible)
StatoolsBot (+http://www.statools.com/bot.html)
stc-crawler/0.84
suchclip
Super graphical search spider/Nutch-1.3
SuperPagesBot/0.1
Suren/Nutch-1.0
SurveyBot/2.3 (Whois Source)
Swami1/Swami-1.1 (Swami Vertical Site Fetcher; devarajaswami at yahoo dot com)
sxit nutch agent/1.0
SygolBot http://www.sygol.com
SymbolHound/Nutch-1.3 (Please contact us if we're being impolite!; www.symbolhound.com; info@symbolhound.com)
Szukacz/1.5 (robot; www.szukacz.pl/html/jak_dziala_robot.html; info@szukacz.pl)
Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl)
T-Mobile Dash Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240; MSNBOT-MOBILE/1.1; +http://search.msn.com/msnbot.htm)
T-Mobile Dash Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240;) MSNBOT-MOBILE/1.0 (+http://search.msn.com/msnbot.htm)
T-Mobile Dash Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240;) MSNBOT-MOBILE/1.1 (+http://search.msn.com/msnbot.htm)
t3hb0t/Nutch-1.0 (ab0t; http://nosite.com; notreal@fake.org)
tags2dir.com/0.8 (+http://tags2dir.com/directory/)
Tagvillage Crawler/Nutch-1.2
TailsweepBlogCrawler/Tailsweep-2.11-SNAPSHOT (http://www.tailsweep.com/; bot at [tailsweep] dot com)
TailsweepBlogCrawler/Tailsweep-2.6-SNAPSHOT (http://www.tailsweep.com/; bot at [tailsweep] dot com)
TailsweepBlogCrawler/Tailsweep-2.9-SNAPSHOT (http://www.tailsweep.com/; bot at [tailsweep] dot com)
tangmiSpider/Nutch-1.1 (search; http://hi.baidu.com/ebyy; dl_tangmi@163.com)
TCG/Nutch-1.0 (TCGspider; http://tcg.com ; TCGEmail)
tea (whz_test_send@163.com)
TeezirAgent/1.0 (Windows; testversie)
tehr0b0t/Nutch-0.9
temaseek.com/Nutch-1.0 (temaseek.com; crawler@temaseek.com)
TemaSEEK/Nutch-1.1 (http://www.temaseek.com; crawler@temaseek.com)
Teoma/1.0
Teoma/Mozilla/5.0 (Ask Jeeves Question and Answer Search; http://about.ask.com/en/docs/about/webmasters.shtml; dr.edison@ask.com)
Teoma/Mozilla/5.0 (Ask Jeeves Question and Answer Search; http://www.ask.com/about/help/webmasters; dr.edison@ask.com)
Teoma/Nutch-1.2 ( Question and Answer Search; Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html); bot@afarm.com)
Teragram/SAS Crawler
Test-Fetcher-0.1/Nutch-0.9 (Awesomeness)
test-name/test-version (test-url)
Test.Buzzz/Nutch-0.8.1 (Test.Buzz; http://test.com; test@test.com)
TEST/hoqBot-1.0 (dummy; http://www.dummy.com; info at dummy dot com)
test/Nutch-0.9 (fortest; http://www.moojoy.com; fuxiang@yingzhitech.com)
Test/Nutch-1.0 (Nutch spiderman; http://www.cosmoage.com ; MyEmail)
test/Nutch-1.0 (test; test; test)
test/Nutch-1.2 (test; www.163.com; xxx@126.com)
testbaidu/Nutch-1.2
TestBot/Nutch-1.1
TestNutch/Nutch-1.2 (testing Nutch; http://nutch.apache.org; info@nutch.apache.org)
The Crawler/Nutch-1.3
TheaSearch Crawler/Nutch-1.4
theasearch.com spider/Nutch-1.4
theasearchCrawler/Nutch-1.4 (http://www.theasearch.com/; hansod1@gmail.com)
TheRarestParser/0.4b (http://rarestblog.com/therarestparser/)
ThumbShots-Bot (+http://thumbshots.in/bot.html)
thumbshots-de-bot (+http://www.thumbshots.de)
thumbshots-de-bot (+http://www.thumbshots.de/)
TigrinyaCrawler(tirawler@gmail.com)
TimKimSearch/2.1
TinEye/1.1 (http://tineye.com/crawler.html)
Tipiweb/Nutch-1.0 (http://www.tipiweb.net)
to-night-Bot (Version: 1.06, powered by www.to-night.de +http://www.to-night.de/pfadzurbotseite/bot.html)
Tobot/Nutch-1.4 ('tobot at eel dot rdc dot toshiba dot co dot jp')
TomekSamcik/Nutch-1.0 (http://tomek.samcik.net; tomek@mail.samcik.net)
toofaan/Nutch-1.0 (http://www.toofaan.com)
Toplistbot
topyx-crawler
Tours/Nutch-1.2
TRAAAVELPIG/Nutch-1.3
tradove.com/1.0
Trends Crawler, Real time trends bot (info@trendscrawler.com)
TrialSpider/Nutch-1.1
tricologia medicina/Nutch-1.0 (tricologia medica; http://www.studioditricologia.it; agatapollicino@live.it)
trovi/1.10 (+http://www.trovi.be)
trovi/1.11 (+http://www.trovi.be)
TrueKnowledgeBot/Nutch-1.0 (A web crawler to extract facts from pages for the True Knowledge knowledge base; http://www.trueknowledge.com; miners -AT- trueknowledge _dot_ com)
TruNorth/Nutch-1.4
TRY ENGINE/Nutch-1.0
TsinghuaLab
TSWebLiveBot/Bot-1.0 (Crawling the Web; http://www.tsweblive.com; admin@tsweblive.com)
tudelft-research/Nutch-1.0
tukangcari
TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html)
TwengaBot-Discover (http://www.twenga.fr/bot-discover.html)
TwengaBot/2.0 (http://www.twenga.com/bot.html)
TwengaBot/2.0 text/*
Twib::Crawler
Twib::Crawler/0.02
Twib::Crawler/0.03
Twitterbot v1.0
Twitterbot/1.0
t_crawler/0.4
ua-bot/1.26 http://www.ua-bot.com/
UberaSiteExplorer/UberaSiteExplorer-1.0
uk_parliament (+http://www.parliament.uk;archives@parliament.uk)
unister-bot (Mozilla/5.0 compatible; crawler@unister-gmbh.de)
UnisterBot (Mozilla/5.0 compatible; crawler@unister-gmbh.de)
university/Nutch-1.0 (research)
UniversityofEdinburgh/Nutch-1.0-dev (Academic; http://www.ed.ac.uk; bhaddow at inf point ed point ac punto uk)
unknownbot (abc@mail.com)
uocCrawler/Nutch-0.9 (uocCrawler; www.uoc.org; info.uoc@gmail.com)
Updownerbot (+http://www.updowner.com/bot)
upictoBot
UrlFetcher/1.0
UrlsListDownloader (larbin2.6.3@unspecified.mail)
url_test (larbin2.6.3@unspecified.mail)
User-Agent: BoardReader-Image-Fetcher /1.0 info@boardreader.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.8) Gecko/2009032609 Firefox/3.0.8 (.NET CLR 3.5.30729)
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)
UserAgent
us_orgedu_viewer (larbin2.6.3@unspecified.mail)
UTPA-GSA-CRAWLER (Enterprise; S5-E9C9SWHTV8JJB; syshelp@utpa.edu,webhelp@utpa.edu)
Utsz.NET Spider/Nutch-1.3
uw_cse_xwc/Nutch-0.9 (University of Washington Computer Science extensible web crawler project
Validator/1
VerbstarBot/1
vik-robot/Nutch-1.0 (vikspider; http://vik.com; chenlibiti@163.com)
VoluniaNutch/Nutch-1.2 (testing VoluniaNutch; http://www.volunia.com; info@volunia.com)
volverine/Nutch-0.9 (agentspider; beast@mail.com)
Voracious/0.1
votre-regard/0.1
voyager/2.0 (http://www.kosmix.com/crawler.html)
VSE/1.0 (fabio.foroni@kisiwa.it)
VSE/1.0 (freddy.parra@legis.com.co)
VSE/1.0 (mann@vivisimo.com)
VSE/1.0 (myunn@vivisimo.com)
VSE/1.0 (notificacionservicios@iconomultimedia.com)
VSE/1.0 (nunzio.d.hayslip@lmco.com)
VSE/1.0 (pagano@vivisimo.com)
VSE/1.0 (turnshek@vivisimo.com)
VSE/1.0 (wall@vivisimo.com)
W3C-checklink/4.5 [4.154] libwww-perl/5.805
W3C-checklink/4.5 [4.154] libwww-perl/5.823
W3C-checklink/4.5 [4.160] libwww-perl/5.823
W3C-checklink/4.6 [4.194] libwww-perl/5.805
W3C-checklink/4.6 [4.194] libwww-perl/5.823
W3C-checklink/4.6 [4.194] libwww-perl/5.836
W3C-checklink/4.7 libwww-perl/5.836
W3C-checklink/4.7 libwww-perl/6.01
W3C-checklink/4.8 libwww-perl/5.836
W3C-checklink/4.8 libwww-perl/5.837
W3C-checklink/4.8 libwww-perl/6.02
W3C-checklink/4.81 libwww-perl/5.836
WaC crawler/Nutch-1.2 (Collecting linguistic data from the Web; http://www.sigwac.org.uk/)
Wada.vn Vietnamese Search/2.1
WAP_Browser/5.0 (compatible; YodaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
waue/Nutch-1.2 (MyTest; gm1.nchc.org.tw; waue@nchc.org.tw)
WBP Ninja Spider/Nutch-1.2 (http://www.webuildpages.com)
WDG_SiteValidator/1.6.4
Web Spider/Nutch-0.9
webcrawler (compatible; heritrix/1.14.3 ++http://www.onb.ac.at/about/webarchivierung.htm)
webcrawler (compatible; heritrix/1.14.4 ++http://www.onb.ac.at/about/webarchivierung.htm)
Webcrawler/1.0 (web spider; )
Webcrawler/Nutch-1.0 (Test crawl; lucene.apache.org/nutch/; a@b.net)
WebCrawler/Nutch-1.2 (WebCrawler; WebCrawler)
WebImages 0.3 ( http://herbert.groot.jebbink.nl/?app=WebImages )
WebIndex (webindex.csv.lt)
webmeasurement-bot, http://rvs.informatik.uni-leipzig.de/bot.php
webmoney.advisor.bot/Nutch-1.0
webmoney.advisor/Nutch-1.0
WebMonitorRobots/Nutch-1.0 (Nutch_spiderman; http://www.csdl.ac.cn; zhang_zhixiong2003@yahoo.com)
WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/)
WebReaper v10.0 - www.webreaper.net
WebReaper v9.8 - www.webreaper.net
WebReaper vWebReaper v7.3 - www,otway.com/webreaper
Webspider/1.0 (web spider; )
WebSurfer text/*
web_bq (larbin2.6.3@unspecified.mail)
WeKnow Spider/Nutch-1.0
welinktech/Nutch-1.3
weneobot/Nutch-1.2 (weneo.de bot; http://www.weneo.de/bot; bot at weneo.de)
WeSEE:Search/0.1 (Alpha, http://www.wesee.com/)
WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)
WeViKa
WEVONZO
wget-1.0 (larbin2.6.3@unspecified.mail)
Wget/0.1 (linux-gnu)
Wget/0.1.3
Wget/1.10
Wget/1.10.1
Wget/1.10.2
Wget/1.10.2 (Red Hat modified)
Wget/1.11.1
Wget/1.11.2
Wget/1.11.3
Wget/1.11.4
Wget/1.11.4 (Red Hat modified)
Wget/1.11.4 Red Hat modified
Wget/1.12 (darwin10.6.0)
Wget/1.12 (freebsd8.1)
Wget/1.12 (linux-gnu)
Wget/1.13.1 (linux-gnu)
Wget/1.13.4 (darwin10.8.0)
Wget/1.13.4 (darwin11.1.0)
Wget/1.13.4 (darwin11.2.0)
Wget/1.13.4 (darwin11.3.0)
Wget/1.13.4 (linux-gnu)
Wget/1.5.3
Wget/1.5.3.1
Wget/1.8.2
Wget/1.9
Wget/1.9+cvs-stable (Red Hat modified)
Wget/1.9.1
whiteiexpres/Nutch-0.9 (whiteiWebBot; whiteiexpress.com; whiteye@kon-x.com)
WhoYougle
wiiy/Nutch-1.0
wiiy/Nutch-1.1 (Nutch Search Engineer; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
wikiwix-bot-3.0
WindowPane1.0
windows xp
wirkon.com (Version: 1.06, powered by www.wirkon.com+http://www.wirkon.com)
WisdomCrawler/1.0 (A crawler from Lab of Wisdom, School of Software, Dalian University of Technology, for testing purpose; http://wisdom.dlut.edu.cn/newsite/Home.aspx; liangliang0007 at 126 dot com)
WMCAI-robot (http://www.topicmaster.jp/wmcai/crawler.html)
woobot/1.1
Woovi/Nutch-1.0 (http://www.woovi.com/bot)
WordPress/3.4-beta1;
WordPress/3.4-beta2-20452;
WoS
Wotbox/2.0 (bot@wotbox.com; http://www.wotbox.com)
Wouah/Nutch-1.0 (http://www.wouah.fr)
wpss_test.pl
wsrwebometer/1.0 (wsrwebometer crawler; http://www.webscienceresearch.org/crawlerInfo.html; crawladmin at webscienceresearch dot com)
wttsearch/Nutch-1.0 (Crawler for SIFE (WTT); http://wtt.beecom.ch/wttsearch.html; wtt@beecom.ch)
ww7bot
www (wang_cyi@163.com)
www.bemash.net/Nutch-1.3 (bemash; www.bemash.net; info@bemash.net)
wwwster/1.4 (Beta, mailto:gue@cis.uni-muenchen.de)
WyoBot 1.0
XP (XP)
xp/Nutch-1.0 (Nutch spiderman; *; MyEmail)
xunuu/Nutch-1.0
Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html )
Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html)
Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0)
Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/blog-search/)
Y!J-PSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Y!J-SRD/1.0
yacy (www.yacy.net; amd64 FreeBSD 7.1-PRERELEASE; java 1.6.0_07; Europe/en) Jakarta Commons-HttpClient/3.1
yacy (www.yacy.net; amd64 FreeBSD 7.1-STABLE; java 1.6.0_07; Europe/en) Jakarta Commons-HttpClient/3.1
yacy (www.yacy.net; amd64 Linux 2.6.24-23-generic; java 1.6.0_07; Europe/de) Jakarta Commons-HttpClient/3.1
yacy (www.yacy.net; amd64 Linux 2.6.24-24-generic; java 1.6.0_07; Europe/en) Jakarta Commons-HttpClient/3.1
yacy (www.yacy.net; amd64 Linux 2.6.28-11-generic; java 1.6.0_0; Europe/de) Jakarta Commons-HttpClient/3.1
yacy (www.yacy.net; amd64 Linux 2.6.28-11-generic; java 1.6.0_0; Europe/en) Jakarta Commons-HttpClient/3.1
yacy (www.yacy.net; amd64 Linux 2.6.28-11-generic; java 1.6.0_13; Europe/en) Jakarta Commons-HttpClient/3.1
yacy (www.yacy.net; amd64 Linux 2.6.28-12-generic; java 1.6.0_0; Europe/de) Jakarta Commons-HttpClient/3.1
yacy (www.yacy.net; amd64 Windows 7 6.1; java 1.6.0_14; Europe/de) Jakarta Commons-HttpClient/3.1
yacy (www.yacy.net; amd64 Windows Vista 6.1; java 1.6.0_13; Europe/de) Jakarta Commons-HttpClient/3.1
yacy (www.yacy.net; i386 Linux 2.6.18-128.1.10.el5xen; java 1.6.0_07; Europe/de) Jakarta Commons-HttpClient/3.1
yacy (www.yacy.net; i386 Linux 2.6.18-6-686; java 1.6.0_10; Europe/de) Jakarta Commons-HttpClient/3.1
yacy (www.yacy.net; i386 Linux 2.6.23; java 1.6.0_06; Europe/en) Jakarta Commons-HttpClient/3.1
yacy (www.yacy.net; i386 Linux 2.6.24-etchnhalf.1-686; java 1.5.0_14; GMT+01:00/de) Jakarta Commons-HttpClient/3.1
yacy (www.yacy.net; i386 Linux 2.6.28-11-generic; java 1.6.0_0; Europe/en) Jakarta Commons-HttpClient/3.1
yacy (www.yacy.net; x86 Windows XP 5.1; java 1.6.0_05; Europe/de) Jakarta Commons-HttpClient/3.1
yacy (www.yacy.net; x86 Windows XP 5.1; java 1.6.0_13; Europe/de) Jakarta Commons-HttpClient/3.1
yacy/0.5910493 (www.yacy.net; x86 Windows Vista 6.0; java 1.6.0_11; Europe/de) Jakarta Commons-HttpClient/3.1
yacybot (allip/any; x86_64 Mac OS X 10.6.7; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
yacybot (amd64 FreeBSD 7.1-PRERELEASE; java 1.6.0_07; Europe/en) http://yacy.net/bot.html
yacybot (amd64 FreeBSD 7.1-STABLE; java 1.6.0_07; Europe/en) http://yacy.net/bot.html
yacybot (amd64 FreeBSD 8.0-RELEASE-p2; java 1.6.0_07; Europe/en) http://yacy.net/bot.html
yacybot (amd64 FreeBSD 8.0-RELEASE-p3; java 1.6.0_07; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.16.46-0.12-smp; java 1.6.0_15; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-128.2.1.el5; java 1.6.0_14; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-128.4.1.el5; java 1.6.0_14; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-128.7.1.el5; java 1.6.0; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-164.11.1.el5; java 1.6.0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-164.15.1.el5; java 1.6.0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-164.15.1.el5xen; java 1.6.0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-164.6.1.el5; java 1.6.0; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-164.6.1.el5; java 1.6.0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-164.6.1.el5; java 1.6.0_16; Asia/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-164.9.1.el5xen; java 1.6.0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-164.el5; java 1.6.0; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-164.el5; java 1.6.0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-194.17.1.el5xen; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-194.3.1.el5.028stab069.6; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-194.3.1.el5; java 1.6.0_0; GB-Eire/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-194.3.1.el5; java 1.6.0_14; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-194.8.1.el5; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-53.el5; java 1.6.0; America/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.18-6-amd64; java 1.5.0_14; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.22-14-server; java 1.6.0_06; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.23.17-dbserv; java 1.6.0_04; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.24-16-server; java 1.6.0_07; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.24-23-generic; java 1.6.0_07; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.24-23-generic; java 1.6.0_07; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.24-23-generic; java 1.6.0_07; GMT/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.24-24-generic; java 1.6.0_07; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.24-24-generic; java 1.6.0_07; GMT/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.24-7-pve; java 1.6.0_13; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.24-etchnhalf.1-amd64; java 1.5.0_14; UTC/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.24-etchnhalf.1-amd64; java 1.6.0_07; UTC/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.24-etchnhalf.1-amd64; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.26-1-amd64; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.26-2-amd64; java 1.5.0_17; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_0; Europe/fr) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_12; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.26-2-xen-amd64; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.26-2-xen-amd64; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.26-2-xen-amd64; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.26-2-xen-amd64; java 1.6.0_21; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.27-11-generic; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.27-11-generic; java 1.6.0_10; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.27-11-server; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.27-14-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.27.25-170.2.72.fc10.x86_64; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.27.39-0.2-default; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-11-generic; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-11-generic; java 1.6.0_13; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-12-generic; java 1.6.0_13; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-12-generic; java 1.6.0_13; GMT/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-14-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-14-generic; java 1.6.0_14; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-15-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-15-generic; java 1.6.0_14; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-15-generic; java 1.6.0_14; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-15-generic; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-16-generic; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-16-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-16-generic; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-16-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-17-generic; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-17-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-17-generic; java 1.6.0_16; GMT/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-18-generic; java 1.6.0_19; GMT/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-18-generic; java 1.6.0_20; GMT/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28-hardened-r7; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28.3-vs2.3.0.36.4; java 1.6.0_12; Universal/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.28.3-vs2.3.0.36.4; java 1.6.0_20; Universal/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.29-ARCH; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.30-1-amd64; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.30.1; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.30.5-43.fc11.x86_64; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-11-rt; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-14-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-14-generic; java 1.6.0_15; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-15-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-15-generic; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-16-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-16-server; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-17-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-17-server; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-19-generic; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-19-server; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-20-generic; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-20-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-20-generic; java 1.6.0_0; US/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-20-generic; java 1.6.0_15; Atlantic/es) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-20-generic; java 1.6.0_15; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-20-generic; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-20-server; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-20-server; java 1.6.0_15; US/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-21-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-21-server; java 1.6.0_0; Asia/ja) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-21-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-22-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-22-server; java 1.6.0_18; Asia/ja) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-22-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-gentoo-r6; java 1.6.0_0; Europe/fr) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-gentoo-r6; java 1.6.0_17; Etc/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31-gentoo-r6; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31.12-0.1-desktop; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31.4-desktop-0.rc2.1mnb; java 1.6.0_0; Europe/fr) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.31.5; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32-18-server; java 1.6.0_18; America/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32-21-generic; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32-22-generic; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32-22-generic; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32-22-generic; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32-22-generic; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32-23-generic; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32-24-generic; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32-24-generic; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32-24-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32-24-server; java 1.6.0_18; America/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32-24-server; java 1.6.0_18; Europe/cs) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32-25-generic; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32-3-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32-3-amd64; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32-gentoo; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32-trunk-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32.2; java 1.6.0_17; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32.2; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32.2; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.32.9-rscloud; java 1.6.0_20; Etc/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.33-ARCH; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.33.8-149.fc13.x86_64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.33; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.34-12-desktop; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.34-ARCH; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.34-rc6-git1; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Linux 2.6.9-023stab048.4-smp; java 1.6.0; GMT/en) http://yacy.net/bot.html
yacybot (amd64 Windows 2003 5.2; java 1.6.0_19; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Windows 7 6.1; java 1.6.0_14; America/en) http://yacy.net/bot.html
yacybot (amd64 Windows 7 6.1; java 1.6.0_14; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Windows 7 6.1; java 1.6.0_15; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Windows 7 6.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Windows 7 6.1; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (amd64 Windows 7 6.1; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Windows Server 2008 6.0; java 1.6.0_14; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Windows Server 2008 R2 6.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Windows Server 2008 R2 6.1; java 1.6.0_21; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Windows Vista 6.0; java 1.6.0_15; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Windows Vista 6.0; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Windows Vista 6.0; java 1.6.0_17; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Windows Vista 6.0; java 1.7.0-ea; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Windows Vista 6.1; java 1.6.0_12; Europe/de) http://yacy.net/bot.html
yacybot (amd64 Windows Vista 6.1; java 1.6.0_13; Europe/de) http://yacy.net/bot.html
yacybot (freeworld-global; amd64 Linux 2.6.32-35-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
yacybot (freeworld-global; amd64 Linux 2.6.32-36-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
yacybot (freeworld-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (freeworld-global; i386 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (freeworld-global; i386 Linux 2.6.38.8-desktop586-6.mga; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
yacybot (freeworld-global; x86 Windows 7 6.1; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
yacybot (freeworld-global; x86_64 Mac OS X 10.6.4; java 1.6.0_22; Europe/es) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 FreeBSD 7.1-STABLE; java 1.6.0_07; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 FreeBSD 8.2-STABLE; java 1.6.0; GMT/pl) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.16.46-0.12-smp; java 1.6.0_15; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.18-194.26.1.el5; java 1.6.0_14; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.26-2-xen-amd64; java 1.6.0_26; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.26-custom; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.26; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.28.3-vs2.3.0.36.4; java 1.6.0_20; Universal/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.31-23-server; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-23-server; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-28-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-30-generic; java 1.6.0_20; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-30-generic; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-30-server; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-31-generic; java 1.7.0; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-31-server; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-32-generic; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-32-generic; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-32-server; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-32-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-33-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-34-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-34-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-342-ec2; java 1.6.0_20; Etc/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-36-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-37-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-37-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-38-generic; java 1.7.0; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-40-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_29; US/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.7.0_147-icedtea; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-6-pve; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32.2; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.34.7-0.5-desktop; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.34.7-0.7-desktop; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.35-23-generic; java 1.6.0_20; Asia/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.35-24-generic; java 1.6.0_20; Asia/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.35-24-generic; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.35-24-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.35.6; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.36.4-vs2.3.0.36.39-netcup; java 1.6.0_18; Etc/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.37.6; java 1.6.0_25; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.37; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.38-10-generic; java 1.6.0_22; Asia/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.38-11-server; java 1.6.0_22; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.38-13-generic; java 1.6.0_22; America/pt) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.38-2-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.38-8-generic; java 1.6.0_22; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.38-8-generic; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.38-8-server; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.38.2-grsec-xxxx-grs-ipv6-64-filer; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.38.2-grsec-xxxx-grs-ipv6-64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-12-generic; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-12-server; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-12-server; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-13-generic; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-13-generic; java 1.7.0_147-icedtea; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-14-generic; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-14-generic; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-15-server; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-15-server; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-16-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-rc4-1-desktop; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-rc5-1-desktop; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.0-1-amd64; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.0-1.2-desktop; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.0-1.2-desktop; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.10-1.9-default; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.10-hardened; java 1.7.0_03-icedtea; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.1; java 1.7.0_147-icedtea; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.2-1.fc16.x86_64; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.2-1.fc16.x86_64; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.4-1-ARCH; java 1.7.0_147-icedtea; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.9-1.4-default; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-1-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-10-generic; java 1.7.0_147-icedtea; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-18-generic; java 1.7.0_03-icedtea; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-2-rt-amd64; java 1.6.0_24; Asia/ja) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-21-generic; java 1.7.0_03-icedtea; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0; java 1.7.0_147-icedtea; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.1-2-ARCH; java 1.7.0_147-icedtea; Europe/fr) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.13-1-ARCH; java 1.7.0_03-icedtea; Europe/fr) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.2-1-ARCH; java 1.7.0_147-icedtea; Europe/fr) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.2-1.fc16.x86_64; java 1.7.0_b147-icedtea; W-SU/ru) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.3.1-1-ARCH; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_25; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_26; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_26; Europe/fr) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_27; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Asia/ru) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/fr) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/no) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/sv) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_01; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_01; Europe/fr) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows Vista 6.0; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.18-028stab070.2; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.18-028stab070.5; java 1.6.0_22; Etc/de) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.18-194.26.1.el5.028stab079.2; java 1.6.0_24; Etc/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.24-1-686; java 1.6.0_0; Europe/cs) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.24-28-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.32-24-generic; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.32-3-pve; java 1.6.0_18; Europe/cs) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.32-33-generic-pae; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.32-36-generic-pae; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.32-39-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.32-4-pve; java 1.6.0_18; Etc/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.32-5-686-bigmem; java 1.6.0_26; Etc/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.6.0_18; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.36; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.37-gentoo-r4-110510; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.37-gentoo-r4-110510; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.38-13-generic; java 1.6.0_20; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.38-2-686; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.38-8-generic-pae; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.38-8-generic; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.38-8-generic; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.38.7-smp; java 1.6.0_29; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.38; java 1.6.0_24; Europe/es) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 3.0.0-12-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 3.0.0-12-generic-pae; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 3.0.0-13-generic-pae; java 1.6.0_23; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 3.0.0-15-generic-pae; java 1.6.0_23; America/fr) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 3.0.0-16-generic; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows 2003 5.2; java 1.6.0_30; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows 7 6.1; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_01; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows XP 5.1; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows XP 5.1; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows XP 5.1; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows XP 5.1; java 1.6.0_29; Europe/es) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows XP 5.1; java 1.6.0_31; Europe/sv) http://yacy.net/bot.html
yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Asia/ru) http://yacy.net/bot.html
yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Europe/ru) http://yacy.net/bot.html
yacybot (i386 FreeBSD 8.0-RELEASE-p2; java 1.7.0; GMT/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.4.18; java 1.5.0_15; America/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.4.35-wt1; java 1.6.0_05; CET/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.16.37-domU-v1; java 1.6.0_12; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.16.46-0.12-smp; java 1.5.0_06; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.18-028stab064.7; java 1.6.0_15; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.18-028stab069.6; java 1.6.0_20; Etc/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.18-128.1.10.el5PAE; java 1.5.0_18; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.18-128.1.10.el5xen; java 1.6.0_07; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.18-128.1.14.el5xen; java 1.6.0_07; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.18-164.2.1.el5; java 1.5.0_17; America/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.18-6-686; java 1.6.0_10; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.18-6-686; java 1.6.0_16; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.18-92.1.22.el5xen; java 1.6.0_10; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.23; java 1.6.0_06; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.23; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.24-1-686; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.24-10-pve; java 1.6.0_12; Etc/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.24-23-generic; java 1.6.0_07; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.24-23-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.24-23-server; java 1.6.0_17; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.24-24-generic; java 1.6.0_07; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.24-24-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.24-27-server; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.24-8-pve; java 1.6.0_12; Zulu/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.24-etchnhalf.1-686; java 1.5.0_14; GMT+01:00/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.24.6; java 1.6.0_13; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.24_UB; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.25-gentoo-r7; java 1.6.0_07; UTC/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.25.20-0.5-pae; java 1.6.0_15; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.26-2-686-bigmem; java 1.6.0_12; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.26-2-686-bigmem; java 1.6.0_14; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.26-2-686; java 1.5.0_17; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_0; America/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_12; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_12; Europe/fr) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.26-2-openvz-686; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.26-2-openvz-686; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.26-2-openvz-686; java 1.6.0_0; UTC/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.26-gentoo-r3-gh0st; java 1.6.0_13; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.26; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.27-11-generic; java 1.5.0_16; GMT+01:00/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.27-11-generic; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.27-9-generic; java 1.6.0_10; Australia/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.27.10-grsec-xxxx-grs-ipv4-32; java 1.6.0_07; Europe/fr) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.27.10-grsec-xxxx-grs-ipv4-32; java 1.6.0_12; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.27.10-grsec-xxxx-grs-ipv6-32; java 1.6.0_12; Europe/fr) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.27.21-0.1-default; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.27.21-0.1-pae; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-11-generic; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-11-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-11-generic; java 1.6.0_0; Europe/fr) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-11-generic; java 1.6.0_13; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-12-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-12-netbook; java 1.6.0_13; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-13-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-13-generic; java 1.6.0_13; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-13-generic; java 1.6.0_14; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-14-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-15-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-15-generic; java 1.6.0_13; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-15-server; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-16-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-16-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-17-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-17-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-18-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-18-generic; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-19-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-19-server; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.28-gentoo-r5; java 1.5.0_18; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.30-2.slh.2-sidux-686; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.30-gentoo-r6-090907; java 1.6.0_17; GMT/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.30-gentoo-r6; java 1.5.0_22; Europe/el) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-14-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-16-generic-pae; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-16-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-16-generic; java 1.6.0_15; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-17-generic-pae; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-17-generic; java 1.6.0_15; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-17-generic; java 1.6.0_15; SystemV/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-17-generic; java 1.6.0_16; Europe/fr) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-19-generic-pae; java 1.6.0_15; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-19-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-19-generic; java 1.6.0_15; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-20-generic-pae; java 1.6.0_0; America/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-20-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-20-generic; java 1.6.0_0; Europe/fr) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-21-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-ARCH; java 1.6.0_17; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31-gentoo; java 1.6.0_17; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31.12-0.1-desktop; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31.12-mfs002-xen0-compat-i586; java 1.6.0_15; Universal/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.31.6; java 1.6.0_15; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-21-generic-pae; java 1.6.0_18; America/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-21-generic; java 1.6.0_18; America/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-22-generic-pae; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-22-generic-pae; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-22-generic; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-23-generic; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-23-generic; java 1.6.0_18; Europe/es) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-23-generic; java 1.6.0_20; Europe/es) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-24-generic; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-24-generic; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-24-generic; java 1.6.0_20; America/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-5-686; java 1.6.0_18; America/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-5-686; java 1.6.0_18; Atlantic/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-gentoo-r7; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-trunk-686-bigmem; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-trunk-686; java 1.6.0_17; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-trunk-686; java 1.6.0_18; America/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32-trunk-686; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32.2-xxxx-std-ipv4-32; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32.2-xxxx-std-ipv4-32; java 1.6.0_17; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.32.2; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.34.6-xxxx-grs-ipv6-32; java 1.6.0_0; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.9-023stab048.6-enterprise; java 1.6.0_07; Europe/de) http://yacy.net/bot.html
yacybot (i386 Linux 2.6.9-5.ELsmp; java 1.6.0_12-rev; Asia/en) http://yacy.net/bot.html
yacybot (i386 Mac OS X 10.5.6; java 1.5.0_16; Europe/de) http://yacy.net/bot.html
yacybot (i386 Mac OS X 10.5.7; java 1.5.0_16; Europe/de) http://yacy.net/bot.html
yacybot (i386 Mac OS X 10.5.7; java 1.5.0_19; Europe/de) http://yacy.net/bot.html
yacybot (i386 Mac OS X 10.5.8; java 1.5.0_19; Europe/de) http://yacy.net/bot.html
yacybot (i386 Mac OS X 10.5.8; java 1.5.0_20; Europe/fr) http://yacy.net/bot.html
yacybot (i386 Mac OS X 10.5.8; java 1.5.0_22; Europe/de) http://yacy.net/bot.html
yacybot (ppc Mac OS X 10.5.6; java 1.5.0_13; Europe/de) http://yacy.net/bot.html
yacybot (ppc Mac OS X 10.5.8; java 1.5.0_22; Europe/nl) http://yacy.net/bot.html
yacybot (sparc Linux 2.6.26-2-sparc64-smp; java 1.6.0_0; America/en) http://yacy.net/bot.html
yacybot (webportal-global; amd64 Linux 2.6.26-2-xen-amd64; java 1.6.0_0; Europe/en) http://yacy.net/bot.html
yacybot (webportal-global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (webportal-global; amd64 Linux 3.0.0-17-generic; java 1.7.0_147-icedtea; Europe/de) http://yacy.net/bot.html
yacybot (webportal-global; amd64 Linux 3.0.4-ck; java 1.6.0_22; America/en) http://yacy.net/bot.html
yacybot (webportal-global; amd64 Windows 7 6.1; java 1.7.0; Europe/de) http://yacy.net/bot.html
yacybot (webportal-global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/pt) http://yacy.net/bot.html
yacybot (webportal/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (webportal/global; amd64 Windows 7 6.1; java 1.7.0_02; Europe/de) http://yacy.net/bot.html
yacybot (webportal/global; i386 Linux 2.6.35-28-generic; java 1.6.0_20; America/en) http://yacy.net/bot.html
yacybot (x86 SunOS 5.11; java 1.6.0_12; Europe/en) http://yacy.net/bot.html
yacybot (x86 SunOS 5.11; java 1.6.0_13; Europe/en) http://yacy.net/bot.html
yacybot (x86 SunOS 5.11; java 1.6.0_15; Europe/en) http://yacy.net/bot.html
yacybot (x86 Windows 2003 5.2; java 1.6.0_14; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows 2003 5.2; java 1.6.0_16; America/en) http://yacy.net/bot.html
yacybot (x86 Windows 2003 5.2; java 1.6.0_16; Europe/en) http://yacy.net/bot.html
yacybot (x86 Windows 2003 5.2; java 1.6.0_17; America/en) http://yacy.net/bot.html
yacybot (x86 Windows 2003 5.2; java 1.6.0_17; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows 2003 5.2; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows 7 6.1; java 1.6.0_16; Asia/ru) http://yacy.net/bot.html
yacybot (x86 Windows 7 6.1; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows 7 6.1; java 1.6.0_17; America/en) http://yacy.net/bot.html
yacybot (x86 Windows 7 6.1; java 1.6.0_17; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows 7 6.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows 7 6.1; java 1.6.0_18; Europe/sv) http://yacy.net/bot.html
yacybot (x86 Windows 7 6.1; java 1.6.0_20; Europe/fr) http://yacy.net/bot.html
yacybot (x86 Windows 7 6.1; java 1.6.0_20; Europe/sv) http://yacy.net/bot.html
yacybot (x86 Windows Server 2008 6.0; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows Server 2008 6.0; java 1.6.0_20; Europe/nl) http://yacy.net/bot.html
yacybot (x86 Windows Vista 6.0; java 1.6.0_04; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows Vista 6.0; java 1.6.0_07; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows Vista 6.0; java 1.6.0_11; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows Vista 6.0; java 1.6.0_13; Europe/en) http://yacy.net/bot.html
yacybot (x86 Windows Vista 6.0; java 1.6.0_14; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows Vista 6.0; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (x86 Windows Vista 6.0; java 1.6.0_21; Europe/en) http://yacy.net/bot.html
yacybot (x86 Windows Vista 6.1; java 1.5.0_13; Europe/sv) http://yacy.net/bot.html
yacybot (x86 Windows Vista 6.1; java 1.6.0_11; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows Vista 6.1; java 1.6.0_12; Australia/en) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.5.0_06; Europe/sv) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_05; Asia/en) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_06; Europe/es) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_11; Europe/fr) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_12; Europe/pl) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_13; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_14; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_14; Europe/ru) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_15; America/en) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_15; America/fr) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_15; Europe/es) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_16; America/en) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_16; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_17; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_18; America/en) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_19; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_20; America/en) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.7.0-ea; Asia/zh) http://yacy.net/bot.html
yacybot (x86_64 Mac OS X 10.6.2; java 1.6.0_17; Europe/de) http://yacy.net/bot.html
yacybot (x86_64 Mac OS X 10.6.3; java 1.6.0_15; Europe/de) http://yacy.net/bot.html
yacybot (x86_64 Mac OS X 10.6.3; java 1.6.0_17; America/en) http://yacy.net/bot.html
yacybot (x86_64 Mac OS X 10.6.3; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
yacybot (x86_64 Mac OS X 10.6.4; java 1.6.0_20; America/en) http://yacy.net/bot.html
yacybot (x86_64 Mac OS X 10.6.4; java 1.6.0_20; Europe/de) http://yacy.net/bot.html
yacybot (x86_64 Mac OS X 10.6.4; java 1.6.0_20; Europe/fr) http://yacy.net/bot.html
yacybot (x86_64 Mac OS X 10.6.7; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
YaDirectBot/1.0
Yahoo Pipes 1.0
Yahoo Pipes 2.0
YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide)
Yandex.Server/2009.5
Yandex/1.01.001 (compatible; Win16; H)
Yandex/1.01.001 (compatible; Win16; I)
Yandex/1.01.001 (compatible; Win16; I)/Nutch-1.2
Yandex/1.01.001 (compatible; Win16; m)
Yandex/1.01.001 (compatible; Win16; P)
Yandex/1.03.003 (compatible; Win16; D)
YandexBlogs
YandexSomething/1.0
Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/)
yanky-nutch/Nutch-0.9 (nothing; nothing; yanky.young@gmail.com)
yechizi/Nutch-1.2
yeetec_dasbot (zhaoqingfeng@yeetec.com)
Yepi/1.0 (NHN Corp.; http://help.naver.com/robots/)
Yeti/0.01 (nhn/1noon, yetibot@naver.com, check robots.txt daily and follows it)
Yeti/1.0 (+http://help.naver.com/robots/)
Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)
yolinkBot
YowedoBot/Yowedo 1.0 (Search Engine crawler for yowedo.com; http://yowedo.com/en/partners.html; crawler@yowedo.com)
YPBot/Raven1.1.3 (compatible; Googlebot/2.1;+http://www.yellowpages.com/about/legal/crawl)
yrspider (Mozilla/5.0 (compatible; YRSpider; +http://www.yunrang.com/yrspider.html))
YUMROO/yumroo.search.bot-1.0 (search bot of http://www.yumroo.az; info at yumroo dot az)
zawc/Nutch-1.0 (http://www.zawc.com)
zawc/Nutch-1.2 (http://www.zawc.com)
zclosure/Nutch-1.3
zclosure/Nutch-1.4
zebra457/Nutch-1.3
Zeus 76869 Webster Pro V2.9 Win32
zhuada/Nutch-1.0 (http://www.zhuada.com/spider/; spider@zhuada.com)
Zhuada/Nutch-1.0 (www.zhuada.com/spider/)
zibber-v0.1(www.zibb.com/crawler/)
Zillabyte/Nutch-1.3
Zillsy.com Search engine /Nutch-1.4
zologo/Nutch-1.1 (zologo; http://www.zologo.com; hibo@zologo.co)
zomba-bot/0.1
ZoomSpider - wrensoft.com [ZSEBOT]
Zscho.de Crawler/Nutch-1.0-Zscho.de-semantic_patch (Zscho.de Crawler, collecting for machine learning; http://zscho.de/ )
Zscho.de/Nutch-0.9-semantic_patch
zschobot/Nutch-0.9-semantic_patch (zschobot indexing; Zscho.de/de/bot.html)
ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)
ZupeeCrawler0.1.1
^Byte (http://CaretByte.com)
_USERAGENT_
#!/usr/bin/env python
# encoding: utf-8
"""Parse the number of robot requests from an Apache log file
The list of robot user agents comes from http://www.user-agents.org/ but has
had some agents added to it since the list was last updated in 2010.
"""
from __future__ import division
import fileinput
import optparse
import os
import re
import sys
UA_RE = re.compile(r'^[^ ]+ [^ ]+ [^ ]+ \[[^\]]+\] "[^"]+" \d+ \d+ "[^"]+" "(.*?)"')
def main(bot_agent_file="user-agents.txt"):
all_requests = 0
bot_requests = 0
bot_count = {}
bot_agents = set()
with open(bot_agent_file, "r") as f:
bot_agents.update(filter(None, (i.strip() for i in f)))
for line in sys.stdin:
all_requests += 1
m = UA_RE.match(line)
ua = m.group(1)
if ua == "-" or ua in bot_agents:
bot_count[ua] = bot_count.get(ua, 0) + 1
bot_requests += 1
bots = bot_count.keys()
bots.sort(key=lambda i: bot_count[i])
for bot in bots:
print "%10d %s" % (bot_count[bot], bot)
print "Bot requests: %d of %d (%0.2f%%)" % (bot_requests, all_requests,
100 * (bot_requests / all_requests))
if __name__ == "__main__":
fmt = optparse.IndentedHelpFormatter(max_help_position=8)
parser = optparse.OptionParser(__doc__.strip(), formatter=fmt)
parser.add_option("--bot-user-agents", metavar="FILE",
default=os.path.join(os.path.dirname(__file__),
"user-agents.txt"),
help="File containing a list of bot user-agents "
"(default=%defaults)")
(options, filenames) = parser.parse_args()
if filenames:
parser.error("Provide log data on stdin")
main(bot_agent_file=options.bot_user_agents)
'Mozilla/5.0 (compatible; BMC/1.0 (Y!J-AGENT))'
( Robots.txt Validator http://www.searchengineworld.com/cgi-bin/robotcheck.cgi )
(DreamPassport/3.0; isao/MyDiGiRabi)
(GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
(Privoxy/1.0)
*/Nutch-0.9-dev
+SitiDi.net/SitiDiBot/1.0 (+Have Good Day)
-
- (larbin2.6.3@unspecified.mail)
-DIE-KRAEHE- META-SEARCH-ENGINE/1.1 http://www.die-kraehe.de
123spider-Bot (Version: 1.02&#44; powered by www.123spider.de
192.comAgent
1st ZipCommander (Net) - http://www.zipcommander.com/
2Bone_LinkChecker/1.0 libwww-perl/5.64
3485 Mozilla/5.0 (compatible; Bender; http://benderthewebrobot.tumblr.com)
4anything.com LinkChecker v2.0
8484 Boston Project v 1.0
: Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1
:robot/1.0 (linux) ( admin e-mail: undefined http://www.neofonie.de/loesungen/search/robot.html )
A-Online Search
A1 Keyword Research/1.0.2 (+http://www.micro-sys.dk/products/keyword-research/) miggibot/2007.03.27
A1 Sitemap Generator/1.0 (+http://www.micro-sys.dk/products/sitemap-generator/) miggibot/2006.01.24
aardvark-crawler
AbachoBOT
AbachoBOT (Mozilla compatible)
ABCdatos BotLink/5.xx.xxx#BBL
Aberja Checkomat
abot/0.1 (abot; http://www.abot.com; abot@abot.com)
Aboundex/0.2 (http://www.aboundex.com/crawler/)
About/0.1libwww-perl/5.47
Abrave Spider v5.3 Robot 5 (http://robot.abrave.com)
Accelatech RSSCrawler/0.4
accoona
Accoona-AI-Agent/1.1.1 (crawler at accoona dot com)
Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com)
Ace Explorer
Ack (http://www.ackerm.com/)
AcoiRobot
Acoon Robot v1.50.001
Acoon Robot v1.52 (http://www.acoon.de)
Acoon v4.1.0 (www.acoon.de)
Acoon v4.10.1 (www.acoon.de)
Acoon v4.10.3 (www.acoon.de)
Acoon v4.10.4 (www.acoon.de)
Acoon v4.10.5 (www.acoon.de)
Acoon v4.9.3 (www.acoon.de)
Acoon v4.9.5 (www.acoon.de)
Acoon-Robot 4.0.x.[xx] (http://www.acoon.de)
Acoon-Robot v3.xx (http://www.acoon.de and http://www.acoon.com)
AcoonBot/4.10.5 (+http://www.acoon.de)
Acorn/Nutch-0.9 (Non-Profit Search Engine; acorn.isara.org; acorn at isara dot org)
ActiveBookmark 1.x
Activeworlds
ActiveWorlds/3.xx (xxx)
Ad Muncher v4.xx.x
Ad Muncher v4x Build xxxxx
Adaxas Spider (http://www.adaxas.net/)
AddThis.com robot tech.support@clearspring.com
Advanced Browser (http://www.avantbrowser.com)
AESOP_com_SpiderMan
agadine/1.x.x (+http://www.agada.de)
Agent
Agent-SharewarePlazaFileCheckBot/2.0+(+http://www.SharewarePlaza.com)
AgentName/0.1 libwww-perl/5.48
Aghaven/Nutch-1.2 (www.aghaven.com)
AIBOT/2.1 By +(www.21seek.com A Real artificial intelligence search engine China)
AideRSS/1.0 (aiderss.com)
aipbot/1.0 (aipbot; http://www.aipbot.com; aipbot@aipbot.com)
aipbot/2-beta (aipbot dev; http://aipbot.com; aipbot@aipbot.com)
Akregator/1.2.9; librss/remnants
Aladin/3.324
Alcatel-BG3/1.0 UP.Browser/5.0.3.1.2
Aleksika Spider/1.0 (+http://www.aleksika.com/)
AlertInfo 2.0 (Powered by Newsbrain)
AlkalineBOT/1.3
AlkalineBOT/1.4 (1.4.0326.0 RTM)
Allesklar/0.1 libwww-perl/5.46
Alligator 1.31 (www.nearsoftware.com)
Allrati/1.1 (+)
AltaVista Intranet V2.0 AVS EVAL search@freeit.com
AltaVista Intranet V2.0 Compaq Altavista Eval sveand@altavista.net
AltaVista Intranet V2.0 evreka.com crawler@evreka.com
AltaVista V2.0B crawler@evreka.com
amaya/x.xx libwww/x.x.x
AmfibiBOT
Amfibibot/0.06 (Amfibi Web Search; http://www.amfibi.com; agent@amfibi.com)
Amfibibot/0.07 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com)
amibot
Amiga-AWeb/3.4.167SE
AmigaVoyager/3.4.4 (MorphOS/PPC native)
AmiTCP Miami (AmigaOS 2.04)
Amoi 8512/R21.0 NF-Browser/3.3
amzn_assoc
animalstxt.org/bot (http://animalstxt.org/)
AnnoMille spider 0.1 alpha - http://www.annomille.it
annotate_google; http://ponderer.org/download/annotate_google.user.js
Anonymized by ProxyOS: http://www.megaproxy.com
Anonymizer/1.1
AnswerBus (http://www.answerbus.com/)
AnswerChase PROve x.0
AnswerChase x.0
ANTFresco/x.xx
antibot-V1.1.5/i586-linux-2.2
AnzwersCrawl/2.0 (anzwerscrawl@anzwers.com.au;Engine)
Apache-HttpClient/4.1.1 (java 1.5)
Apache-HttpClient/4.1.2 (java 1.5)
Apexoo Spider 1.x
Aplix HTTP/1.0.1
Aplix_SANYO_browser/1.x (Japanese)
Aplix_SEGASATURN_browser/1.x (Japanese)
Aport
appie 1.1 (www.walhello.com)
Apple iPhone v1.1.4 CoreMedia v1.0.0.4A102
Apple-PubSub/65.1.1
Applied-Technologies-Inc-Spider/Nutch-1.4
ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby.com;)
ArachBot
Arachnoidea (arachnoidea@euroseek.com)
AramaM/Nutch-2.0-dev
araneo
aranhabot
ArchitextSpider
archive.org_bot
Argus/1.1 (Nutch; http://www.simpy.com/bot.html; feedback at simpy dot com)
Arikus_Spider
Arquivo-web-crawler (compatible; heritrix/1.12.1 +http://arquivo-web.fccn.pt)
Arquivo-web-crawler (compatible; heritrix/1.14.3 +http://arquivo.pt/faq-crawling)
ASAHA Search Engine Turkey V.001 (http://www.asaha.com/)
Asahina-Antenna/1.x
Asahina-Antenna/1.x (libhina.pl/x.x ; libtime.pl/x.x)
ask.24x.info
AskAboutOil/0.06-rcp (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@askaboutoil.com)
asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com)
ASPSeek/1.2.5
ASPseek/1.2.9d
ASPSeek/1.2.x
ASPSeek/1.2.xa
ASPseek/1.2.xx
ASPSeek/1.2.xxpre
ASPseek/1.4
ASSORT/0.10
asterias/2.0
AtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html)
Atomic_Email_Hunter/4.0
Atomz/1.0
atSpider/1.0
Attentio/Nutch-0.9-dev (Attentio's beta blog crawler; www.attentio.com; info@attentio.com)
AU-MIC/2.0 MMP/2.0
AUDIOVOX-SMT5600
augurfind
augurnfind V-1.x
autoemailspider
autohttp
autowebdir 1.1 (www.autowebdir.com)
AV Fetch 1.0
Avant Browser (http://www.avantbrowser.com)
AVSearch-1.0(peter.turney@nrc.ca)
AVSearch-2.0-fusionIdx-14-CompetitorWebSites
AVSearch-3.0(AltaVista/AVC)
AWeb
axadine/ (Axadine Crawler; http://www.axada.de/; )
AxmoRobot - Crawling your site for better indexing on www.axmo.com search engine.
Azureus 2.x.x.x
A_TEST_CRALWER_BY_ISSEL/Nutch-1.4 (http://issel.ee.auth.gr; kvavliak at issel dot ee dot auth dot gr)
BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si)
BaboomBot/1.x.x (+http://www.baboom.us)
BacklinkCrawler (http://www.backlinktest.com/crawler.html)
BackStreet Browser 3.x
BaiduImagespider+(+http://www.baidu.jp/search/s308.html)
BaiDuSpider
Baiduspider+(+http://help.baidu.jp/system/05.html)
Baiduspider+(+http://www.baidu.com/search/spider.htm)
Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
Balihoo/Nutch-1.0-dev (Crawler for Balihoo.com search engine - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com)
BanBots/1.2 (spider@banbots.com)
Barca/2.0.xxxx
BarcaPro/1.4.xxxx
BarraHomeCrawler (albertof@barrahome.org)
bCentral Billing Post-Process
bdcindexer_2.6.2 (research@bdc)
BDFetch
BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html) (X11; I; Linux 2.0.44 i686)
BeamMachine/0.5 (dead link remover of www.beammachine.net)
beautybot/1.0 (+http://www.uchoose.de/crawler/beautybot/)
BebopBot/2.5.1 ( crawler http://www.apassion4jazz.net/bebopbot.html )
BeebwareDirectory/v0.01
berlin-fu-cow (+http://hpsg.fu-berlin.de/cow)
Big Brother (http://pauillac.inria.fr/~fpottier/)
Big Fish v1.0
BigBrother/1.6e
BigCliqueBOT/1.03-dev (bigclicbot; http://www.bigclique.com; bot@bigclique.com)
BIGLOTRON (Beta 2;GNU/Linux)
Bigsearch.ca/Nutch-x.x-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.ca/; info@enhancededge.com)
Bilbo/2.3b-UNIX
BilgiBetaBot/0.8-dev (bilgi.com (Beta) ; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
BilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com)
billbot wjj@cs.cmu.edu
binlar_2.6.3 (binlar2.6.3@unspecified.mail)
binlar_2.6.3 (larbin2.6.3@unspecified.mail)
binlar_2.6.3 (test@mgmt.mic)
Bitacle bot/1.1
Bitacle Robot (V:1.0;) (http://www.bitacle.com)
Biyubi/x.x (Sistema Fenix; G11; Familia Toledo; es-mx)
BlackBerry7520/4.0.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/5.0.3.3 UP.Link/5.1.2.12 (Google WAP Proxy/1.0)
BlackBerry9630/5.0.0.732 Profile/MIDP-2.1 Configuration/CLDC-1.1 VendorID/102 ips-agent
BlackWidow
Blaiz-Bee/1.0 (+http://www.blaiz.net)
Blaiz-Bee/2.00.8222 (BE Internet Search Engine http://www.rawgrunt.com)
Blaiz-Bee/2.00.xxxx (+http://www.blaiz.net)
bldbCrawler/Nutch-1.4
BlevintronBot version 2012-04-02 contact http://en.wikipedia.org/wiki/User_talk:Blevintron
BlevintronBot version 2012-04-03 contact http://en.wikipedia.org/wiki/User_talk:Blevintron
BlevintronBot version 2012-04-04 contact http://en.wikipedia.org/wiki/User_talk:Blevintron
BlevintronBot version 2012-04-09 contact http://en.wikipedia.org/wiki/User_talk:Blevintron
BlevintronBot version 2012-05-01 contact http://en.wikipedia.org/wiki/User_talk:Blevintron
BlitzBOT@tricus.net
BlitzBOT@tricus.net (Mozilla compatible)
BlockNote.Net
BlogBot/1.x
BlogBridge 2.13 (http://www.blogbridge.com/)
Bloglines Title Fetch/1.0 (http://www.bloglines.com)
Bloglines-Images/0.1 (http://www.bloglines.com)
Bloglines/3.1 (http://www.bloglines.com)
BlogMap (http://www.feedmap.net)
Blogpulse (info@blogpulse.com)
BlogPulseLive (support@blogpulse.com)
BlogSearch/1.x +http://www.icerocket.com/
blogsearchbot-pumpkin-3
BlogsNowBot&#44; V 2.01 (+http://www.blogsnow.com/)
BlogVibeBot-v1.1 (spider@blogvibe.nl)
blogWatcher_Spider/0.1 (http://www.lr.pi.titech.ac.jp/blogWatcher/)
BlogzIce/1.0 (+http://icerocket.com; rhodes@icerocket.com)
BlogzIce/1.0 +http://www.icerocket.com/
BloobyBot
Bloodhound/Nutch-0.9 (Testing Crawler for Research - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com)
BLT/1.0.4 (http://www.bltformac.com/)
bluefish 0.6 HTML editor
BMCLIENT
BMLAUNCHER
Bobby/4.0.x RPT-HTTPClient/0.3-3E
boitho.com-dc/0.xx (http://www.boitho.com/dcbot.html)
boitho.com-robot/1.x
boitho.com-robot/1.x (http://www.boitho.com/bot.html)
Bookdog/x.x
Bookmark Buddy bookmark checker (http://www.bookmarkbuddy.net/)
Bookmark Renewal Check Agent [http://www.bookmark.ne.jp/]
Bookmark Renewal Check Agent [http://www.bookmark.ne.jp/] (Version 2.0beta)
BookmarkBase(2/;http://bookmarkbase.com)
Bot mailto:craftbot@yahoo.com
BotLaju v2.0 (http://laju.my/)
BPImageWalker/2.0 (www.bdbrandprotect.com)
BravoBrian bstop.bravobrian.it
BravoBrian SpiderEngine MarcoPolo
BrightCrawler (http://www.brightcloud.com/brightcrawler.asp)
BrokenLinkBot version 2012-03-31 contact http://en.wikipedia.org/wiki/User_talk:Blevintron
BrokenLinkBot version 20120319.102558 contact http://en.wikipedia.org/wiki/User_talk:Blevintron
BrokenLinkBot version 20120320.224751 contact http://en.wikipedia.org/wiki/User_talk:Blevintron
BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html)
BSDSeek/1.0
BStop.BravoBrian.it Agent Detector
BTbot/0.x (+http://www.btbot.com/btbot.html)
Btsearch/1.1 (Btsearch; http://www.test.com/test.html; test@test.com)
BTWebClient/180B(9704)
BuildCMS crawler (http://www.buildcms.com/crawler)
Bulkfeeds/r1752 (http://bulkfeeds.net/)
BullsEye
bumblebee@relevare.com
BunnySlippers
BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawler@burstfind.com)
buscador-age (Enterprise; T2-C9QZ24BXJWWK2; gep@ti.eptisa.com)
Buscaplus Robi/1.0 (http://www.buscaplus.com/robi/)
businessdbbot/v1.2 (http://www.businessdb.com/bot.php)
BW-C-2.0
bwh3_user_agent
Cabot/Nutch-0.9 (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; agent@amfibi.com)
Cabot/Nutch-1.0-dev (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; agent@amfibi.com)
CamelHttpStream/1.0
Cancer Information and Support International;
carleson/1.0
Carnegie_Mellon_University_Research_WebBOT-->PLEASE READ-->http://www.andrew.cmu.edu/~brgordon/webbot/index.html http://www.andrew.cmu.edu/~brgordon/webbot/index.html
Carnegie_Mellon_University_WebCrawler http://www.andrew.cmu.edu/~brgordon/webbot/index.html
Catall Spider
CatchBot/1.0; +http://www.catchbot.com
CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com)
CCBot/1.0 (+http://www.commoncrawl.org/bot.html)
ccubee/x.x
CDR/1.7.1 Simulator/0.7(+http://timewe.net) Profile/MIDP-1.0 Configuration/CLDC-1.0
CE-Preload
CentiverseBot
CentiverseBot - investigator
CentiverseBot/3.0 (http://www.centiverse-project.net)
Ceramic Tile Installation Guide (http://www.floorstransformed.com)
CERN-LineMode/2.15
cfetch/1.0
CFNetwork/x.x
cg-eye interactive
Charon/1.x (Amiga)
Chat Catcher/1.0
Checkbot/1.xx LWP/5.xx
CheckLinks/1.x.x
CheckUrl
CheckWeb
Chilkat/1.0.0 (+http://www.chilkatsoft.com/ChilkatHttpUA.asp)
China Local Browse 2.6
Chitika ContentHit 1.0
ChristCRAWLER 2.0
CHttpClient by Open Text Corporation
cinvestav/Nutch-1.4
CipinetBot (http://www.cipinet.com/bot.html)
Cityreview Robot (+http://www.cityreview.org/crawler/)
CJ Spider/
CJB.NET Proxy
CjxSearch/Nutch-1.4
ClariaBot/1.0
Claymont.com
CloakDetect/0.9 (+http://fulltext.seznam.cz/)
cloud-valley/bigdata-1.0 (http://www.cloud-valley.com/; news@mobier.mobi)
CloudACL/Nutch-1.4
Clushbot/2.x (+http://www.clush.com/bot.html)
Clushbot/3.x-BinaryFury (+http://www.clush.com/bot.html)
Clushbot/3.xx-Ajax (+http://www.clush.com/bot.html)
Clushbot/3.xx-Hector (+http://www.clush.com/bot.html)
Clushbot/3.xx-Peleus (+http://www.clush.com/bot.html)
COAST WebMaster Pro/4.x.x.xx (Windows NT)
CoBITSProbe
coccoc
Cocoal.icio.us/1.0 (v36) (Mac OS X; http://www.scifihifi.com/cocoalicious)
Code Search Crawler/Nutch-1.2 (Code Search Crawler; www.iai.uni-bonn.de)
Cogentbot/1.X (+http://www.cogentsoftwaresolutions.com/bot.html)
ColdFusion
ColdFusion (BookmarkTracker.com)
collage.cgi/1.xx
combine/0.0
Combine/2.0 http://combine.it.lth.se/
Combine/3 http://combine.it.lth.se/
Combine/x.0
cometrics-bot&#44; http://www.cometrics.de
Commerce Browser Center
Comodo-Certificates-Spider
COMODOSpider/Nutch-1.2
complex_network_group/Nutch-0.9-dev (discovering the structure of the world-wide-web; http://cantor.ee.ucla.edu/~networks/crawl; nimakhaj@gmail.com)
Computer_and_Automation_Research_Institute_Crawler crawler@ilab.sztaki.hu
Comrite/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
Contact
ContactBot/0.2
Content Crawler
ContentSmartz
contype
Convera Internet Spider V6.x
ConveraCrawler/0.2
ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl)
ConveraMultiMediaCrawler/0.1 (+http://www.authoritativeweb.com/crawl)
CoolBot
Cooliris/1.5 CFNetwork/459 Darwin/10.0.0d3
CoralWebPrx/0.1.1x (See http://coralcdn.org/)
coruscan/Nutch-1.4
cosmos/0.8_(robot@xyleme.com)
cosmos/0.9_(robot@xyleme.com)
CoteoNutchCrawler/Nutch-0.9 (info [at] coteo [dot] com)
CougarSearch/0.x (+http://www.cougarsearch.com/faq.shtml)
Covac TexAs Arachbot
CoverScout%203/3.0.1 CFNetwork/339.5 Darwin/9.5.0 (i386) (iMac5&#44;1)
Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
Cowbot-0.1.x (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
CrawlConvera0.1 (CrawlConvera@yahoo.com)
Crawler
Crawler (cometsearch@cometsystems.com)
Crawler admin@crawler.de
Crawler V 0.2.x admin@crawler.de
crawler4j (http://code.google.com/p/crawler4j/)
crawler@alexa.com
CrawlerBoy Pinpoint.com
Crawllybot/0.1 (Crawllybot; +http://www.crawlly.com; crawler@crawlly.com)
CreativeCommons/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)
Cricket-A100/1.0 UP.Browser/6.3.0.7 (GUI) MMP/2.0
CrocCrawler vx.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686)
CRW1/Nutch-1.2
csci_b659/0.13
CSE HTML Validator Professional (http://www.htmlvalidator.com/)
Cuam Ver0.050bx
Cuasarbot/0.9b http://www.cuasar.com/spider_beta/
curl/7.10.x (i386-redhat-linux-gnu) libcurl/7.10.x OpenSSL/0.9.7a ipv6 zlib/1.1.4
curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
curl/7.7.x (i386--freebsd4.3) libcurl 7.7.x (SSL 0.9.6) (ipv6 enabled)
curl/7.8 (i686-pc-linux-gnu) libcurl 7.8 (OpenSSL 0.9.6)
curl/7.9.x (win32) libcurl 7.9.x
CurryGuide SiteScan 1.1
Custo x.x (www.netwu.com)
Custom Spider www.bisnisseek.com /1.0
Cyberdog/2.0 (Macintosh; 68k)
CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp)
CyberSpyder Link Test/2.1.12 (admin@mspennyworth.com)
CyberSpyder Link Test/2.2.1
CydralSpider/1.x (Cydral Web Image Search; http://www.cydral.com)
CydralSpider/3.0 (Cydral Image Search; http://www.cydral.com)
CydralSpider/3.2.6 (Cydral Image Search; http://www.cydral.com)
DA 3.5 (www.lidan.com)
DA 4.0
DA 4.0 (www.downloadaccelerator.com)
DA 5.0
DA 7.0
DAP x.x
Dart Communications PowerTCP
DataCha0s/2.0
DataFountains/DMOZ Downloader
DataFountains/Dmoz Downloader (http://ivia.ucr.edu/useragents.shtml)
DataFountains/DMOZ Feature Vector Corpus Creator (http://ivia.ucr.edu/useragents.shtml)
DataparkSearch/4.47 (+http://dataparksearch.org/bot)
DataparkSearch/4.xx (http://www.dataparksearch.org/)
DataSpear/1.0 (Spider; http://www.dataspear.com/spider.html; spider@dataspear.com)
DataSpearSpiderBot/0.2 (DataSpear Spider Bot; http://dssb.dataspear.com/bot.html; dssb@dataspear.com)
DatenBot( http://www.sicher-durchs-netz.de/bot.html)
DaviesBot/1.7 (www.wholeweb.net)
daypopbot/0.x
dbDig(http://www.prairielandconsulting.com)
DBrowse 1.4b
DBrowse 1.4d
DC-Sakura/x.xx
dCSbot/1.1
DDD
dds explorer v1.0 beta
de.searchengine.comBot 1.2 (http://de.searchengine.com/spider)
DeadLinkCheck/0.4.0 libwww-perl/5.xx
Deep Link Calculator v1.0
deepak-USC/ISI
DeepIndex
DeepIndex ( http://www.zetbot.com )
DeepIndex (www.en.deepindex.com)
DeepIndexer.ca
del.icio.us-thumbnails/1.0 Mozilla/5.0 (compatible; Konqueror/3.4; FreeBSD) KHTML/3.4.2 (like Gecko)
DeleGate/9.0.5-fix1
Demo Bot DOT 16b
Demo Bot Z 16b
Denmex websearch (http://search.denmex.com)
Der gro&#223;e BilderSauger 2.00u
dev-spider2.searchpsider.com/1.3b
DevComponents.com HtmlDocument Object
DiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html)
Diamond/x.0
DiamondBot
Digger/1.0 JDK/1.3.0rc3
DigOut4U
DIIbot/1.2
Dillo/0.8.5-i18n-misc
Dillo/0.x.x
disastrous/1.0.5 (running with Python 2.5.1; http://www.bortzmeyer.org/disastrous.html; archangel77@del.icio.us)
DISCo Pump x.x
disco/Nutch-0.9 (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com)
disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com)
DittoSpyder
dlab spider/Nutch-1.3
dlman
dloader(NaverRobot)/1.0
DNSRight.com WebBot Link Ckeck Tool. Report abuse to: dnsr@dnsright.com
DoCoMo/1.0/Nxxxi/c10
DoCoMo/1.0/Nxxxi/c10/TB
DoCoMo/1.0/P502i/c10 (Google CHTML Proxy/1.0)
DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)'''
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)
DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)
DoCoMo/2.0 P900iV(c100;TB;W24H11)
DoCoMo/2.0 SH901iS(c100;TB;W24H12)&#44;gzip(gfe) (via translate.google.com)
DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html)
DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html)
DocZilla/1.0 (Windows; U; WinNT4.0; en-US; rv:1.0.0) Gecko/20020804
dodgebot/experimental
DogSniffer
Domnutch-Bot/Nutch-1.0 (Domnutch; http://www.Nutch.de/)
DonutP; Windows98SE
Doubanbot/1.0 (bot@douban.com http://www.douban.com)
Download Demon/3.x.x.x
Download Druid 2.x
Download Express 1.0
Download Master
Download Ninja 3.0
Download Wonder
Download-Tipp Linkcheck (http://download-tipp.de/)
Download.exe(1.1) (+http://www.sql-und-xml.de/freeware-tools/)
DownloadDirect.1.0
Dr.Web (R) online scanner: http://online.drweb.com/
Dragonfly File Reader
Drecombot/1.0 (http://career.drecom.jp/bot.html)
DrRajendra/Nutch-0.9 (Senior Researcher; http://cse.iitkgp.ac.in/~rpr; rajendra@cse.iitkgp.ernet.in)
Drupal (+http://drupal.org/)
DSurf15a 01
DSurf15a 71
DSurf15a 81
DSurf15a VA
DTAAgent
dtSearchSpider
Dual Proxy
DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)
Dumbot(version 0.1 beta - dumbfind.com)
Dumbot(version 0.1 beta - http://www.dumbfind.com/dumbot.html)
Dumbot(version 0.1 beta)
e-sense 1.0 ea(www.vigiltech.com/esensedisclaim.html)
e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/)
eApolloBot/2.0 (compatible; heritrix/2.0.0-SNAPSHOT-20071024.170148 +http://www.eapollo-opto.com)
EARTHCOM.info/1.x [www.earthcom.info]
EARTHCOM.info/1.xbeta [www.earthcom.info]
EasyDL/3.xx
EasyDL/3.xx http://keywen.com/Encyclopedia/Bot
EBrowse 1.4b
EC2LinkFinder
eCatch/3.0
EchO!/2.0
EdisterBot (http://www.edister.com/bot.html)
edu.pjwstk.focusedcrawler.training-data-gatherer
Educate Search VxB
egothor/3.0a (+http://www.xdefine.org/robot.html)
EgotoBot/4.8 (+http://www.egoto.com/about.htm)
ejupiter.com
EldoS TimelyWeb/3.x
Element/Nutch-2.0-dev
elfbot/1.0 (+http://www.uchoose.de/crawler/elfbot/)
ELI/20070402:2.0 (DAUM RSS Robot&#44; Daum Communications Corp.; +http://ws.daum.net/aboutkr.html)
ELinks (0.x.x; Linux 2.4.20 i586; 132x60)
ELinks/0.x.x (textmode; NetBSD 1.6.2 sparc; 132x43)
EmailSiphon
EmailSpider
EmailWolf 1.00
emefgebot/beta (+http://emefge.de/bot.html)
EmeraldShield.com WebBot
EmeraldShield.com WebBot (http://www.emeraldshield.com/webbot.aspx)
EMPAS_ROBOT
EnaBot/1.x (http://www.enaball.com/crawler.html)
endo/1.0 (Mac OS X; ppc i386; http://kula.jp/endo)
Enfish Tracker
Enterprise_Search/1.0
Enterprise_Search/1.0.xxx
Enterprise_Search/1.00.xxx;MSSQL (http://www.innerprise.net/es-spider.asp)
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html)
envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php)
envolk[ITS]spider/1.6(+http://www.envolk.com/envolkspider.html)
EPiServer Link Checker
EroCrawler
ES.NET_Crawler/2.0 (http://search.innerprise.net/)
eseek-larbin_2.6.2 (crawler@exactseek.com)
ESISmartSpider
eStyleSearch 4 (compatible; MSIE 6.0; Windows NT 5.0)
ESurf15a 15
EuripBot/0.x (+http://www.eurip.com) GetFile
EuripBot/0.x (+http://www.eurip.com) GetRobots
EuripBot/0.x (+http://www.eurip.com) PreCheck
Eurobot/1.0 (http://www.ayell.eu)
EvaalSE - bot@evaal.com
eventax/1.0 (http://www.eventax.de)
eventax/1.3 (eventax; http://www.eventax.de/; info@eventax.de)
EventMachine HttpClient
Everest-Vulcan Inc./0.1 (R&amp;D project; host=e-1-24; http://everest.vulcan.com/crawlerhelp)
Everest-Vulcan Inc./0.1 (R&amp;D project; http://everest.vulcan.com/crawlerhelp)
Exabot-Images/1.0
Exabot-Test/1.0
Exabot/2.0
Exabot/3.0
ExactSearch
ExactSeek Crawler (http://www.exactseek.com/)/Nutch-1.4
ExactSeek Crawler (nutch 1.4)/Nutch-1.4
ExactSeek Crawler/0.1
exactseek-crawler-2.63 (crawler@exactseek.com)
exactseek-pagereaper-2.63 (crawler@exactseek.com)
exactseek.com
Exalead NG/MimeLive Client (convert/http/0.120)
Excalibur Internet Spider V6.5.4
Execrawl/1.0 (Execrawl; http://www.execrawl.com/; bot@execrawl.com)
exooba crawler/exooba crawler (crawler for exooba.com; http://www.exooba.com/; info at exooba dot com)
exooba/exooba crawler (exooba; exooba)
ExperimentalHenrytheMiragoRobot
Expired Domain Sleuth
Express WebPictures (www.express-soft.com)
ExtractorPro
Extreme Picture Finder
EyeCatcher (Download-tipp.de)/1.0
Factbot 1.09 (see http://www.factbites.com/webmasters.php)
factbot : http://www.factbites.com/robots
FaEdit/2.0.x
FairAd Client
FANGCrawl/0.01
FARK.com link verifier
Fast Crawler Gold Edition
FAST Enterprise Crawler 6 (Experimental)
FAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com/srsapp/contactus/
FAST Enterprise Crawler 6 used by Cobra Development (admin@fastsearch.com)
FAST Enterprise Crawler 6 used by Comperio AS (sts@comperio.no)
FAST Enterprise Crawler 6 used by FAST (FAST)
FAST Enterprise Crawler 6 used by Pages Jaunes (pvincent@pagesjaunes.fr)
FAST Enterprise Crawler 6 used by Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au)
FAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg)
FAST Enterprise Crawler 6 used by WWU (wardi@uni-muenster.de)
FAST Enterprise Crawler/6 (www.fastsearch.com)
FAST Enterprise Crawler/6.4 (helpdesk at fast.no)
FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0)
FAST MetaWeb Crawler (helpdesk at fastsearch dot com)
Fast PartnerSite Crawler
FAST-WebCrawler/2.2.10 (Multimedia Search) (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2.6 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)
FAST-WebCrawler/2.2.7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)http://www.fast.no
FAST-WebCrawler/2.2.8 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)http://www.fast.no
FAST-WebCrawler/3.2 test
FAST-WebCrawler/3.3 (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)
FAST-WebCrawler/3.4/Nirvana (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)
FAST-WebCrawler/3.4/PartnerSite (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)
FAST-WebCrawler/3.5 (atw-crawler at fast dot no; http://fast.no/support.php?c=faqs/crawler)
FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
FAST-WebCrawler/3.6/FirstPage (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler)
FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)
FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
FAST-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
FAST-WebCrawler/3.x Multimedia
FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no)
fastbot crawler beta 2.0 (+http://www.fastbot.de)
FastBug http://www.ay-up.com
FastCrawler 3.0.1 (crawler@1klik.dk)
FastSearch Web Crawler for Verizon SuperPages (kevin.watters@fastsearch.com)
Favcollector/2.0 (info@favcollector.com http://www.favcollector.com/)
FavIconizer
favo.eu crawler/0.6 (http://www.favo.eu)
FavOrg
Favorites Checking (http://campulka.net)
Favorites Sweeper v.2.03
Faxobot/1.0
FDM 1.x
FDM 2.x
Feed Seeker Bot (RSS Feed Seeker http://www.MyNewFavoriteThing.com/fsb.php)
Feed24.com
Feed::Find/0.0x
Feedable/0.1 (compatible; MSIE 6.0; Windows NT 5.1)
FeedChecker/0.01
FeedDemon/2.7 (http://www.newsgator.com/; Microsoft Windows XP)
Feedfetcher-Google-iGoogleGadgets; (+http://www.google.com/feedfetcher.html)
Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
FeedForAll rss2html.php v2
FeedHub FeedDiscovery/1.0 (http://www.feedhub.com)
FeedHub MetaDataFetcher/1.0 (http://www.feedhub.com)
Feedjit Favicon Crawler 1.0
Feedreader 3.xx (Powered by Newsbrain)
Feedshow/x.0 (http://www.feedshow.com; 1 subscriber)
FeedshowOnline (http://www.feedshow.com)
Feedster Crawler/3.0; Feedster&#44; Inc.
FeedZcollector v1.x (Platinum) http://www.feeds4all.com/feedzcollector
Felix - Mixcat Crawler (+http://mixcat.com)
fetch libfetch/2.0
fetch/Nutch-1.2 (TCGfetch; http://fetch.thecyberguardian.com; TCGEmail)
FFC Trap Door Spider
Filangy/0.01-beta (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)
Filangy/1.0x (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; filangy-agent@filangy.com)
Filangy/1.0x (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com)
fileboost.net/1.0 (+http://www.fileboost.net)
FileHound x.x
Filtrbox/1.0
FindAnISP.com_ISP_Finder_v99a
Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece)
findfiles.net/0.98 (Robot;test_robot@gmx-topmail.de)
findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0.4 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.0.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.1.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.2 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/2.5 (+http://wortschatz.uni-leipzig.de/findlinks/)
findlinks/x.xxx (+http://wortschatz.uni-leipzig.de/findlinks/)
FineBot
Finjan-prefetch
Firefly/1.0
Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5)
Firefox (kastaneta03@hotmail.com)
Firefox_1.0.6 (kasparek@naparek.cz)
FirstGov.gov Search - POC:firstgov.webmasters@gsa.gov
firstsbot
Flapbot/0.7.2 (Flaptor Crawler; http://www.flaptor.com; crawler at flaptor period com)
FlashGet
FLATARTS_FAVICO
Flexum spider
Flexum/2.0
FlickBot 2.0 RPT-HTTPClient/0.3-3
flunky
fly/6.01 libwww/4.0D
flyindex.net 1.0/http://www.flyindex.net
FnooleBot/2.5.2 (+http://www.fnoole.com/addurl.html)
FocusedSampler/1.0
Folkd.com Spider/0.1 beta 1 (www.folkd.com)
FollowSite Bot ( http://www.followsite.com/bot.html )
FollowSite.com ( http://www.followsite.com/b.html )
Fooky.com/ScorpionBot/ScoutOut; http://www.fooky.com/scorpionbots
Francis/1.0 (francis@neomo.de http://www.neomo.de/)
Franklin Locator 1.8
free-downloads.net download-link validator /0.1
FreeFind.com-SiteSearchEngine/1.0 (http://freefind.com; spiderinfo@freefind.com)
Frelicbot/1.0 +http://www.frelic.com/
FreshDownload/x.xx
FreshNotes crawler&lt; report problems to crawler-at-freshnotes-dot-com
FSurf15a 01
FTB-Bot http://www.findthebest.co.uk/
Full Web Bot 0416B
Full Web Bot 0516B
Full Web Bot 2816B
Fumae2Oo/Nutch-1.4
FuseBulb.Com
FyberSpider (+http://www.fybersearch.com/fyberspider.php)
FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php)
Gagglebot
GAIS Robot/1.0B2
Gaisbot/3.0 (indexer@gais.cs.ccu.edu.tw; http://gais.cs.ccu.edu.tw/robot.php)
Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php)
GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)
Gallent Search Spider v1.4 Robot 2 (http://robot.GallentSearch.com)
gamekitbot/1.0 (+http://www.uchoose.de/crawler/gamekitbot/)
Gamespy_Arcade
GammaSpider/1.0
gazz/x.x (gazz@nttrd.com)
geckobot
Generic Mobile Phone (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
generic_crawler/01.0217/
GenesisBrowser (HTTP 1.1; 0.9; XP SP2; .NET CLR 2.0.50727)
genieBot (http://64.5.245.11/faq/faq.html)
geniebot wgao@genieknows.com
GeoBot/1.0
geobucket/Nutch-1.4
GeonaBot 1.x; http://www.geona.com/
geourl/2.0b2
GeoURLBot 1.0 (http://geourl.org)
GetBot
GetRight/3.x.x
GetRight/4.5xx
GetRight/4.x
GetRight/4.x[a-e]
GetRight/6.1 (Pro)
GetRightPro/6.0beta2
GetWeb/0.1 libwww-perl/5.16
GhostRouteHunter/20021130 (https://www.sixxs.net/tools/grh/; info@sixxs.net)
gigabaz/3.1x (baz@gigabaz.com; http://gigabaz.com/gigabaz/)
Gigabot/2.0 (gigablast.com)
Gigabot/2.0/gigablast.com/spider.html
Gigabot/2.0; http://www.gigablast.com/spider.html
Gigabot/2.0att
Gigabot/3.0 (http://www.gigablast.com/spider.html)
Gigabot/x.0
GigabotSiteSearch/2.0 (sitesearch.gigablast.com)
GNODSPIDER (www.gnod.net)
Go!Zilla 3.x (www.gozilla.com)
Go!Zilla/4.x.x.xx
Go-Ahead-Got-It/1.1
Goblin/0.9 (http://www.goguides.org/)
Goblin/0.9.x (http://www.goguides.org/goblin-info.html)
GoForIt.com
GOFORITBOT ( http://www.goforit.com/about/ )
GoGuides.Org Link Check
GoldenFeed Spider 1.0 (http://www.goldenfeed.com)
Goldfire Server
gonzo1[P] +http://www.suchen.de/popups/faq.jsp
gonzo2[P] +http://www.suchen.de/faq.html
Goofer/0.2
Google Talk
googlebot (larbin2.6.0@unspecified.mail)
Googlebot-Image/1.0
Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html)
Googlebot/2.1 ( http://www.google.com/bot.html)
Googlebot/2.1 ( http://www.googlebot.com/bot.html)
Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Googlebot/2.1+(+http://www.google.com/bot.html)
Googlebot/Googlebot-1.0
Googlebot/Nutch-1.4
Googlebot/Test ( http://www.googlebot.com/bot.html)
Gordon's Spider/Nutch-0.9 (http://www.sharethis.com; gordon@sharethis.com)
GrapeFX/0.3 libwww/5.4.0
great-plains-web-spider/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot.php; jason@flatlandindustries.com)
GreatNews/1.0
GreenBrowser
gridwell (http://search.gridwell.com)
GrigorBot 0.8 (http://www.grigor.biz/bot.html)
Gromit/1.0
grub crawler(http://www.grub.org)
grub-client
gsa-crawler (Enterprise; GID-01422; jplastiras@google.com)
gsa-crawler (Enterprise; GID-01742;gsatesting@rediffmail.com)
gsa-crawler (Enterprise; GIX-02057; dm@enhesa.com)
gsa-crawler (Enterprise; GIX-03519; cknuetter@stubhub.com)
gsa-crawler (Enterprise; GIX-0xxxx; enterprise-training@google.com)
gsa-crawler (Enterprise; M2-B3X6EDEBUC6JB; serveradm@dickinson.edu)
gsa-crawler (Enterprise; M2-B49PXEELPJSAA; webmaster@bethel.edu)
gsa-crawler (Enterprise; M2-JGHL8PEKVA2JS; webmaster@bethel.edu)
gsa-crawler (Enterprise; S5-RLAPLGRTU6NAT; networkadmin@figleaf.com)
gsa-crawler (Enterprise; T3-GNGZGLT73YWB2; search@byu.edu)
gsa-crawler (Enterprise; T3-L39FLJP3RYWGY; dlfr-france.fr-system@atos.net)
gsa-crawler (Enterprise; T3-LLNR7LKTRYSQY; dlfr-france.fr-system@atos.net)
gsa-crawler (Enterprise; T3-MRHJJGX73YWBJ; estauthamer@emidconsult.com)
GSiteCrawler/v1.xx rev. xxx (http://gsitecrawler.com/)
GSLFbot
Guestbook Auto Submitter
Gulliver/1.3
Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)
Gungho/0.08004 (http://code.google.com/p/gungho-crawler/wiki/Index)
GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html)
GurujiImageBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html)
gva/europe (Geneva Web Explorer.; http://www.genevasearch.com/; info@genevasearch.com)
HablaCentro/1.0 (HablaCentro Crawler; http://hablacentro.com/; webmaster@hablacentro.com)
Haier-T10C/1.0 iPanel/2.0 WAP2.0 (compatible; UP.Browser/6.2.2.4; UPG1; UP/4.0; Embedded)
HappyFunBot/1.1
Harvest-NG/1.0.2
Haste/0.12 (HOME: http://haste.kytoon.com/)
Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot)
Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)
Hatena Mobile Gateway/1.0
Hatena Pagetitle Agent/1.0
Hatena RSS/0.3 (http://r.hatena.ne.jp)
HatenaScreenshot/1.0 (checker)
HAZY.SPIDER/Nutch-1.4
hbtronix.spider.2 -- http://hbtronix.de/spider.php
HeinrichderMiragoRobot
HeinrichderMiragoRobot (http://www.miragorobot.com/scripts/deinfo.asp)
Helix/1.x ( http://www.sitesearch.ca/helix/)
HenriLeRobotMirago (http://www.miragorobot.com/scripts/frinfo.asp)
HenrytheMiragoRobot
HenryTheMiragoRobot (http://www.miragorobot.com/scripts/mrinfo.asp)
hgrepurl/1.0
Hi! I'm CsCrawler my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss2005/googlespam/crawler.html RPT-HTTPClient/0.3-3
HiDownload
Hippias/0.9 Beta
HitList
Hitwise Spider v1.0 http://www.hitwise.com
HLoader
holmes/3.11 (http://morfeo.centrum.cz/bot)
holmes/3.9 (onet.pl)
holmes/3.xx (OnetSzukaj/5.0; +http://szukaj.onet.pl)
holmes/x.x
HolmesBot (http://holmes.ge)
HomePageSearch(hpsearch.uni-trier.de)
Homerbot: www.homerweb.com
Honda-Search/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; search@honda-search.com)
HooWWWer/2.1.3 (debugging run) (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-info&lt;at>hiit.fi)
HooWWWer/2.1.x ( http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-info&lt;at>hiit.fi)
HopperBot/1.1 (compatible; heritrix/3.1.0 +http://www.hopper.com)
HotJava/1.0.1/JRE1.1.x
Hotzonu/x.0
HPL/Nutch-0.9 -
htdig/3.1.6 (http://computerorgs.com)
htdig/3.1.6 (unconfigured@htdig.searchengine.maintainer)
htdig/3.1.x (root@localhost)
Html Link Validator (www.lithopssoft.com)
HTML2JPG Blackbox&#44; http://www.html2jpg.com
HTML2JPG Enterprise
HTMLParser/1.x
HTTP Retriever
http://Anonymouse.org/ (Unix)
http://Ask.24x.Info/ (http://narres.it/)
http://hilfe.acont.de/bot.html ACONTBOT
http://OzySoftware.com/Index.html
http://SiteIntel.net Bot
http://www.almaden.ibm.com/cs/crawler
http://www.almaden.ibm.com/cs/crawler [rc1.wf.ibm.com]
http://www.almaden.ibm.com/cs/crawler [wf216]
http://www.baidu.com/Nutch-1.2
http://www.erotis.it/sexyshop (erotis sexy shop - Migliaia di articoli nel nostro store erotico
http://www.eroxxx.it/sexyshop (sexy shop - eroxxx il sex shop per i tuoi perversi desideri. Migliaia di articoli erotici, sex toys come vibratori, bambole gonfibili e tanto altro; http://www.eroxxx.it; info@eroxxx.it)
http://www.ip2location.com
http://www.istarthere.com_spider@istarthere.com
http://www.miosexy.it/sexyshop (sexy shop - miosexy, sexshop, BDSM, vibratori, falli, vagine, lubrificanti, dvd porno, film hard, lingerie - Migliaia di articoli nel nostro sexy shop online.; http://www.miosexy.it; info@miosexy.it)
http://www.monogol.de
http://www.orgasmosexyshop.it/sexyshop (sexy shop - orgasmo sexy shop è l'erotic shop che cercavi, sex toys, we vibe, giochi erotici, bambole gonfibili e tanto altro; http://www.orgasmosexyshop.it; info@orgasmosexyshop.it)
http://www.sexa.it/sexyshop (sexa erotic shop, sexa.it sexy shop BDSM, vibratori, falli, vagine, lubrificanti, dvd porno, film hard, lingerie - Migliaia di articoli nel nostro sexy shop online.; http://www.sexa.it; info@sexa.it)
http://www.sexypepe.it/sexyshop (sexy shop - sexy toys, BDSM, vibratori, falli, vagine, lubrificanti, dvd porno, film hard, lingerie - Migliaia di articoli nel nostro sexy shop online.; http://www.sexypepe.it; info@sexypepe.it)
http://www.sofiasexyshop.it/sexyshop (sexy shop - sexy toys, BDSM, vibratori, falli, vagine, lubrificanti, dvd porno, film hard, lingerie - Migliaia di articoli nel nostro sexy shop online.; http://www.sofiasexyshop.it; marketing@sofiasexyshop.it)
http://www.spaziosexy.com/sexyshop (spazio sexy erotic sexy shop - Tutti i sexy toys, vibratori, bambole gonfiabili, masturbatori, tenga e tanto altro; http://www.spaziosexy.com; info@spaziosexy.com)
http://www.timelyweb.com/
http://www.trendtech.dk/spider.asp)
HTTP::Lite/2.x.x
HTTPEyes
HTTPResume v. 1.x
httpunit/1.5
httpunit/1.x
Huaweisymantecspider (compatible; MSIE 8.0; DSE-support@huaweisymantec.com)
HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/cn/IRL/spider)
HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/en/IRL/spider)
Hybrid/1.2 [en] (OS Independent)
HyperEstraier/1.x.xx
i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search.com)
IAArchiver-1.0
iaskspider
iaskspider2 (iask@staff.sina.com.cn)
ia_archiver
ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)
ia_archiver(OS-Wayback)
ia_archiver-web.archive.org
ia_archiver/1.6
IBrowse/2.2 (AmigaOS 3.5)
IBrowse/2.2 (Windows 3.1)
iCab/2.5.2 (Macintosh; I; PPC)
ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(at)ml(dot)nict(dot)go(dot)jp)
ICC-Crawler(Mozilla-compatible;http://kc.nict.go.jp/icc/crawl.html;icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp)
ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html)
iCCrawler (http://www.iccenter.net)
ICCrawler - ICjobs (http://www.icjobs.de/bot.htm)
ICE Browser/5.05 (Java 1.4.0; Windows 2000 5.0 x86)
ichiro/3.0 (http://help.goo.ne.jp/help/article/1142)
ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)
ichiro/6.0 (http://help.goo.ne.jp/door/crawler.html)
ichiro/x.0 (http://help.goo.ne.jp/door/crawler.html)
ichiro/x.0 (ichiro@nttr.co.jp)
IconSurf/2.0 favicon finder (see http://iconsurf.com/robot.html)
IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html)
ICOO Loader v.x.x.x
ICRA_label_spider/x.0
icsbot-0.1
IDA
ideare - SignSite/1.x
iearthworm/1.0&#44; iearthworm@yahoo.com.cn
IEFav172Free
iFeed.jp/2.0 (www.psychedelix.com/agents/agents.rss; 0 subscribers)
igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html)
iGetter/1.x (Macintosh;G;PPC)
iGetter/2 (Macintosh; U; PPC Mac OS X; en)
IIITBOT/1.1 (Indian Language Web Search Engine; http://webkhoj.iiit.net; pvvpr at iiit dot ac dot in)
ilial/Nutch-0.9 (Ilial&#44; Inc. is a Los Angeles based Internet startup company. For more information please visit http://www.ilial.com/crawler; http://www.ilial.com/crawler; crawl@ilial.com)
ilial/Nutch-0.9-dev
IlseBot/1.x
IlTrovatore-Setaccio ( http://www.iltrovatore.it)
Iltrovatore-Setaccio/0.3-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)
IlTrovatore-Setaccio/1.2 ( http://www.iltrovatore.it/aiuto/faq.html)
Iltrovatore-Setaccio/1.2 (It-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it)
iltrovatore-setaccio/1.2-dev (spidering; http://www.iltrovatore.it/aiuto/.....)
IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it)
ImageVisu/v4.x.x
ImageWalker/2.0 (www.bdbrandprotect.com)
Incutio HttpClient v0.x
IncyWincy data gatherer(webmaster@loopimprovements.com
IncyWincy page crawler(webmaster@loopimprovements.com
IncyWincy(http://www.look.com)
IncyWincy(http://www.loopimprovements.com/robot.html)
IncyWincy/2.1(loopimprovements.com/robot.html)
IndexTheWeb.com Crawler7
Industry Program 1.0.x
Inet library
InetURL/1.0
info@pubblisito.com- (http://www.pubblisito.com) il Sud dei Motori di Ricerca
Infoaxe./Nutch-0.9
infoConveraCrawler/0.8 ( http://www.authoritativeweb.com/crawl)
InfoFly/1.0 (http://www.versions-project.org/)
InfoLink/1.x
INFOMINE/8.0 Adders
INFOMINE/8.0 RemoteServices
INFOMINE/8.0 VLCrawler (http://infomine.ucr.edu/useragents)
InfoNaviRobot(F107)
InfoSeek Sidewinder/0.9
InfoSeek Sidewinder/1.0A
InfoSeek Sidewinder/1.1A
Infoseek SideWinder/1.45 (Compatible; MSIE 10.0; UNIX)
Infoseek SideWinder/2.0B (Linux 2.4 i686)
INGRID/3.0 MT (webcrawler@NOSPAMexperimental.net; http://webmaster.ilse.nl/jsp/webmaster.jsp)
Inktomi Search
InnerpriseBot/1.0 (http://www.innerprise.com/)
innoventage/Nutch-1.0 (poc; www.google.com; proof of concept)
innoventage/Nutch-1.0 (poc; www.innoventage.com; proof of concept)
Insitor.com search and find world wide!
Insitornaut
InstallShield DigitalWizard
integrity/1.6
Intelix/0.x (cs; http://www.microton.cz/intelix/; microton@@microton.cz)
Interarchy/x.x.x (InterarchyCrawler)
Internet Ninja x.0
InternetArchive/0.8-dev(Nutch;http://lucene.apache.org/nutch/bot.html;nutch-agent@lucene.apache
InternetLinkAgent/3.1
InternetSeer.com
intraVnews/1.x
IOI/2.0 (ISC Open Index crawler; http://index.isc.org/; bot@index.isc.org)
IP*Works! V5 HTTP/S Component - by /n software - www.nsoftware.com
IP2MapBot/1.1 &lt;a href=http://www.ip2map.com>http://www.ip2map.com&lt;/a>
IPiumBot laurion(dot)com
IpselonBot/0.xx-beta (Ipselon; http://www.ipselon.com; ipselonbot@ipselon.com)
Iria/1.xxa
IRLbot/1.0 ( http://irl.cs.tamu.edu/crawler)
IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler/)
IrssiUrlLog/0.2
Irvine/1.x.x
ISC Systems iRc Search 2.1
iSiloX/4.xx Windows/32
isurf (tszhu@canada.com)
itim
iTunes/x.x.x
IUPUI Research Bot v 1.9a
iVia Page Fetcher (http://ivia.ucr.edu/useragents.shtml)
iVia/4.0 CanonizeUrl (http://infomine.ucr.edu/iVia/useragents.shtml
IWAgent/ 1.0 - www.brandprotect.com
J-PHONE/3.0/J-SH07
Jabot/6.x (http://odin.ingrid.org/)
Jabot/7.x.x (http://odin.ingrid.org/)
Jabse.com Crawler v.1.0 www.jabse.com/crawler.php//imagecrawler
Jack
jadu-crawler (Enterprise; M2-ANLVCZEKDC2AA; admin@jadu.net)
Jakarta Commons-HttpClient/2.0xxx
Jakarta Commons-HttpClient/3.0-rcx
Jambot/0.1.x (Jambot; http://www.jambot.com/blog; crawler@jambot.com)
Jambot/0.2.1 (Jambot; http://www.jambot.com/blog/static.php?page=webmaster-robot; crawler@jambot.com)
Java 1.1
Java/1.4.1_01
Java/1.6.0_07
Java/1.6.0_10
Java/1.6.0_19
Java/1.6.0_20
Java/1.6.0_24
Java/1.6.0_26
Java/1.7.0_02
Java1.0.21.0
Java1.1.xx.x
Java1.3.0rc1
Java1.3.x
Java1.4.0
JavaCrawler/1.1
Jayde Crawler. http://www.jayde.com
JBH Agent 2.0
jBrowser/J2ME Profile/MIDP-1.0 Configuration/CLDC-1.0 (Google WAP Proxy/1.0)
JCheckLinks/0.1 RPT-HTTPClient/0.3-1
jcrobot
JDK/1.1
Jeode/1.x.x
Jet/Nutch-1.4
Jetbot/1.0
JetBrains Omea Reader 1.0.x (http://www.jetbrains.com/omea_reader/)
JetBrains Omea Reader 2.0 Release Candidate 1 (http://www.jetbrains.com/omea_reader/)
JetCar
Jigsaw/2.2.x W3C_CSS_Validator_JFouffa/2.0
JoBo/1.3 (http://www.matuschek.net/jobo.html)
JoBo/1.x (http://www.matuschek.net/jobo.html)
JoBo/@JOBO_VERSION@(http://www.matuschek.net/jobo.html)
JobSpider_BA/1.1
JOC Web Spider
JordoMedia/1.0 RSS File Reader (http://www.jordomedia.com)
Journster [alpha] (http://journster.com/)
Journster.com RSS/Atom aggregator 0.5 (http://www.journster.com/bot.phtml)
JRTS Check Favorites Utility
JRTwine Software Check Favorites Utility
Jyxobot/x
K-Meleon/0.6 (Windows; U; Windows NT 5.1; en-US; rv:0.9.5) Gecko/20011011
k2spider
kacst/Nutch-1.4
KAIST AITrc Crawler
KakleBot - www.kakle.com/0.1 (KakleBot - www.kakle.com; http:// www.kakle.com/bot.html; support@kakle.com)
kalooga/kalooga-4.0-dev-datahouse (Kalooga; http://www.kalooga.com; info@kalooga.com)
kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler; crawler@kalooga.com)
Kapere (http://www.kapere.com)
kapooha/Nutch-1.4 (crawler; http://www.kapooha.com; aeroganic@gmail.com)
Kavande Crawler 1.0/Nutch-1.4 ( Iranian National Web Crawler ; kh3rad@gmail.com)
Kavande Crawler 1.0/Nutch-1.4 (Iranian National Web Crawler)
Kazehakase/0.x.x.[x]
KDDI-SN22 UP.Browser/6.0.7 (GUI) MMP/1.1 (Google WAP Proxy/1.0)
Kenjin Spider
Kevin http://dznet.com/kevin/
Kevin http://websitealert.net/kevin/
KeywordSearchTool.co/Nutch-1.4 (http://KeywordSearchTool.co/robot)
KE_1.0/2.0 libwww/5.2.8
KFSW-Bot (Version: 1.01 powered by KFSW www.kfsw.de)
kinja-imagebot (http://www.kinja.com/)
kinjabot (http://www.kinja.com)
Kiodia Spider/Nutch-1.4
KIT-Fireball/2.0
KIT-Fireball/2.0 (compatible; Mozilla 4.0; MSIE 5.5)
KiwiStatus/0.3 (NZS.com New Zealand Search; http://www.nzs.com/kiwi-status/)
Klondike/1.50 (WSP Win32) (Google WAP Proxy/1.0)
KnowItAll(knowitall@cs.washington.edu)
Knowledge.com/0.x
Kontiki Client x.xx
Krugle/Krugle&#44;Nutch/0.8+ (Krugle web crawler; http://www.krugle.com/crawler/info.html; webcrawler@krugle.com)
KSbot/1.0 (KnowledgeStorm crawler; http://www.knowledgestorm.com/resources/content/crawler/index.html; crawleradmin@knowledgestorm.com)
KSCrawler/Nutch-1.0 (http://www.kindsight.net/en/kscrawler; crawler@kindsight.net)
kuloko-bot/0.x
kulokobot www.kuloko.com kuloko@backweave.com
kulturarw3/0.1
KummHttp/1.1 (compatible; KummClient; Linux rulez)
KWC-KX9/1109 UP.Browser/6.2.3.9.g.1.107 (GUI) MMP/2.0 UP.Link/6.3.0.0.0
Labrador/0.2; http://ir.dcs.gla.ac.uk/labrador; craigm@dcs.gla.ac.uk
Lachesis
lanshanbot/1.0
lanshanbot/1.0 (+http://search.msn.com/msnbot.htm)
LapozzBot/1.4 ( http://robot.lapozz.com)
LapozzBot/1.5 (+http://robot.lapozz.hu)
larbin (samualt9@bigfoot.com)
LARBIN-EXPERIMENTAL (efp@gmx.net)
larbin_2.1.1 larbin2.1.1@somewhere.com
larbin_2.2.0 (crawl@compete.com)
larbin_2.2.1_de_Viennot (Laurent.Viennot@inria.fr)
larbin_2.2.2 (sugayama@lab7.kuis.kyoto-u.ac.jp)
larbin_2.2.2_guillaume (guillaume@liafa.jussieu.fr)
larbin_2.6.0 (larbin2.6.0@unspecified.mail)
larbin_2.6.1 (larbin2.6.1@unspecified.mail)
larbin_2.6.2 (hamasaki@grad.nii.ac.jp)
larbin_2.6.2 (larbin2.6.2@unspecified.mail)
larbin_2.6.2 (listonATccDOTgatechDOTedu)
larbin_2.6.2 (pimenas@systems.tuc.gr)
larbin_2.6.2 (tom@lemurconsulting.com)
larbin_2.6.2 (vitalbox1@hotmail.com)
larbin_2.6.3 (fuxiang90@gamil.com)
larbin_2.6.3 (gordon0640@gmail.com)
larbin_2.6.3 (hhgg20000@21cn.com)
larbin_2.6.3 (hwq_super@126.com)
larbin_2.6.3 (larbin2.6.3@unspecified.mail)
larbin_2.6.3 (ltaa_web_crawler@groupes.epfl.ch)
larbin_2.6.3 (wgao@genieknows.com)
larbin_2.6.3_for_(http://cosco.hiit.fi/search/) tsilande@hiit.fi
larbin_2.6_basileocaml (basile.starynkevitch@cea.fr)
larbin_devel (http://pauillac.inria.fr/~ailleret/prog/larbin/)
lawinfo-crawler/Nutch-0.9-dev (Crawler for lawinfo.com pages; http://www.lawinfo.com; webmaster@lawinfo.com)
lb-spider/Mozilla/5.0 Gecko/20100101 Firefox/10.0.2 (lb-spider; http://www.linkbutler.de/spider; spider@linkbutler.de)
lc/$ROADS::Version libwww-perl/5.00
lcabotAccept: */*
LeapTag/0.8.1.beta081.r3750 (compatible; Mozilla 4.0; MSIE 5.5; robot@yoriwa.com)
LECodeChecker/3.0 libgetdoc/1.0
LeechGet 200x (www.leechget.de)
LEIA/2.90
LEIA/3.01pr (LEIAcrawler; [SNIP])
LetsCrawl.com/1.0 +http://letscrawl.com/
LexiBot/1.00
LG-LX260 POLARIS-LX260/2.0 MMP/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.1
LG/U8138/v1.0
Libby_1.1/libwww-perl/5.47
libcurl-agent/1.0
LibertyW (+http://www.lw01.com)
librabot/2.0 (+http://academic.research.microsoft.com/)
libWeb/clsHTTP -- hiongun@kt.co.kr
libwww-perl/5.41
libwww-perl/5.45
libwww-perl/5.48
libwww-perl/5.50
libwww-perl/5.52 FP/2.1
libwww-perl/5.52 FP/4.0
libwww-perl/5.53
libwww-perl/5.63
libwww-perl/5.64
libwww-perl/5.65
libwww-perl/5.800
libwww/5.3.2
Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/)
Liferea/1.x.x (Linux; es_ES.UTF-8; http://liferea.sf.net/)
LightningDownload/1.0beta2
LightningDownload/1.x.x
LightningDownload/1.x.x [Accelerated x]
Lijit Crawler (+http://www.lijit.com/robot/crawler)
LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/; info(a)lijit(d)com)
Lincoln State Web Browser
Link Valet Online 1.x
LinkAlarm/2.x
Linkbot
Linkbot x.0
LinkCheck (linkcheck@inter7.com http://www.inter7.com/linkcheck)
LinkChecker/3.4 (+http://linkchecker.sourceforge.net/)
LinkChecker/4.9 (+http://linkchecker.sourceforge.net/)
LinkChecker/5.1 (+http://linkchecker.sourceforge.net/)
LinkChecker/7.0 (+http://linkchecker.sourceforge.net/)
LinkChecker/7.2 (+http://linkchecker.sourceforge.net/)
LinkChecker/7.3 (+http://linkchecker.sourceforge.net/)
LinkChecker/7.4 (+http://linkchecker.sourceforge.net/)
LinkChecker/7.5 (+http://linkchecker.sourceforge.net/)
LinkLint-checkonly/2.x.x
LinkLint-spider/2.x.x
linknzbot
LinkPimpin v1.0
LinkProver 2.1
Links (0.9x; Linux 2.4.7-10 i686)
Links (0.9xpre12; Linux 2.2.14-5.0 i686; 80x24)
Links (2.xpre7; Linux 2.4.18 i586; x)
Links - http://gossamer-threads.com/scripts/links/
Links 2.0 (http://gossamer-threads.com/scripts/links/)
Links SQL (http://gossamer-threads.com/scripts/links-sql/)
Links4US-Crawler&#44; (+http://links4us.com/)
LinkScan/11.0beta2 UnixShareware robot from Elsop.com (used by Indiafocus/Indiainfo)
LinkScan/9.0g Unix
LinkScan/x.x Unix
LinksManager.com (http://linksmanager.com/linkchecker.html)
LinkSonar/1.35
LinkSweeper/1.x
LinkWalker
link_check3.plx libwww-perl/5.65
ListBidBot (freelance job spider http://listbid.com)&lt;a href=http://listbid.com>Freelance&lt;/a>
LitlrBot (http://litlr.me/bot.html)
LiveTrans/Nutch-0.9 (maintainer: cobain at iis dot sinica dot edu dot tw; http://wkd.iis.sinica.edu.tw/LiveTrans/)
Llaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html)
LMQueueBot/0.2
lmspider (lmspider@scansoft.com)
lmspider/Nutch-0.9-dev (For research purposes.; www.nuance.com; lmspider@nuance.com)
LNSpiderguy
LocalBot/1.0 ( http://www.localbot.co.uk/)
LocalcomBot/1.2.x ( http://www.local.com/bot.htm)
LOCIndexer/Nutch-1.0
Lockstep Spider/1.0
Look.com
Lotus-Notes/4.5 ( Windows-NT )
LotusDiscovery/x.0 (compatible; Mozilla 4.0; MSIE 4.01; Windows NT)
Lovel as 1.0 ( +http://www.everatom.com)
LTI/LemurProject Nutch Spider/Nutch-1.0-dev (lti crawler for CMU; http://www.lti.cs.cmu.edu; changkuk at cmu dot edu)
LTI/LemurProject Nutch Spider/Nutch-1.0-dev (Research spider using Nutch; http://www.lemurproject.org; mhoy@cs.cmu.edu)
luchs.at URL checker
Lunascape
lwp-trivial/1.32
lwp-trivial/1.34
lwp-trivial/1.35
LWP::Simple/5.22
LWP::Simple/5.36
LWP::Simple/5.48
LWP::Simple/5.50
LWP::Simple/5.51
LWP::Simple/5.53
LWP::Simple/5.63
LWP::Simple/5.803
Lycos_Spider_(modspider)
Lycos_Spider_(T-Rex)
Lynx/2-4-2 (Bobcat/0.5 [DOS] Jp Beta04)
Lynx/2.6 libwww-FM/2.14
Lynx/2.8 (;http://seebot.org)
Lynx/2.8.3dev.9 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.6
Lynx/2.8.4rel.1 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.6c (human-guided@lerly.net)
Lynx/Lynx(2.0)
Lynx/Lynx(2.2) (Lynx)
Lynx/Lynx(8899)
Lynx/Nutch-1.4
Mac Finder 1.0.xx
Mackster( http://www.ukwizz.com )
Mag-Net
MagicWML/1.0 (forcewml)
magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
magpie-crawler/1.1 (U; Linux i386; en-GB; +http://www.brandwatch.net)
MagpieRSS/0.7x (+http://magpierss.sf.net)
Mahiti.Com/Mahiti Crawler-1.0 (Mahiti.Com; http://mahiti.com ; mahiti.com)
mahonie, neofonie search:robot (http://spider.neofonie.de)/Nutch-1.4 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to spider@neofonie.de; http://spider.neofonie.de)
mahonie, neofonie search:robot/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to spider@neofonie.de; http://spider.neofonie.de; spider@neofonie.de)
Mail.Ru/1.0
Mail.RU/2.0
mailto:webcraft@bea.com
mammoth/1.0 ( http://www.sli-systems.com/)
MANTIS/Nutch-1.4
MantraAgent
MapoftheInternet.com ( http://MapoftheInternet.com)
Mariner/5.1b [de] (Win95; I ;Kolibri gncwebbot)
Marketwave Hit List
Martini
Marvin v0.3
MaSagool/1.0 (MaSagool; http://sagool.jp/; info@sagool.jp)
Mass Downloader 2.x
MasterSeek
Mata Hari/2.00
Matrix S.p.A. - FAST Enterprise Crawler 6 (Unknown admin e-mail address)
maxomobot/dev-20051201 (maxomo; http://67.102.134.34:4047/MAXOMO/MAXOMObot.html; maxomobot@maxomo.com)
McBot/5.001 (windows; U; NT4.0; en-us)
MDbot/1.0 (+http://www.megadownload.net/bot.html)
Media Player Classic
MediaCrawler-1.0 (Experimental)
Mediapartners-Google/2.1 ( http://www.googlebot.com/bot.html)
MediaSearch/0.1
MegaSheep v1.0 (www.searchuk.com internet sheep)
Megite2.0 (http://www.megite.com)
Mercator-1.x
Mercator-2.0
Mercator-Scrub-1.1
merocrawl/Nutch-1.4 (merobase crawler; www.merobase.com; info@merobase.com)
Metaeuro Web Crawler/0.2 (MetaEuro Web Search Clustering Engine; http://www.metaeuro.com; crawler at metaeuro dot com)
MetaGer-LinkChecker
MetagerBot/0.8-dev (MetagerBot; http://metager.de; )
MetaGer_PreChecker0.1
MetaProducts Download Express/1.x
Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner.de/)
metatagsdir/0.7 (+http://metatagsdir.com/directory/)
MFC Foundation Class Library 4.0
MFC_Tear_Sample
MFHttpScan
MicroBaz
Microsoft Data Access Internet Publishing Provider Cache Manager
Microsoft Data Access Internet Publishing Provider DAV
Microsoft Data Access Internet Publishing Provider Protocol Discovery
Microsoft Log Parser 2.2
Microsoft Small Business Indexer
Microsoft URL Control - 6.00.8xxx
MicrosoftPrototypeCrawler (How's my crawling? mailto:newbiecrawler@hotmail.com)
Microsoft_Internet_Explorer_5.00.438 (fjones@isd.net)
MIIxpc/4.2
Mindjet MindManager
minibot
miniRank/1.6 (Website ranking; www.minirank.com; robot)
MiracleAlphaTest
MireoBot
Missauga Locate 1.0.0
Missigua Locator 1.9
Missouri College Browse
Mister Pix II 2.02a
Mister PiX version.dll
Misterbot-Nutch/0.7.1 (Misterbot-Nutch; http://www.misterbot.fr; admin@misterbot.fr)
Miva (AlgoFeedback@miva.com)
Mizzu Labs 2.2
MJ12bot/vx.x.x (http://majestic12.co.uk/bot.php?+)
MJ12bot/vx.x.x (http://www.majestic12.co.uk/projects/dsearch/mj12bot.php)
MJBot (SEO assessment)
MLBot (www.metadatalabs.com)
MLBot (www.metadatalabs.com/mlbot)
MnogoSearch/3.2.xx
MnoGoSearch/3.3.9
Mo College 1.9
moget/x.x (moget@goo.ne.jp)
mogimogi/1.0
moiNAG 0.02
MojeekBot/0.x (archi; http://www.mojeek.com/bot.html)
MOMspider/1.00 libwww-perl/0.40
monkeyagent
MoonBrowser (version 0.41 Beta4)
Moreoverbot/x.00 (+http://www.moreover.com)
Morris - Mixcat Crawler ( http://mixcat.com)
Motoricerca-Robots.txt-Checker/1.0 (http://tool.motoricerca.info/robots-checker.phtml)
Motorola-V3m Obigo
Mouse-House/7.4 (spider_monkey spider info at www.mobrien.com/sm.shtml)
MovableType/x.x
mozDex/0.xx-dev (mozDex; http://www.mozdex.com/en/bot.html; spider@mozdex.com)
Mozi!
Mozilla
Mozilla (libwhisker/2.4)
Mozilla (Mozilla@somewhere.com)
Mozilla 4.0(compatible; BotSeer/1.0; +http://botseer.ist.psu.edu)
Mozilla/1.1 (compatible; MSPIE 2.0; Windows CE)
Mozilla/1.10 [en] (Compatible; RISC OS 3.70; Oregano 1.10)
Mozilla/1.22 (compatible; MSIE 2.0; Windows 95)
Mozilla/1.22 (compatible; MSIE 2.0d; Windows NT)
Mozilla/1.22 (compatible; MSIE 5.01; PalmOS 3.0) EudoraWeb 2
Mozilla/2.0
Mozilla/2.0 (compatible; AOL 3.0; Mac_PowerPC)
Mozilla/2.0 (Compatible; AOL-IWENG 3.0; Win16)
Mozilla/2.0 (compatible; Ask Jeeves)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/webmasters.shtml)
Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://sp.ask.com/docs/about/tech_crawling.html)
Mozilla/2.0 (compatible; EZResult -- Internet Search Engine)
Mozilla/2.0 (compatible; MS FrontPage x.0)
Mozilla/2.0 (compatible; MSIE 2.1; Mac_PowerPC)
Mozilla/2.0 (compatible; MSIE 3.02; Update a; AK; Windows NT)
Mozilla/2.0 (compatible; MSIE 3.02; Update a; AOL 3.0; Windows 95)
Mozilla/2.0 (compatible; MSIE 3.0; AK; Windows 95)
Mozilla/2.0 (compatible; MSIE 3.0; Windows 3.1)
Mozilla/2.0 (compatible; MSIE 3.0B; Win32)
Mozilla/2.0 (compatible; NEWT ActiveX; Win32)
Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E)
Mozilla/2.0 compatible; Check&amp;Get 1.1x (Windows 98)
Mozilla/2.01 (Win16; I)
Mozilla/2.02Gold (Win95; I)
Mozilla/3.0 (compatible)
Mozilla/3.0 (compatible; AvantGo 3.2)
Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com)
Mozilla/3.0 (compatible; HP Web PrintSmart 04b0 1.0.1.34)
Mozilla/3.0 (compatible; Indy Library)
Mozilla/3.0 (compatible; Linkman)
Mozilla/3.0 (compatible; MuscatFerret/1.5.4; claude@euroferret.com)
Mozilla/3.0 (compatible; MuscatFerret/1.5; olly@muscat.co.uk)
Mozilla/3.0 (compatible; MuscatFerret/1.6.x; claude@euroferret.com)
Mozilla/3.0 (compatible; netart generator/1.0; libwww-perl/5.64)
Mozilla/3.0 (compatible; NetPositive/2.2)
Mozilla/3.0 (compatible; Opera/3.0; Windows 3.1) v3.1
Mozilla/3.0 (compatible; Opera/3.0; Windows 95/NT4) 3.2
Mozilla/3.0 (compatible; PerMan Surfer 3.0; Win95)
Mozilla/3.0 (compatible; REL Software Web Link Validator 2.x)
Mozilla/3.0 (compatible; scan4mail (advanced version) http://www.peterspages.net/?scan4mail)
Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com)
Mozilla/3.0 (compatible; Web Link Validator 2.x)Web Link Validator http://www.relsoftware.com/ link validation software
Mozilla/3.0 (compatible; WebCapture x.x; Auto; Windows)
Mozilla/3.0 (compatible; Webinator-DEV01.home.iprospect.com/2.56)
Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56)
Mozilla/3.0 (compatible;)
Mozilla/3.0 (Compatible;Viking/1.8)
Mozilla/3.0 (DreamPassport/3.0)
Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)
Mozilla/3.0 (Liberate DTV 1.1)
Mozilla/3.0 (Planetweb/2.100 JS SSL US; Dreamcast US)
Mozilla/3.0 (Slurp.so/Goo; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/3.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/3.0 (Vagabondo/1.1 MT; webcrawler@NOSPAMwise-guys.nl; http://webagent.wise-guys.nl/)
Mozilla/3.0 (Vagabondo/1.x MT; webagent@wise-guys.nl; http://webagent.wise-guys.nl/)
Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)
Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMwise-guys.nl; http://webagent.wise-guys.nl/)
Mozilla/3.0 (Win16; I)
Mozilla/3.0 (Win95; I)
Mozilla/3.0 (WinNT; I)
Mozilla/3.0 (WorldGate Gazelle 3.5.1 build 11; FreeBSD2.2.8-STABLE)
Mozilla/3.0 (X11; I; OSF1 V4.0 alpha)
Mozilla/3.0 NAVIO_AOLTV (11; 13; Philips; PH200; 1; R2.0C36_AOL.0110OPTIK; R2.0.0139d_OPTIK)
Mozilla/3.0 WebTV/1.2 (compatible; MSIE 2.0)
Mozilla/3.01 (compatible; AmigaVoyager/2.95; AmigaOS/MC680x0)
Mozilla/3.01 (Compatible; Links2Go Similarity Engine)
Mozilla/3.01 (compatible; Netbox/3.5 R92; Linux 2.2)
Mozilla/3.01-C-MACOS8 (Macintosh; I; PPC)
Mozilla/3.01Gold (X11; I; Linux 2.0.32 i486)
Mozilla/3.01Gold (X11; I; SunOS 5.5.1 sun4m)
Mozilla/3.01SGoldC-SGI (X11; I; IRIX 6.3 IP32)
Mozilla/3.04 (compatible; ANTFresco/2.13; RISC OS 4.02)
Mozilla/3.04 (compatible; NCBrowser/2.35; ANTFresco/2.17; RISC OS-NC 5.13 Laz1UK1309)
Mozilla/3.04 (compatible;QNX Voyager 2.03B ;Photon)
Mozilla/3.x (I-Opener 1.1; Netpliance)
Mozilla/4.0
Mozilla/4.0 (agadine3.0) www.agada.de
Mozilla/4.0 (Compatible); URLBase 6
Mozilla/4.0 (compatible: AstraSpider V.2.1 : astrafind.com)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; .NET CLR 2.0;)
Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
Mozilla/4.0 (compatible; &lt;a href=http://www.reget.com>ReGet Deluxe 5.1&lt;/a>; Windows NT 5.1)
Mozilla/4.0 (compatible; Advanced Email Extractor v2.xx)
Mozilla/4.0 (compatible; Arachmo)
Mozilla/4.0 (compatible; BorderManager 3.0)
Mozilla/4.0 (compatible; BOTW Spider; +http://botw.org)
Mozilla/4.0 (compatible; B_L_I_T_Z_B_O_T)
Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0)
Mozilla/4.0 (compatible; Check&amp;Get 3.0; Windows NT)
Mozilla/4.0 (compatible; ChristCrawler.com ChristCrawler@ChristCENTRAL.com)
Mozilla/4.0 (compatible; crawlx&#44; crawler@trd.overture.com)
Mozilla/4.0 (compatible; DAUMOA-video; +http://ws.daum.net/aboutkr.html)
Mozilla/4.0 (compatible; DepSpid/5.0x; +http://about.depspid.net)
Mozilla/4.0 (compatible; DnloadMage 1.0)
Mozilla/4.0 (compatible; FastCrawler3 support-fastcrawler3@fast.no)
Mozilla/4.0 (compatible; FDSE robot)
Mozilla/4.0 (compatible; GPU p2p crawler http://gpu.sourceforge.net/search_engine.php)
Mozilla/4.0 (compatible; grub-client-0.2.x; Crawl your stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-0.3.x; Crawl your own stuff with http://grub.org)
Mozilla/4.0 (compatible; grub-client-2.x)
Mozilla/4.0 (compatible; http://search.thunderstone.com/texis/websearch/about.html)
Mozilla/4.0 (compatible; ibisBrowser)
Mozilla/4.0 (compatible; ICS 1.2.xxx)
Mozilla/4.0 (compatible; IE-Favorites-Check-0.5)
Mozilla/4.0 (compatible; Iplexx Spider/1.0 http://www.iplexx.at)
Mozilla/4.0 (compatible; KeepNI web site monitor)
Mozilla/4.0 (compatible; Link Utility; http://net-promoter.com)
Mozilla/4.0 (compatible; Lotus-Notes/5.0; Windows-NT)
Mozilla/4.0 (compatible; MSIE 4.01; AOL 4.0; Windows 98)
Mozilla/4.0 (compatible; MSIE 4.01; Mac_PowerPC)
Mozilla/4.0 (compatible; MSIE 4.01; MSIECrawler; Windows 95)
Mozilla/4.0 (compatible; MSIE 4.01; Vonna.com b o t)
Mozilla/4.0 (compatible; MSIE 4.01; Windows 95)
Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; MSN Companion 2.0; 800x600; Compaq)
Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19.123.2.733) OrangeBot-Mobile 2008.0 (mobilesearch.support@orange-ftgroup.com)
Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPS; 240x320)
Mozilla/4.0 (compatible; MSIE 4.01; Windows NT Windows CE)
Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)
Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot) Microsoft
Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 5.0 Robot)
Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 6.0 Robot)
Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) ACR
Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Indonesia Interactive
Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) WebQuest Designs
Mozilla/4.0 (compatible; MSIE 5.01; Windows 95) via &lt;B>Avirt Gateway Server&lt;/B> v4.0
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) (samualt9@bigfoot.com)
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; FeedFinder-2.0; http://bloggz.se/crawler)
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; NetCaptor 6.5.0RC1)
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT)
Mozilla/4.0 (compatible; MSIE 5.0; AOL 5.0; Windows 95; DigExt; Gateway2000; sureseeker.com)
Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC; AtHome021)
Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience: http://www.netnose.com)
Mozilla/4.0 (compatible; MSIE 5.0; Win32) via proxy gateway CERN-HTTPD/3.0 libwww/2.17
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/)
Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6
Mozilla/4.0 (compatible; MSIE 5.0; Windows ME) Opera 5.11 [en]
Mozilla/4.0 (compatible; MSIE 5.0; Windows ME; Link Checker 2.x.xx http://www.kyosoft.com)
Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)
Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt)
Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent
Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com)
Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; www.psychedelix.com)
Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; www.psychedelix.com/; http://www.galaxy.com/info/crawler.html)
Mozilla/4.0 (compatible; MSIE 5.0; YANDEX)
Mozilla/4.0 (compatible; MSIE 5.5; AOL 4.0; Windows 98; GoBeez (www.gobeez.com))
Mozilla/4.0 (compatible; MSIE 5.5; Windows 95; Transmission Segment; Hotbar 2.0)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Crazy Browser 1.x.x)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; KITV4.7 Wanadoo)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; SAFEXPLORER TL)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; SYMPA; Katiesoft 7; SimulBrowse 3.0)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; BTinternet V8.1)
Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; QXW03018)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Active Cache Request
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Fetch API Request
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; .NET CLR 1.0.3705)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; AIRF)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; AspTear 1.5)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; N_o_k_i_a)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461) RPT-HTTPClient/0.3-3E
Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.singingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revision: 3.10
Mozilla/4.0 (compatible; MSIE 6.0; AOL 9.0; Windows 98; .NET CLR 1.1.4322; MEGAUPLOAD 2.0)
Mozilla/4.0 (compatible; MSIE 6.0; AOL 9.0; Windows NT 5.1; SV1; HbTools 4.7.2)
Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Skampy/0.9.x [en]
Mozilla/4.0 (compatible; MSIE 6.0; TargetSeek/1.0; +http://www.targetgroups.net/TargetSeek.html)
Mozilla/4.0 (compatible; MSIE 6.0; Win32) WebWasher 3.0
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98) REL Software Web Link Validator 2.x)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98) Web Link Validator 2.x)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Net M@nager V3.02 - www.vinn.com.au)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; support@illumit.com; http://www.illumit.com/Products/weblight/)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; http://www.Abolimba.de)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 1.1.4322; MEGAUPLOAD 2.0; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) (larbin2.6.3@unspecified.mail)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; Lunascape 2.1.3)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Google Wireless Transcoder;)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP entries t_st; http://tuezilla.de/t_st-odp-entries-agent.html)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP links test; http://tuezilla.de/test-odp-links-agent.html)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ZoomSpider.net bot; .NET CLR 1.1.4322)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) (dns_admin@c-a-s-h.com)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 1.0.3705)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Covac UPPS Cathan 1.2.5;)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Crayon Crawler; snprtz|T04056566514940; (R1 1.5))
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Deepnet Explorer)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FunWebProducts; SV1; .NET CLR 1.0.3705; Media Center PC 4.0; InfoPath.1; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; msn OptimizedIE8;ESMX; Alexa Toolbar; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; heritrix/1.3.0 http://www.cs.washington.edu/research/networking/websys/)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 3.0)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; iOpus-I-M)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; iRider 2.21.1108; FDM)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; KKman3.0)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MathPlayer2.0)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Maxthon)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; PeoplePal 3.0; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461; IOpener Release 1.1.04)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QihooBot 1.0 qihoobot@qihoo.net)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SIMBAR Enabled; InfoPath.1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; StumbleUpon.com 1.760; .NET CLR 1.1.4322)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Embedded Web Browser from: http://bsalsa.com/; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html )
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AskTbMYC-ST/5.14.1.20007; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; BTRS1088340; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; BTRS28059; GTB7.2; .NET CLR 2.0.50727; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; BTRS28059; GTB7.3; .NET CLR 2.0.50727; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; DX-Browser 5.0.0.0)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts; ezPeer+ v1.0 Beta (0.4.1.98); ezPeer+ v1.0 (0.5.0.00); .NET CLR 1.1.4322; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; IBP; .NET CLR 1.1.4322)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.1; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.2; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; AskTB5.6; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Media Center PC 3.0; .NET CLR 1.0.3705; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MRA 4.3 (build 01218))
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MSIECrawler)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)
Mozilla/4.0 (compatible; MSIE 6.1; Windows NT)
Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)/Nutch-1.3
Mozilla/4.0 (compatible; MSIE 7.0; Win32) Link Commander 4.0
Mozilla/4.0 (compatible; MSIE 7.0; Windows 98; Maxthon; ZangoToolbar 7.3.85; SpamBlockerUtility 7.9.95)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 3.0.04506.30; .NET CLR 3.5.30729)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; bgft)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB5; User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://bsalsa.com) ; .NET CLR 2.0.50727)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; IPMS/4669080A-14AF91D9EAE; TCO_20091110170859; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; Maxthon; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0)
Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.1; .NET CLR 1.1.4322; InfoPath.1; .NET CLR 2.0.50727)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET CLR 1.1.4322; AskTbCLM/5.8.0.12304)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; AskTbNRO/5.14.1.20007; InfoPath.2)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; BRI/1)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS122237; SIMBAR={1B09D51F-67CD-4798-A3B7-04E7920571B8}; GTB7.2; EasyBits GO v1.0; InfoPath.1; .NET CLR 2.0.50727; .NET CLR 1.1.4322; MSN Optimized;ES; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.5; OfficeLivePatch.1.3; BRI/2; .NET4.0C; AskTB5.6)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS122288; GTB7.2; AskTbSTC-US/5.13.1.18107; FDM; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS122529; GTB7.3; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.3)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS122701; GTB7.3; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E; InfoPath.3; BRI/2)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS5841; GTB7.3; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.2; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.2; .NET CLR 2.0.50727; InfoPath.2; AskTbFF/5.9.1.14019)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.3; (R1 1.6); .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.3; EasyBits GO v1.0; .NET CLR 2.0.50727)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.1; .NET CLR 2.0.50727; BRI/1)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; MRSPUTNIK 2, 4, 0, 491; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; Orange 8.0; GTB6.3; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from: http://bsalsa.com/; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30618; OfficeLiveConnector.1.3; OfficeLivePatch.1.3)
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; .NET CLR 3.5.30729)
Mozilla/4.0 (compatible; MSIE enviable; DAUMOA 2.0; DAUM Web Robot; Daum Communications Corp.&#44; Korea; +http://ws.daum.net/aboutkr.html)
Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.1; DAUM Web Robot; Daum Communications Corp.&#44; Korea)
Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/delete_main.asp)
Mozilla/4.0 (compatible; Netcraft Web Server Survey)
Mozilla/4.0 (compatible; NetPromoter Spider;http://www.net-promoter.com/)
Mozilla/4.0 (compatible; Opera/3.0; Windows 4.10) 3.51 [en]
Mozilla/4.0 (compatible; Powermarks/3.5; Windows 95/98/2000/NT)
Mozilla/4.0 (compatible; RSS Popper)
Mozilla/4.0 (compatible; SiteKiosk 4.0; MSIE 5.0; Windows 98; SiteCoach 1.0)
Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)
Mozilla/4.0 (compatible; SPENG)
Mozilla/4.0 (compatible; SuperCleaner 2.xx; Windows 98)
Mozilla/4.0 (compatible; Synapse)
Mozilla/4.0 (compatible; WebCapture 3.0; Windows)
Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)
Mozilla/4.0 (compatible; WSN Links)
Mozilla/4.0 (compatible; www.euro-directory.com; urlchecker1.0)
Mozilla/4.0 (compatible; www.galaxy.com)
Mozilla/4.0 (compatible; www.linkguard.com Linkguard Online 1.0; Windows NT)
Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid)
Mozilla/4.0 (compatible; Yahoo Japan; for robot study; kasugiya)
Mozilla/4.0 (compatible;)
Mozilla/4.0 (compatible;MSIE 6.0; Windows NT 5.0; H010818)
Mozilla/4.0 (fantomBrowser)
Mozilla/4.0 (fantomCrew Browser)
Mozilla/4.0 (hhjhj@yahoo.com)
Mozilla/4.0 (JemmaTheTourist;http://www.activtourist.com)
Mozilla/4.0 (MobilePhone PM-8200/US/1.0) NetFront/3.x MMP/2.0
Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible; Googlebot/2.1; http://www.google.com/bot.html)
Mozilla/4.0 (Mozilla; http://www.mozilla.org/docs/en/bot.html; master@mozilla.com)
Mozilla/4.0 (Sleek Spider/1.2)
Mozilla/4.0 (Toread-Crawler/1.1; +http://news.toread.cc/crawler.php)
Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; wn.furlbot@looksmart.net)
Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)
Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net; http://www.WISEnutbot.com)
Mozilla/4.0 compatible ZyBorg/1.0 for Homepage (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)
Mozilla/4.0 efp@gmx.net
Mozilla/4.0 WebTV/2.6 (compatible; MSIE 4.0)
Mozilla/4.0 [en] (Ask Jeeves Corporate Spider)
Mozilla/4.0(compatible; Zealbot 1.0)
Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1)
Mozilla/4.01 (compatible; NORAD National Defence Network)
Mozilla/4.01 [en](Win95;I)
Mozilla/4.02 [en] (X11; I; SunOS 5.6 sun4u)
Mozilla/4.03 (111111)
Mozilla/4.04 (compatible; Dulance bot; +http://www.dulance.com/bot.jsp)
Mozilla/4.04 [en] (X11; I; HP-UX B.10.20 9000/712)
Mozilla/4.04 [en] (X11; I; IRIX 5.3 IP22)
Mozilla/4.05 (Macintosh; I; 68K Nav)
Mozilla/4.05 (Macintosh; I; PPC Nav)
Mozilla/4.05 [en] (X11; I; SunOS 4.1.4 sun4m)
Mozilla/4.08 [en] (Win98; U ;Nav)
Mozilla/4.08 [en] (WinNT; U)
Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_TrueRobot/1.4 libwww/5.2.8
Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_VoilaBot/1.6 libwww/5.3.2
Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)
Mozilla/4.5 (compatible; iCab 2.5.3; Macintosh; I; PPC)
Mozilla/4.5 (compatible; OmniWeb/4.0.5; Mac_PowerPC)
Mozilla/4.5 (compatible; OmniWeb/4.1-beta-1; Mac_PowerPC)
Mozilla/4.5 RPT-HTTPClient/0.3-2
Mozilla/4.5 [en]C-CCK-MCD {RuralNet} (Win98; I)
Mozilla/4.5b1 [en] (X11; I; Linux 2.0.35 i586)
Mozilla/4.6 [en] (http://www.cnet.com/)
Mozilla/4.61 [de] (OS/2; I)
Mozilla/4.61 [en] (X11; U; ) - BrowseX (2.0.0 Windows)
Mozilla/4.7
Mozilla/4.7 (compatible; http://eidetica.com/spider)
Mozilla/4.7 (compatible; Intelliseek; http://www.intelliseek.com)
Mozilla/4.7 (compatible; OffByOne; Windows 98) Webster Pro V3.2
Mozilla/4.7 (compatible; Whizbang)
Mozilla/4.7 (compatible; WhizBang; http://www.whizbang.com/crawler)
Mozilla/4.7 [en](BecomeBot@exava.com)
Mozilla/4.7 [en](Exabot@exava.com)
Mozilla/4.7 [en]C-CCK-MCD {Yahoo;YIP052400} (Win95; I)
Mozilla/4.72 [en] (BACS http://www.ba.be)
Mozilla/4.72C-CCK-MCD Caldera Systems OpenLinux [en] (X11; U; Linux 2.2.14 i686)
Mozilla/4.75C-ja [ja] (X11; U; OSF1 V5.1 alpha)
Mozilla/4.76 (Windows 98; U) Opera 5.12 [en]
Mozilla/4.76 [en] (X11; U; FreeBSD 4.4-STABLE i386)
Mozilla/4.76 [en] (X11; U; SunOS 5.7 sun4u)
Mozilla/4.77C-SGI [en] (X11; U; IRIX 6.5 IP32)
Mozilla/5.0 ((compatible;+Baiduspider/2.0;++http://www.baidu.com/search/spider.html))
Mozilla/5.0 (+http://www.eurekster.com/mammoth) Mammoth/0.1
Mozilla/5.0 (+http://www.sli-systems.com/) Mammoth/0.1
Mozilla/5.0 (Clustered-Search-Bot/1.0; support@clush.com; http://www.clush.com/)
Mozilla/5.0 (compatible) GM RSS Panel X
Mozilla/5.0 (compatible; +http://www.evri.com/evrinid)
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620
Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620
Mozilla/5.0 (compatible; Abonti/0.8 - http://www.abonti.com)
Mozilla/5.0 (compatible; AcoonBot/4.10.6; +http://www.acoon.de/robot.asp)
Mozilla/5.0 (compatible; AcoonBot/4.10.7; +http://www.acoon.de/robot.asp)
Mozilla/5.0 (compatible; AhrefsBot/1.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/2.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsBot/3.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/)
Mozilla/5.0 (compatible; aiHitBot/1.1; +http://www.aihit.com/)
Mozilla/5.0 (compatible; AnsearchBot/1.x; +http://www.ansearch.com.au/)
Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; archive.org_bot/1.10.0 +http://www.loc.gov/minerva/crawl.html)
Mozilla/5.0 (compatible; archive.org_bot/1.13.1x http://crawler.archive.org)
Mozilla/5.0 (compatible; archive.org_bot/1.5.0-200506132127 http://crawler.archive.org) Hurricane Katrina
Mozilla/5.0 (compatible; archive.org_bot/3.1.1 +http://pandora.nla.gov.au/crawl.html)
Mozilla/5.0 (compatible; archive.org_bot/3.1.1 +http://www.bne.es/es/LaBNE/PreservacionDominioES/AvisoWebmasters/index.html)
Mozilla/5.0 (compatible; archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html)
Mozilla/5.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/webmasters.shtml)
Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://www.ask.com/about/help/webmasters)
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Mozilla/5.0 (compatible; BanBots/2.0b; Fetch; +http://www.banbots.com)
Mozilla/5.0 (compatible; BecomeBot/1.23; http://www.become.com/webmasters.html)
Mozilla/5.0 (compatible; BecomeBot/1.xx; MSIE 6.0 compatible; http://www.become.com/webmasters.html)
Mozilla/5.0 (compatible; BecomeBot/2.0beta; http://www.become.com/webmasters.html)
Mozilla/5.0 (compatible; BecomeBot/2.x; MSIE 6.0 compatible; http://www.become.com/site_owners.html)
Mozilla/5.0 (compatible; BecomeJPBot/2.3; MSIE 6.0 compatible; +http://www.become.co.jp/site_owners.html)
Mozilla/5.0 (compatible; BeetleBot; )
Mozilla/5.0 (compatible; Bender; http://benderthewebrobot.tumblr.com)
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
Mozilla/5.0 (compatible; Birubot/1.0) Gecko/2009032608 Firefox/3.0.8
Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)
Mozilla/5.0 (compatible; BlogRefsBot/0.1; http://www.blogrefs.com/about/bloggers)
Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)
Mozilla/5.0 (compatible; Bot; +http://pressemitteilung.ws/spamfilter
Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8
Mozilla/5.0 (compatible; BuzzRankingBot/1.0; +http://www.buzzrankingbot.com/)
Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html)
Mozilla/5.0 (compatible; cdlwas_bot; heritrix/1.14.1 +http://webarchives.cdlib.org/p/webmasters)
Mozilla/5.0 (compatible; Charlotte/1.0b; charlotte@betaspider.com)
Mozilla/5.0 (compatible; Charlotte/1.0b; http://www.searchme.com/support/)
Mozilla/5.0 (compatible; crawler/3.1.0-SNAPSHOT-20110914.213704 +http://www.notconfigured.com/)
Mozilla/5.0 (compatible; Crawling jpeg; http://www.yama.info.waseda.ac.jp)
Mozilla/5.0 (compatible; Custo 3 (Netwu.com); Windows NT 5.1)
Mozilla/5.0 (compatible; DCPbot/1.0; +http://domains.checkparams.com/)
Mozilla/5.0 (compatible; DCPbot/1.1; +http://domains.checkparams.com/)
Mozilla/5.0 (compatible; de/1.13.2 +http://www.de.com)
Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com)
Mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)
Mozilla/5.0 (compatible; discobot/2.0; +http://discoveryengine.com/discobot.html)
Mozilla/5.0 (compatible; DNS-Digger-Explorer/1.0; +http://www.dnsdigger.com)
Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com)
Mozilla/5.0 (compatible; Dow Jones Searchbot)
Mozilla/5.0 (compatible; EARTHCOM.info/2.01; http://www.earthcom.info)
Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu)
Mozilla/5.0 (compatible; egothor/8.0g; +http://ego.ms.mff.cuni.cz/)
Mozilla/5.0 (compatible; en-US; ReverseGet/1.0; http://reverseget.com/; robot@reverseget.com)
Mozilla/5.0 (compatible; en-US; RGAnalytics/1.0)
Mozilla/5.0 (compatible; en-US; RGAnalytics/1.0; )
Mozilla/5.0 (compatible; EventGuruBot/1.0; +http://www.eventguru.com/spider.html)
Mozilla/5.0 (compatible; Exabot Test/3.0; +http://www.exabot.com/go/robot)
Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
Mozilla/5.0 (compatible; ExaleadCloudView/5;)
Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)
Mozilla/5.0 (compatible; Falconsbot; +http://ws.nju.edu.cn/falcons/)
Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/main/CrawlerFAQs.fhtml)
Mozilla/5.0 (compatible; Galbot/1.0; +http://www.galbot.com/bot.html)
mozilla/5.0 (compatible; genevabot http://www.healthdash.com)
Mozilla/5.0 (compatible; Google Desktop) Paros/3.2.12
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Hailoobot/1.2; +http://www.hailoo.com/spider.html)
mozilla/5.0 (compatible; heritrix/1.0.4 http://innovationblog.com)
Mozilla/5.0 (compatible; heritrix/1.10.2 +http://i.stanford.edu/)
Mozilla/5.0 (compatible; heritrix/1.12.1 +http://newstin.com/)
Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com)
Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com) [email:paul@page-store.com]
Mozilla/5.0 (compatible; heritrix/1.12.1b +http://netarkivet.dk/website/info.html)
Mozilla/5.0 (compatible; heritrix/1.14.1 +http://www.nlc.gov.cn/)
Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.accelobot.com)
Mozilla/5.0 (compatible; heritrix/1.14.3.r6601+http://www.yellowpages.com/about/legal/crawl)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://ce.yazduni.ac.ir)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://censoweb.speed.dcc.ufmg.br)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://susanszky.tatk.elte.hu)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.attrakt.com)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.test.test)
Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.youscribe.com)
mozilla/5.0 (compatible; heritrix/1.3.0 http://archive.crawler.org)
Mozilla/5.0 (compatible; heritrix/1.4.0 +http://www.chepi.net)
Mozilla/5.0 (compatible; heritrix/1.4t http://www.truveo.com/)
Mozilla/5.0 (compatible; heritrix/1.5.0 http://www.l3s.de/~kohlschuetter/projects/crawling/)
Mozilla/5.0 (compatible; heritrix/1.5.0-200506231921 http://pandora.nla.gov.au/crawl.html)
Mozilla/5.0 (compatible; heritrix/1.6.0 http://www.worio.com/)
Mozilla/5.0 (compatible; heritrix/1.7.0 +http://www.greaterera.com/)
Mozilla/5.0 (compatible; heritrix/1.8.0 +http://yourwebsite.domain.com/)
Mozilla/5.0 (compatible; Heritrix/1.8.0 http://www.hanzoarchives.com)
Mozilla/5.0 (compatible; heritrix/1.x.x +http://www.accelobot.com)
Mozilla/5.0 (compatible; heritrix/2.0.0-RC1 +http://www.aol.com)
Mozilla/5.0 (compatible; heritrix/3.1.0 +http://mith.umd.edu)
Mozilla/5.0 (compatible; heritrix/3.1.0 +http://sid.uncu.edu.ar/sid/cdh)
Mozilla/5.0 (compatible; heritrix/3.1.0 +http://www.library.unt.edu/digitalprojects/web-archiving)
Mozilla/5.0 (compatible; heritrix/3.1.0 +http://www.northgrid.co.jp)
Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 +http://www.archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120320.221924 +http://archive.org/details/archive.org_bot)
Mozilla/5.0 (compatible; Hermit Search. Com; +http://www.hermitsearch.com)
Mozilla/5.0 (compatible; HomeTags/1.0; +http://www.hometags.nl/bot)
Mozilla/5.0 (compatible; http://www.IsMySiteUp.Net/bot/ )
Mozilla/5.0 (compatible; http://www.UptimeAuditor.com/bot/ )
Mozilla/5.0 (compatible; HyperixScoop/1.3; +http://www.hyperix.com)
Mozilla/5.0 (compatible; iaskspider/1.0; MSIE 6.0)
Mozilla/5.0 (compatible; IDBot/1.0; +http://www.id-search.org/bot.html)
Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/
Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/)
Mozilla/5.0 (compatible; InterseekWeb/3.x)
Mozilla/5.0 (compatible; IstellaBot/1.01.18 +http://www.tiscali.it/)
Mozilla/5.0 (compatible; Jim +http://www.hanzoarchives.com)
Mozilla/5.0 (compatible; KaloogaBot; http://kalooga.com/crawler)
Mozilla/5.0 (compatible; KLSH-PageGet/1.03; http://www.kondo-net.gr.jp/klsh/)
Mozilla/5.0 (compatible; Konqueror/2.0.1; X11); Supports MD5-Digest; Supports gzip encoding
Mozilla/5.0 (compatible; Konqueror/2.1.1; X11)
Mozilla/5.0 (compatible; Konqueror/2.2.2)
Mozilla/5.0 (compatible; Konqueror/2.2.2; Linux 2.4.14-xfs; X11; i686)
Mozilla/5.0 (compatible; Konqueror/3.2; Linux)
Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
Mozilla/5.0 (compatible; LemSpider 0.1)
Mozilla/5.0 (compatible; lemurwebcrawler admin@lemurproject.org; +http://boston.lti.cs.cmu.edu/crawler_12/)
Mozilla/5.0 (compatible; linkfluence/0.9; http://linkfluence.net/)
Mozilla/5.0 (compatible; linkfluence; http://linkfluence.net/)
Mozilla/5.0 (compatible; LinksManager.com_bot http://linksmanager.com/linkchecker.html)
Mozilla/5.0 (compatible; LinkStash Bookmark Manager; http://www.xrayz.co.uk/)
Mozilla/5.0 (compatible; ltbot/1.3; bot@language-tools.com)
Mozilla/5.0 (compatible; LucidWorks/; +; crawler at example dot com)
Mozilla/5.0 (compatible; LuminateBot/1.0; +http://www.luminate.com/bot/)
Mozilla/5.0 (compatible; Mail.RU/2.0)
Mozilla/5.0 (compatible; MetamojiCrawler/1.0; +http://www.metamoji.com/jp/crawler.html
Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)
Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html)
Mozilla/5.0 (compatible; MojeekBot/2.0; http://www.mojeek.com/bot.html)
Mozilla/5.0 (compatible; MOSBookmarks/v2.6-Plus; Link Checker)
Mozilla/5.0 (compatible; Mozilla/5.0 +http://www.up.ru)
Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; crawler_admin@podtech.net)
Mozilla/5.0 (compatible; MSIE 7.0 +http://internetmemory.org/en/); crawling on behalf of The National Archives UK Government Web Archive: http://www.nationalarchives.gov.uk/webarchive/; email: webarchive@nationalarchives.gsi.gov.uk
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0
Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/2.0
Mozilla/5.0 (compatible; Najdi.si/3.1)
Mozilla/5.0 (compatible; Netseer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com)
Mozilla/5.0 (compatible; Nigma.ru/3.0; crawler@nigma.ru)
Mozilla/5.0 (compatible; oBot/2.3.1; +http://www-935.ibm.com/services/us/index.wss/detail/iss/a1029077?cntxt=a1027244)
Mozilla/5.0 (compatible; OnetSzukaj/5.0; http://szukaj.onet.pl)
Mozilla/5.0 (compatible; OntoPort/; +; admin@ontoport.de)
Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html)
Mozilla/5.0 (compatible; PagestackerBot; http://www.pagestacker.com)
Mozilla/5.0 (compatible; PalmeraBot; http://www.links24h.com/help/palmera) Version 0.001
Mozilla/5.0 (compatible; PEAR HTTP_Request class; http://feed.moo.jp/)
Mozilla/5.0 (compatible; Phonifier; +http://www.phonifier.com)
Mozilla/5.0 (compatible; Plukkie/1.4; http://www.botje.com/plukkie.htm)
Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm)
Mozilla/5.0 (compatible; pogodak.ba/3.x)
Mozilla/5.0 (compatible; Pogodak.hr/3.1)
Mozilla/5.0 (compatible; Proximic crawler; +http://www.proximic.com/en/about-us/contact-us.html)
Mozilla/5.0 (compatible; proximic; +http://www.proximic.com)
Mozilla/5.0 (compatible; PWeBot/3.1; http://www.programacionweb.net/robot.php)
Mozilla/5.0 (compatible; Quantcastbot/1.0; www.quantcast.com)
Mozilla/5.0 (compatible; Robo/1.0b; http://potaru.com/robo.html)/Nutch-1.2
Mozilla/5.0 (compatible; robtexbot/1.0; http://www.robtex.com/ )
Mozilla/5.0 (compatible; RTSbot/1.0; +http://adamcaudill.com/rtsbot/)
Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)
Mozilla/5.0 (compatible; ScribdReader/1.0; +http://www.float.com)
Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/)
Mozilla/5.0 (compatible; Semager/1.4; http://www.semager.de/blog/semager-bots/)
Mozilla/5.0 (compatible; Semager/1.4c; +http://www.semager.de/blog/semager-bots/)
Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html)
Mozilla/5.0 (compatible; ShunixBot/1.x.x +http://www.shunix.com/robot.htm)
Mozilla/5.0 (compatible; ShunixBot/1.x; http://www.shunix.com/bot.htm)
Mozilla/5.0 (compatible; sindice-fetcher/0.1.0 +http://sindice.com/developers/bot)
Mozilla/5.0 (compatible; sindice-site-manager/0.1.0 +http://sindice.com/developers/bot)
Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)
Mozilla/5.0 (compatible; SkreemRBot +http://skreemr.com)
Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9
Mozilla/5.0 (compatible; spbot/3.0; +http://www.seoprofiler.com/bot )
Mozilla/5.0 (compatible; special_archiver/1.5.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html)
Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); http://nlp.fi.muni.cz/projects/biwec/)
Mozilla/5.0 (compatible; SpurlBot/0.2)
Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/)
Mozilla/5.0 (compatible; SummizeBot +http://www.summize.com)
Mozilla/5.0 (compatible; SYCLIKControl/LinkChecker;)
Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot.html)
Mozilla/5.0 (compatible; Theophrastus/x.x; http://users.cs.cf.ac.uk/N.A.Smith/theophrastus.php)
Mozilla/5.0 (compatible; TridentSpider/3.1)
Mozilla/5.0 (compatible; U; Linux i686 (x86_64); de-DE; <a href=http://wikimpress.org/>Wikimpress</a>) Wikimpress/1.0
Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)
Mozilla/5.0 (compatible; Voluniabot/1.0; spider@volunia.com)
Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/)
Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)
Mozilla/5.0 (compatible; Web CEO Online robot)
Mozilla/5.0 (compatible; Webduniabot/1.0; +http://search.webdunia.com/bot.aspx)
Mozilla/5.0 (compatible; WebmasterCoffee/0.7; +http://webmastercoffee.com/about)
Mozilla/5.0 (compatible; Windows NT 5.0; phpwebbrainBot/0.1 - http://www.monsterli.ch/phpwebbrain/)
Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com)
Mozilla/5.0 (compatible; woriobot +http://worio.com)
Mozilla/5.0 (compatible; WoW Lemmings Kathune/2.0;http://www.wowlemmings.com/kathune.html)
Mozilla/5.0 (compatible; XTbot/1.0v; +http://www.externaltest.com)
Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (compatible; Yahoo! Slurp Asia; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)
Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html)
Mozilla/5.0 (compatible; Yeti/1.0; NHN Corp.; http://help.naver.com/robots/)
Mozilla/5.0 (compatible; YioopBot +http://www.yioop.com/bot.php)
Mozilla/5.0 (compatible; YioopBot; +http://www.yioop.com/bot.php)
Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; )
Mozilla/5.0 (compatible; Yoono; http://www.yoono.com/)
Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
Mozilla/5.0 (compatible; Zenbot/1.3; +http://zen.co.za/webmasters/)
Mozilla/5.0 (compatible; zermelo +http://www.powerset.com) [email:paul@page-store.com&#44;crawl@powerset.com]
Mozilla/5.0 (compatible;archive.org_bot/1.7.1; collectionId=316; Archive-It; +http://www.archive-it.org)
Mozilla/5.0 (compatible;archive.org_bot/heritrix-1.9.0-200608171144 +http://pandora.nla.gov.au/crawl.html)
Mozilla/5.0 (compatible;FindITAnswersbot/1.0;+http://search.it-influentials.com/bot.htm)
Mozilla/5.0 (compatible;MAINSEEK_BOT)
Mozilla/5.0 (compatible;picmole/1.0 +http://www.picmole.com)
Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;)
Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; http://mozshot.nemui.org/)
Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) (compatible; Yeti-Mobile/0.1; +http://help.naver.com/robots/)
mozilla/5.0 (larbin2.6.3@unspecified.mail)
Mozilla/5.0 (rbzt.net)
Mozilla/5.0 (research@mediatrec.com)
Mozilla/5.0 (Sage)
Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Mozilla/5.0 (SunOS 5.8 sun4u; U) Opera 5.0 [en]
Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html)
Mozilla/5.0 (Version: xxxx Type:xx)
Mozilla/5.0 (wgao@genieknows.com)
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.7) NimbleCrawler 1.11 obeys UserAgent NimbleCrawler For problems contact: crawler_at_dataalchemy.com
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1 (http://www.voila.com/)
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11
Mozilla/5.0 (Windows; U;XMPP Tiscali Communicator v.10.0.1; Windows NT 5.1; it; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3
Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For problems contact: crawler@health
Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.com
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/20021130
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19
Mozilla/5.0 gURLChecker/0.x.x (Linux)
Mozilla/5.0 URL-Spider
Mozilla/5.0 usww.com-Spider-for-w8.net
Mozilla/5.0 wgao@genieknows.com
Mozilla/5.0 whoiam [http://www.axxus.de/]
Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)
Mozilla/5.0 [en] (X11; U; Linux 2.2.15 i686 +http://www.yahoo.com/index.html)
Mozilla/5.0+(compatible;+CukBot;+Not+a+spammer;+++http://www.companiesintheuk.co.uk/bot.html)
Mozilla/5.0/heritrix/3.1.0 (compatible;; http://vancouver.cs.washington.edu:8080/)
Mozilla/5.0/Nutch-1.1 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13)
Mozilla/6.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Mozilla/Nutch-1.2 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.11)
MQbot metaquerier.cs.uiuc.edu/crawler
MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://falcon.cs.uiuc.edu; mqbot@cs.uiuc.edu)
MSFrontPage/4.0
MSIE 4.0 (Win95)
MSIE-5.13 (larbin@unspecified.mail)
msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)
msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)
MSNBOT/0.xx (http://search.msn.com/msnbot.htm)
msnbot/2.0b (+http://search.msn.com/msnbot.htm)
msnbot/2.0b (+http://search.msn.com/msnbot.htm)._
msnbot/x.xx ( http://search.msn.com/msnbot.htm)
MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default)
MSNPTC/1.0
MSProxy/2.0
MSRBOT
MSRBOT (http://research.microsoft.com/research/sv/msrbot)
Mulder&#44; VCR-1.0
multiBlocker browser
multicrawler ( http://sw.deri.org/2006/04/multicrawler/robots.html)
MultiText/0.1
MusicWalker2.0 ( http://www.somusical.com)
MVAClient
My Nutch Spider/Nutch-1.2
My Nutch Spider/Nutch-1.4
My Nutch Spider/Nutch-1.5-SNAPSHOT
my nutch/Nutch-1.4
My Spider/Nutch-1.4
My WinHTTP Connection
myCrawl/Nutch-1.3
myDaemon
MyGetRight/1.0.0
MyGetRight/1.0b
Mylinea.com Crawler 2.0
mylinkcheck/1.02
Naamah 1.0.1/Blogbot (http://blogbot.de/)
Naamah 1.0a/Blogbot (http://blogbot.de/)
naba/Nutch-1.4
NABOT/5.0
nabot_1.0
NameOfAgent (CMS Spider)
naoFavicon4IE/1.xx
NASA Search 1.0
NationalDirectory-WebSpider/1.3
NationalDirectoryAddURL/1.0
NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com)
NaverBot_dloader/1.5
NavissoBot
NavissoBot/1.7 (+http://navisso.com/)
NCSA Beta 1 (http://vias.ncsa.uiuc.edu/viasarchivinginformation.html)
Nebullabot/2.2 (http://bot.nebulla.info)
NEC Research Agent -- compuman at research.nj.nec.com
NEC-Hayek/1.0
Net-Seekr Bot/Net-Seekr Bot V1 (http://www.net-seekr.com)
NetAnts/1.2x
NETCOMplete/x.xx
netEstate NE Crawler (+http://www.sengine.info/)
NetinfoBot/1.0 (http://netinfo.bg/netinfobot.html)
NetLookout/2.24
Netluchs/0.8-dev ( ; http://www.netluchs.de/; ___don't___spam_me_@netluchs.de)
NetMechanic Vx.0
NetNewsWire/2.x (Mac OS X; http://ranchero.com/netnewswire/)
NetNoseCrawler/v1.0
Netprospector JavaCrawler
NetPumper/x.xx
NetResearchServer(http://www.look.com)
NetResearchServer/x.x(loopimprovements.com/robot.html)
NetSeer/Nutch-0.9 (NetSeer Crawler; http://www.netseer.com; crawler@netseer.com)
NetSprint -- 2.0
NetSprint -- 3.0
NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com; support@netwhat.com)
NetZippy
NeuralBot/0.2
newsearchengine (ThisUser@unspecified.mail)
NewsGator FetchLinks extension/0.2.0 (http://graemef.com)
NewsGatorOnline/2.0 (http://www.newsgator.com; 1 subscribers)
NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot)
NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx)
NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v0.x
NG-Search/0.90 (NG-SearchBot; http://www.ng-search.com; )
NG/1.0
NG/2.0
NG/4.0.1229
NgapaBot/Nutch-1.3 (NgapaBot is crawler for Indonesia Search Engine; http://www.ngapa.com; admin at ngapa dot com)
nicebot
NICO/1.0
Nikita the Spider (http://NikitaTheSpider.com/)
NISE/Nutch-2.0-dev
NITLE Blog Spider/0.01
Nitro Downloader 1.x (www.klsofttools.com)
Noago Spider
Nocilla/1.0
Nokia-WAPToolkit/1.2 googlebot(at)googlebot.com
Nokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi http://find.mobi/bot.html abuse@mtld.mobi)
Nokia6610/1.0 (3.09) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html)
Nokia7110/1.0 (05.01) (Google WAP Proxy/1.0)
NokodoBot/1.x (+http://nokodo.com/bot.htm)
Norbert the Spider(Burf.com)
noxtrumbot/1.0 (crawler@noxtrum.com)
noyona_0_1
NP/0.1 (NP; http://www.nameprotect.com; npbot@nameprotect.com)
NPBot (http://www.nameprotect.com/botinfo.html)
NPBot-1/2.0
Nsauditor/1.x
NSPlayer/10.0.0.xxxx WMFSDK/10.0
nsyght.com/Nutch-1.0-dev (nsyght.com; Nsyght.com)
nsyght.com/Nutch-x.x (nsyght.com; search.nsyght.com)
nttdirectory_robot/0.9 (super-robot@super.navi.ocn.ne.jp)
Nucleus SiteList LinkChecker/1.1
nuSearch Spider &lt;a href='http://www.nusearch.com'>www.nusearch.com&lt;/a> (compatible; MSIE 4.01)
NuSearch Spider (compatible; MSIE 6.0)
NuSearch Spider www.nusearch.com
Nutch
Nutch crawler/Nutch-0.9 (picapage.com; admin@picapage.com)
Nutch Experimental Crawler/Nutch-1.4
Nutch Spider Evaluation/Nutch-1.4
Nutch test crawler/Nutch-1.4-dev (Crawler testing; crawler/a/t/luminouslabs.com)
Nutch, Leistungsschutz-Debatte Crawler/Nutch-1.4
nutch-1.4/Nutch-1.4
Nutch-1/Nutch-1.4
nutch-solr-integration-test/Nutch-1.2 (FreeCode Norway Web Crawler using Nutch 1.0; http://www.freecode.no/; reza.keshmir.90@gmail.com)
Nutch/Nutch-0.9 (Eurobot; http://www.ayell.eu )
Nutch/Nutch-1.4 (A semantic crawler for a PhD thesis; http://issel.ee.auth.gr/doku.php/research; kvavliak at issel dot ee dot auth dot gr)
Nutch/Nutch-1.4 (A semantic crawler for a PhD thesis; http://issel.ee.auth.gr; kvavliak at issel dot ee dot auth dot gr)
NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)
NutchCVS/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)
NutchCVS/0.7.1 (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycrawl@cs.washington.edu)
NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.org/nutch/bot.html; ec2test at lucene.com)
NutchOrg/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net)
nutchsearch/Nutch-0.9 (Nutch Search 1.0; herceg_novi at yahoo dot com)
NutchVinegarCrawl/Nutch-0.8.1 (Vinegar; http://www.cs.washington.edu; eytanadar at gmail dot com)
Nutraspace/Nutch-1.2 (www.nutraspace.com)
obidos-bot (just looking for books.)
ObjectsSearch/0.01-dev (ObjectsSearch;http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com)
ObjectsSearch/0.0x (ObjectsSearch; http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com)
oBot ((compatible;Win32))
Ocelli/1.x (http://www.globalspec.com/Ocelli)
Octopus
Octora Beta - www.octora.com
Octora Beta Bot - www.octora.com
Offline Explorer 1.*
OKArabia search engine/Nutch-1.4
OliverPerry
OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Internet CategorizerOmniExplorer http://www.omni-explorer.com/ car &amp; shopping search (64.62.175.xxx)
OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Job Crawler
OmniExplorer_Bot/1.1x (+http://www.omni-explorer.com) Torrent Crawler
OmniExplorer_Bot/x.xx (+http://www.omni-explorer.com) WorldIndexer
onCHECK Linkchecker von www.scientec.de fuer www.onsinn.de
onCHECK-Robot&#44; www.onsearch.de
Onet.pl SA- http://szukaj.onet.pl
online link validator (http://www.dead-links.com/)
Online24-Bot (Version: 1.0x&#44; powered by www.online24.de)
OntoSpider/1.0 libwww-perl/5.65
OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com )
OpenAcoon v4.0.x (www.openacoon.de)
OpenAcoon v4.10.5 (www.openacoon.de)
Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)
OpenCalaisSemanticProxy
Openfind data gatherer- Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html)
Openfind Robot/1.1A2
OpenISearch/1.x (www.openisearch.com)
OpenTaggerBot (http://www.opentagger.com/opentaggerbot.htm)
OpenTextSiteCrawler/2.9.2
OpenWebSpider/0.x.x (http://www.openwebspider.org)
OpenWebSpider/x
Opera/5.0 (Linux 2.0.38 i386; U) [en]
Opera/5.11 (Windows ME; U) [ru]
Opera/5.12 (Windows 98; U) [en]
Opera/6.01 (larbin@unspecified.mail)
Opera/6.x (Linux 2.4.8-26mdk i686; U) [en]
Opera/6.x (Windows NT 4.0; U) [de]
Opera/7.x (Windows NT 5.1; U) [en]
Opera/8.xx (Windows NT 5.1; U; en)
Opera/9.0 (Windows NT 5.1; U; en)
Opera/9.00 (Windows NT 5.1; U; de)
Opera/9.60 (Windows NT 5.1; U; de) Presto/2.1.1
Opera/9.80 (Macintosh; Intel Mac OS X; U; en)
Opera/9.80 (Windows NT 5.1; U; en) Presto/2.5.22 Version/10.50
Opera/9.80 (Windows NT 6.1; U; cs) Presto/2.10.229 Version/11.61
Opera/9.80 (Windows NT 6.1; U; en-US) Presto/2.7.62 Version/11.01
OpidooBOT (larbin2.6.3@unspecified.mail)
OPWV-SDK UP.Browser/7.0.2.3.119 (GUI) MMP/2.0 Push/PO
Oracle Application Server Web Cache 10g
Oracle iMTCrawler
Oracle Ultra Search
OrangeCrawler/Nutch-1.0 (ldorange.crawler@orange-ftgroup.com)
OrangeSpider
Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm)
OrbitSpiderBot/1
Orca Browser (http://www.orcabrowser.com)
OSSProxy 1.3.305.321 (Build 305.321 Win32 en-us)(Dec 21 2005 16:30:54)
OutfoxBot/0.x (For internet experiments; http://; outfox.agent@gmail.com)
OutfoxMelonBot/0.5 (for internet experiments; http://; outfoxbot@gmail.com)
Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)
OWR_Crawler 0.1
ozelot/2.7.3 (Search engine indexer; www.flying-cat.de/ozelot; ozelot@flying-cat.de)
PADLibrary Spider
PageBitesHyperBot/600 (http://www.pagebites.com/)
Pagebull http://www.pagebull.com/
PagePeeker.com
PagePeeker.com (info: http://pagepeeker.com/robots)
Pagestacker Bot
page_verifier (http://www.securecomputing.com/goto/pv)
PagmIEDownload
Pangalactic Gargleblaster/Nutch-1.2 (Very intoxicating; http://circuitbomb.com; DigitalMail)
parallelContextFocusCrawler1.1parallelContextFocusCrawler1.1
ParaSite/1.0b (http://www.ianett.com/parasite/)
PArchiveCrawler/1.0 (PArchiveCrawler)
Patwebbot (http://www.herz-power.de/technik.html)
pavuk/0.9pl29b i686-pc-linux-gnu
PBrowse 1.4b
pd02_1.0.0 pd02_1.0.0@dzimi@post.sk
PEAR HTTP_Request class ( http://pear.php.net/ )
PEERbot www.peerbot.com
PeopleChat/Search_Engine
percbotspider
percbotspider <ops@percolate.com>
PEval 1.4b
pflab (cboc-test@lab.ntt.co.jp)
PHP/3.x.xx
PHP/4.0.4pl1
PHP/4.0.6
PHP/4.1.1
PHP/4.1.2
PHP/5.2.17p1
PicklBot 1.0
PicoSearch/1.0
Piffany_Web_Scraper_v0.x
Piffany_Web_Spider_v0.x
PigeonBot1.0 BETA
PingALink Monitoring Services 1.0
PingALink Monitoring Services 1.0 (http://www.pingalink.com)
Pingdom GIGRIB (http://www.pingdom.com)
Pingthesemanticweb-robotvalidator/3.0
pipeLiner/0.3a (PipeLine Spider;http://www.pipeline-search.com/webmaster.html; webmaster'at'pipeline-search.com)
pipeLiner/0.xx (PipeLine Spider; http://www.pipeline-search.com/webmaster.html)
Pita
Pixray-Seeker/1.1 (Pixray-Seeker; http://www.pixray.com/pixraybot; crawler@pixray.com)
Pizilla++ ver 2.45
PJspider/3.0 (pjspider@portaljuice.com; http://www.portaljuice.com)
Plagger/0.x.xx (http://plagger.org/)
PlagiarBot/1.0
PlantyNet_WebRobot_V1.9 dhkang@plantynet.com
plinki/0.1 (you got plinked! (thats a good thing..); http://www.plinki.com; crawl@plinki.com)
PluckFeedCrawler/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com; 1 subscribers)
Pluggd/Nutch-0.9 (automated crawler http://www.pluggd.com;support at pluggd dot com)
Pockey-GetHTML/4.11.6 (Win32; GUI; ix86)
Pockey-GetHTML/4.12.0 (Win32; GUI; ix86)
Pockey-GetHTML/x.xx
Pockey/x.x.x
Pockey7.x.x(WIN32GUI)
POE-Component-Client-HTTP/0.64 (perl; N; POE; en; rv:0.640000)
Poirot
polybot 1.0 (http://cis.poly.edu/polybot/)
Pompos/1.x http://dir.com/pompos.html
Pompos/1.x pompos@iliad.fr
Popdexter/1.0
Port Huron Labs
PortalBSpider/2.0 (spider@portalb.com)
portalmmm/2.0 S500i(c20;TB)
PortalNG-Crawler/Nutch-1.4 (Crawler des Portals der Uni Ulm.; http://portal.uni-ulm.de; christian.aschoff at uni-ulm.de)
PostFavorites
PostPost/1.0 (+http://postpost.com/crawlers)
potbot 1.0
PRCrawler/Nutch-0.9 (data mining development project; crawler@projectrialto.com)
Primo Web Spider/Nutch-1.4
PrivacyFinder Cache Bot v1.0
PrivacyFinder/1.1
private test crawler curl-based
Privoxy/3.0 (Anonymous)
Production Bot 0116B
Production Bot 2016B
Production Bot DOT 3016B
Program Shareware 1.0.2
Progressive Download
Progressive Download HTTP check
Project XP5 [2.03.07-111203]
PROve AnswerBot 4.0
ProWebGuide Link Checker (http://www.prowebguide.com)
psbot/0.1 (+http://www.picsearch.com/bot.html)
PSurf15a 11
PSurf15a 51
PSurf15a VA
psycheclone
PubCrawl (pubcrawl.stanford.edu)
puf/0.91beta6a (Linux 2.2.18; i686)
puf/0.93.2a (Linux 2.4.18; i686)
pulseBot (pulse Web Miner)
PureSight
PuxaRapido v1.0
PWeBot/1.2 Inspector (http://www.programacionweb.net/robot.php)
PycURL
PycURL/7.xx.x
Python-urllib/1.17
Python-urllib/1.1x
Python-urllib/2.0a1
Python-urllib/2.6
Qango.com Web Directory (http://www.qango.com/)
QEAVis Agent/Nutch-0.9 (Quantitative Evaluation of Academic Websites Visibility; http://nlp.uned.es/qeavis
QleeQ1/Nutch-1.4
QPCreep Test Rig ( We are not indexing- just testing )
QuepasaCreep ( crawler@quepasacorp.com )
QuepasaCreep v0.9.1x
QueryN Metasearch
quickobot/quickobot-1 (Quicko Labs; http://quicko.co; robot at quicko dot co)
Quicksilver (Blacktree&#44;MacOSX)
QuickTime\xaa.7.0.4 (qtver=7.0.4;cpu=PPC;os=Mac 10.3.9)
QweeryBot/3.01 ( http://qweerybot.qweery.nl)
Qweery_robot.txt_CheckBot/3.01 (http://qweerybot.qweery.com)
R6_CommentReader(www.radian6.com/crawler)
R6_CommentReader_(www.radian6.com/crawler)
R6_FeedFetcher_(www.radian6.com/crawler)
rabaz (rabaz at gigabaz dot com)
RaBot/1.0 Agent-admin/phortse@hanmail.net
RADaR-Bot/Nutch-1.4 (http://radar-bot.com/)
Rainbot1.1
ramBot xtreme x.x
RAMPyBot - www.giveRAMP.com/0.1 (RAMPyBot - www.giveRAMP.com; http://www.giveramp.com/bot.html; support@giveRAMP.com)
RAMPyBot/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
Rank Exec (rankexec.com) Reciprocal Link Manager 1.x/bot
Rankivabot/3.2 (www.rankiva.com; 3.2; vzmxikn)
Rational SiteCheck (Windows NT)
RAYSPIDER/Nutch-0.9
ReadABlog Spider (compatible; 1.1; feed update; www.readablog.com)
RealDownload/4.0.0.4x
REAP-crawler Nutch/Nutch-1.0-dev (Reap Project; http://reap.cs.cmu.edu/REAP-crawler/; Reap Project)
Reaper [2.03.10-031204] (http://www.sitesearch.ca/reaper/)
Reaper/2.0x (+http://www.sitesearch.ca/reaper)
REBOL Core 2.x.x.x.x
REBOL View 1.x.x.x.x
RebusnetBot (+http://www.rebusnet.biz)
RebusnetPADBot/1.5x (+http://www.rebusnet.biz)
reciprocal links checker (http://www.recip-links.com/)
RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com)
RedCarpet/1.2 (http://www.redcarpet-inc.com/robots.html)
RedCell/0.1 (InfoSec Search Bot (Coming Soon); http://www.telegenetic.net/bot.html; lhall@telegenetic.net)
RedCell/0.1 (RedCell; telegenetic.net/bot.html; lhall_at_telegenetic.net)
RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/)
REL Link Checker Lite x.x
RepoMonkey Bait &amp; Tackle/v1.01
Rewebber/1.2 libwww-perl/5.41
rico/0.1
RISSBot/dev (http://www.tkolb.de/rissbot.txt)
RixBot (http://babelserver.org/rix)
RMA/1.0 (compatible; RealMedia)
rmatie.com spider/Nutch-1.4
rmatie.com/Nutch-1.4
RoboCrawl (http://www.canadiancontent.net)
RoboCrawl (www.canadiancontent.net)
RoboPal (http://www.findpal.com/)
Robot/www.pj-search.com
Robot: NutchCrawler- Owner: wdavies@acm.org
Robot@SuperSnooper.Com
Robozilla/1.0
Rome Client (http://tinyurl.com/64t5n) Ver: 0.9
Ronzoobot/1.5 (http://www.ronzoo.com/about/)
Rotondo/3.1 libwww/5.3.1
RPT-HTTPClient/0.3-x
RRC (crawler_admin@bigfoot.com)
RSpider/1.0
RssBandit/1.5.0.10 (.NET CLR 1.1.4322.2407; WinNT 5.1.2600.0; http://www.rssbandit.org) (.NET CLR 1.1.4322.2407; WinNT 5.1.2600.0; )
RSSMicro.com RSS/Atom Feed Robot
RSSOwl/1.2.3 2006-11-26 (Windows; U; zhtw)
RSSOwl/1.2.4 Preview Release 2007-04-15 (Windows; U; zhtw)
RssReader/1.0.xx.x (http://www.rssreader.com) Microsoft Windows NT 5.1.2600.0
RSurf15a 41
RSurf15a 51
RSurf15a 81
Rubbot/1.0 (+http://rubhub.com/)
RufusBot (Rufus Web Miner; http://64.124.122.252/feedback.html)
RufusBot (Rufus Web Miner; http://www.webaroo.com/rooSiteOwners.html)
ruijie (larbin@unspecified.mail)
Ruky-Roboter (Version: 1.06, powered by www.ruky.de +http://www.ruky.de/zeige/bot.html)
Rumours-Agent
RX Bar
S&amp;L Spider (http://search.hirners.com/)
S.T.A.L.K.E.R. (http://www.seo-tools.net/en/bot.aspx)
S2-Crawler/1.1 (your search engine; http://www.search2.net/; info at search2 dot net)
S2Robot
sadsa/Nutch-1.4
SafariBookmarkChecker (+http://www.coriolis.ch/)
sait/Nutch-0.9 (SAIT Research; http://www.samsung.com)
SaladSpoon/ShopSalad 1.0 (Search Engine crawler for ShopSalad.com; http://shopsalad.com/en/partners.html; crawler AT shopsalad.com)
SandCrawler - Compatibility Testing
SapphireWebCrawler/1.0 (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu)
SapphireWebCrawler/Nutch-1.0-dev (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu)
savvybot/0.2
SBIder/0.7 (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html)
SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html)
SBL-BOT (http://sbl.net)
ScanWeb
ScholarScope/Nutch-1.5 (Scholar Research Engine)
ScholarUniverse/0.8 (Nutch;+http://scholaruniverse.com/bot.jsp; fetch-agent@scholaruniverse.com)
schwarzmann.biz-Spider_for_paddel.org+(http://www.innerprise.net/usp-spider.asp)
Science Traveller International 1X/1.0
ScollSpider/2.0 (+http://www.webwobot.com/ScollSpider.php)
Scooter-3.0.EU
Scooter-3.0.FS
Scooter-3.0.HD
Scooter-3.0.VNS
Scooter-3.0QI
Scooter-3.2
Scooter-3.2.BT
Scooter-3.2.DIL
Scooter-3.2.EX
Scooter-3.2.JT
Scooter-3.2.NIV
Scooter-3.2.SF0
Scooter-3.2.snippet
Scooter-3.3dev
Scooter-ARS-1.1
Scooter-ARS-1.1-ih
scooter-venus-3.0.vns
Scooter-W3-1.0
Scooter-W3.1.2
Scooter/1.0
Scooter/1.0 scooter@pa.dec.com
Scooter/1.1 (custom)
Scooter/2.0 G.R.A.B. V1.1.0
Scooter/2.0 G.R.A.B. X2.0
Scooter/3.3
Scooter/3.3.QA.pczukor
Scooter/3.3.vscooter
Scooter/3.3_SF
Scooter2_Mercator_x-x.0
Scooter_bh0-3.0.3
Scooter_trk3-3.0.3
Scope (Mars+)
ScoutAbout
ScoutAnt/0.1; +http://www.ant.com/what_is_ant.com/
scoutmaster
Screaming Frog SEO Spider/1.80
Screaming Frog SEO Spider/1.90
Scrubby/2.x (http://www.scrubtheweb.com/)
Scrubby/3.0 (+http://www.scrubtheweb.com/help/technology.html)
Search+
Search-Engine-Studio
search.ch V1.4
search.ch V1.4.2 (spiderman@search.ch; http://www.search.ch)
Search/1.0 (http://www.innerprise.net/es-spider.asp)
searchbot admin@google.com
SearchByUsa/2 (SearchByUsa; http://www.SearchByUsa.com/bot.html; info@SearchByUsa.com)
SearchdayBot
SearchExpress Spider0.99
SearchGuild/DMOZ/Experiment (searchguild@gmail.com)
SearchGuild_DMOZ_Experiment (chris@searchguild.com)
Searchit-Now Robot/2.2 (+http://www.searchit-now.co.uk)
Searchmee! Spider v0.98a
SearchSight/2.0 (http://SearchSight.com/)
SearchSpider.com/1.1
Searchspider/1.2 (SearchSpider; http://www.searchspider.com; webmaster@searchspider.com)
SearchTone2.0 - IDEARE
Seegul/Nutch-1.4
SeekBot/1.0 RobotsTxtFetcher/1.3
Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3
Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF)
Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2
Seeker.lookseek.com
semaforo.net
Semager/1.1 (http://www.semager.de/blog/semager-bots/)
Semager/1.x (http://www.semager.de)
semanticdiscovery/0.x
SemrushBot/0.91
SemrushBot/0.92
SemrushBot/Nutch-1.5-SNAPSHOT
Sensis Web Crawler (search_comments\at\sensis\dot\com\dot\au)
Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au)
SeoBot
SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm)
service-public-indexeur
SeznamBot/1.0
SeznamBot/1.0 (+http://fulltext.seznam.cz/)
SeznamBot/2.0-test (+http://fulltext.sblog.cz/)
SeznamBot/3.0 (+http://fulltext.sblog.cz/)
sGroup crawler 1/Nutch-1.3
ShablastBot 1.0
Shareaza v1.x.x.xx
SharewarePlazaFileCheckBot/1.0+(+http://www.SharewarePlaza.com)
Shim Crawler
Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawler/; crawl@logos.ic.i.u-tokyo.ac.jp)
ShopWiki/1.0 ( +http://www.shopwiki.com/)
ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot)
Shoula.com Crawler 2.0
SietsCrawler/1.1 (+http://www.siets.biz)
Sigram/Nutch-1.0-dev (Test agent for Nutch development; http://www.sigram.com/bot.html; bot at sigram dot com)
Siigle Orumcex v.001 Turkey (http://www.siigle.com)
silk/1.0
silk/1.0 (+http://www.slider.com/silk.htm)/3.7
SimpleFavPanel/1.2
Simpy 1.x; http://www.simpy.com/
Simpy/1.x (Simpy; http://www.simpy.com/?ref=bot; feedback at simpy dot com)
sindice.net sitemap fetcher
Sirketcebot/v.01 (http://www.sirketce.com/bot.html)
SiteBar/3.x.x (Bookmark Server; http://sitebar.org/)
SiteBar/x.x
SiteBar/x.x.x (Bookmark Server; http://sitebar.org/)
sitecheck.internetseer.com
sitecheck.internetseer.com (For more info see: http://sitecheck.internetseer.com)
SiteRecon+(xx)
SiteSnagger
SiteSpider +(http://www.SiteSpider.com/)
SiteSucker/1.x.x
SiteSucker/2.2.4
SiteSucker/2.3.2
SiteSucker/2.3.3
SiteSucker/2.3.5
SiteSucker/2.3.6
SiteTaggerBot (http://www.sitetagger.com/bot.htm)
SiteTruth.com site rating system
SiteWinder
SiteXpert
Skampy/0.9.x (http://www.skaffe.com/skampy-info.html)
SkimBot/1.0 (www.skimlinks.com <dev@skimlinks.com>)
SkimBot/1.0 (www.skimlinks.com)
Skimpy/0.x (http://www.skaffe.com/skampy-info.html)
Skywalker/0.1 (Skywalker; anonymous; anonymous)
Slarp/0.1
Sleipnir
Sleipnir Version 1.xx
Sleipnir Version2.x
Sleipnir/2.xx
Slider_Search_v1-de
SlimBrowser
Slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Slurp/2.0-KiteWeekly (slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Slurp/si (slurp@inktomi.com; http://www.inktomi.com/slurp.html)
Slurpy Verifier/1.0
SlySearch (slysearch@slysearch.com)
SlySearch/1.0 http://www.plagiarism.org/crawler/robotinfo.html
SlySearch/1.x http://www.slysearch.com
SmartDownload/1.2.67 (Win32; Jan 12 1999)
SmartDownload/1.2.77 (Win32; Feb 1 2000)
SmartDownload/1.2.77 (Win32; Jun 19 2001)
smartwit.com
SmiffyDCMetaSpider/1.0
sna-0.0.1 (mikemuzio@msn.com)
sna-0.0.1 mikeelliott@hotmail.com
snap.com beta crawler v0
Snapbot/1.0
Snapbot/1.0 (Snap Shots&#44; +http://www.snap.com)
Snappy/1.1 ( http://www.urltrends.com/ )
Snarfer/0.x.x (http://www.snarfware.com/)
SnoopRob/x.x
Snoopy v1.xx
Snoopy v1.xx- : User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MyIE2)
Snoopy_v0.xx
SnykeBot/0.6 (http://www.snyke.com)
SocSciBot ()
SoftBank/1.0/812SH/SHJ001 Browser/NetFront/3.3 Profile/MIDP-2.0 Configuration/CLDC-1.1
SoftHypermarketFileCheckBot/1.0+(+http://www.softhypermaket.com)
Softizerbot (http://www.softizer.com)
SoftNet Nutch Spider/Nutch-1.4
sogou develop spider
Sogou Orion spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
sogou spider
Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
sohu agent
sohu-search
Sosospider+(+http://help.soso.com/webspider.htm)
Space Bison/0.02 [fu] (Win67; X; SK)
SpeedDownload/1.x
speedfind ramBot xtreme 8.1
Speedy Spider (Beta/x.x; speedy@entireweb.com)
Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/speedyspider/)
Speedy_Spider (http://www.entireweb.com)
Sphere Scout&#38;v4.0 - scout at sphere dot com
Sphider
Spida/0.1
Spider-Sleek/2.0 (+http://search-info.com/linktous.html)
spider.batsch.com
Spider.TerraNautic.net - v:1.04
spider.yellopet.com - www.yellopet.com
Spider/maxbot.com admin@maxbot.com
spider/Nutch-1.3 (spider; http://www.xxx.com)
SpiderKU/0.x
SpiderMan
SpiderMonkey/7.0x (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml)
Spinne/2.0
Spinne/2.0 med
Spinne/2.0 med_AH
Spock Crawler (http://www.spock.com/crawler)
sportsuchmaschine.de-Robot (Version: 1.02- powered by www.sportsuchmaschine.de)
spotinfluence/Nutch-1.2 (Spot Influence crawler; http://spotinfluence.com; hello at spotinfluence dot com)
spotinfluence/Nutch-1.4 (Spot Influence crawler; http://spotinfluence.com; hello at spotinfluence dot com)
sproose/0.1-alpha (sproose crawler; http://www.sproose.com/bot.html; crawler@sproose.com)
SQ Webscanner
Squid-Prefetch
squidclam
SquidClamAV_Redirector 1.x.x
Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu)
Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu)
Sqworm/2.9.89-BETA (beta_release; 20020130-839; i686-pc-linux-gnu)
SS/Nutch-1.5 (SS)
ssearch_bot (sSearch Crawler; http://www.semantissimo.de)
SSurf15a 11
StackRambler/x.x
Stamina/1.4
Star Downloader
StarDownloader/1.xx
stat statcrawler@gmail.com
stc-crawler/0.84
Steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)
Strategic Board Bot (+http://www.strategicboard.com)
Submission Spider at surfsafely.com
suchbaer.de
suchbaer.de (CrawlerAgent v0.103)
suchbot
Suchknecht.at-Robot
suchpadbot/1.0 (+http://www.suchpad.de)
Summon
Sunrise XP/2.x
Sunrise/0.42g (Windows XP)
SuperBot/x.x (Win32)
SuperBot/x.x.x.xx (Windows XP)
Superdownloads Spiderman
SURF
SurferF3 1/0
SurfMaster
SurveyBot/2.2 &lt;a href='http://www.whois.sc'>Whois Source&lt;/a>
SurveyBot/2.3 (Whois Source)
suzuran
SWB/V1.4 (HP)
swbot/0.9c libwww/5.3.1
Swooglebot/2.0. (+http://swoogle.umbc.edu/swooglebot.htm)
SWSBot-Images/1.2 http://www.smartwaresoft.com/swsbot12.html
SygolBot http://www.sygol.net
Sylera/1.2.x
SyncBot
SyncIT/x.x
Syndirella/0.91pre
SynoBot
Syntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawler
Szukacz/1.x
Szukacz/1.x (robot; www.szukacz.pl/jakdzialarobot.html; szukacz@proszynski.pl)
T-Online Browser
tags2dir.com/0.8 (+http://tags2dir.com/directory/)
Tagword (http://tagword.com/dmoz_survey.php)
Tagyu Agent/1.0
Talkro Web-Shot/1.0 (E-mail: webshot@daumsoft.com- Home: http://222.122.15.190/webshot)
TAMU_CS_IRL_CRAWLER/1.0
targetblaster.com/0.9k
TargetYourNews.com bot
TCDBOT/Nutch-0.8 (PhD student research;http://www.tcd.ie; mcgettrs at t c d dot IE)
TE
TeamSoft WinInet Component
TECOMAC-Crawler/0.x
Tecomi Bot (http://www.tecomi.com/bot.htm)
Teemer (NetSeer&#44; Inc. is a Los Angeles based Internet startup company.; http://www.netseer.com/crawler.html; crawler@netseer.com)
Teleport Pro/1.2x(.1xxx)
Teoma MP
teomaagent crawler-admin@teoma.com
teomaagent1 [crawler-admin@teoma.com]
teoma_agent1
Teradex Mapper; mapper@teradex.com; http://www.teradex.com
TeragramCrawler
terraminds-bot/1.0 (support@terraminds.de)
TerrawizBot/1.0 (+http://www.terrawiz.com/bot.html)
Test spider
TestCrawler/Nutch-0.9 (Testing Crawler for Research ; http://balihoo.com/index.aspx; tgautier at balihoo dot com)
The Expert HTML Source Viewer (http://www.expert-html.com)
theasearchCrawler/Nutch-1.4 (http://www.theasearch.com/; hansod1@gmail.com)
TheRarestParser/0.2a (http://therarestwords.com/)
TheSuBot/0.1 (www.thesubot.de)
thumbshots-de-bot (+http://www.thumbshots.de/)
thumbshots-de-Bot (Version: 1.02&#44; powered by www.thumbshots.de)
thumbshots-de-Bot (Version: 1.02- powered by www.thumbshots.de)
TigrinyaCrawler(tirawler@gmail.com)
timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html
TinEye/1.1 (http://tineye.com/crawler.html)
tivraSpider/1.0 (crawler@tivra.com)
TJG/Spider
TJvMultiHttpGrabber Component
Tkensaku/x.x(http://www.tkensaku.com/q.html)
Tobot/Nutch-1.4 ('tobot at eel dot rdc dot toshiba dot co dot jp')
toCrawl/UrlDispatcher
Topodia/1.2-dev (Topodia - Crawler for HTTP content indexing; http://www.topodia.com/; support@topodia.com)
TOPOS robot/1.1 (http://www.topos.com.ua/)
Toutatis x-xx.x (hoppa.com)
Toutatis x.x (hoppa.com)
Toutatis x.x-x
traazibot/testengine (+http://www.traazi.de)
Trailfire-bot/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
Trailfire-bot/0.7.1 (Trailfire page content analyzer; http://trailfire.com; info@trailfire.com)
Trailfire/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)
Trampelpfad-Spider
Trampelpfad-Spider-v0.1
tricosMetaCheck 1.2216-08-1999 (http://www.tricos.com/metacheck)
TruNorth/Nutch-1.4
TsinghuaLab
TSurf15a 11
TulipChain/5.x (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://java.sun.com/) Linux/2.4.17
TulipChain/5.xx (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://apple.com/) Mac_OS_X/10.2.8
Tumblr/1.0 RSS syndication (+http://www.tumblr.com/) (support@tumblr.com)
TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html)
TurnitinBot/x.x (http://www.turnitin.com/robot/crawlerinfo.html)
Turnpike Emporium LinkChecker/0.1
TutorGig/1.5 (+http://www.tutorgig.com/crawler)
Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler)
Twiceler www.cuill.com/robots.html
Twiceler-0.9 http://www.cuill.com/twiceler/robot.html
Twisted PageGetter
Twitterbot/1.0
Twitturly / v0.x
Twotrees Reactive Filter V2.0
Tycoon Agent/Nutch-1.0-dev
TygoBot
TygoProwler
UCmore
UCMore Crawler App
UCWEB5.1
UDM
UdmSearch/3.1.x
UIowaCrawler/1.0
UKWizz/Nutch-0.8.1 (UKWizz Nutch crawler; http://www.ukwizz.com/)
Ultraseek
Under the Rainbow 2.2
UnisterBot (Mozilla/5.0 compatible; crawler@unister-gmbh.de)
unknownbot (abc@mail.com)
unknownght.com Web Server IIS vs Apache Survey. See Results at www.DNSRight.com
UofTDB_experiment (leehyun@cs.toronto.edu)
UP.Browser/3.01-IG01 UP.Link/3.2.3.4
updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com)
updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.om)
Updownerbot (+http://www.updowner.com/bot)
UPG1 UP/4.0 (compatible; Blazer 1.0)
upictoBot
Uptimebot
UptimeBot(www.uptimebot.com)
URI::Fetch/0.06
URL Spider Pro/x.xx (innerprise.net)
URLBase/6.x
URLBlaze
urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html
URLGetFile
URL_Spider_Pro/x.x
URL_Spider_Pro/x.x+(http://www.innerprise.net/usp-spider.asp)
User-Agent: BoardReader Favicon Fetcher /1.0 info@boardreader.com
User-Agent: BoardReader Image Fetcher /1.0 info@boardreader.com
User-Agent: BoardReader-Image-Fetcher /1.0 info@boardreader.com
User-Agent: LjSEEK Picture-Bot /1.0 contact@ljseek.com
User-Agent: FileHeap! file downloader (http://www.fileheap.com)
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
User-Agent: Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE.com)
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)
user-agent=Mozilla/3.01Gold
USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html)
UtilMind HTTPGet
Utopia WebWasher 3.0
uTorrent/1500
VadixBot
Vagabondo-WAP/2.0 (webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)/1.0 Profile
Vagabondo/1.x MT (webagent@wise-guys.nl)
Vagabondo/2.0 MT
Vagabondo/2.0 MT (webagent at wise-guys dot nl)
Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl)
Vagabondo/3.0 (webagent at wise-guys dot nl)
Vakes/0.01 (Vakes; http://www.vakes.com/; search@vakes.com)
VayalaCreep-v0.0.1 (haploid@haploid.com)
Vayala|Creep-v0.0.1 (codepoet@wildties.com)
vb wininet
versus 0.2 (+http://versus.integis.ch)
versus crawler eda.baykan@epfl.ch
Verticrawlbot
VeryGoodSearch.com.DaddyLongLegs
verzamelgids.nl - Networking4all Bot/x.x
Verzamelgids/2.2 (http://www.verzamelgids.nl)
Vespa Crawler
virus_detector (virus_harvester@securecomputing.com)
VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com)
Visicom Toolbar
Vision Research Lab image spider at vision.ece.ucsb.edu
VLC media player - version 0.8.5 Janus - (c) 1996-2006 the VideoLAN team
VMBot/0.x.x (VMBot; http://www.VerticalMatch.com/; vmbot@tradedot.com)
Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/)
voyager-hc/1.0
voyager/1.0
voyager/2.0 (http://www.kosmix.com/html/crawler.html)
VSE/1.0 (mann@vivisimo.com)
VSE/1.0 (myunn@vivisimo.com)
VSE/1.0 (testcrawler@hotmail.com)
VSE/1.0 (testcrawler@vivisimo.com)
vspider
vspider/3.x
VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu;+vwbot@cs.uiuc.edu
W3C-checklink/3.x.x.x libwww-perl/5.xx
W3C-checklink/4.81 libwww-perl/5.836
W3C-checklink/4.x [4.xx] libwww-perl/5.xxx
W3C-WebCon/5.x.x libwww/5.x.x
W3CLineMode/5.4.0 libwww/5.x.x
W3CRobot/5.4.0 libwww/5.4.0
W3C_Validator/1.xxx libwww-perl/5.xx
w3m/0.x.xx
W3SiteSearch Crawler_v1.1 http://www.w3sitesearch.de
Wada.vn Vietnamese Search/2.1
wadaino.jp-crawler 0.2 (http://wadaino.jp/)
WannaBe (Macintosh; PPC)
WapOnWindows 1.0
Watchfire WebXM 1.0
WAVcheck 1.0.x (http://www.webbanalys.se/apps/WAVcheck/)
Wavefire/0.8-dev (Wavefire; http://www.wavefire.com; info@wavefire.com)
Waypath development crawler - info at waypath dot com
Waypath Scout v2.x - info at waypath dot com
WDG_Validator/1.1
Web Image Collector
Web Link Validator 1.5
Web Snooper
web-bekannt (Version: 1.02&#44; powered by www.internetservice-franken.de)
web-bekannt (Version: 1.02&#44; powered by www.web-bekannt.de)
Web-Bot V1.03
Web-Robot/5.0 (en-US; web-robot.com/policy.html) Web-Robot Crawler/2.0.3
web2express.org/Nutch-0.9-dev (leveled playing field; http://web2express.org/; info at web2express.org)
WebAlta Crawler/1.2.1 (http://www.webalta.ru/bot.html)
WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html)
WebarooBot (Webaroo Bot; http://www.webaroo.com/rooSiteOwners.html)
WebAuto/3.4xxx (WinNT; I)
webbandit/4.xx.0
WebBug/5.x
Webclipping.com
webcollage/1.xx
WebCompass 2.0
WebCopier vx.x
WebCopier vx.xa
WebCorp/1.0
webcrawl.net
webcrawler (compatible; heritrix/1.14.4 ++http://www.onb.ac.at/about/webarchivierung.htm)
WebDownloader for X x.xx
Webdup/0.9
WebFetch
webfetch/5.x.x
WebFilter Robot 1.0
WebFilter Robot 1.x
WebFindBot(http://www.web-find.com)
Webglimpse 2.xx.x (http://webglimpse.net)
webGobbler/1.x.x
webhack
WebImages 0.3 ( http://herbert.groot.jebbink.nl/?app=WebImages )
WebLight/4.x.x (support@illumit.com; http://www.illumit.com/Products/weblight/)
Weblink's checker/
Weblog Attitude Diffusion 1.0
webmeasurement-bot&#44; http://rvs.informatik.uni-leipzig.de
WebMiner/x.x [en] (Win98; I)
WeBoX/0.xx
WebPix 1.0 (www.netwu.com)
WebQL
WebRACE/1.1 (University of Cyprus- Distributed Crawler)
WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/)
WebReaper vx.x - www.webreaper.net
WebReaper [info@webreaper.net]
WebReaper [webreaper@webreaper.net]
WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; Search@WebSearch.COM.AU)
WebSearchBench WebCrawler v0.1(Experimental)
WebSearchBench WebCrawler V1.0 (Beta)- Prof. Dr.-Ing. Christoph Lindemann- Universit&#228;t Dortmund- cl@cs.uni-dortmund.de- http://websearchbench.cs.uni-dortmund.de/
Website Explorer/0.9.x.x
Website eXtractor
WebsiteWorth v1.0
Webspinne/1.0 webmaster@webspinne.de
Websquash.com (Add url robot)
WebStat/1.0 (Unix; beta; 20040314)
Webster v0.3 ( http://webster.healeys.net/ )
webster-internet.de pad browser
WebStripper/2.xx
WebTrafficExpress/x.0
WebTrends/3.0 (WinNT)
WebVac (webmaster@pita.stanford.edu)
WebVal/1.0
Webverzeichnis.de - Telefon: 01908 / 26005
WebVulnCrawl.unknown/1.0 libwww-perl/5.803
WebWatcherMonitor/2.01
WebZIP/x.x (http://www.spidersoft.com)
Wells Search II
WEP Search 00
WeSEE:Search/0.1 (Alpha, http://www.wesee.com/)
WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)
West Wind Internet Protocols 4.xx
WeViKa
WFARC
Wget/1.10.2
Wget/1.10.2 (Red Hat modified)
Wget/1.11.1
Wget/1.11.4
Wget/1.11.4 Red Hat modified
Wget/1.12 (linux-gnu)
Wget/1.13.4 (darwin10.8.0)
Wget/1.13.4 (darwin11.1.0)
Wget/1.13.4 (darwin11.2.0)
Wget/1.13.4 (darwin11.3.0)
Wget/1.13.4 (linux-gnu)
Wget/1.x(.x)GNU wget http://www.gnu.org/software/wget/wget.html - file downloader
Wget/1.x+cvs-stable (Red Hat modified)
Wget/1.x.x+cvs
Whatsup/x.x
whatUseek_winona/3.0
WhizBang! Lab
wikiwix-bot-3.0
Wildsoft Surfer
Willow Internet Crawler by Twotrees V2.1
WinampMPEG/2.00 (larbin@unspecified.mail)
WincerSong Agent v1.0
Windows-Media-Player/10.00.00.xxxx
WinGet 1.1
WinHTTP Example/1.0
WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkBot; winkbot@wink.com)
WinPodder (http://winpodder.com)
WinWAP/3.x (3.x.x.xx; Win32) (Google WAP Proxy/1.0)
Wir sind die Borg (Version: 1.03&#44; Sie wurden Assimiliert +http://www.yammba.com/suchmaschine/bot.html)
WIRE/0.11 (Linux; i686; Bot&#44;Robot&#44;Spider&#44;Crawler&#44;aromano@cli.di.unipi.it)
WIRE/0.x (Linux; i686; Bot&#44;Robot&#44;Spider&#44;Crawler)
wirkon.com (Version: 1.06, powered by www.wirkon.com+http://www.wirkon.com)
WISEbot/1.0 (WISEbot@koreawisenut.com; http://wisebot.koreawisenut.com)
WiseWire-Spider2
wish-project (http://wish.slis.tsukuba.ac.jp/)
woobot/1.1
WordChampBot
WordPress/3.4-beta1;
WordPress/3.4-beta2-20452;
WordPress/x.x.x.x PHP/4.x.xx
worio heritrix bot (+http://worio.com/)
woriobot ( http://www.worio.com/)
WorldLight
WorQmada/1.0
Wotbox/2.0 (bot@wotbox.com; http://www.wotbox.com)
Wotbox/alpha0.6 (bot@wotbox.com; http://www.wotbox.com)
Wotbox/alpha0.x.x (bot@wotbox.com; http://www.wotbox.com) Java/1.4.1_02
WSB WebCrawler V1.0 (Beta)- cl@cs.uni-dortmund.de
WSB&#44; http://websearchbench.cs.uni-dortmund.de
wttsearch/Nutch-1.0 (Crawler for SIFE (WTT); http://wtt.beecom.ch/wttsearch.html; wtt@beecom.ch)
wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/)
Wusage/x.0@boutell.com
Wwlib/Linux
WWSBOT 1.x [--- http://www.analyzer.nu ---]
WWW-Mechanize/1.1x
www.arianna.it
www.business-socket.com registry verify/1.x
www.doweb.co.uk crawler
www4mail/2.x libwww-FM/2.14 (Unix; I)
WWWC/1.0x
WWWeasel Robot v1.00 (http://wwweasel.de)
WWWOFFLE/2.x
wwwster/1.x (Beta- mailto:gue@cis.uni-muenchen.de)
wxDownload Fast
X-Crawler
Xaldon WebSpider
Xenu Link Sleuth 1.xx
Xenu's Link Sleuth 1.x[a-z]
Xerka WebBot v1.0.0 [UPVOpenDir]
xine/1.0
xirq/0.1-beta (xirq; http://www.xirq.com; xirq@xirq.com)
XMLSlurp/0.1 libwww-perl/5.805
XRL/2.00b1 (Linux; i686; en-us) (+http://metamark.net/about)
Xylix
xyro_(xcrawler@cosmos.inria.fr)
Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html )
Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html)
Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/blog-search/)
Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Y!J-SRD/1.0
Y!J/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)
Y!OASIS/TEST no-ad Mozilla/4.08 [en] (X11; I; FreeBSD 2.2.8-STABLE i386)
Y!TunnelPro
yacy (www.yacy.net; v20040602; i386 Linux 2.4.26-gentoo-r13; java 1.4.2_06; MET/en)
yacybot (freeworld-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 FreeBSD 8.2-STABLE; java 1.6.0; GMT/pl) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.16.46-0.12-smp; java 1.6.0_15; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-31-generic; java 1.7.0; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-34-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-342-ec2; java 1.6.0_20; Etc/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-37-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-37-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-38-generic; java 1.7.0; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-40-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_29; US/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-6-pve; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.37.6; java 1.6.0_25; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 2.6.38-13-generic; java 1.6.0_22; America/pt) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-12-generic; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-12-server; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-12-server; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-14-generic; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-15-server; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-15-server; java 1.6.0_26; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.0.0-16-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.0-1.2-desktop; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.0-1.2-desktop; java 1.6.0_22; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.10-1.9-default; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.10-hardened; java 1.7.0_03-icedtea; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.1; java 1.7.0_147-icedtea; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.2-1.fc16.x86_64; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.1.9-1.4-default; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-1-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-10-generic; java 1.7.0_147-icedtea; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-18-generic; java 1.7.0_03-icedtea; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-2-rt-amd64; java 1.6.0_24; Asia/ja) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0-21-generic; java 1.7.0_03-icedtea; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.0; java 1.7.0_147-icedtea; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.1-2-ARCH; java 1.7.0_147-icedtea; Europe/fr) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_24; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.13-1-ARCH; java 1.7.0_03-icedtea; Europe/fr) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.2-1-ARCH; java 1.7.0_147-icedtea; Europe/fr) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.2.2-1.fc16.x86_64; java 1.7.0_b147-icedtea; W-SU/ru) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Linux 3.3.1-1-ARCH; java 1.6.0_24; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_25; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_26; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_26; Europe/fr) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/no) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_03; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.32-39-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.32-4-pve; java 1.6.0_18; Etc/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 2.6.37-gentoo-r4-110510; java 1.6.0_22; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 3.0.0-12-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 3.0.0-12-generic-pae; java 1.6.0_23; Europe/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 3.0.0-13-generic-pae; java 1.6.0_23; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 3.0.0-15-generic-pae; java 1.6.0_23; America/fr) http://yacy.net/bot.html
yacybot (freeworld/global; i386 Linux 3.0.0-16-generic; java 1.6.0_23; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows 2003 5.2; java 1.6.0_30; America/en) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_01; Europe/de) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows XP 5.1; java 1.6.0_29; Europe/es) http://yacy.net/bot.html
yacybot (freeworld/global; x86 Windows XP 5.1; java 1.6.0_31; Europe/sv) http://yacy.net/bot.html
yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Asia/ru) http://yacy.net/bot.html
yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Europe/ru) http://yacy.net/bot.html
yacybot (webportal-global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html
yacybot (webportal-global; amd64 Linux 3.0.0-17-generic; java 1.7.0_147-icedtea; Europe/de) http://yacy.net/bot.html
yacybot (webportal/global; amd64 Windows 7 6.1; java 1.7.0_02; Europe/de) http://yacy.net/bot.html
yacybot (webportal/global; i386 Linux 2.6.35-28-generic; java 1.6.0_20; America/en) http://yacy.net/bot.html
yacybot (x86 Windows XP 5.1; java 1.5.0_06; Europe/de) yacy.net
Yahoo Pipes 1.0
Yahoo! Mindset
Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html )
Yahoo-MMAudVid/1.0 (mms dash mmaudvidcrawler dash support at yahoo dash inc dot com)
Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.com ;Mozilla 4.0 compatible; MSIE 7.0;Windows NT 5.0; .NET CLR 2.0)
Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com)
Yahoo-Test/4.0
Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler
YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide)
YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/)
YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/)
YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/yahooseeker.html)
YahooSeeker/1.1 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/)
YahooSeeker/bsv3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html )
YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD ;cafekelsa-dev-webmaster@yahoo-inc.com )
Yandex/1.01.001 (compatible; Win16; I)
YandexBlogs
Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/)
yarienavoir.net/0.2
Yepi/1.0 (NHN Corp.; http://help.naver.com/robots/)
Yeti
Yeti/0.01 (nhn/1noon&#44; yetibot@naver.com&#44; check robots.txt daily and follows it)
Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)
yggdrasil/Nutch-0.9 (yggdrasil biorelated search engine; www dot biotec dot tu minus dresden do de slash schroeder; heiko dot dietze at biotec dot tu minus dresden dot de)
YodaoBot/1.0 (http://www.yodao.com/help/webmaster/spider/; )
yolinkBot
yoofind/yoofind-0.1-dev (yoono webcrawler; http://www.yoono.com ; MyEmail)
yoogliFetchAgent/0.1
yoono/1.0 web-crawler/1.0
YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine
YottaShopping_Bot/4.12 (+http://www.yottashopping.com) Shopping Search Engine
YTunnelPro
Z-Add Link Checker (http://w3.z-add.co.uk/linkcheck/)
Zao-Crawler
Zao-Crawler 0.2b
Zao/0.1 (http://www.kototoi.org/zao/)
ZBot/1.00 (icaulfield@zeus.com)
zclosure/Nutch-1.3
zclosure/Nutch-1.4
Zearchit
ZeBot_lseek.net (bot@ze.bz)
ZeBot_www.ze.bz (ze.bz@hotmail.com)
zedzo.digest/0.1 (http://www.zedzo.com/)
Zend_Http_Client
zermelo Mozilla/5.0 compatible; heritrix/1.12.1 (+http://www.powerset.com) [email:crawl@powerset.com&#44;email:paul@page-store.com]
zerxbot/Version 0.6 libwww-perl/5.79
Zeus ThemeSite Viewer Webster Pro V2.9 Win32
Zeus xxxxx Webster Pro V2.9 Win32
Zeusbot/0.07 (Ulysseek's web-crawling robot; http://www.zeusbot.com; agent@zeusbot.com)
Ziggy -- The Clown From Hell!!
Zillsy.com Search engine /Nutch-1.4
ZipppBot/0.xx (ZipppBot; http://www.zippp.net; webmaster@zippp.net)
ZIPPPCVS/0.xx (ZipppBot/.xx;http://www.zippp.net; webmaster@zippp.net)
Zippy v2.0 - Zippyfinder.com
Zoo Tycoon 2 Client -- http://www.zootycoon.com
ZoomSpider - wrensoft.com
ZoomSpider - wrensoft.com [ZSEBOT]
Zscho.de Crawler/Nutch-1.0-Zscho.de-semantic_patch (Zscho.de Crawler
zspider/0.9-dev http://feedback.redkolibri.com/
ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry)
ZyBorg/1.0 (ZyBorg@WISEnut.com; http://www.WISEnut.com)
_USERAGENT_
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment