-
-
Save acdha/2579174 to your computer and use it in GitHub Desktop.
Report the percentage of your site traffic generated by robots
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 'Mozilla/5.0 (compatible; BMC/1.0 (Y!J-AGENT))' | |
| (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.talktalk.co.uk/products/virus-alerts/) | |
| (TalkTalk Virus Alerts Scanning Engine) | |
| */Nutch-0.9 | |
| */Nutch-1.0 | |
| */Nutch-1.0 (*; *; *) | |
| */Nutch-1.2 | |
| +http://hilfe.acont.de/bot.html ACONTBOT | |
| - | |
| - (larbin2.6.3@unspecified.mail) | |
| 08_800w_web_an (larbin2.6.3@unspecified.mail) | |
| 1.2 beast/Nutch-1.0 (agentspider; bt@mail.com) | |
| 123peoplebot/1.0 | |
| 163tester (tester@163.com) | |
| 2000_sites/Nutch-1.0 (Small_crawldb) | |
| 20090608_web_aq (larbin2.6.3@unspecified.mail) | |
| 222.243.160.126/Nutch-1.0 | |
| 312/Nutch-1.0 | |
| 50.nu V0.01 beta ( http://www.50.nu/bot.html ) | |
| 50.nu/0.01 ( +http://50.nu/bot.html ) | |
| 8dorm/Nutch-1.0 (8dorm; http://www.8dorm.com; webmaster@8dorm.com) | |
| : Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1 | |
| =Mozilla/4.0 | |
| ???/Nutch-1.0 (MyTest; 192.168.143.171; lai) | |
| A1 Website Download/2.0.1 (+http://www.microsystools.com/products/website-download/) miggibot | |
| AAfter.com Crawler/AAfter-1.0 (This bot is very focused, well-behaved, and wants to do good to internet community. For any questions, please call collect +1 214-714-2224. Team AAfter, Dallas, TX, USA; http://aafter.com/crawler.htm; crawler at aafter.com) | |
| AAfter.com Crawler/Nutch-1.0 (This bot is very focused, well-behaved, and wants to do good to internet community. For any questions, please call collect +1 214-714-2224. Team AAfter, Dallas, TX, USA; http://aafter.com/crawler.htm; crawler at aafter.com) | |
| Abortion.sg/Nutch-1.1 (www.Abortion.sg; crawler@Abortion.sg) | |
| Aboundex/0.2 (http://www.aboundex.com/crawler/) | |
| Abrave Spider v5.3 Robot 5 (http://robot.abrave.com) | |
| ACME Corporation/Nutch-1.0 (ACME Spider; http://www.acme.com; test123@spam.la) | |
| Acoon v4.1.0 (www.acoon.de) | |
| Acoon v4.10.1 (www.acoon.de) | |
| Acoon v4.10.3 (www.acoon.de) | |
| Acoon v4.10.4 (www.acoon.de) | |
| Acoon v4.10.5 (www.acoon.de) | |
| Acoon v4.9.3 (www.acoon.de) | |
| Acoon v4.9.5 (www.acoon.de) | |
| AcoonBot/4.10.5 (+http://www.acoon.de) | |
| acquia-crawler | |
| acquia-crawler (detected bad behaviour? please tell us at it@acquia.com) | |
| Acushare_Spider | |
| AddThis.com robot tech.support@clearspring.com | |
| adidxbot/1.1 (+http://search.msn.com/msnbot.htm) | |
| AdnormCrawler www.adnorm.com/crawler | |
| AdultShop.Sg/Nutch-1.1 (www.AdultShop.Sg; crawler@AdultShop.Sg) | |
| AdultsVisit.us/Nutch-1.0 (www.AdultsVisit.us; crawler@AdultsVisit.us) | |
| AdultsVisitUs/Nutch-1.1 (www.AdultsVisitUs.com; crawler@AdultsVisitUs.com) | |
| agbot/1.0 (AgHaven.com search engine crawler; http://search.aghaven.com; bot@aghaven.com) | |
| Agent | |
| agentname/Nutch-1.0-dev | |
| Aghaven/Nutch-1.2 | |
| Aghaven/Nutch-1.2 (www.aghaven.com) | |
| Ahacc.Ru bot | |
| AIR1/Nutch-0.9 | |
| AISearchBot (Email: aisearchbot@gmail.com; If your web site doesn't want to be crawled, please send us a email.) | |
| akhomraven/Nutch-1.2 ( using Nutch 1.2; http://www.creativewires.com/; akhomraven@gmail.com) | |
| AlexionResearchBot/Nutch-1.3 | |
| alhiqma 0.1 (Al-Hiqma Engine; http://labs.unrealasia.net/engine/; admin@unrealasia.net) | |
| all_web3 (larbin2.6.3@unspecified.mail) | |
| al_org_net_viewer (larbin2.6.3@unspecified.mail) | |
| al_org_viewer (larbin2.6.3@unspecified.mail) | |
| al_viewer (larbin2.6.3@unspecified.mail) | |
| amibot - http://www.amidalla.de - tech@amidalla.com libwww-perl/5.831 | |
| andrologia medicina/Nutch-1.0 (andrologia medica; http://www.studiodiandrologia.it; profscandurra@alice.it) | |
| Anemone/0.6.1 | |
| animalstxt.org/bot (http://animalstxt.org/) | |
| AnnunciPazzi/Wauuu 0.1 (AnnunciPazzi; http://www.annuncipazzi.it/; admin@annuncipazzi.it) | |
| Ant/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com) | |
| Antbot/1.0 | |
| AntBot/Ant-Nutch-1.1 (Ant Nutch Crawler; http://www.ant.com; crawler@ant.com) | |
| Apache-HttpClient/4.1.1 (java 1.5) | |
| Apache-HttpClient/4.1.2 (java 1.5) | |
| AppEngine-Google; (+http://code.google.com/appengine; appid: thedomaintips) | |
| Apple iPhone OS v2.2 CoreMedia v1.0.0.5G77 | |
| AppleWebKit/531.21.10 | |
| Applied-Technologies-Inc-Spider/Nutch-1.4 | |
| aquarelabot (no-reply@noemail.com) | |
| Ara.com.tr/Nutch-1.2 | |
| AramaM/Nutch-2.0-dev | |
| araneo | |
| Argh!/Nutch-1.2 | |
| arianna.libero.it | |
| Arquivo-web-crawler (compatible; heritrix/1.12.1 +http://arquivo-web.fccn.pt) | |
| Arquivo-web-crawler (compatible; heritrix/1.14.3 +http://arquivo.pt) | |
| Arquivo-web-crawler (compatible; heritrix/1.14.3 +http://arquivo.pt/faq-crawling) | |
| asearcher/Nutch-1.2 (asearcher; www.asearcher.com; asearcher@yeah.net) | |
| asearcher/Nutch-1.2 (asearcher; www.asearcher.com; asearcher@yearch.net) | |
| ASER-ORNL/Nutch-1.0 | |
| Aspire-Tech Nutch/Nutch-1.1-dev (Aspire-Tech Nutch Robot Agent) | |
| ASPseek/1.4 | |
| Atomic_Email_Hunter/4.0 | |
| Atomz/1.0 | |
| Atrax Solutions atraxbot/0.3; http://www.atraxsolutions.com/atraxbot | |
| atraxbot/0.2 | |
| Attentio/Nutch-0.9-dev (Attentio's beta blog crawler; www.attentio.com; info@attentio.com) | |
| avishayResearchSpider/Nutch-1.2 | |
| axtar.az/axtar-1 (axtar crawler; http://axtar.az; admin@axtar.az) | |
| A_TEST_CRALWER_BY_ISSEL/Nutch-1.4 (http://issel.ee.auth.gr; kvavliak at issel dot ee dot auth dot gr) | |
| B656_testing/Nutch-1.0 (Nutch spiderman) | |
| BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si) | |
| BacklinkCrawler (http://www.backlinktest.com/crawler.html) | |
| BaiDuSpider | |
| Baiduspider+(+http://help.baidu.jp/system/05.html) | |
| Baiduspider+(+http://www.baidu.com/search/spider.htm) | |
| Baiduspider+(+http://www.baidu.com/search/spider_jp.html) | |
| Baiduspider+(+http://www.baidu.jp/spider/) | |
| Balihoo/Nutch-1.0-dev (Crawler for Balihoo.com search engine - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com) | |
| baypup/1.1 (Baypup; http://www.baypup.com/; jason@baypup.com) | |
| baypup/1.1 (Baypup; http://www.baypup.com; jason@baypup.com) | |
| baypup/colbert (Baypup; http://sf.baypup.com/webmasters; jason@baypup.com) | |
| baypup/realtime (Baypup Realtime; http://sf.baypup.com/webmasters; jason@baypup.com) | |
| BDFetch | |
| beast/Nutch-0.9 (agentspider; beast@mail.com) | |
| beast/Nutch-1.0 (agentspider; beast@mail.com) | |
| Bee | |
| belief,*/Nutch-1.2 (belief; belief; belief) | |
| belief/belief | |
| berlin-fu-cow (+http://hpsg.fu-berlin.de/cow) | |
| betaBot | |
| BigTMBot/1.0 | |
| Biible/Nutch-1.2 (Biible; http://www.biible.info ; raymond.balmes@gmail.com) | |
| BilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com) | |
| bim-zone (+http://www.bim-zone.com) | |
| binlar_2.6.3 (binlar2.6.3@unspecified.mail) | |
| binlar_2.6.3 (larbin2.6.3@unspecified.mail) | |
| binlar_2.6.3 (phanendra_kalapala@McAfee.com) | |
| binlar_2.6.3 (test@mgmt.mic) | |
| bitlybot | |
| BizwikiBot/0.3 (http://www.bizwiki.com/bizwikibot.htm) | |
| BlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent | |
| BlackBerry9630/5.0.0.732 Profile/MIDP-2.1 Configuration/CLDC-1.1 VendorID/102 ips-agent | |
| blackcrawl/Nutch-0.9 (crawl for fun; www.zipfelchappe.com; zipfelchappe@localhost) | |
| bldbCrawler/Nutch-1.4 | |
| BlevintronBot version 2012-04-02 contact http://en.wikipedia.org/wiki/User_talk:Blevintron | |
| BlevintronBot version 2012-04-03 contact http://en.wikipedia.org/wiki/User_talk:Blevintron | |
| BlevintronBot version 2012-04-04 contact http://en.wikipedia.org/wiki/User_talk:Blevintron | |
| BlevintronBot version 2012-04-09 contact http://en.wikipedia.org/wiki/User_talk:Blevintron | |
| BlevintronBot version 2012-05-01 contact http://en.wikipedia.org/wiki/User_talk:Blevintron | |
| Blips Bot (www.dailyradar.com) | |
| BlogPulseLive (support@blogpulse.com) | |
| bloo/Nutch-1.0 (bloo; wiki.bloo.com; bloo at bloo dot com) | |
| BLT/1.0.3 | |
| BLT/1.0.4 (http://www.bltformac.com/) | |
| bobcrawler (contact bobcrawler@gmail.com with questions or concerns) | |
| bobcrawler/1.0 (respects robots.txt including crawl-delay, send concerns to bobcrawler@gmail.com) | |
| boitho.com-dc/0.86 ( http://www.boitho.com/dcbot.html ) | |
| bookagent/Nutch-1.0 | |
| boosker/Nutch-1.2 | |
| bot (Java Hotspot(TM) Server 64-Bit VM; +http://www.daniel-nowak-ug.de) Resource Description Engine | |
| bot/1.0 (bot; http://; bot@bot.bot) | |
| BotBot | |
| BotLaju v2.0 (http://laju.my/) | |
| BrokenLinkBot version 2012-03-31 contact http://en.wikipedia.org/wiki/User_talk:Blevintron | |
| BrokenLinkBot version 20120319.102558 contact http://en.wikipedia.org/wiki/User_talk:Blevintron | |
| BrokenLinkBot version 20120320.224751 contact http://en.wikipedia.org/wiki/User_talk:Blevintron | |
| Browsershots | |
| bss (meng1986626@163.com) | |
| Btsearch/1.1 (Btsearch; http://www.baotongsoft.com/search.html; Btsearch-agent@baotongsoft.com) | |
| Btsearch/1.1 (Btsearch; http://www.test.com/test.html; test@test.com) | |
| Bubble Spider/Nutch-1.0 (Bubble Bot; http://f-desk/hwa; frank@verogem.com) | |
| buscador-age (Enterprise; T2-C9QZ24BXJWWK2; gep@ti.eptisa.com) | |
| businessdbbot/v1.2 (http://www.businessdb.com/bot.php) | |
| Cabot/Nutch-1.0 (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; help at amfibi dot com) | |
| Cabot/Nutch-1.0-dev (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; help at amfibi dot com) | |
| Cabot/Nutch-1.2 (Amfibi's webcrawler robot; http://www.amfibi.com/cabot; cabot@amfibi.com) | |
| CamontSpider/1.0 +http://epweb2.ph.bham.ac.uk/user/slater/camont/info.html | |
| CatchBot/1.0; +http://www.catchbot.com | |
| CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot.php; cbot@cazoodle.com) | |
| cc/Nutch-1.0 | |
| CCBot/1.0 (+http://www.commoncrawl.org/bot.html) | |
| cdacp/Nutch-0.9 (IIT Kharagpur; http://www.cdac.in; asoni@cdac.in) | |
| cds.robot/Nutch-1.2 | |
| CE-Preload | |
| CEIT/Nutch-1.1 | |
| Cercador AOC / Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)/Nutch-1.0 | |
| CervantesRIBot | |
| Charlotte | |
| Charlotte/0.05 | |
| check.rdfa/0.0.1 (+http://check.rdfa.info/) RDF-RDFa-Parser/1.09_09 RDF-RDFa-Linter/0.04 RDF-RDFa-Generator/0.04 libwww-perl/5.831 | |
| Chemical Search Engine/Nutch-1.2 (ChemSE crawler, Build a chemical sourcing database; '; info123@chemse.com) | |
| Chen Li/Nutch-1.0 (Nutch spiderman; http://chenli.com.cn; chenlibiti@163.com) | |
| chklinks-robot/3.09 (using libwww-perl/5.837) | |
| cierzo-development/Nutch-1.1-dev | |
| cierzo/Nutch-0.9 | |
| cinvestav/Nutch-1.1 | |
| cinvestav/Nutch-1.4 | |
| citeseerxbot (compatible; citeseerxbot; +http://citeseer.ist.psu.edu/) | |
| Cityreview Robot (+http://www.cityreview.org/crawler/) | |
| cizilla.com/Cizilla-1.0 (CIZILLA - Complete Internet Search Engine; http://www.cizilla.com; crawl@cizilla.com) | |
| CjxSearch/Nutch-1.4 | |
| ck's crawler/Nutch-1.2 | |
| CKC Spider/Nutch-1.4 | |
| CLARAbot | |
| CligooRobot/1.0 | |
| clj-httpc | |
| cloud-valley/bigdata-1.0 (http://www.cloud-valley.com/; news@mobier.mobi) | |
| CloudACL/Nutch-1.4 | |
| CloudCrawler606/Nutch-1.2 | |
| cn_1 (larbin2.6.3@unspecified.mail) | |
| cn_1_viewer (larbin2.6.3@unspecified.mail) | |
| cn_ab_viewer (larbin2.6.3@unspecified.mail) | |
| cn_org_viewer (larbin2.6.3@unspecified.mail) | |
| cn_org_viewer2 (larbin2.6.3@unspecified.mail) | |
| coccoc | |
| Code Search Crawler/Nutch-1.2 (Code Search Crawler; www.iai.uni-bonn.de) | |
| Codeton Software RSS Bot/1.0 | |
| colkbrowser/0.7; Linux OS | |
| colorgo agent/1.0 | |
| colorgo.com/Nutch-1.2 | |
| Combine/3 http://combine.it.lth.se/ | |
| Comodo SSL Checker | |
| Comodo-Certificates-Spider | |
| COMODOspider/Nutch-1.0 | |
| COMODOSpider/Nutch-1.2 | |
| compatible; Mozilla 4.0; MSIE 5.5; (SqwidgeBot b1 - http://www.sqwidge.com/bot/) | |
| Content Crawler | |
| Contextured Site Crawler/Nutch-1.0 (http://contextured.com/features/crawler; crawler@contextured.com) | |
| copyright sheriff (+http://www.copyrightsheriff.com/) | |
| core-I-Bot/1.0 (+http://www.core.at/core-i-bot/) | |
| coruscan/Nutch-1.4 | |
| Covario IDS/Nutch-1.2 | |
| Covario-IDS/1.0 (Covario; http://www.covario.com/ids; support at covario dot com) | |
| Covario-IDS/1.0 (Covario; http://www.covario.com; support at covario dot com) | |
| CovarioIDS/1.1 (http://www.covario.com/ids; support at covario dot com) | |
| CowCrawler/CowCrawler-dev (+http://beta.cow.com) | |
| CPSE Crawler(http://www.clusterpoint.com) | |
| crabbyBot/1.0 | |
| Crawl Test/Nutch-1.2 | |
| Crawl/0.01 libcrawl/0.3 | |
| crawler | |
| Crawler 0.1 | |
| Crawler/Nutch-1.2 | |
| crawler/Nutch-1.3 | |
| Crawler0.1 | |
| crawler01.DiverseBooks.com/Nutch-1.0 (Web crawler aimed at collecting information about Books. Mostly SF for now; http://diversebooks.com/crawler; info at diversebooks tot kom) | |
| crawler4j (http://code.google.com/p/crawler4j/) | |
| CrawlerProcess (http://www.PowerMapper.com) /3.3.300.0 | |
| CrawlerTest/Nutch-1.0-dev | |
| crawly_2.6.3 (crawly@commandcom.com) | |
| CRW1/Nutch-1.2 | |
| crwal_zkzy (chenchaoyin@software.ict.ac.cn) | |
| CSimpleSpider Robot | |
| CSVNutch/Nutch-1.3 (Academic test crawl; pshopcrawl at gmail) | |
| curl/7.15.5 (i686-suse-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8d zlib/1.2.3 libidn/0.6.8 | |
| curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 | |
| curl/7.19.0 (x86_64-suse-linux-gnu) libcurl/7.19.0 OpenSSL/0.9.8h zlib/1.2.5 libidn/1.10 | |
| curl/7.19.7 (x86_64-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15 | |
| curl/7.21.0 (x86_64-apple-darwin10.4.0) libcurl/7.21.0 OpenSSL/0.9.8l zlib/1.2.3 libidn/1.19 | |
| curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5 | |
| CyberSpyder Link Test/2.2.1 | |
| CydralSpider/1.9 (Cydral Web Image Search; http://www.cydral.com) | |
| CydralSpider/3.2 (Cydral Image Search; http://www.cydral.com) | |
| CydralSpider/3.2.6 (Cydral Image Search; http://www.cydral.com) | |
| DAgent | |
| daodao (larbin@unspecified.mail) | |
| DatenschutzRobot/0.01 | |
| Dawwer/1.0 | |
| Deafindex.se/Nutch-1.3 | |
| DealGates Bot/1.1 by Luc Michalski (http://spider.dealgates.com/bot.html) | |
| del.icio.us-thumbnails/1.0 Mozilla/5.0 (compatible; Konqueror/3.4; FreeBSD) KHTML/3.4.2 (like Gecko) | |
| DelvuBot/1.0 | |
| Deneme/Nutch-0.9 | |
| Denodo IECrawler/4.5 | |
| deva/deva-1.0 (Deva Fetcher; devarajaswami at yahoo dot com) | |
| Digital Dialogue Media Pty Ltd Perth Australia /Nutch-1.2 (Nutch spiderman; www.amnet.net.au ; bingatron@digitaldialogue.com.au ) | |
| DigitalPebble/Nutch-1.2 (DigitalPebble's robot; http://www.digitalpebble.com; robot@digitalpebble.com) | |
| DiscoverED by ccLearn/Nutch-0.9 | |
| Dit/1 (Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; YPC 3.0.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727); http://www.dit.unitn.it; bot@dit.unitn.it) | |
| dlab spider/Nutch-1.3 | |
| dmzg/Nutch-1.0 | |
| DNSTallyBot/0.2 | |
| DoCoMo/1.0/N505i/c20/TB/W20H10 (compatible; RFCrawler-Mobile/1.0; +http://www.rfms.jp/crawler.html) | |
| DoCoMo/2.0 D903i(c100;TB;W28H20) (compatible; Hatena-Mobile-Gateway/1.2; +http://mgw.hatena.ne.jp/help) | |
| DoCoMo/2.0 i-robot(c10;TC) | |
| DoCoMo/2.0 N902iS(c100;TB;W24H12)(compatible; moba-crawler; http://crawler.dena.jp/) | |
| DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) | |
| DoCoMo/2.0 P05A(c100;TB;W24H15) (compatible; BaiduMobaider/1.0; +http://www.baidu.jp/spider/) | |
| DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/) | |
| DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/) | |
| DogSniffer | |
| Domnutch-Bot/Nutch-1.0 (Domnutch; http://www.Nutch.de/) | |
| Doot/1.0/Nutch-1.2 | |
| doraemon (wangzheng088@gmail.com) | |
| DownloadIpTxtFile | |
| drone | |
| droreSpider (gaowenbinmarr@gmail.com) | |
| DrRajendra/Nutch-0.9 (Senior Researcher; http://cse.iitkgp.ac.in/~rpr; rajendra@cse.iitkgp.ernet.in) | |
| dtSearchSpider | |
| DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html) | |
| ealbum/Nutch-1.0 (ealbum crawler; www.ealbum.com; steley2000 at yahoo dot com) | |
| easyDigGet/1.0 | |
| ebooksearchtool | |
| ebsearch (larbin@unspecified.mail) | |
| EC2LinkFinder | |
| ecxi/Nutch-1.0 (esCERT-UPC-ecxi; http://escert.upc.edu/; admin escert edu) | |
| EdisterBot (http://www.edister.com/bot.html) | |
| edu.pjwstk.focusedcrawler.training-data-gatherer | |
| eFills/Nutch-0.9 | |
| Elblindo the Blind Bot (Version: 1.06, powered by www.elblindo.com +http://www.elblindo.com/pfadzurbotseite/bot.html) | |
| Element/Nutch-2.0-dev | |
| emBot-GalaBuzz/Nutch-1.0 (http://emining.jp/; em@galabuzz.jp) | |
| emefgebot/BETA (+http://emefge.de/bot.html) | |
| envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html) | |
| EoLarbin_2.6.3.1 (ns6934@ovh.net) | |
| EPiServer Link Checker | |
| ePochta_Extractor/4.0 | |
| Eric Osgood/Nutch-1.0 (Nutch spiderman; http://www.calpoly.edu/~eosgood ; MyEmail) | |
| es_com_viewer (larbin2.6.3@unspecified.mail) | |
| es_org_viewer (larbin2.6.3@unspecified.mail) | |
| es_org_web (larbin2.6.3@unspecified.mail) | |
| Eurobot/1.1 (http://eurobot.ayell.eu) | |
| Eurobot/Nutch-1.0-dev (1.0) | |
| Eurosoft-Bot (Version: 1.06, powered by www.eurosoftware.de +http://www.eurosoftware.de/zeige/bot.html) | |
| eventax/1.0 (http://www.eventax.de) | |
| EventMachine HttpClient | |
| EventusCrawler/0.1 (Collects event information) | |
| ExactSeek Crawler (http://www.exactseek.com/)/Nutch-1.4 | |
| ExactSeek Crawler (nutch 1.4)/Nutch-1.4 | |
| example/Nutch-0.9 (listallabout; http://www.listallabout.com; contact@listallabout.com) | |
| Facebook Labs/Nutch-1.0 (Test crawler for facebook.; http://www.facebook.com/kma; kma at facebook dot com) | |
| Facebook share follower | |
| facebookexternalhit/1.0 (http://www.facebook.com/externalhit_uatext.php) | |
| facebookscraper/facebookscraper/1.0 | |
| fairshare2.0 test crawler/fairshare2.0 (Check us out at http://fairshare.cc ) | |
| fangdudu_1.0 | |
| FAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg) | |
| fc2bots (http://fc2.com/) | |
| FeedChecker/0.01 | |
| fetch/Nutch-1.0 (TCGfetch; http://fetch.thecyberguardian.com; TCGEmail) | |
| fetch/Nutch-1.2 (TCGfetch; http://fetch.thecyberguardian.com; TCGEmail) | |
| Filter/Nutch-1.2 | |
| findfiles.net/0.96 (Robot;test_robot@gmx-topmail.de) | |
| findfiles.net/0.97 (Robot;test_robot@gmx-topmail.de) | |
| findfiles.net/0.98 (Robot;test_robot@gmx-topmail.de) | |
| findlinks/1.1.5-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/1.1.6-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/1.1.6-beta3 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/1.1.6-beta5 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/2.0.4 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/2.0.5 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/2.1.3 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/2.1.5 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/2.2 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/2.5 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| FINLY | |
| Firefox | |
| Firefox/3.0.10 es-ES U Ubuntu jaunty 9.04 | |
| firesignbranding/mokum (A friendly experimental http fetcher.; www.firesignbranding.com; info@firesignbranding.com) | |
| firtex2 agent/1.0 | |
| Flaptor RT/Nutch-1.0 (Flaptor test; http://www.flaptor.com; rodriguez arroba gmail punto com) | |
| Flaptor/Nutch-1.0 (Flaptor crawling; http://www.flaptor.com; info@flaptor.com) | |
| flatlandbot/baypup (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot; jason@flatlandindustries.com) | |
| Flexum/2.0 | |
| FlightDeckReportsBot/2.0 (http://www.flightdeckreports.com/pages/bot) | |
| Focal/Focal-1.0-dev (Syllabs focused web crawler; http://www.syllabs.com/crawler.html; crawler@syllabs.com) | |
| FOIIABot/1.1 (http://foiia.com/bot/; info at foiia.com) | |
| foxman/Nutch-1.2 (Nutch spiderman; http://www.163.com/; MyEmail) | |
| Free Link Checker - http://www.Control-F.com - based on W3C-checklink/4.3 [4.42] libwww-perl/5.801 | |
| FreeNutch/Nutch-1.2 | |
| FrequencyAnalyzer/0.2 | |
| frobee.com Link Checker/0.7 | |
| fr_org_viewer (larbin2.6.3@unspecified.mail) | |
| Fumae2Oo/Nutch-1.4 | |
| FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php) | |
| g10_132_yy_x11/110510 (Mesin Pencari bahasa Indonesia; http://www.nguber.com; websearch at nguber dot com) | |
| Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) | |
| GarlikCrawler/1.1 (http://garlik.com/) | |
| GarlikCrawler/1.1 (http://garlik.com/, crawler@garik.com) | |
| GarlikCrawler/1.1 (http://garlik.com/, crawler@garlik.com) | |
| gaurav/Nutch-0.9 | |
| GematchCrawler/1.8 (http://www.gematch.com/crawler.html) | |
| GematchCrawler/2 (http://www.gematch.com/crawler.html) | |
| Generic | |
| geobucket/Nutch-1.4 | |
| GeoHasher/Nutch-1.0 (GeoHasher Web Search Engine; geohasher.gotdns.org; geo_hasher at yahoo * com) | |
| gettor/0.1 | |
| GFJ-Hosting.de_PerformanceCrawler | |
| gh-index-bot/Nutch-1.0 (GH Web Search.; lucene.apache.org; gh_email at someplace dot com) | |
| Gigabot/3.0 (http://www.gigablast.com/spider.html) | |
| Gimmie60/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.11) | |
| GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com) | |
| Gizmick/1.0 (A non-profit search engine created for the benefit of charity.; http://www.i-s-a-r-a.org/search/faq/ (take out the | |
| GMI sentiment crawler/Nutch-1.0 (GMI sentiment crawler; http://GMI.googlepages.com ; MyEmail) | |
| Gnam Gnam Spider | |
| Go http package | |
| GOFORITBOT ( http://www.goforit.com/about/ ) | |
| gold crawler | |
| Goldfire Server | |
| gonzo/1[P] (+http://www.suchen.de/faq.html) | |
| gonzo1[D] +http://www.suchen.de/faq.html | |
| gonzo1[P] +http://www.suchen.de/faq.html | |
| gooblogsearch/2.0 (http://help.goo.ne.jp/contact/) | |
| Google/4.0 | |
| google/Nutch-1.1 | |
| googlebot | |
| Googlebot (larbin2.6.3@unspecified.mail) | |
| googlebot (search@socbay.com) | |
| Googlebot-Image/1.0 | |
| Googlebot-Image/1.0 (larbin@unspecified.mail) | |
| Googlebot-Trojan/0.1.1/Nutch-1.1-dev (Trojan Scanner v.0.7; http://www.google.com/bot.html) | |
| Googlebot/1.0 (Googlebot; http://www.google.com/; googlebot@google.com) | |
| Googlebot/2.1 (+http://www.googlebot.com/bot.html) | |
| Googlebot/2.1+(+http://www.google.com/bot.html) | |
| Googlebot/2.1/Nutch-1.1 | |
| Googlebot/Googlebot-1.0 | |
| Googlebot/Nutch-1.0 | |
| Googlebot/Nutch-1.0 (Prototype; http://en.wikipedia.org/wiki/Web_crawler; donotreply at prototype dot com) | |
| Googlebot/Nutch-1.4 | |
| googleru_orgnet_viewer (larbin2.6.3@unspecified.mail) | |
| google_web (larbin2.6.3@unspecified.mail) | |
| goole.crawler/Nutch-1.0 | |
| Gperp (Gperp@163.com) | |
| Gperp (UserAgent) | |
| GreenIvory/Nutch-0.9 (GreenIvory-BlueCrane; http://greenivory.fr; v.taesch@greenivory.com) | |
| Grewatsch/Nutch-1.2 (Hochschule Lausitz(; http://www.HS-Lausitz.de ; Morris.Grewatsch@HS-Lausitz.de) | |
| gsa-bizkaia (Enterprise; T2-B84TKEFXKWWBK; gsa@bizkaia.net) | |
| gsa-crawler (Enterprise; C5-QP8LD5K6A2JJA; me@mycompany.com) | |
| gsa-crawler (Enterprise; GIX-04247; me@mycompany.com) | |
| gsa-crawler (Enterprise; M2-ANLVCZEKDC2AA; admin@jadu.net) | |
| gsa-crawler (Enterprise; M2-B3X6EDEBUC6JB; serveradm@dickinson.edu) | |
| gsa-crawler (Enterprise; M2-B49PXEELPJSAA; webmaster@bethel.edu) | |
| gsa-crawler (Enterprise; M2-FLTWN9EKUC6JS; uriah.liggett@etegent.com) | |
| gsa-crawler (Enterprise; M2-FZL5CXEKDA2AA; google_mini@tribalddb.de) | |
| gsa-crawler (Enterprise; M2-JGHL8PEKVA2JS; webmaster@bethel.edu) | |
| gsa-crawler (Enterprise; M2-JS5323WKVA6JA; dana.champine@slcc.edu) | |
| gsa-crawler (Enterprise; M2-KKXEPREKUC2AT; biggersj@hillsboroughcounty.org) | |
| gsa-crawler (Enterprise; M2-KQMM4KAKDA6JS; sesame@u.washington.edu) | |
| gsa-crawler (Enterprise; M2-KWZGA5SKCA2AB; googlemini@bvt.org.uk,helpdesk@bvt.org.uk) | |
| gsa-crawler (Enterprise; M2-PA8WZ3SKDA2AS; dana.champine@slcc.edu) | |
| gsa-crawler (Enterprise; MID-00215; jreeve@weber.k12.ut.us) | |
| gsa-crawler (Enterprise; S5-DPJCVFSVSEWAS; itsunixadmins@vanderbilt.edu) | |
| gsa-crawler (Enterprise; S5-F5KK3P63Y6NAS; datacenter2@digitalinsight.com,noc@digitalinsight.com,kenneth.lemoine@digitalinsight.com) | |
| gsa-crawler (Enterprise; S5-GE8RVQN3J4NJS; nobody@google.com) | |
| gsa-crawler (Enterprise; S5-HX5HMEAVAESJS; marcos@justdigital.com.br,marco@justdigital.com.br,aseri@justdigital.com.br) | |
| gsa-crawler (Enterprise; S5-HX5HMEAVAESJS; marcos@justdigital.com.br,mteixeira@justdigital.com.br,aseri@justdigital.com.br) | |
| gsa-crawler (Enterprise; S5-HYU2NGRTV6JJA; wwwadm@ua.es) | |
| gsa-crawler (Enterprise; S5-LRNEWUP3EUJAT; nobody@google.com) | |
| gsa-crawler (Enterprise; S5-Q7C5KR6TY8JAS; datacenter2@digitalinsight.com,noc@digitalinsight.com,kenneth.lemoine@digitalinsight.com) | |
| gsa-crawler (Enterprise; S5-RLAPLGRTU6NAT; networkadmin@figleaf.com) | |
| gsa-crawler (Enterprise; T1-EW7TFYE5SGSAS; allan.s.stilwell@nasa.gov) | |
| gsa-crawler (Enterprise; T2-A6WTP66PZJWAS; webmaster@uva.es) | |
| gsa-crawler (Enterprise; T2-CBUNHYW9ENWAB; webservices@calstate.edu) | |
| gsa-crawler (Enterprise; T2-CVDXLNS9ENWAB; julien.marchet@sword-group.com) | |
| gsa-crawler (Enterprise; T2-K6FCMUW9WQWJT; webservices@calstate.edu) | |
| gsa-crawler (Enterprise; T2-KX74XYNZFQSAS; gsa@searchtechnologies.com) | |
| gsa-crawler (Enterprise; T2-LXRKXYNZENSAA; gsa4.admin@searchtechnologies.com) | |
| gsa-crawler (Enterprise; T3-A3BDGUFT8YSGH; rgerry@fsc.follett.com) | |
| gsa-crawler (Enterprise; T3-GNGZGLT73YWB2; search@byu.edu) | |
| gsa-crawler (Enterprise; T3-H5CRHE772YWKK; tech@typhon.com) | |
| gsa-crawler (Enterprise; T3-L39FLJP3RYWGY; dlfr-france.fr-system@atos.net) | |
| gsa-crawler (Enterprise; T3-LLNR7LKTRYSQY; dlfr-france.fr-system@atos.net) | |
| gsa-crawler (Enterprise; T3-MRHJJGX73YWBJ; estauthamer@emidconsult.com) | |
| gsa-crawler (Enterprise; T3-Q3A8GJTXKYWBJ; tech@typhon.com) | |
| gsa-crawler-ARD (Enterprise; S5-C6H8MTSZNSJJB; webmaster@ard.de) | |
| gsa-crawler-ARD (Enterprise; T1-NZSVYYSPYJSJS; Technik-Support@ard.de) | |
| gsa-crawler-ARD (Enterprise; T1-NZSVYYSPYJSJS; webmaster@ard.de) | |
| gsa-crawler1 (Enterprise; S5-DPJCVFSVSEWAS; itsunixadmins@vanderbilt.edu) | |
| gsa-crawler2 (Enterprise; S5-DWXMDHSVBGSAS; itsunixadmins@vanderbilt.edu) | |
| GSLFbot | |
| GurujiImageBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html) | |
| gva/europe (Geneva Web Explorer.; http://www.genevasearch.com/; info@genevasearch.com) | |
| gyplayer/1.0 | |
| HablaCentro/1.0 (HablaCentro Crawler; http://hablacentro.com/; webmaster@hablacentro.com) | |
| HaDoot/1.0/Nutch-1.2 (Hadoot Open Source Search Engine; http://www.hadoot.com/) | |
| hao123_viewer (larbin2.6.3@unspecified.mail) | |
| Harmonia/Nutch-1.2 | |
| HarvestMan 1.4 | |
| Hatena Antenna/0.5 (http://a.hatena.ne.jp/help) | |
| HatenaScreenshot/1.0 (checker) | |
| HAZY.SPIDER/Nutch-1.4 | |
| hclsreport-crawler (+http://hclsreport.com/ | |
| HD nutch agent/1.0 | |
| HD nutch agent/1.2 | |
| HD nutch agent/Nutch-1.2 | |
| HD search agent/1.0 | |
| HD search agent/1.1 | |
| HeartRails Robot/0.1 (http://www.heartrails.com) | |
| hitcrawler_0.1 (ringken@gmail.com) | |
| hitspider (larbin2.6.3@unspecified.mail) | |
| hoge (co2h2onacl@gmail.com) | |
| hogna v0.1 | |
| holmes/3.12 (OnetSzukaj/5.0; +http://szukaj.onet.pl) | |
| holmes/3.12.4 (http://morfeo.centrum.cz/bot) | |
| holmes/3.9 | |
| HolmesBot (http://holmes.ge) | |
| HopperBot/1.1 (compatible; heritrix/3.1.0 +http://www.hopper.com) | |
| HostHarvest/0.4.28 (http://www.raskgaard.dk/~cara/hostharvest.html) | |
| HounderCrawl/Nutch-0.9 (Hounder Search Bot; http://www.flaptor.com/) | |
| http://ask.24x.info/ | |
| http://domino.research.ibm.com/comm/research_projects.nsf/pages/sai-crawler.callingcard.html | |
| http://SiteIntel.net Bot | |
| http://www.amagit.com/ | |
| http://www.baidu.com/Nutch-1.2 | |
| http://www.sidusoft.cn/Nutch-1.0 (http://www.sidusoft.cn) | |
| http://www.sidusoft.com.cn/Nutch-1.0 (http://www.sidusoft.com.cn) | |
| http://www.sina.com.cn/Nutch-1.0 (http://www.sina.com.cn) | |
| http://www.stero.pl/splywy-kajakowe | kajaki@onet.pl | tel. 601 41 88 60 | |
| http://www.uni-koblenz.de/~flocke/robot-info.txt | |
| https_page_test (larbin2.6.3@unspecified.mail) | |
| https_url_test (larbin2.6.3@unspecified.mail) | |
| HTTrack Website Copier/3.0x (offline browser; web mirror utility) | |
| HuaweiSymantec spider (compatible; MSIE 8.0; wangdanfeng@huaweisymantec.com) | |
| Huaweisymantecspider (compatible; MSIE 8.0; DSE-support@huaweisymantec.com) | |
| Huaweisymantecspider (compatible; MSIE 8.0; wangdanfeng@huaweisymantec.com) | |
| HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/cn/IRL/spider) | |
| HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/en/IRL/spider) | |
| HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.talktalk.co.uk/products/virus-alerts/) | |
| I am a friendly (BETA) R.O.B.O.T. I am crawling the web. If I do not respect your robots.txt, e-mail me. Soon I will have a website. More info at ggggggg73@hotmail.com | |
| ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com) | |
| ia_archiver(OS-Wayback) | |
| ia_archiver-web.archive.org | |
| iBot | |
| ibuena (robot@ibuena.net) | |
| ICC-Crawler (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html) | |
| ICC-Crawler(Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html) | |
| ICC-Crawler(Mozilla-compatible; icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp; http://kc.nict.go.jp/project1/crawl.html) | |
| ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html) | |
| ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.csk.com/sys-nishi/) | |
| Iceweasel2.0.0.16/Nutch-1.0 (Webbrowser; http://iceweasel.com; info@iceweasel.com) | |
| Iceweasel2.0.0.16/Nutch-1.1 (Webbrowser; http://iceweasel.com; info@iceweasel.com) | |
| ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html) | |
| ichiro/3.0 (http://help.goo.ne.jp/help/article/1142) | |
| ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html) | |
| ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html) | |
| ichiro/6.0 (http://help.goo.ne.jp/door/crawler.html) | |
| ict/Nutch-1.1 (ict; http://idoc.ict.ac.cn) | |
| iDEAL/Nutch-1.2 | |
| IE-9 | |
| IE/6.0 (compatible; heritrix/1.14.4 +http://202.206.249.110) | |
| IE/8.0 (compatible; heritrix/1.14.3+http://179.39.11.19/) | |
| IE/Nutch-1.0 | |
| IE6_IE7_IE8 (larbin2.6.3@unspecified.mail) | |
| IIT Bombay CFILT NLP Bot/Nutch-1.1 (IITB CFILT Crawler) | |
| IITB-CFILT-Bot/Nutch-1.1 (This is the crawler of IIT Bombay, India. The data will be used for research purposes.; http://www.cfilt.iitb.ac.in/; pb@cse.iitb.ac.in) | |
| ikubinfo/Nutch-1.2 (research) | |
| ilocal company search | |
| IlseBot/1.1 | |
| imrd-1.2 (walker@imrd.com) | |
| IMRD/1.0 (robot@imrd.com) | |
| imrdbot1.0 (imrdbot@imrd.com) | |
| INDEXACIONEDUCAREX/Nutch-1.0 | |
| InetURL:/1.0 | |
| Infoaxe./Nutch-0.9 | |
| Infoaxe./Nutch-1.0 | |
| informerabot/Nutch-1.2 (test crawler; http://www.informera.de; info at informera dot de) | |
| innocrawl/Nutch-1.4 | |
| innoventage/Nutch-1.0 (poc; www.google.com; proof of concept) | |
| innoventage/Nutch-1.0 (poc; www.innoventage.com; proof of concept) | |
| inonsh/Nutch-1.1 | |
| inonsh/Nutch-1.1 (Web Crawler; www.inonsh.com; webmaster@inonsh.com) | |
| inonsh/Nutch-1.2 | |
| Internet Explorer | |
| Internet Explorer/8.0 (compatible; heritrix/1.10.1+http://192.168.0.1) | |
| intumit/Nutch-1.1 | |
| intumit/Nutch-1.2 | |
| intumit/Nutch-1.2 (intumit) | |
| intumit/Nutch-1.3 | |
| intumit/Nutch-1.4 | |
| ipec/Nutch-1.2 (ipec; ipec.cn; info@ipec.cn) | |
| IPv6'n/0.1; +http://draft.scyphus.co.jp/ipv6.html | |
| ir1.0 (jessievop@hotmail.com) | |
| IS Alpha/Nutch-1.0 | |
| IS Alpha/Nutch-1.1 | |
| IS Alpha/zcspider-0.1 | |
| IS Alpha/zcspider-0.1 (Wisshow Network Technology LTD.CO) | |
| Isara/Isara-1.0 (A non-profit search engine benefiting charity.; http://www.isara.org; webmaster@isara.org) | |
| Isara/Isara-1.0 (A non-profit search engine for the benefit of charity.; http://www.isara.org; search@isara.org) | |
| Isara/Nutch-1.2 (A non-profit search engine for the benefit of charity.; http://www.isara.org; community@isara.org) | |
| Itah/1.1 (Itah; http://www.itah.com; support@itah.com) | |
| itim | |
| iVia/5.0 Adders (http://infomine.ucr.edu:80/iVia/useragents.shtml) | |
| Jabse.com Crawler v.1.0 www.jabse.com/crawler.php//imagecrawler | |
| jadu-crawler (Enterprise; M2-ANLVCZEKDC2AA; admin@jadu.net) | |
| Jakarta Commons-HttpClient/3.0.1 | |
| Jakarta Commons-HttpClient/3.1 | |
| James Prabhu/Nutch-1.0 (Nutch spiderman; http://www.cosmoage.com ; MyEmail) | |
| Jasmine/Nutch-1.2 (searching for The One; www.eCartesia.com; Robin.Hood@SherwoodForest.com) | |
| Java/1.5.0 | |
| Java/1.5.0_01 | |
| Java/1.5.0_13 | |
| Java/1.5.0_14 | |
| Java/1.5.0_15 | |
| Java/1.5.0_16 | |
| Java/1.5.0_17 | |
| Java/1.5.0_18 | |
| Java/1.5.0_19 | |
| Java/1.6.0 | |
| Java/1.6.0_01 | |
| Java/1.6.0_03 | |
| Java/1.6.0_06 | |
| Java/1.6.0_07 | |
| Java/1.6.0_10 | |
| Java/1.6.0_12 | |
| Java/1.6.0_13 | |
| Java/1.6.0_15 | |
| Java/1.6.0_16 | |
| Java/1.6.0_17 | |
| Java/1.6.0_18 | |
| Java/1.6.0_19 | |
| Java/1.6.0_20 | |
| Java/1.6.0_21 | |
| Java/1.6.0_22 | |
| Java/1.6.0_24 | |
| Java/1.6.0_26 | |
| Java/1.6.0_27 | |
| Java/1.7.0_02 | |
| JavaCrawler/1.1 | |
| Jawler Beta - arman(dot)vartan(at)itwm(dot)fhg(dot).com | |
| Jawler Beta - arman(dot)vartan(at)itwm(dot)fhg(dot)de | |
| jcrawler4/Nutch-1.3 (bot) | |
| jcrobot | |
| Jennifer/Nutch-0.9 (Jennifer; Jennifer; Jennifer) | |
| Jet/Nutch-1.4 | |
| jetbot(+http://premium.jetrun.jp/help/robot.html) | |
| JikeSpider Mozilla/5.0 (compatible; JikeSpider; +http://shoulu.jike.com/spider.html) | |
| JoBo/1.3 (http://www.matuschek.net/jobo.html) | |
| Joy CMU/Nutch-1.2 (Data collection for research ; http://mlt.sv.cmu.edu/joy; joy@cs.cmu.edu) | |
| jrobot (jazy333@gmail.com) | |
| Just another academic search engine/Nutch-1.2 (search; fcprogs (gmail)) | |
| JUST-CRAWLER(+http://www.justsystems.com/jp/tech/crawler/) | |
| Jyxobot/1 | |
| k2spider | |
| kacst/Nutch-1.4 | |
| KAIST AITrc Crawler( crawler . crawling . | |
| kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler) | |
| kapooha/Nutch-1.4 (crawler; http://www.kapooha.com; aeroganic@gmail.com) | |
| KAS/Nutch-1.2 (Nutch ARGUS; MyEmail) | |
| Kavande Crawler 1.0/Nutch-1.4 ( Iranian National Web Crawler ; kh3rad@gmail.com) | |
| Kavande Crawler 1.0/Nutch-1.4 (Iranian National Web Crawler) | |
| Kavande Crawler 1.0/Nutch-1.4 (Kavande) | |
| Kavande Crawler 1.0/Nutch-1.4-dev (Iranian National Web Crawler) | |
| Keybot Translation-Search-Machine | |
| KeywordSearchTool.co/Nutch-1.4 (http://KeywordSearchTool.co/robot) | |
| kindsight/Nutch-1.0 (kscrawler; www.projectrialto.com; crawler@projectrialto.com) | |
| Kiodia Spider/Nutch-1.4 | |
| KiwiStatus (NZS.com)/0.2 (NZS.com KiwiStatus Spider, Local Search New Zealand; http://www.nzs.com; bot-at-nzs dot com) | |
| KiwiStatus/0.3 (NZS.com New Zealand Search; http://www.nzs.com/kiwi-status/) | |
| KoopitBot/0.1 (http://www.koopit.com/) | |
| KS Crawler/Nutch-1.0 (http://www.kindsight.net/kscrawler; crawler@kindsight.net) | |
| KS Spider/Nutch-0.9 | |
| KSCrawler/Nutch-1.0 (http://www.kindsight.net/en/kscrawler; crawler@kindsight.net) | |
| Kyoto-Crawler/2.0 (Mozilla-compatible; kyoto-crawler-contact(at)nlp(dot)kuee(dot)kyoto-u(dot)ac(dot)jp; http://nlp.ist.i.kyoto-u.ac.jp/) | |
| L.webis/0.19 | |
| L.webis/0.32 (http://webalgo.iit.cnr.it/index.php?pg=lwebis) | |
| L.webis/0.35 (http://webalgo.iit.cnr.it/index.php?pg=lwebis) | |
| L.webis/0.40 (http://webalgo.iit.cnr.it/index.php?pg=lwebis) | |
| L.webis/0.42 (http://webalgo.iit.cnr.it/index.php?pg=lwebis) | |
| L.webis/0.44 (http://webalgo.iit.cnr.it/index.php?pg=lwebis) | |
| L.webis/0.47 (http://webalgo.iit.cnr.it/index.php?pg=lwebis) | |
| L.webis/0.50 (http://webalgo.iit.cnr.it/index.php?pg=lwebis) | |
| L.webis/0.53 (http://webalgo.iit.cnr.it/index.php?pg=lwebis) | |
| L.webis/0.78 (http://webalgo.iit.cnr.it/index.php?pg=lwebis) | |
| L.webis/0.85 (http://webalgo.iit.cnr.it/index.php?pg=lwebis) | |
| L.webis/posttool (http://webalgo.iit.cnr.it/index.php?pg=lwebis) | |
| larbin (larbin@unspecified.mail) | |
| larbin (xieys19860101@163.com) | |
| larbin-m (larbin@magellan.cc) | |
| larbin_2.6.3 | |
| larbin_2.6.3 (airborne08@gmail.com) | |
| larbin_2.6.3 (changzhiwin@163.com) | |
| larbin_2.6.3 (chowtin@qq.com) | |
| larbin_2.6.3 (cswchli@polyu.edu.hk) | |
| larbin_2.6.3 (fuxiang90@gamil.com) | |
| larbin_2.6.3 (gordon0640@gmail.com) | |
| larbin_2.6.3 (h0451bct@rz.hu-berlin.de) | |
| larbin_2.6.3 (hhgg20000@21cn.com) | |
| larbin_2.6.3 (hustguoguo@qq.com) | |
| larbin_2.6.3 (hwq_super@126.com) | |
| larbin_2.6.3 (kevinew.1221@gmail.com) | |
| larbin_2.6.3 (laomadadupi@gmail.com) | |
| larbin_2.6.3 (larbin2.6.3@unspecified.mail) | |
| larbin_2.6.3 (larbin263@126.com) | |
| larbin_2.6.3 (larbincrawler@gmail.com) | |
| larbin_2.6.3 (myemail@address.co.uk) | |
| larbin_2.6.3 (pcwang@pchome.com.tw) | |
| larbin_2.6.3 (talex@astaro.com) | |
| larbin_2.6.3 (tmdss@163.com) | |
| larbin_2.6.3 (wangqun09@mails.gucas.ac.cn) | |
| larbin_2.6.3 (whoismickey@gmail.com) | |
| larbin_2.6.3 (ymyang2010@hotmail.com) | |
| larbin_2.6.4 (larbin2.6.4@unspecified.mail) | |
| LargeSmall Crawler (LargeSmall; http://onespot.com; info@onespot.com) | |
| larsrc (nobody@dreamland.com) | |
| lb-spider/Mozilla/5.0 Gecko/20100101 Firefox/10.0.2 (lb-spider; http://www.linkbutler.de/spider; spider@linkbutler.de) | |
| Leo Chow/Nutch-0.9 (Nutch spiderman; http://www-u1.icil.net; MyEmail) | |
| lessonupgrades/Nutch-1.2 | |
| LexxeBot/1.0 (lexxebot@lexxe.com) | |
| lhsoso.com/1.0 | |
| libcurl-agent/1.0 | |
| libiViaCore/5.0 generate_infomine_category_classifiers (http://changesiv.ucr.edu/useragents.html) | |
| libiViaCore/5.0 generate_infomine_category_classifiers (http://irsdev.intute.ac.uk:8999/useragents.html) | |
| librabot/1.0 (+http://search.msn.com/msnbot.htm) | |
| librabot/1.0 (compatible; MSIE 6.0; Windows NT 5.0) | |
| librabot/2.0 (+http://academic.research.microsoft.com/) | |
| librabot/2.0 (+http://search.msn.com/msnbot.htm) | |
| Libri amici, libri magistri | |
| libwww-perl/5.805 | |
| libwww-perl/5.831 | |
| libwww-perl/5.834 | |
| LICrawler licrawler@gmail.com | |
| Lijit Crawler (+http://www.lijit.com/robot/crawler) | |
| Lijit Crawler/Nutch-0.9 (Reports crawler; http://www.lijit.com/robot/crawler; info(a)lijit(d)com) | |
| LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/robot/crawler; info(a)lijit(d)com) | |
| LinguaBot/v0.001-dev (MultiLinual Sarch Engine v0.001; LinguaSeek; admin at linguaseek dot com) | |
| Linguee Bot (bot@linguee.com) | |
| Linguee Bot (http://www.linguee.com/bot; bot@linguee.com) | |
| LinkAider (http://linkaider.com/crawler/) | |
| LinkAlarm/3.0 | |
| LinkChecker/3.4 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/4.8 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/4.9 (+http://linkchecker.s/urceforge.net/) | |
| LinkChecker/4.9 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/5.1 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/5.2 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/5.3 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/6.2 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/6.3 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/6.4 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/6.5 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/6.6 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/6.7 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/6.8 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/6.9 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/7.0 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/7.1 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/7.2 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/7.3 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/7.4 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/7.5 (+http://linkchecker.sourceforge.net/) | |
| LinkStar Research Crawler (http://linkstar.com/) | |
| LinkWalker/2.0 | |
| linky | |
| Lisboa/Nutch-1.2 | |
| LitlrBot (http://litlr.me/bot.html) | |
| livedoor ScreenShot/0.10 | |
| li_viewer (larbin2.6.3@unspecified.mail) | |
| lmspider/Nutch-0.9-dev (For research purposes.; www.nuance.com; lmspider@nuance.com) | |
| lmspider/Nutch-1.2 (For research purposes.; www.nuance.com; tegic-SMS atdomain nuance dot com) | |
| lobot/0.3 (http://www.lofoso.com/robot.html) | |
| local/Nutch-1.2 | |
| LOCIndexer/Nutch-1.0 | |
| LookBot | |
| loosos_bot/Loos-1.2 (loosos_crawler_bot; loosos.com; bot@loosos.com) | |
| loosos_bot/loosos v1.0 (loosos.com crawler; loosos.com; loosos@126.com) | |
| LucidMedia ClickSense/4.0 (support@lucidmedia.com; http://www.lucidmedia.com/) | |
| Lycos_Spider_(modspider) | |
| LynnBot/1 | |
| Lynx/2.8.4rel.1 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.6c () | |
| Lynx/Lynx(2.0) | |
| Lynx/Lynx(2.2) (Lynx) | |
| Lynx/Lynx(8899) | |
| Lynx/Nutch-1.4 | |
| m0cca.com/Nutch-1.1 (www.m0cca.com; crawler@m0cca.com) | |
| M42bot/Nutch-1.0 (Prototype; http://en.wikipedia.org/wiki/Web_crawler; donotreply at prototype dot com) | |
| magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net) | |
| magpie-crawler/1.1 (U; Linux i386; en-GB; +http://www.brandwatch.net) | |
| mahonie, neofonie search:robot (http://spider.neofonie.de)/Nutch-1.4 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to spider@neofonie.de; http://spider.neofonie.de) | |
| mahonie, neofonie search:robot/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to spider@neofonie.de; http://spider.neofonie.de; spider@neofonie.de) | |
| Mail.Ru/1.0 | |
| Mail.RU/2.0 | |
| male.com.sg/Nutch-1.1 (http://www.male.com.sg; crawler@male.com.sg) | |
| male.com.sg/Nutch-1.1 (www.male.com.sg; crawler@male.com.sg) | |
| MANTIS/Nutch-1.4 | |
| Mediapartners-Google | |
| Medio HttpUrlSearchProvider(medio.com) | |
| MegaSpider | |
| MELANI/Nutch-0.9 (Nutch spiderman; http://www.melani.admin.ch; MyEmail) | |
| memkompreneble | |
| mercury/Nutch-1.2 | |
| merocrawl/Nutch-1.4 (merobase crawler; www.merobase.com; info@merobase.com) | |
| mesiuma/Nutch-1.0 | |
| MetaGeneratorCrawler/1.1 (www.metagenerator.info) | |
| Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner.de/) | |
| Metaspinner/1.0 (Metaspinner Search Engine; http://www.meta-spinner.de/; support@meta-spinner.de) | |
| MetaTagRobot/1.9 (http://www.widexl.com/remote/search-engines/metatag-analyzer.html) | |
| MetaTagRobot/2.1 (http://www.seocentro.com/tools/search-engines/metatag-analyzer.html) | |
| Metz/Nutch-1.2 | |
| Microsoft Bing Mobile SocialStreams Bot | |
| Midas/Nutch-1.0 (Nutch Spider; www.unal.edu.co; yamile.jaime@gmail.com) | |
| MireoBot | |
| mixi-mobile-converter/1.0 (http://mixi.jp/) | |
| miyatai/Nutch-1.4 | |
| MLBot (www.metadatalabs.com/mlbot) | |
| mmonitor/Nutch-1.0 (MarkMonitor Robots; http://www.markmonitor.com; tlin@markmonitor.com) | |
| mnoGoSearch (http://www.truthsearch.us/) | |
| MnoGoSearch/3.3.8 | |
| MnoGoSearch/3.3.9 | |
| Mobier/Mobier-1.0 (http://mobier.mobi/; abc@mobier.mobi) | |
| Mobier/Mobier-1.0 (http://mobier.mobi/; service@mobier.mobi) | |
| Modiphibot/0.9 (Rev.Nagayama1; http://www.modiphi.com/) | |
| Mojolicious (Perl) | |
| MOMspider/1.00 libwww-perl/0.40 | |
| Monkeyworx Spider/Nutch-1.2 (http://example.com; noreply@example.com) | |
| mootlaw/Nutch-1.2 (MootLaw Searc Bot; mootlaw.com; eric at moot law dot com) | |
| Motoricerca-Robots.txt-Checker/1.0 (http://tool.motoricerca.info/robots-checker.phtml) | |
| Mozilla crawl/5.0 (compatible; fairshare.cc +http://fairshare.cc) | |
| mozilla firefox | |
| Mozilla incompatible Wget (http://www.artiesoft.com/lexxbot.php mailto:info-429artiesoft.com) | |
| Mozilla/4.0 (+http://www.up.ru) | |
| Mozilla/4.0 (compatible: FDSE robot) | |
| Mozilla/4.0 (compatible; Coosto linkresolver 1.0; http://www.coosto.nl/) | |
| Mozilla/4.0 (compatible; Vagabondo/4.0; http://webagent.wise-guys.nl/) | |
| Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) | |
| Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) | |
| Mozilla/4.0 (compatible; ApocalXExplorerBot/0.9; +http://explorer.apocalx.com/bot.php) | |
| Mozilla/4.0 (compatible; http://search.thunderstone.com/texis/websearch/about.html) | |
| Mozilla/4.0 (compatible; Kwaclebot/0.8; http://www.kwacle.com/kbot.html) | |
| Mozilla/4.0 (compatible; MSIE 4.01; Windows 95; http://www.freefind.com/spider.html ) | |
| Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19.123.2.733) OrangeBot-Mobile 2008.0 (mobilesearch.support@orange-ftgroup.com) | |
| Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 5.0 Robot) | |
| Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 6.0 Robot) | |
| Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Instituto Cervantes | |
| Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; FeedFinder-2.0; http://bloggz.se/crawler) | |
| Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com) | |
| Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot) | |
| mozilla/4.0 (compatible; msie 6.0; windows 2000) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) (larbin2.6.3@unspecified.mail) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) (larbin@unspecified.mail) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Embedded Web Browser from: http://bsalsa.com/; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322; +http://www.archive.org) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 2.0.50727; CIBA; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 2.0.50727; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Embedded Web Browser from: http://bsalsa.com/; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts; GTB5; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 2.0.50727; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; InfoPath.1; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB5; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 1.1.4322; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB6.4; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 2.0.50727; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 2.0.50727; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; TuneUp HTML Client Embedded Web Browser from: http://bsalsa.com/; .NET CLR 1.1.4322; InfoPath.2; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; QQDownload 1.7; EmbeddedWB 14.52 from: http://www.bsalsa.com/ EmbeddedWB 14.52; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; TencentTraveler 4.0; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 2.0.50727; CIBA; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot) | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; FDM; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; FunWebProducts; GTB6; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 1.0.3705; Media Center PC 4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 1.1.4322; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; AskTB5.3) | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; FunWebProducts; MindQuizSearchToolbar 1.1; GTB6.5; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 1.0.3705; Media Center PC 4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 1.1.4322; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; AskTB5.3) | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6.4; FunWebProducts; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 1.0.3705; Media Center PC 4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 1.1.4322; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; AskTB5.3) | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; livebot-searchsense/0.1; +http://search.msn.com/msnbot.htm) | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; TencentTraveler 4.0; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; IE7Pro) | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Webaroo/1.3.940; MSDigitalLocker; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Wysigot 6.01; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; FDM; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; MS-RTC LM 8; InfoPath.2; OfficeLiveConnector.1.3; OfficeLivePatch.1.3; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; WOW64; User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://bsalsa.com) ; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; +http://www.europarchive.org) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; Embedded Web Browser from: http://bsalsa.com/; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30; IE7Pro; .NET CLR 3.0.04506.648; .NET CLR 1.1.4322; .NET CLR 3.5.21022) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://bsalsa.com) ; FDM; .NET CLR 2.0.50727) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://bsalsa.com) ; Tablet PC 1.7; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR 2.0.50727; IE7Pro; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; InfoPath.2; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) | |
| Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.jsp) | |
| Mozilla/4.0 (larbin2.6.3@unspecified.mail) | |
| Mozilla/4.0 (Toread-Crawler/1.1; +http://news.toread.cc/crawler.php) | |
| Mozilla/4.0 http://referencement.abaabaa.com/add_url.php : Easy referencement (version 1.0) | |
| Mozilla/4.0 http://referencement.abaabaa.com/add_url_GRATUIT.php : Easy referencement (version 1.0) | |
| Mozilla/4.0(compatible;MSIE7.0) (larbin2.6.3@unspecified.mail) | |
| Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html ) | |
| Mozilla/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html ; agentname at setooz dot_com ) | |
| Mozilla/5.0 ((compatible;+Baiduspider/2.0;++http://www.baidu.com/search/spider.html)) | |
| Mozilla/5.0 (compatable; FNRBot/1.0; http://www.fairwaysandroughs.com/bot.php) | |
| Mozilla/5.0 (compatible; heritrix/3.1.0 +http://mozilla.org) | |
| Mozilla/5.0 (compatible; +http://pk.com) | |
| Mozilla/5.0 (compatible; +http://www.evri.com/evrinid) | |
| Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620 | |
| Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620 | |
| Mozilla/5.0 (compatible; 80bot/0.71; http://www.80legs.com/spider.html;) Gecko/2008032620 | |
| Mozilla/5.0 (compatible; abby/1.0; +http://www.ellerdale.com/crawler.html) | |
| Mozilla/5.0 (compatible; AcoonBot/4.10.6; +http://www.acoon.de/robot.asp) | |
| Mozilla/5.0 (compatible; AcoonBot/4.10.7; +http://www.acoon.de/robot.asp) | |
| Mozilla/5.0 (compatible; AhrefsBot/1.0; +http://ahrefs.com/robot/) | |
| Mozilla/5.0 (compatible; AhrefsBot/2.0; +http://ahrefs.com/robot/) | |
| Mozilla/5.0 (compatible; AhrefsBot/3.0; +http://ahrefs.com/robot/) | |
| Mozilla/5.0 (compatible; aiHitBot/1.0-DS; +http://www.aihit.com/) | |
| Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/) | |
| Mozilla/5.0 (compatible; aiHitBot/1.1; +http://www.aihit.com/) | |
| Mozilla/5.0 (compatible; akula/k311; +http://k311.fd.cvut.cz/) | |
| Mozilla/5.0 (compatible; AntBot/1.0; +http://www.ant.com/) | |
| Mozilla/5.0 (compatible; ApptusBot/1.0; +http://www.apptus.com) | |
| Mozilla/5.0 (compatible; archive.org_bot +http://pandora.nla.gov.au/crawl.html) | |
| Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot) | |
| Mozilla/5.0 (compatible; archive.org_bot +http://www.bne.es/es/LaBNE/PreservacionDominioES/AvisoWebmasters/index.html) | |
| Mozilla/5.0 (compatible; archive.org_bot/1.14.1 +http://www.loc.gov/minerva/crawl.html) | |
| Mozilla/5.0 (compatible; archive.org_bot/1.15.4 +http://www.loc.gov/minerva/crawl.html) | |
| Mozilla/5.0 (compatible; archive.org_bot/1.5.0 +http://www.bne.es/es/LaBNE/PreservacionDominioES/AvisoWebmasters/index.html) | |
| Mozilla/5.0 (compatible; archive.org_bot/1.5.0 +http://www.loc.gov/minerva/crawl.html) | |
| Mozilla/5.0 (compatible; archive.org_bot/1.5.4 +http://www.loc.gov/minerva/crawl.html) | |
| Mozilla/5.0 (compatible; archive.org_bot/1.8.0 +http://www.loc.gov/minerva/crawl.html) | |
| Mozilla/5.0 (compatible; archive.org_bot/3.1.0 +http://www.archive.org/details/archive.org_bot) | |
| Mozilla/5.0 (compatible; archive.org_bot/3.1.1 +http://pandora.nla.gov.au/crawl.html) | |
| Mozilla/5.0 (compatible; archive.org_bot/3.1.1 +http://www.bne.es/es/LaBNE/PreservacionDominioES/AvisoWebmasters/index.html) | |
| Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.archive.org) | |
| Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.bne.es/es/LaBNE/PreservacionDominioES/AvisoWebmasters/index.html) | |
| Mozilla/5.0 (compatible; archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html) | |
| Mozilla/5.0 (compatible; archive_crawler/3.0.0-SNAPSHOT-20091205.013431 +http://www.archive.org/details/archive_crawler) | |
| Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml) | |
| Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://www.ask.com/about/help/webmasters) | |
| Mozilla/5.0 (compatible; Atomic5 Search; +http://atomic5.com/) | |
| Mozilla/5.0 (compatible; attributor/1.13.2 +http://www.attributor.com) | |
| Mozilla/5.0 (compatible; Ayna +http://www.ayna.com) | |
| Mozilla/5.0 (compatible; ayna-crawler +http://www.ayna.com) | |
| Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) | |
| Mozilla/5.0 (compatible; BeatrixxX/1.14.4 +http://beerwurstonkelz.org) | |
| Mozilla/5.0 (compatible; Bender; http://benderthewebrobot.tumblr.com) | |
| Mozilla/5.0 (compatible; Bender; http://sites.google.com/site/bendercrawler) | |
| Mozilla/5.0 (compatible; bingbot-media/2.0 +http://www.bing.com/bingbot.htm) | |
| Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) | |
| Mozilla/5.0 (compatible; Birubot/1.0) Gecko/2009032608 Firefox/3.0.8 | |
| Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; crawler@bixolabs.com) | |
| Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; crawler@mail.bixolabs.com) | |
| Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot) | |
| Mozilla/5.0 (compatible; BlinkaCrawler/1.0; +http://www.blinka.jp/crawler/) | |
| Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/) | |
| Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/bnf.html) | |
| Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html) | |
| Mozilla/5.0 (compatible; bush-crawler +http://www.arkevista.com) | |
| Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly.html) Gecko/2009032608 Firefox/3.0.8 | |
| Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8 | |
| Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html) | |
| Mozilla/5.0 (compatible; cdlwas_bot; heritrix/1.14.1 +http://webarchives.cdlib.org/p/webmasters) | |
| Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/) | |
| Mozilla/5.0 (compatible; CligooRobot/2.0; +http://www.cligoo.de/wk/technik.php) | |
| Mozilla/5.0 (compatible; crawler/3.0.0 +http://www.notconfigured.com/) | |
| Mozilla/5.0 (compatible; crawler/3.1.0-SNAPSHOT-20110914.213704 +http://www.notconfigured.com/) | |
| Mozilla/5.0 (compatible; cuil.Com/1.14.3 +http://www.cuil.com) | |
| Mozilla/5.0 (compatible; Cyberz Communication Agent; +http://www.cyberz.co.jp/) | |
| Mozilla/5.0 (compatible; DCPbot/1.0; +http://domains.checkparams.com/) | |
| Mozilla/5.0 (compatible; DCPbot/1.1; +http://domains.checkparams.com/) | |
| Mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html | |
| Mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html) | |
| Mozilla/5.0 (compatible; discobot/2.0; +http://discoveryengine.com/discobot.html) | |
| Mozilla/5.0 (compatible; Dolphin/1.0; +http://tele-house.ru/crawler.html) | |
| Mozilla/5.0 (compatible; Dolphin/1.0; +http://telehouse.ru/crawler.html) | |
| Mozilla/5.0 (compatible; dombot +http://domlog.org) | |
| Mozilla/5.0 (compatible; dooplr/2.3.5 +http://www.yahoo.com) | |
| Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org) | |
| Mozilla/5.0 (compatible; dotSemantic/1.0; +http://www.dotsemantic.org) | |
| Mozilla/5.0 (compatible; Dow Jones Searchbot) | |
| Mozilla/5.0 (compatible; EllerdaleBot/ 1.0; +http://www.ellerdale.com/crawler.html) | |
| Mozilla/5.0 (compatible; en-US; ReverseGet/1.0; http://reverseget.com/; robot@reverseget.com) | |
| Mozilla/5.0 (compatible; Estbot/1.0; http://www.estsoft.com/) | |
| Mozilla/5.0 (compatible; EventGuruBot/1.0; +http://www.eventguru.com/spider.html) | |
| Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot) | |
| Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot) | |
| Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot) | |
| Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com) | |
| Mozilla/5.0 (compatible; Falconsbot; +http://iws.seu.edu.cn/services/falcons/contact_us.jsp) | |
| Mozilla/5.0 (compatible; Falconsbot; +http://ws.nju.edu.cn/falcons/) | |
| Mozilla/5.0 (compatible; Feedtrace-bot/0.2; bot@feedtrace.com) | |
| Mozilla/5.0 (compatible; FlipboardBrowserProxy/1.1; +http://flipboard.com/browserproxy) | |
| Mozilla/5.0 (compatible; FlipboardProxy/1.1; +http://flipboard.com/browserproxy) | |
| Mozilla/5.0 (compatible; FMSH; http://ticmigrations.fr/) | |
| Mozilla/5.0 (compatible; foobot/0.0.1 +http://foofind.com) | |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | |
| Mozilla/5.0 (compatible; GurujiBot/1.0; +http://www.guruji.com/en/WebmasterFAQ.html) | |
| Mozilla/5.0 (compatible; Hailoobot/1.2; +http://www.hailoo.com/spider.html) | |
| Mozilla/5.0 (compatible; HatoPoppo/1.0b; http://potaru.com/HatoPoppo.html)/Nutch-1.2 | |
| Mozilla/5.0 (compatible; heritrix/ 1.14.3+http://202.200.48.59) | |
| Mozilla/5.0 (compatible; heritrix/${pom.version} +http://crawler.archive.org) | |
| Mozilla/5.0 (compatible; heritrix/${pom.version} +http://seekda.com) | |
| Mozilla/5.0 (compatible; heritrix/${pom.version} +http://webservices.seekda.com) | |
| Mozilla/5.0 (compatible; heritrix/${pom.version} +http://www.adsafemedia.com) | |
| Mozilla/5.0 (compatible; heritrix/1.10.0+http://bbs.ustc.edu.cn/) | |
| Mozilla/5.0 (compatible; heritrix/1.10.1 +http://www2.zzu.edu.cn/xxgl) | |
| Mozilla/5.0 (compatible; heritrix/1.10.2 +http://netarkivet.dk/website/info.html; http://eia.libis.lt) | |
| Mozilla/5.0 (compatible; heritrix/1.12.0 +http://www.webcorp.org.uk/) | |
| Mozilla/5.0 (compatible; heritrix/1.12.1 +http://archive.aueb.gr) | |
| Mozilla/5.0 (compatible; heritrix/1.12.1 +http://curiousgeorge.info) | |
| Mozilla/5.0 (compatible; heritrix/1.12.1 +http://eia.libis.lt) | |
| Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.webarchiv.cz) | |
| Mozilla/5.0 (compatible; heritrix/1.12.1b +http://netarkivet.dk/website/info.html) | |
| Mozilla/5.0 (compatible; heritrix/1.14.0 +http://10.0.11.109) | |
| Mozilla/5.0 (compatible; heritrix/1.14.0 +http://lib.csu.edu.cn/pub/zndxtsg/index.htm) | |
| Mozilla/5.0 (compatible; heritrix/1.14.0 +http://webarchive.nlc.gov.cn) | |
| Mozilla/5.0 (compatible; heritrix/1.14.0 +http://www.banq.qc.ca/archivageweb/carte.html) | |
| Mozilla/5.0 (compatible; heritrix/1.14.0 +http://www.cienciapt.net) | |
| Mozilla/5.0 (compatible; heritrix/1.14.0+http://www.nationallibrary.fi/) | |
| Mozilla/5.0 (compatible; heritrix/1.14.1 +http://www.cdlib.org/) | |
| Mozilla/5.0 (compatible; heritrix/1.14.1 +http://www.enxnc.com.cn) | |
| Mozilla/5.0 (compatible; heritrix/1.14.1 +http://www.nlc.gov.cn/) | |
| Mozilla/5.0 (compatible; heritrix/1.14.1 +http://www.padi.cat) | |
| Mozilla/5.0 (compatible; heritrix/1.14.1 +http://www.webarchive.org.uk/) | |
| Mozilla/5.0 (compatible; heritrix/1.14.1 +http://www.widernet.org) | |
| Mozilla/5.0 (compatible; heritrix/1.14.1+http://192.168.0.1) | |
| Mozilla/5.0 (compatible; heritrix/1.14.2 +http://www.novinky.cz) | |
| Mozilla/5.0 (compatible; heritrix/1.14.2 yptrino +http://www.buddybuzz.net/yptrino) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://127.0.0.1) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://192.168.0.1) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://192.168.0.91) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://202.200.49.123/) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://archive.org) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://badcheese.com) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://dtn.net) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://ece.ut.ac.ir) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://europarchive.org) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://mytest.com) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://nipg.inf.elte.hu) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://web.archive.org) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.accelobot.com) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.cs.ucl.ac.uk/staff/C.Raiciu/crawl.html) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.google.com) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.it.kmitl.ac.th) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.webarchiv.cz) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.xbmu.edu.cn/) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3+http://202.200.59.7) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3.r6601 +http://www.buddybuzz.net/yptrino) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3.r6601 +http://www.yp.com) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3.r6601+http://www.buddybuzz.net/yptrino) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3.r6601+http://www.yellowpages.com/about/legal/crawl) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3.r6601+http://www.yp.com) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://192.168.1.11) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://221.204.244.37) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://ce.yazduni.ac.ir) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://censoweb.speed.dcc.ufmg.br) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://graphseeker.com) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://hpsg.fu-berlin.de) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://iran.ir/) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://it.usyd.edu.au/~smerity/schwa/crawler/) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://lib.ccut.edu.cn/) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://susanszky.tatk.elte.hu) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://swr.de) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.attrakt.com) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.cacak-dis.rs) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.ccert.edu.cn/) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.dlsu.edi.ph) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.findthatfile.com) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.google.com) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.iran.ir/) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.kb.se) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.speed.dcc.ufmg.br) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.test.test) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.youscribe.com) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4+http://today.hit.edu.cn/) | |
| Mozilla/5.0 (compatible; heritrix/1.15.1-200807172326 +http://www.accelobot.com) | |
| Mozilla/5.0 (compatible; heritrix/1.15.4-200909230304 +http://pirun.ku.ac.th/~g5165280) | |
| Mozilla/5.0 (compatible; heritrix/1.15.5-201006012335 +http://yandex.ru) | |
| Mozilla/5.0 (compatible; heritrix/1.5 +http://www.metacarta.com) | |
| Mozilla/5.0 (compatible; heritrix/1.6.0 +http://www.offis.de) | |
| Mozilla/5.0 (compatible; heritrix/1.6.0 +http://www.schluetersche.de) | |
| Mozilla/5.0 (compatible; heritrix/1.6.0-OFFIS +http://www.test.test) | |
| Mozilla/5.0 (compatible; heritrix/1.8.0 +http://www.bl.uk/) | |
| Mozilla/5.0 (compatible; heritrix/1.8.0 +http://www.cs.vt.edu) | |
| Mozilla/5.0 (compatible; heritrix/1.8.0 +http://yourwebsite.domain.com/) | |
| Mozilla/5.0 (compatible; heritrix/2.0.0 +http://www.aol.com) | |
| Mozilla/5.0 (compatible; heritrix/2.0.0 +http://www.google.com) | |
| Mozilla/5.0 (compatible; heritrix/2.0.1 +http://tibesti.research.att.com/research-crawler.html) | |
| Mozilla/5.0 (compatible; heritrix/2.0.2 +http://getprotected.asu.edu/files/wass.pdf) | |
| Mozilla/5.0 (compatible; heritrix/2.0.2 +http://localhost) | |
| Mozilla/5.0 (compatible; heritrix/2.0.2 +http://seekda.com) | |
| Mozilla/5.0 (compatible; heritrix/2.0.2 +http://www.loc.gov/webcapture/) | |
| Mozilla/5.0 (compatible; heritrix/2.0.2 +http://www.yell.com) | |
| Mozilla/5.0 (compatible; heritrix/3.0.0 +http://dataclip.com/crawler.html) | |
| Mozilla/5.0 (compatible; heritrix/3.0.0 +http://vefsofnun.bok.hi.is) | |
| Mozilla/5.0 (compatible; heritrix/3.0.0 +http://webarchive.loc.gov) | |
| Mozilla/5.0 (compatible; heritrix/3.0.0 +http://www.bl.uk/aboutus/stratpolprog/digi/domresproj/index.html) | |
| Mozilla/5.0 (compatible; heritrix/3.0.0 +http://www.dataclip.com) | |
| Mozilla/5.0 (compatible; heritrix/3.0.0 +http://www.webarchive.org.uk/) | |
| Mozilla/5.0 (compatible; heritrix/3.0.0-beta +http://webarchive.loc.gov) | |
| Mozilla/5.0 (compatible; heritrix/3.0.0-beta +http://webarchives.loc.gov) | |
| Mozilla/5.0 (compatible; heritrix/3.0.1-SNAPSHOT-20110209.020021 +http://newsconsole.com) | |
| Mozilla/5.0 (compatible; heritrix/3.0.1-SNAPSHOT-20110328.124355 +http://www.neoscio.de) | |
| Mozilla/5.0 (compatible; heritrix/3.1.0 +http://mith.umd.edu) | |
| Mozilla/5.0 (compatible; heritrix/3.1.0 +http://sid.uncu.edu.ar/sid/cdh) | |
| Mozilla/5.0 (compatible; heritrix/3.1.0 +http://www.library.unt.edu/digitalprojects/web-archiving) | |
| Mozilla/5.0 (compatible; heritrix/3.1.0 +http://www.northgrid.co.jp) | |
| Mozilla/5.0 (compatible; heritrix/3.1.0-SNAPSHOT-20110614.230024 +http://www.sina.com.cn) | |
| Mozilla/5.0 (compatible; heritrix/3.1.0-SNAPSHOT-20110920.235706 +http://www.archive.org/details/archive.org_bot) | |
| Mozilla/5.0 (compatible; heritrix/3.1.0-SNAPSHOT-20111013.155210 +http://archive.org/details/archive.org_bot) | |
| Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20100429.232622 +http://crawler.archive.org) | |
| Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20100915.040035 +http://www.archive.bibalex.org) | |
| Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20100928.230034 +http://www.dataclip.com) | |
| Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20110411.223526 +http://efca.pl/crawler) | |
| Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 +http://www.archive.org/details/archive.org_bot) | |
| Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120320.221924 +http://archive.org/details/archive.org_bot) | |
| Mozilla/5.0 (compatible; heritrix/UNKNOWN +http://www.adsafemedia.com) | |
| Mozilla/5.0 (compatible; heritrix/UNKNOWN +http://www.google.com) | |
| Mozilla/5.0 (compatible; HomeTags/1.0; +http://www.hometags.nl/bot) | |
| Mozilla/5.0 (compatible; http://summarity.com/) | |
| Mozilla/5.0 (compatible; http; +http://www.idrc.ca/cp/ev-136691-201-1-DO_TOPIC.html; tangyang9@gmail.com) | |
| Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/ | |
| Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/) | |
| mozilla/5.0 (compatible; INA dlweb; +http://www.ticm.msh-paris.fr/spip.php?article6) | |
| Mozilla/5.0 (compatible; IntelCSbot/0.2.1beta) | |
| Mozilla/5.0 (compatible; IstellaBot/1.01.18 +http://www.tiscali.it/) | |
| Mozilla/5.0 (compatible; janforman.com/2.4; +http://janforman.com/) | |
| Mozilla/5.0 (compatible; KaloogaBot; http://kalooga.com/crawler) | |
| Mozilla/5.0 (compatible; KaloogaBot; http://www.kalooga.com/info.html?page=crawler) | |
| Mozilla/5.0 (compatible; KLSH-PageGet/1.03; http://www.kondo-net.gr.jp/klsh/) | |
| Mozilla/5.0 (compatible; KoDe/0.2; +http://www.knoova.com/) | |
| Mozilla/5.0 (compatible; kulturarw3 +http://www.kb.se/soka/internet/sv-webbsidor/) | |
| Mozilla/5.0 (compatible; lemurwebcrawler admin@lemurproject.org; +http://boston.lti.cs.cmu.edu/crawler_12/) | |
| Mozilla/5.0 (compatible; linkfluence/0.9; http://linkfluence.net/) | |
| Mozilla/5.0 (compatible; linkfluence; http://linkfluence.net/) | |
| Mozilla/5.0 (compatible; loc-crawler/0.11.0 +http://archive.org) | |
| Mozilla/5.0 (compatible; loc-crawler/0.11.0 +http://loc.gov) | |
| Mozilla/5.0 (compatible; loc-crawler/3.0.0 +http://webarchive.loc.gov) | |
| Mozilla/5.0 (compatible; loc-crawler/3.0.1-SNAPSHOT-20110412.010027 +http://webarchive.loc.gov) | |
| Mozilla/5.0 (compatible; loku; +http://loku.com/; development@loku.com) | |
| Mozilla/5.0 (compatible; ltbot/1.3; bot@language-tools.com) | |
| Mozilla/5.0 (compatible; LucidWorks/; +; crawler at example dot com) | |
| Mozilla/5.0 (compatible; LuminateBot/1.0; +http://www.luminate.com/bot/) | |
| Mozilla/5.0 (compatible; MendeleyBot +http://mendeley.com)) | |
| Mozilla/5.0 (compatible; MetamojiCrawler/1.0; +http://www.metamoji.com/jp/crawler.html | |
| Mozilla/5.0 (compatible; METASpider; +http://meta.ua/spider) | |
| Mozilla/5.0 (compatible; Microsoft MSIE 7.0 +http://www.europarchive.org) | |
| Mozilla/5.0 (compatible; MJ12bot/v1.2.1; http://www.majestic12.co.uk/bot.php?+) | |
| Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+) | |
| Mozilla/5.0 (compatible; MJ12bot/v1.2.4; http://www.majestic12.co.uk/bot.php?+) | |
| Mozilla/5.0 (compatible; MJ12bot/v1.2.5; http://www.majestic12.co.uk/bot.php?+) | |
| Mozilla/5.0 (compatible; MJ12bot/v1.3.0; http://www.majestic12.co.uk/bot.php?+) | |
| Mozilla/5.0 (compatible; MJ12bot/v1.3.1; http://www.majestic12.co.uk/bot.php?+) | |
| Mozilla/5.0 (compatible; MJ12bot/v1.3.2; http://www.majestic12.co.uk/bot.php?+) | |
| Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+) | |
| Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+),gzip(gfe),gzip(gfe),gzip(gfe) AppEngine-Google; (+http://code.google.com/appengine; appid: pr0xy--me) | |
| Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+) | |
| Mozilla/5.0 (compatible; MJ12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+) | |
| Mozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+) | |
| Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+) | |
| Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html#relaunch) | |
| Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html) | |
| Mozilla/5.0 (compatible; Mozilla/5.0 +http://www.up.ru) | |
| Mozilla/5.0 (compatible; Mp3Bot/0.4; +http://mp3realm.org/mp3bot/) | |
| Mozilla/5.0 (compatible; Mp3Bot/0.6; +http://mp3realm.org/mp3bot/) | |
| Mozilla/5.0 (compatible; Mp3Bot/0.7; +http://mp3realm.org/mp3bot/) | |
| Mozilla/5.0 (compatible; MSIE 6.0 Win 5.1 +http://www.europarchive.org) | |
| Mozilla/5.0 (compatible; MSIE 7.0 +http://internetmemory.org/en/) | |
| Mozilla/5.0 (compatible; MSIE 7.0 +http://internetmemory.org/en/); crawling on behalf of The National Archives UK Government Web Archive: http://www.nationalarchives.gov.uk/webarchive/; email: webarchive@nationalarchives.gsi.gov.uk | |
| Mozilla/5.0 (compatible; MSIE 7.0 +http://www.europarchive.org) | |
| Mozilla/5.0 (compatible; MSIE 7.0 Win32 +http://internetmemory.org/en/) | |
| Mozilla/5.0 (compatible; MSIE 7.0 Win32 +http://www.europarchive.org/) | |
| Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0 | |
| Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/2.0 | |
| Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/3.0 | |
| Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html) | |
| Mozilla/5.0 (compatible; NerdByNature.Bot; http://www.nerdbynature.net/bot) | |
| Mozilla/5.0 (compatible; Netseer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com) | |
| Mozilla/5.0 (compatible; Nigma.ru/3.0; crawler@nigma.ru) | |
| Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/) | |
| Mozilla/5.0 (compatible; oBot/2.3.1; +http://www-935.ibm.com/services/us/index.wss/detail/iss/a1029077?cntxt=a1027244) | |
| Mozilla/5.0 (compatible; OGSearchSpider 3508 tom@lemurconsulting.com) | |
| Mozilla/5.0 (compatible; OntoPort/; +; admin@ontoport.de) | |
| Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html) | |
| Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html) | |
| Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/) | |
| Mozilla/5.0 (compatible; Plukkie/1.1; http://www.botje.com/plukkie.htm) | |
| Mozilla/5.0 (compatible; Plukkie/1.2; http://www.botje.com/plukkie.htm) | |
| Mozilla/5.0 (compatible; Plukkie/1.3; http://www.botje.com/plukkie.htm) | |
| Mozilla/5.0 (compatible; Plukkie/1.4; http://www.botje.com/plukkie.htm) | |
| Mozilla/5.0 (compatible; Popular-Bookmark-Bot/1.0; +http://www.popular-bookmark.com/) | |
| Mozilla/5.0 (compatible; Presansbot/1.0 +http://www.presans.com) | |
| Mozilla/5.0 (compatible; proximic; +http://www.proximic.com) | |
| Mozilla/5.0 (compatible; ptd-crawler; +http://bixolabs.com/crawler/ptd/; crawler@bixolabs.com) | |
| Mozilla/5.0 (compatible; Purebot/1.1; +http://www.puritysearch.net/) | |
| Mozilla/5.0 (compatible; q1; +http://www.qleeq.com; info@qleeq.com) | |
| Mozilla/5.0 (compatible; Robo/1.0b; http://potaru.com/robo.html)/Nutch-1.2 | |
| Mozilla/5.0 (compatible; RTGI; http://rtgi.fr/) | |
| Mozilla/5.0 (compatible; RTSbot/1.0; +http://adamcaudill.com/rtsbot/) | |
| Mozilla/5.0 (compatible; Scarlett/ 1.0; +http://www.ellerdale.com/crawler.html) | |
| Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/) | |
| Mozilla/5.0 (compatible; ScribdReader/1.0; +http://www.float.com) | |
| Mozilla/5.0 (compatible; ScribdReader/1.0; +http://www.scribd.com/reader.html) | |
| Mozilla/5.0 (compatible; Search17Bot/1.1; http://www.search17.com/bot.php) | |
| Mozilla/5.0 (compatible; Semager/1.4; http://www.semager.de/blog/semager-bots/) | |
| Mozilla/5.0 (compatible; Semager/1.4c; +http://www.semager.de/blog/semager-bots/) | |
| Mozilla/5.0 (compatible; SEODat/0.1 http://crawler.seodat.com) | |
| Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html) | |
| Mozilla/5.0 (compatible; sgbot v0.01a, sgcrawlerbot@gmail.com) | |
| Mozilla/5.0 (compatible; sindice-fetcher/0.1.0 +http://sindice.com/developers/bot) | |
| Mozilla/5.0 (compatible; sindice-site-manager/0.1.0 +http://sindice.com/developers/bot) | |
| Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/) | |
| Mozilla/5.0 (compatible; SiteBot/0.1; +http://www.sitebot.org/robot/) | |
| Mozilla/5.0 (compatible; SkreemRBot +http://skreemr.com) | |
| Mozilla/5.0 (compatible; socketcrawler; http://nlp.fi.muni.cz/projects/biwec/) | |
| Mozilla/5.0 (compatible; spbot/1.0; +http://www.seoprofiler.com/bot/ ) | |
| Mozilla/5.0 (compatible; spbot/1.1; +http://www.seoprofiler.com/bot/ ) | |
| Mozilla/5.0 (compatible; spbot/1.2; +http://www.seoprofiler.com/bot/ ) | |
| Mozilla/5.0 (compatible; spbot/1.5; +http://www.seoprofiler.com/bot/ ) | |
| Mozilla/5.0 (compatible; spbot/2.0.1; +http://www.seoprofiler.com/bot/ ) | |
| Mozilla/5.0 (compatible; spbot/2.0.2; +http://www.seoprofiler.com/bot/ ) | |
| Mozilla/5.0 (compatible; spbot/2.0.3; +http://www.seoprofiler.com/bot/ ) | |
| Mozilla/5.0 (compatible; spbot/2.0.4; +http://www.seoprofiler.com/bot ) | |
| Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ ) | |
| Mozilla/5.0 (compatible; spbot/2.1; +http://www.seoprofiler.com/bot ) | |
| Mozilla/5.0 (compatible; spbot/3.0; +http://www.seoprofiler.com/bot ) | |
| Mozilla/5.0 (compatible; special_archiver/1.5.0 +http://www.bne.es/es/LaBNE/PreservacionDominioES/AvisoWebmasters/index.html) | |
| Mozilla/5.0 (compatible; special_archiver/1.5.0 +http://www.loc.gov/minerva/crawl.html) | |
| Mozilla/5.0 (compatible; special_archiver/1.5.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html) | |
| Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/) | |
| Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); http://nlp.fi.muni.cz/projects/biwec/) | |
| Mozilla/5.0 (compatible; Steeler/3.5; http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) | |
| Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/) | |
| Mozilla/5.0 (compatible; suggybot v0.01a, sgcrawlerbot@gmail.com) | |
| Mozilla/5.0 (compatible; Swarm; +swarm@webscaled.com) | |
| Mozilla/5.0 (compatible; Tagoobot/3.0; +http://www.tagoo.ru) | |
| Mozilla/5.0 (compatible; TopBlogsInfo/2.0; +topblogsinfo@gmail.com) | |
| Mozilla/5.0 (compatible; U; Linux i686 (x86_64); de-DE; <a href=http://wikimpress.org/>Wikimpress</a>) Wikimpress/1.0 | |
| Mozilla/5.0 (compatible; Urlfilebot/2.2; +http://urlfile.com/bot.html) | |
| Mozilla/5.0 (compatible; VideoSurf_bot +http://www.videosurf.com/bot.html) | |
| Mozilla/5.0 (compatible; Voluniabot/1.0; spider@volunia.com) | |
| Mozilla/5.0 (compatible; WASALive Bot ; http://blog.wasalive.com/wasalive-bots/) | |
| Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/) | |
| Mozilla/5.0 (compatible; Wazzup1.0.1028; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.1446; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.1630; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.1830; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.2280; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.2496; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.2926; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.3265; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.3635; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.4087; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.4172; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.5321; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.6644; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.6811; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.6957; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.7005; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.7225; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.8567; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.9277; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.9813; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; Wazzup1.0.9989; http://75.126.176.112/Wazzup) | |
| Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html) | |
| Mozilla/5.0 (compatible; Web CEO Online robot) | |
| Mozilla/5.0 (compatible; WebCensusProjectBot/0.03 +http://www.webcensusproject.org/) | |
| Mozilla/5.0 (compatible; WebmasterCoffee/0.7; +http://webmastercoffee.com/about) | |
| Mozilla/5.0 (compatible; woriobot +http://worio.com) | |
| Mozilla/5.0 (compatible; XmarksFetch/1.0; +http://www.xmarks.com/about/crawler; info@xmarks.com) | |
| Mozilla/5.0 (compatible; xrss; robot; http://www.xrss.eu/robot; version 2.0) | |
| Mozilla/5.0 (compatible; Yahoo! Slurp Asia; http://help.yahoo.com/help/us/ysearch/slurp) | |
| Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html) | |
| Mozilla/5.0 (compatible; Yahoo! Slurp/si-emb; http://help.yahoo.com/help/us/ysearch/slurp) | |
| Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | |
| Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; R; +http://yandex.com/bots) | |
| Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | |
| Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots) | |
| Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots) | |
| Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots) | |
| Mozilla/5.0 (compatible; Yeti/1.0; NHN Corp.; http://help.naver.com/robots/) | |
| Mozilla/5.0 (compatible; YioopBot +http://www.yioop.com/bot.php) | |
| Mozilla/5.0 (compatible; YioopBot +http://yioop.com/bot.php) | |
| Mozilla/5.0 (compatible; YioopBot +http://www.yioop.com/bot.html) | |
| Mozilla/5.0 (compatible; YioopBot; +http://www.yioop.com/bot.php) | |
| Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; ) | |
| Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; ) | |
| Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; ) | |
| Mozilla/5.0 (compatible;+LolongBot/1.0; +http://www.yisou.us/help/webmaster/) | |
| Mozilla/5.0 (compatible;archive.org_bot/1.7.0; Archive-It; +http://archive-it.org/files/site-owners.html) | |
| Mozilla/5.0 (compatible;archive.org_bot/1.7.0; Archive-It; +http://www.archive-it.org) | |
| Mozilla/5.0 (compatible;archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html) Firefox | |
| Mozilla/5.0 (compatible;archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html) Firefox/0.0 | |
| Mozilla/5.0 (compatible;picmole/1.0 +http://www.picmole.com) | |
| Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;) | |
| Mozilla/5.0 (http://textdigger.com) Gecko/20061208 Firefox/2.0.0.1 | |
| Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) (compatible; Yeti-Mobile/0.1; +http://help.naver.com/robots/) | |
| Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0_1 like Mac OS X; fr-fr) OrangeBot-Mobile AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8A306 Safari/(mobilesearch.support@orange-ftgroup.com) | |
| mozilla/5.0 (larbin2.6.3@unspecified.mail) | |
| Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; ko; rv:1.9.1b2) Gecko/20081201 Firefox/3.1b2/Nutch-0.9 (It's a Mozilla based browser; http://www.mozilla.org/products/firefox/; no@mail.com) | |
| Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardBrowserProxy/0.0.5; +http://flipboard.com/browserproxy) | |
| Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/0.0.5; +http://flipboard.com/browserproxy) | |
| Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy) | |
| Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.8; flipboard.com/crawler rv:0.0.5) Gecko Firefox | |
| Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us; +http://stanford.edu) AppleWebKit/533.21.1 (KHTML, like Gecko) Version/5.0.5 Safari/533.21.1 | |
| Mozilla/5.0 (MetaCarta Labs GeoWebCrawler; +http://labs.metacarta.com/geowebcrawler) | |
| Mozilla/5.0 (someone@somewhere.any) | |
| Mozilla/5.0 (Thriceler-0.1 http://www.kuill.com/robots/spider.html) | |
| Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html) | |
| Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0a2 +http://mozilla.org) | |
| Mozilla/5.0 (Windows; N; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)/Nutch-1.0 (Crawler; lucene.apache.org/nutch/; a@b.net) | |
| Mozilla/5.0 (Windows; N; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)/Nutch-1.1 (Crawler; lucene.apache.org/nutch/; a@b.net) | |
| Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/) | |
| Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6/Nutch-1.1 (browser; http://www.g.cn; google@gmail.com) | |
| Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0/1.0 (bot; http://) | |
| Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.12) Gecko/2009070611 Firefox/3.0.12 (.NET CLR 3.5.30729) AutoPager/0.5.2.2 (http://www.teesoft.info/) | |
| Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 AutoPager/0.5.2.2 (http://www.teesoft.info/) | |
| Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com) | |
| Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1 (http://www.voila.com/) | |
| Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.4.2; trendiction media ssppiiddeerr; http://www.trendiction.com/bot/; please let us know of any problems; ssppiiddeerr at trendiction.com) Gecko/20071127 Firefox/2.0.0.11 | |
| Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 | |
| Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1 GTB5 (.NET CLR 3.5.30729) AutoPager/0.5.2.2 (http://www.teesoft.info/) | |
| Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1) Gecko/20090624 Firefox/3.5 AutoPager/0.5.2.2 (http://www.teesoft.info/) | |
| Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.112 Safari/535.1/Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.112 Safari/535.1 (Mozilla; Mozilla; mozilla@mozilla.org) | |
| Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.12) Gecko/2009070811 Ubuntu/9.04 (jaunty) Firefox/3.0.12/Nutch-1.0-dev (imcs; http://imcs.ro; admin@imcs.ro) | |
| Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19 | |
| Mozilla/5.0 (Yahoo-MMCrawler/4.0; mailto:vertical-crawl-support@yahoo-inc.com) | |
| Mozilla/5.0 (Yahoo-Test/4.0; mailto:vertical-crawl-support@yahoo-inc.com) | |
| Mozilla/5.0 GurujiBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html) | |
| Mozilla/5.0 [en] (X11; U; Linux 2.2.15 i686 +http://www.yahoo.com/index.html) | |
| Mozilla/5.0(compatible;MSIE/6.0+http://www.myspace.com) | |
| Mozilla/5.0+(compatible;+CukBot;++like+Googlebot/2.1;+Not+a+dirty+rotten+spammer;+++http://www.companiesintheuk.co.uk/bot.html) | |
| Mozilla/5.0+(compatible;+CukBot;+Not+a+spammer;+++http://www.companiesintheuk.co.uk/bot.html) | |
| Mozilla/5.0-Firefox/3.0.15 (Firefox@mozilla.com) | |
| Mozilla/5.0/heritrix/3.1.0 (compatible;; http://vancouver.cs.washington.edu:8080/) | |
| Mozilla/5.0/Nutch-1.0-dev ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html ; agentname at setooz dot_com ) | |
| Mozilla/6.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | |
| MozillaTest/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; ) | |
| Mozzila/5.0 (compatible; YamanaLab-Robot/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html) | |
| MPICrawler/0.1 +http://kermit.news.cs.nyu.edu/crawler.html | |
| MR Crawler/Nutch-1.3 | |
| msbot (http://www.malaysiansearch.com/faq.php) | |
| MSIE 5.0 | |
| MSIE 7.0/Nutch-1.0-dev | |
| MSIndianWebcrawl | |
| MSMOBOT/1.1 (+http://search.msn.com/msnbot.htm) | |
| msnbot | |
| msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) | |
| msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) | |
| msnbot-NewsBlogs/1.1 (+http://search.msn.com/msnbot.htm) | |
| msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm) | |
| msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm) | |
| msnbot/1.0 (+http://search.msn.com/msnbot.htm) | |
| msnbot/1.1 (+http://search.msn.com/msnbot.htm) | |
| msnbot/2.0b (+http://search.msn.com/msnbot.htm) | |
| msnbot/2.0b (+http://search.msn.com/msnbot.htm). | |
| msnbot/2.0b (+http://search.msn.com/msnbot.htm)._ | |
| MSNPTC/1.0 (compatible; MSIE 6.0; Windows NT 5.2; MyIE2; .NET CLR 1.1.4322; .NET CLR 1.0.3705) | |
| MSR-ISRCCrawler | |
| MSRBOT | |
| MSRBOT (http://research.microsoft.com/research/sv/msrbot/ | |
| MSU Libraries Search Engine/Nutch-0.9 (MSU Libraries Search Engine Description; none; webmaster at library dot msstate dot edu) | |
| MT-Soft (http://www.mt-soft.com.ar) | |
| multicrawler (+http://sw.deri.org/2006/04/multicrawler/robots.html) | |
| MULTIPRISE/Nutch-1.0 (Robot d'indexation; http://www.multiprise.biz; admin@multiprise.fr) | |
| my nutch agent/1.5 | |
| My Nutch Spider/Nutch-1.2 | |
| My Nutch Spider/Nutch-1.3 | |
| My Nutch Spider/Nutch-1.4 | |
| My Nutch Spider/Nutch-1.5-SNAPSHOT | |
| my nutch/Nutch-1.4 | |
| My Spider/Nutch-1.4 | |
| my-robot/0.1 | |
| my-robot/0.2 | |
| mybot/Nutch-1.0 (mybot; http://mybot.com; mybot@mybot.com) | |
| myCrawl/Nutch-1.3 | |
| myfirsttest/Nutch-1.1 (myfirsttest; myfirsttest.com; test@test.com) | |
| MyNutchTest/Nutch-1.4-dev (Testing some algorithms) | |
| MySearch/feiseek-0.09 | |
| MySearch/Nutch-1.0 | |
| MySpider/1.0 | |
| MyThesisCrawler/Nutch-1.3 | |
| myWebCrawler (Sharpsword@esearch.com) | |
| N/A (farmer.tw@gmail.com) | |
| naba/Nutch-1.3 | |
| naba/Nutch-1.4 | |
| NameOfAgent (CMS Spider) | |
| NameSpider/1.0 | |
| NameSpier/1.0 | |
| nametest/test-version (http://google.com/robots.html) | |
| naturaxalli/Nutch-1.2 (naturaxalli) (A bot for our semantic search engine, look at http://naturaxalli.com/sse; http://naturaxalli.com/sse; info at naturallaxi.com) | |
| Naveen/Nutch-1.0 (Naveen; http://www.google.org; root@me) | |
| netEstate FOAF crawler (+http://www.foaf-search.net/) | |
| netEstate NE Crawler | |
| netEstate NE Crawler (+http://www.sengine.info/) | |
| NetinfoBot/1.1 (http://netinfo.bg/netinfobot.html) | |
| NetSprint -- 3.0 | |
| NetSrcherP/0.2.3 | |
| NetStudyBot1.0 | |
| NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx) | |
| NG/2.0 | |
| NgapaBot/Nutch-1.3 (NgapaBot is crawler for Indonesia Search Engine; http://www.ngapa.com; admin at ngapa dot com) | |
| Nica Spider/Nutch-1.1 | |
| NicheBot (http://nb.bretlester.com/) | |
| NIF/Nutch-1.0-dev (Neuroscience Information Framework Crawl; http://neuinfo.org/; support at neuinfo dot org) | |
| nightsuns.nutch/Nutch-1.0 | |
| NISE/NISE-1.0 (NISE) | |
| NISE/Nutch-2.0-dev | |
| Nokia (larbin2.6.3@unspecified.mail) | |
| Nokia3650/1.0 SymbianOS/6.1 Series60/1.2 Profile/MIDP-1.0 Configuration/CLDC-1.0/ (compatible; YodaoBot-Mobile/1.0; http://www.youdao.com/help/webmaster/spider/; ) | |
| Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html) | |
| NokiaN70/. FASTMobileCrawl/6.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 | |
| nolosear/Nutch-1.2 (Nolosearch.com is searching the web for law related material to index on its server. This is built by a law student who is tired of not finding the law.; http://nolosearch.com/node6; eric at law brief case dot com) | |
| nolosearch/nolo2.0 (law student search engine; nolosearch.com; eric@nolosearch.com) | |
| NoNSTOPRobot/Nutch-1.3 | |
| noopsis Spider/Nutch-1.1 (noopsis crawler) | |
| NoteworthyBot/0.1 | |
| nrsbot/5.0(loopip.com/robot.html) | |
| nrsbot/6.0(loopip.com/robot.html) | |
| NSE/Nutch-1.2 | |
| NSE/Nutch-1.3 | |
| NSE/Nutch-1.3 (saeed@mci.ir) | |
| Nuhk/2.4 (+http://www.neti.ee/cgi-bin/abi/otsing.html) | |
| Nutch 1.0/Nutch-1.0 | |
| Nutch 1.2/Nutch-1.2 (Facet Engine Nutch Crawler; spider@facetengine.com) | |
| Nutch Crawler/Nutch-1.1 | |
| Nutch crawler/Nutch-1.2 | |
| Nutch Experimental Crawler/Nutch-1.4 | |
| nutch search engine 1.2/Nutch-1.2 (Nutch Search Engine) | |
| Nutch Spider 1.1-dev/Nutch-1.0 | |
| Nutch Spider Evaluation/Nutch-1.4 | |
| Nutch Spider/Nutch-1.3 | |
| Nutch test crawler/Nutch-1.4-dev (Crawler testing; crawler/a/t/luminouslabs.com) | |
| nutch test/Nutch-1.0 (nutch test) | |
| Nutch, Leistungsschutz-Debatte Crawler/Nutch-1.4 | |
| Nutch-1.0/Nutch-1.0 (Nutch-1.0; http://lucene.apache.org/nutch/; admin@lucene.apache.org) | |
| nutch-1.1/Nutch-1.1 | |
| nutch-1.2/Nutch-1.2 | |
| Nutch-1.2/Nutch-1.2 (http://lucene.apache.org/nutch/; admin@lucene.apache.org) | |
| nutch-1.3/Nutch-1.3 | |
| nutch-1.4/Nutch-1.4 | |
| Nutch-1/Nutch-1.4 | |
| nutch-buibui/0.0.1/Nutch-1.0 | |
| nutch-crawl/Nutch-1.0-dev (imcs; http://imcs.ro; admin@imcs.ro) | |
| nutch-crawler/Nutch-1.2 | |
| nutch-solr-integration-test/Nutch-1.1 (FreeCode Norway Web Crawler using Nutch 1.0; http://www.freecode.no/; mrskyhorse@163.com) | |
| nutch-solr-integration-test/Nutch-1.2 (ADR Software Web Crawler using Nutch 1.2; http://www.adrsoftware.gr/; info@adrsoftware.gr) | |
| nutch-solr-integration-test/Nutch-1.2 (FreeCode Norway Web Crawler using Nutch 1.0; http://www.bupt.edu.cn/; guorui15@sohu.com) | |
| nutch-solr-integration-test/Nutch-1.2 (FreeCode Norway Web Crawler using Nutch 1.0; http://www.freecode.no/; reza.keshmir.90@gmail.com) | |
| nutch-solr-integration-test/Nutch-1.2 (MoonValley Web Crawler using Nutch 1.2; http://www.moonvalley.com/; cwoolum@moonvalley.com) | |
| nutch-solr-integration-test/Nutch-1.2 (Tagvillage Web Crawler using Nutch 1.0; http://www.tagvillage.com/; admin@tagvillage.com) | |
| nutch-solr-integration/1.0 | |
| nutch-solr-integration/1.1 | |
| nutch-solr-integration/Nutch-1.0 | |
| nutch-solr-integration/Nutch-1.2 | |
| nutch-solr-integration/Nutch-2.0-dev | |
| nutch-solr-integration/Nutch-2.0-dev (aeq123dec; www.aequor.com; gopal.dega@aequor.com) | |
| nutch.biz/Nutch-1.0 (nutch.biz; crawler@nutch.biz) | |
| nutch.us/Nutch-1.0 (nutch.us; crawler@nutch.us) | |
| nutch.us/Nutch-1.0 (www.nutch.us; crawler@nutch.us) | |
| Nutch/1.2/Nutch-1.2 | |
| Nutch/Nutch-0.8.1 (Nutch; http://www.nutch.es; email) | |
| Nutch/Nutch-0.8.1 (Nutch; Nutch; Nutch) | |
| Nutch/Nutch-0.9 (Nutch Search Engineer; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) | |
| Nutch/Nutch-1.0 | |
| Nutch/Nutch-1.0 (academic purpose; cats.kaist.ac.kr; smiler82@naver.com) | |
| Nutch/Nutch-1.0 (academic purpose; cats2.kaist.ac.kr; smiler82@naver.com) | |
| nutch/Nutch-1.0 (MyTest; 172.18.5.30; crawler@crawlzilla) | |
| nutch/Nutch-1.0 (MyTest; set.your.ip.or.domain; your@email) | |
| nutch/Nutch-1.0 (none; http://www.163.com; some@some.com) | |
| Nutch/Nutch-1.0 (Nutch Search Engineer; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) | |
| nutch/Nutch-1.0 (nutch) | |
| nutch/Nutch-1.0 (nutch; http://www.hp.com; nutch@apache.org) | |
| Nutch/Nutch-1.0-dev (A Nutch-based crawler.; http://lucene.apache.org/nutch/bot.html; nutch-agent AT lucene.apache.org) | |
| Nutch/Nutch-1.1 (Nutch Search Engineer; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) | |
| Nutch/Nutch-1.1-dev | |
| Nutch/Nutch-1.2 | |
| nutch/Nutch-1.2 (MyTest; set.your.ip.or.domain; your@email) | |
| Nutch/Nutch-1.2 (Nutch Search Engineer; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) | |
| Nutch/Nutch-1.3 | |
| Nutch/Nutch-1.4 (A semantic crawler for a PhD thesis; http://issel.ee.auth.gr/doku.php/research; kvavliak at issel dot ee dot auth dot gr) | |
| Nutch/Nutch-1.4 (A semantic crawler for a PhD thesis; http://issel.ee.auth.gr; kvavliak at issel dot ee dot auth dot gr) | |
| nutch/Nutch-1.5-SNAPSHOT (Viterbi Web Crawler using Nutch 1.0) | |
| nutch1.1/Nutch-1.0 | |
| nutch1/huntsman1.0 (huntsman@mailguard.com.au) | |
| NutchBot/Nutch-1.2 (Nutch Tester) | |
| NUTCHCRAWLER/Nutch-1.0 | |
| NUTCHCRAWLER/Nutch-1.2 | |
| NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net) | |
| NutchCVS/0.7 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) | |
| NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) | |
| NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) | |
| NutchCVS/Nutch-1.0 (http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) | |
| NutchRobot/Nutch-1.3 | |
| NutchRobot/Nutch-2.0-dev | |
| NutchTest/Nutch-1.0 | |
| NutchTestJb UB Heidelberg, Germany/Nutch-0.9 (b-a-r-t-h-at-u-b-punkt-u-n-i---h-e-i-d-e-l-b-e-r-g-punkt-d-e) | |
| NutchTestRobot/Nutch-1.2 | |
| nutchwax.com/Nutch-1.0 (nutchwax.com; crawler@nutchwax.com) | |
| nutchy/Nutch-1.0 | |
| Nutch_Crawler/Nutch-1.3 | |
| nutch_MR/Nutch-1.0 (nutch_MR; http://www.hp.com; nutch@apache.org) | |
| nutch_solr/Nutch-1.2 | |
| nutch_test/Nutch-0.9 (nutch_test; http://www.8dorm.com; webmaster@8dorm.com) | |
| Nutraspace/Nutch-1.2 (www.nutraspace.com) | |
| nu_tch-princeton/Nu_tch-1.0-dev (princeton crawler for cass project; http://www.cs.princeton.edu/cass/; zhewang a_t cs ddot princeton dot edu) | |
| nWormFeedFinder (http://www.nworm.com) | |
| nyobot/1.1 (Noyb.com search engine crawler; http://www.noyb.com; bot@noyb.com) | |
| NYU CS Nutch 1.0/Nutch-0.9 (NYU CS Nutch 1.0; http://lucene.apache.org/nutch/; webmaster at cs dot nyu dot edu) | |
| OAPS-Document-Fetcher | |
| oBot | |
| Obvius external linkcheck/1.3 | |
| oco/Nutch-1.3 (oco crawler; oco.com; jgrey@oco.com) | |
| OfelioBot | |
| oh,fuck (larbin2.6.3@unspecified.mail) | |
| OKArabia search engine/Nutch-1.4 | |
| OMGCrawler 1.0 | |
| OmniWeb | |
| online-intel/Nutch-1.0 | |
| OOZBOT/0.20 ( -- ; http://www.setooz.com/oozbot.html ; agentname at setooz dot_com ) | |
| OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com ) | |
| OpenAcoon v4.1.0 (www.openacoon.de) | |
| OpenAcoon v4.10.5 (www.openacoon.de) | |
| OpenCalaisSemanticProxy | |
| OpenIntelligenceData/1.1 (+http://www.worldwideweb-x.com/openData.html) | |
| Openstat/0.1 | |
| OpenWebSpider (link collector; http://links.port30.se/cia.html) v (http://www.openwebspider.org/) | |
| OpenWebSpider v0.1.2.B (http://www.openwebspider.org/) | |
| OpenWebSpider v0.1.4 (http://www.openwebspider.org/) | |
| Opera/6.05 (Windows 2000; U) | |
| Opera/9.00 (Windows NT 5.1; U; zh-cn) | |
| Opera/9.26 (Windows NT 5.1; U; zh-cn) | |
| Opera/9.63 (Windows NT 5.1; U; en) Presto/2.1.1 | |
| Opera/9.64 (Windows NT 5.1; U; en) Presto/2.1.1 | |
| Opera/9.80 | |
| Opera/9.80 (Macintosh; Intel Mac OS X; U; en) | |
| Opera/9.80 (Windows NT 5.1; U; en) Presto/2.5.22 Version/10.50 | |
| Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.2.15 Version/10.10 | |
| Opera/9.80 (Windows NT 6.1; U; cs) Presto/2.10.229 Version/11.61 | |
| Opera/9.80 (Windows NT 6.1; U; en-US) Presto/2.7.62 Version/11.01 | |
| opera_a (lostall@gmail.com) | |
| oracle-cag-http-agent/Nutch-1.0 (Crawler to check site html; http://www.oracle.com; arist at oracle dot com) | |
| OrangeCrawler/Nutch-1.0 (ldorange.crawler@orange-ftgroup.com) | |
| OrbitSpiderBot/1 | |
| orgbybot/OrgbyBot v1.2 (Spidering for Orgby; http://www.orgby.com/ ; orgby.com search engine) | |
| Orgbybot/OrgbyBot v1.2 (Spidering the net for Orgby; http://www.orgby.com/ ; Orgby.com Search Engine) | |
| Orgbybot/OrgbyBot v1.3 (Spider; http://orgby.com/bot/ ; Orgby.com Search Engine) | |
| ORISBot | |
| ossse.com spider/Nutch-1.2 (ossse.com spider; http://ossse.com; ossse email address) | |
| PadosAttilaCrawler/Nutch-1.0 (Ozi,PolandWiz,AustriaWiz,WiennaWiz crawlers, Attila Pados, attila.pados@gmail.com; www.ozi.hu, www.polandwiz.com,www.wiennawiz.com,www.austriawiz.com; attila dot pados at gmail dot com) | |
| PagePeeker.com | |
| PagePeeker.com (info: http://pagepeeker.com/robots) | |
| page_test (larbin2.6.3@unspecified.mail) | |
| page_verifier (http://www.securecomputing.com/goto/pv) | |
| page_viewer (larbin2.6.3@unspecified.mail) | |
| Paleoweb.org Friendly Bot ;-)/1 | |
| panelbot | |
| Pangalactic Gargleblaster/Nutch-1.2 (Very intoxicating; http://circuitbomb.com; DigitalMail) | |
| panscient.com | |
| PArchiveCrawler/1.0 (PArchiveCrawler) | |
| parsa/Nutch-1.1 (parsa; http://parsa.epfl.ch; stavros.volos@epfl.ch) | |
| pavuk/0.9.35 linux-gnu | |
| Pawler Beta - pawler(dot)net | |
| Paxbot/1.0 (http://www.pax.com/paxbot.html) | |
| PEAR HTTP_Request class ( http://pear.php.net/ ) | |
| Pentafocus.com pentafocusbot/1.0 (+http://www.pentafocus.com/pentafocusbot) | |
| PenTest.sg/Nutch-1.1 (www.PenTest.sg; crawler@PenTest.sg) | |
| percbotspider | |
| percbotspider <ops@percolate.com> | |
| Peter Wang/Nutch-0.9 (Nutch spiderman; http://peterpuwang.googlepages.com ; MyEmail) | |
| Peter Wang/Nutch-1.0 (Nutch spiderman; http://peterpuwang.googlepages.com ; MyEmail) | |
| Peter Wang/Nutch-1.0 (Nutch spiderman; local host ; saharelkazaz@hotmail.com) | |
| Peter Wang/Nutch-1.2 (Nutch spiderman; http://peterpuwang.googlepages.com ; MyEmail) | |
| pflab (cboc-test@lab.ntt.co.jp) | |
| pflab (co2h2onacl@gmail.com) | |
| Phex 2.02.8.2.92 Java/1.6.0_10 | |
| Phex 2.02.8.2.92 Java/1.6.0_26 | |
| PHP version tracker (http://www.nexen.net/phpversion/bot.php) | |
| PHP/5.2.17p1 | |
| PHPCrawl | |
| PicklBot 1.0 | |
| Pills.com.sg/Nutch-1.1 (Pills.com.sg; crawler@Pills.com.sg) | |
| pinakilabs/Nutch-1.0 | |
| Pingthesemanticweb-robotvalidator/3.0 | |
| Pixray-Seeker/1.1 (Pixray-Seeker; http://www.pixray.com/pixraybot; crawler@pixray.com) | |
| PizzaFace Crawler/Nutch-1.2 | |
| Pockey-GetHTML/4.11.6 (Win32; GUI; ix86) | |
| Pockey-GetHTML/5.1.1 (Win32; GUI; ix86) | |
| Ponyfish | |
| PornoBug Crawler 0.8 * www.pornobug.com | |
| PortalNG-Crawler/Nutch-1.4 (Crawler des Portals der Uni Ulm.; http://portal.uni-ulm.de; christian.aschoff at uni-ulm.de) | |
| PostPost/1.0 (+http://postpo.st/crawlers) | |
| PostPost/1.0 (+http://postpost.com/crawlers) | |
| PR Crawler/Nutch-1.0 ( ; crawler@projectrialto.com) | |
| PR Crawler/Nutch-1.0 (data mining develpment project; crawler@projectrialto.com) | |
| PRBot/0.1 (+http://www.paulridgway.co.uk/prbot) | |
| PRCrawler/Nutch-0.9 (data mining development project; crawler@projectrialto.com) | |
| Primo Web Spider/Nutch-1.4 | |
| PrivacyFinder/1.1 | |
| private test crawler curl-based | |
| Proto_Crawler/Nutch-0.9 (Prototype of a Crawler used in order to test Nutch's limits and performances.; --; lorack@gmail.com) | |
| PRUEBA/Nutch-1.0 | |
| psbot/0.1 (+http://www.picsearch.com/bot.html) | |
| PShop-Nutch/Nutch-1.1 (Academic test crawl; livne at umich) | |
| PShop-Nutch/Nutch-1.1 (Academic test crawl; pshopcrawl at gmail) | |
| PSS-Bot (+http://www.dcs.shef.ac.uk/~aca07pdr/pss/) | |
| pt_viewer (larbin2.6.3@unspecified.mail) | |
| pt_viewer_org (larbin2.6.3@unspecified.mail) | |
| pt_viewer_page (larbin2.6.3@unspecified.mail) | |
| Python-urllib/1.16 | |
| Python-urllib/1.17 | |
| Python-urllib/1.17 AppEngine-Google; (+http://code.google.com/appengine; appid lemmatica) | |
| Python-urllib/2.1 | |
| Python-urllib/2.4 | |
| Python-urllib/2.5 | |
| Python-urllib/2.6 | |
| Python-urllib/2.7 | |
| QCF Testname/Nutch-0.9 (QCF Test description; QCF Test url; QCF Test email ) | |
| Qirina Hurdler v. 1.05 10.11.01 (+http://www.qirina.com/hurdler.html) | |
| QleeQ1/Nutch-1.4 | |
| QrobotW | |
| Qryos | |
| Qseero v1.0.0 | |
| quickobot/quickobot-1 (Quicko Labs; http://quicko.co; robot at quicko dot co) | |
| R6_CommentReader(www.radian6.com/crawler) | |
| R6_FeedFetcher(www.radian6.com/crawler) | |
| R6_Primer(www.radian6.com/crawler) | |
| RADaR-Bot/Nutch-1.4 (http://radar-bot.com/) | |
| radian6_default_(www.radian6.com/crawler) | |
| random house/Nutch-1.0 (Lucene Random House Crawler; http://www.randomhouse.com/; somewhere@neverexisted.com) | |
| random house/Nutch-1.0 (Lucene Random House Crawler; http://www.randomhouse.com/; unixadmin@randomhouse.com) | |
| RankurBot/Rankur2.1 (http://rankur.com; info at rankur dot com) | |
| Raymond Balm s/Biible 1.0 (Biible; http://www.biible.info ; raymond.balmes@gmail.com) | |
| Raymond Balm s/Bible-Ref: (spiderman; http://www.balmes.com ; raymond.balmes@gmail.com) | |
| Raymond Balm s/Biible 1.0 (Biible; http://www.bibleref.com ; raymond.balmes@gmail.com) | |
| Raymond Balm s/Biible 1.0 (Biible; http://www.biible.info ; raymond.balmes@gmail.com) | |
| Raymond Balm s/Biible 1.0 (spiderman; http://www.bibleref.com ; raymond.balmes@gmail.com) | |
| Raymond Balm s/Nutch-1.0 (spiderman; http://www.balmes.com ; raymond.balmes@gmail.com) | |
| Raymond Balm s/Nutch-1.0 (spiderman; http://www.bibleref.com ; raymond.balmes@gmail.com) | |
| RDFaS-Bot/1.0 (+http://www.rdfas.com/bot) | |
| rdfbot/1.0 ( Indian Language Web Search Engine ; Rediff.com ; rdfbotsupport AT rediffmailpro DOT com ) | |
| rdfbot/1.0 (Indian Language Web Search Engine; Rediff.com; rdfbotsupport AT rediffmailpro DOT com) | |
| rdfbot/1.0 (rdfbotsupport AT rediffmailpro DOT com) | |
| research spider[AT]mikelab.net | |
| research-scan-bot/Nutch-1.0 | |
| rganalytics | |
| RhubarbPIe/Nutch-1.3 | |
| RiceComputerArchitecture Crawler/Nutch-1.0 | |
| rincewind/rincewind-0.1 (http://labs.ischroedi.de) | |
| RISSBot/dev (http://www.tkolb.de/rissbot.txt) | |
| RiverglassScanner | |
| rmatie.com spider/Nutch-1.4 | |
| rmatie.com/Nutch-1.4 | |
| RoadRunner/Nutch-1.0 (webmaster@fieldtech.org) | |
| robis-webmasterportal-Bot (Version: 1.06, powered by www.robis-webmasterportal.de +http://www.robis-webmasterportal.de/bot.html) | |
| roboo/Nutch-1.0 (roboo; http://wap.roboo.com; winter.pi@roboo.com) | |
| roboobot/1.0 (roboo; http://wap.roboo.com; winter.pi@roboo.com) | |
| roboobot/Nutch-1.0 (roboobot; http://wap.roboo.com; winter.pi@roboo.com) | |
| robotgenius (http://robotgenius.net) | |
| Robots Bot | |
| Ronbot/Nutch-1.2 | |
| Ronzoobot/1.4 (http://www.ronzoo.com/about.php) | |
| Ronzoobot/1.4 (http://www.ronzoo.com/about/) | |
| Ronzoobot/1.5 (http://www.ronzoo.com/about/) | |
| RSpider/1.0 | |
| ruijie (larbin@unspecified.mail) | |
| Ruky-Roboter (Version: 1.06, powered by www.ruky.de +http://www.ruky.de/zeige/bot.html) | |
| ru_org_net_viewer (larbin2.6.3@unspecified.mail) | |
| ru_org_viewer (larbin2.6.3@unspecified.mail) | |
| ru_prgnet_viewer (larbin2.6.3@unspecified.mail) | |
| rwack/Nutch-1.0 (learning about distributed computing; http://rdsd.com; jhires at gmail dot com) | |
| rwack/Nutch-1.0 (learning about distributed computing; http://rwack.com; jhires at gmail dot com) | |
| S2-Crawler/1.1 (your search engine; http://www.search2.net/; info at search2 dot net) | |
| S2Bot/1.0 (http://search2.net) | |
| S2Bot/1.0 (http://search2.net; bot[at]search2[dot]net) | |
| S2Robot | |
| sadsa/Nutch-1.4 | |
| SaladSpoon/ShopSalad 1.0 (Search Engine crawler for ShopSalad.com; http://shopsalad.com/en/partners.html; crawler AT shopsalad.com) | |
| SaladSpoon/ShopSalad 1.0 (Search Engine crawler for ShopSalad.com; http://shopsalad.com/en/partners.html; crawler@shopsalad.com) | |
| SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) | |
| SAP-KM/WebRepository 1.2 | |
| SBIder/Nutch-1.0-dev (http://www.sitesell.com/sbider.html) | |
| SBIder/SBIder-1.0 (http://www.sitesell.com/sbider.html) | |
| SBIRobobot/0.9 (Rev.Nagayama1; http://news.sbibusiness.com/) | |
| Schloerbot/Nutch-1.0 (Schloer consulting bot; http://schloerconsulting.com/schloerbot) | |
| ScholarScope/Nutch-1.2 (Scholar Research Engine) | |
| ScholarScope/Nutch-1.5 (Scholar Research Engine) | |
| SciCrawl/Nutch-1.1 | |
| ScooperBot www.customscoop.com | |
| Scooter/3.3 | |
| Screaming Frog SEO Spider/1,30 | |
| Screaming Frog SEO Spider/1,40 | |
| Screaming Frog SEO Spider/1,60 | |
| Screaming Frog SEO Spider/1,70 | |
| Screaming Frog SEO Spider/1,80 | |
| Screaming Frog SEO Spider/1.30 | |
| Screaming Frog SEO Spider/1.40 | |
| Screaming Frog SEO Spider/1.80 | |
| Screaming Frog SEO Spider/1.90 | |
| SD Crawler/Nutch-0.9 (automated Crawler) | |
| search.ch V1.4.2 (spiderman@search.ch; http://www.search.ch) | |
| search.KumKie.com | |
| SearchBot | |
| SearchEngine/Nutch-1.4 | |
| SearchEngineVerificationCrawler/Nutch-1.0 (The purpose of this crawling is to collect web pages for verifying search engines.; http://www.yama.info.waseda.ac.jp/~takuya/en/aboutSRVC.html; srvc at yama dot info dot waseda dot ac dot jp) | |
| searcher/Nutch-1.1 | |
| Searchmee! Spider v0.98a | |
| Seegul/Nutch-1.4 | |
| SeekBot/1.0 RobotsTxtFetcher/1.3 | |
| Seengine/1.0/Nutch-1.3-dev | |
| Seengine/1.0/Nutch-2.0-dev | |
| Semager/1.3 http://www.semager.de/blog/semager-bots/ | |
| Semantifire/0.20 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html ; agentname at setooz dot_com ) | |
| SemrushBot/0.9 | |
| SemrushBot/0.91 | |
| SemrushBot/0.92 | |
| SemrushBot/Nutch-1.5-SNAPSHOT | |
| SemtoBot/0.1 | |
| SentinelCrawler 0.2/Nutch-1.0-dev | |
| SeoBot | |
| SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm) | |
| service-public-indexeur | |
| Setooz/Nutch-1.0 (http://www.setooz.com) | |
| setoozbot/0.30 ( compatible; SETOOZBOT/0.30 ; setooz.com ; bot AT setooz DOT com ) | |
| SETOOZBOT/5.0 ( compatible; SETOOZBOT/0.30 ; http://www.setooz.com/bot.html ) | |
| SexSearchBot (+http://gou-sex.com/) | |
| SexShop.Sg/Nutch-1.1 (www.SexShop.Sg; crawler@SexShop.Sg) | |
| SexShop.sg/Nutch-1.1 (www.SexShop.sg; crawler@www.sexshop.sg) | |
| SexyBoom/Nutch-1.2 (SexyBoom; http://www.sexyboom.it/; service@sexyboom.it) | |
| SexyBoom/Wauuu 0.1 (AnnunciPazzi; http://www.sexyboom.it/; service@sexyboom.it) | |
| SeznamBot/2.0 (+http://fulltext.sblog.cz/robot/) | |
| SeznamBot/2.0 (+http://fulltext.seznam.cz/) | |
| SeznamBot/3.0 (+http://fulltext.sblog.cz/) | |
| SeznamBot/3.0-alpha (+http://fulltext.sblog.cz/) | |
| SeznamBot/3.0-beta (+http://fulltext.sblog.cz/) | |
| SeznamBot/3.0-test (+http://fulltext.sblog.cz/) | |
| sGroup crawler 1/Nutch-1.3 | |
| Shared Content Fetcher/Nutch-0.9 (http://www.sharethis.com; gordon@sharethis.com) | |
| SHC/Nutch-1.0 (SemanticHacker Crawler; http://www.semantichacker.com/crawler-info; abuse@semantichacker.com) | |
| SheenBot/SheenBot-1.0.1 (Sheen web crawler) | |
| SheenBot/SheenBot-1.0.2 (Sheen web crawler) | |
| SheenBot/SheenBot-1.0.3 (Sheen web crawler) | |
| SheenBot/SheenBot-1.0.4 (Sheen web crawler) | |
| Shelob (shelob@gmx.net) | |
| ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot) | |
| sihuifeng/Nutch-1.2 | |
| silversurfer.cc | |
| SimilarPages/Nutch-1.0-dev (SimilarPages Nutch Crawler; http://www.similarpages.com; info@similarpages.com) | |
| sindice.net sitemap fetcher | |
| SindiceBot (heritrix/2.0.2 +http://sindice.com/developers/bot) | |
| Sirketcebot/v1.0 (http://www.sirketce.com/bot.php) | |
| SiteGuardBot (support@@siteguard.com) | |
| SiteGuardBot (support@siteguard.com) | |
| SiteSucker/1.6.9 | |
| SiteSucker/2.2.2 | |
| SiteSucker/2.2.2 CFNetwork/454.11.5 Darwin/10.6.0 (i386) (iMac7%2C1) | |
| SiteSucker/2.2.3 | |
| SiteSucker/2.2.4 | |
| SiteSucker/2.3 | |
| SiteSucker/2.3.1 | |
| SiteSucker/2.3.2 | |
| SiteSucker/2.3.3 | |
| SiteSucker/2.3.4 | |
| SiteSucker/2.3.4 CFNetwork/454.11.12 Darwin/10.7.0 (i386) (MacBookPro5%2C5) | |
| SiteSucker/2.3.5 | |
| SiteSucker/2.3.6 | |
| SiteSucker2.2.3 CFNetwork/438.12 Darwin/9.7.0 (i386) (MacBookPro5%2C1) | |
| sixtoCrawler/Nutch-0.9 (sixtoCrawler; www.sixto.org; info.sequic@gmail.com) | |
| SkimBot/1.0 (www.skimlinks.com <dev@skimlinks.com>) | |
| SkimBot/1.0 (www.skimlinks.com) | |
| SkyGrid/1.0 (+http://skygrid.com/partners) | |
| SlimBot (SlimBot; http://www.slimbot.com; info@slimbot.com) | |
| SlugBug Spider/0.1 beta (SlugBug.com search engine; http://www.slugbug.com) | |
| SmallBlue/Synergy-2.0; Nutch-1.0-rc1 | |
| Smarte Bot v1.0 | |
| smartview (larbin2.6.3@unspecified.mail) | |
| SMXEchoCrawl/Nutch-0.9 (Social Metrix Bot; http://www.socialmetrix.com/; bot@socialmetrix.com) | |
| Snapbot/1.0 (Snap Shots, +http://www.snap.com) | |
| SnapOn Long-Crawl/Nutch-0.9 (CCPV Crawler; http://thecrawlerproject.com; aherres@gmail.com) | |
| Snoopy v1.2.4 | |
| SNSpider_0.0.1 (snspider@salatti.net) | |
| socbot (search@socbay.com) | |
| SocSciBot | |
| SoftNet Nutch Spider/Nutch-1.4 | |
| Sogou Video /3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) | |
| Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) | |
| Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98) | |
| soni/Nutch-0.9 (RnD; www.cdac.in; asoni@cdac.in) | |
| Soonxun/1.0 (web spider; ) | |
| Sosospider+(+http://help.soso.com/webspider.htm) | |
| sou300-spider/1.0 (www.sou300.com) | |
| sou300-spider/Nutch-1.0 | |
| sou300-spider/sou300 (www.sou300.com) | |
| sou300/1.0 (www.sou300.com) | |
| sou300/1.1 (www.sou300.com) | |
| Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/) | |
| Sphider | |
| sphsearch.org/Nutch-1.0 (sphsearch.org; crawler@sphsearch.org) | |
| Spider | |
| Spider (Java EE; Windows NT; +http://www.daniel-nowak-ug.de) Resource Description Engine | |
| Spider/5.0 | |
| Spider/Nutch-1.0 | |
| spider/Nutch-1.1 (Nutch; http://lucene.apache.org/nutch/; nutch-agent@lucene.apache.org) | |
| spider/Nutch-1.3 (spider; http://www.xxx.com) | |
| spiderbot119 | |
| SpiderMan/1.0 | |
| Spiprabot | |
| SpokeSpider/1.0 (http://support.spoke.com/webspider/) Mozilla/5.0 (not really) | |
| spotinfluence/Nutch-1.2 (Spot Influence crawler; http://spotinfluence.com; hello at spotinfluence dot com) | |
| spotinfluence/Nutch-1.4 (Spot Influence crawler; http://spotinfluence.com; hello at spotinfluence dot com) | |
| Squid Prefetch 1.0 | |
| ss (meng1985636@163.com) | |
| SS/Nutch-1.5 (SS) | |
| ssearch_bot (sSearch Crawler; http://www.semantissimo.de) | |
| ssearch_bot/Nutch-1.0 (SSearch Crawler; http://www.semantissimo.de) | |
| SSSE/0.08 | |
| StackRambler/2.0 (MSIE incompatible) | |
| StatoolsBot (+http://www.statools.com/bot.html) | |
| stc-crawler/0.84 | |
| suchclip | |
| Super graphical search spider/Nutch-1.3 | |
| SuperPagesBot/0.1 | |
| Suren/Nutch-1.0 | |
| SurveyBot/2.3 (Whois Source) | |
| Swami1/Swami-1.1 (Swami Vertical Site Fetcher; devarajaswami at yahoo dot com) | |
| sxit nutch agent/1.0 | |
| SygolBot http://www.sygol.com | |
| SymbolHound/Nutch-1.3 (Please contact us if we're being impolite!; www.symbolhound.com; info@symbolhound.com) | |
| Szukacz/1.5 (robot; www.szukacz.pl/html/jak_dziala_robot.html; info@szukacz.pl) | |
| Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl) | |
| T-Mobile Dash Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240; MSNBOT-MOBILE/1.1; +http://search.msn.com/msnbot.htm) | |
| T-Mobile Dash Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240;) MSNBOT-MOBILE/1.0 (+http://search.msn.com/msnbot.htm) | |
| T-Mobile Dash Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; Smartphone; 320x240;) MSNBOT-MOBILE/1.1 (+http://search.msn.com/msnbot.htm) | |
| t3hb0t/Nutch-1.0 (ab0t; http://nosite.com; notreal@fake.org) | |
| tags2dir.com/0.8 (+http://tags2dir.com/directory/) | |
| Tagvillage Crawler/Nutch-1.2 | |
| TailsweepBlogCrawler/Tailsweep-2.11-SNAPSHOT (http://www.tailsweep.com/; bot at [tailsweep] dot com) | |
| TailsweepBlogCrawler/Tailsweep-2.6-SNAPSHOT (http://www.tailsweep.com/; bot at [tailsweep] dot com) | |
| TailsweepBlogCrawler/Tailsweep-2.9-SNAPSHOT (http://www.tailsweep.com/; bot at [tailsweep] dot com) | |
| tangmiSpider/Nutch-1.1 (search; http://hi.baidu.com/ebyy; dl_tangmi@163.com) | |
| TCG/Nutch-1.0 (TCGspider; http://tcg.com ; TCGEmail) | |
| tea (whz_test_send@163.com) | |
| TeezirAgent/1.0 (Windows; testversie) | |
| tehr0b0t/Nutch-0.9 | |
| temaseek.com/Nutch-1.0 (temaseek.com; crawler@temaseek.com) | |
| TemaSEEK/Nutch-1.1 (http://www.temaseek.com; crawler@temaseek.com) | |
| Teoma/1.0 | |
| Teoma/Mozilla/5.0 (Ask Jeeves Question and Answer Search; http://about.ask.com/en/docs/about/webmasters.shtml; dr.edison@ask.com) | |
| Teoma/Mozilla/5.0 (Ask Jeeves Question and Answer Search; http://www.ask.com/about/help/webmasters; dr.edison@ask.com) | |
| Teoma/Nutch-1.2 ( Question and Answer Search; Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html); bot@afarm.com) | |
| Teragram/SAS Crawler | |
| Test-Fetcher-0.1/Nutch-0.9 (Awesomeness) | |
| test-name/test-version (test-url) | |
| Test.Buzzz/Nutch-0.8.1 (Test.Buzz; http://test.com; test@test.com) | |
| TEST/hoqBot-1.0 (dummy; http://www.dummy.com; info at dummy dot com) | |
| test/Nutch-0.9 (fortest; http://www.moojoy.com; fuxiang@yingzhitech.com) | |
| Test/Nutch-1.0 (Nutch spiderman; http://www.cosmoage.com ; MyEmail) | |
| test/Nutch-1.0 (test; test; test) | |
| test/Nutch-1.2 (test; www.163.com; xxx@126.com) | |
| testbaidu/Nutch-1.2 | |
| TestBot/Nutch-1.1 | |
| TestNutch/Nutch-1.2 (testing Nutch; http://nutch.apache.org; info@nutch.apache.org) | |
| The Crawler/Nutch-1.3 | |
| TheaSearch Crawler/Nutch-1.4 | |
| theasearch.com spider/Nutch-1.4 | |
| theasearchCrawler/Nutch-1.4 (http://www.theasearch.com/; hansod1@gmail.com) | |
| TheRarestParser/0.4b (http://rarestblog.com/therarestparser/) | |
| ThumbShots-Bot (+http://thumbshots.in/bot.html) | |
| thumbshots-de-bot (+http://www.thumbshots.de) | |
| thumbshots-de-bot (+http://www.thumbshots.de/) | |
| TigrinyaCrawler(tirawler@gmail.com) | |
| TimKimSearch/2.1 | |
| TinEye/1.1 (http://tineye.com/crawler.html) | |
| Tipiweb/Nutch-1.0 (http://www.tipiweb.net) | |
| to-night-Bot (Version: 1.06, powered by www.to-night.de +http://www.to-night.de/pfadzurbotseite/bot.html) | |
| Tobot/Nutch-1.4 ('tobot at eel dot rdc dot toshiba dot co dot jp') | |
| TomekSamcik/Nutch-1.0 (http://tomek.samcik.net; tomek@mail.samcik.net) | |
| toofaan/Nutch-1.0 (http://www.toofaan.com) | |
| Toplistbot | |
| topyx-crawler | |
| Tours/Nutch-1.2 | |
| TRAAAVELPIG/Nutch-1.3 | |
| tradove.com/1.0 | |
| Trends Crawler, Real time trends bot (info@trendscrawler.com) | |
| TrialSpider/Nutch-1.1 | |
| tricologia medicina/Nutch-1.0 (tricologia medica; http://www.studioditricologia.it; agatapollicino@live.it) | |
| trovi/1.10 (+http://www.trovi.be) | |
| trovi/1.11 (+http://www.trovi.be) | |
| TrueKnowledgeBot/Nutch-1.0 (A web crawler to extract facts from pages for the True Knowledge knowledge base; http://www.trueknowledge.com; miners -AT- trueknowledge _dot_ com) | |
| TruNorth/Nutch-1.4 | |
| TRY ENGINE/Nutch-1.0 | |
| TsinghuaLab | |
| TSWebLiveBot/Bot-1.0 (Crawling the Web; http://www.tsweblive.com; admin@tsweblive.com) | |
| tudelft-research/Nutch-1.0 | |
| tukangcari | |
| TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html) | |
| TwengaBot-Discover (http://www.twenga.fr/bot-discover.html) | |
| TwengaBot/2.0 (http://www.twenga.com/bot.html) | |
| TwengaBot/2.0 text/* | |
| Twib::Crawler | |
| Twib::Crawler/0.02 | |
| Twib::Crawler/0.03 | |
| Twitterbot v1.0 | |
| Twitterbot/1.0 | |
| t_crawler/0.4 | |
| ua-bot/1.26 http://www.ua-bot.com/ | |
| UberaSiteExplorer/UberaSiteExplorer-1.0 | |
| uk_parliament (+http://www.parliament.uk;archives@parliament.uk) | |
| unister-bot (Mozilla/5.0 compatible; crawler@unister-gmbh.de) | |
| UnisterBot (Mozilla/5.0 compatible; crawler@unister-gmbh.de) | |
| university/Nutch-1.0 (research) | |
| UniversityofEdinburgh/Nutch-1.0-dev (Academic; http://www.ed.ac.uk; bhaddow at inf point ed point ac punto uk) | |
| unknownbot (abc@mail.com) | |
| uocCrawler/Nutch-0.9 (uocCrawler; www.uoc.org; info.uoc@gmail.com) | |
| Updownerbot (+http://www.updowner.com/bot) | |
| upictoBot | |
| UrlFetcher/1.0 | |
| UrlsListDownloader (larbin2.6.3@unspecified.mail) | |
| url_test (larbin2.6.3@unspecified.mail) | |
| User-Agent: BoardReader-Image-Fetcher /1.0 info@boardreader.com | |
| User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.122 Safari/534.30 | |
| User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.8) Gecko/2009032609 Firefox/3.0.8 (.NET CLR 3.5.30729) | |
| User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 | |
| User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729) | |
| UserAgent | |
| us_orgedu_viewer (larbin2.6.3@unspecified.mail) | |
| UTPA-GSA-CRAWLER (Enterprise; S5-E9C9SWHTV8JJB; syshelp@utpa.edu,webhelp@utpa.edu) | |
| Utsz.NET Spider/Nutch-1.3 | |
| uw_cse_xwc/Nutch-0.9 (University of Washington Computer Science extensible web crawler project | |
| Validator/1 | |
| VerbstarBot/1 | |
| vik-robot/Nutch-1.0 (vikspider; http://vik.com; chenlibiti@163.com) | |
| VoluniaNutch/Nutch-1.2 (testing VoluniaNutch; http://www.volunia.com; info@volunia.com) | |
| volverine/Nutch-0.9 (agentspider; beast@mail.com) | |
| Voracious/0.1 | |
| votre-regard/0.1 | |
| voyager/2.0 (http://www.kosmix.com/crawler.html) | |
| VSE/1.0 (fabio.foroni@kisiwa.it) | |
| VSE/1.0 (freddy.parra@legis.com.co) | |
| VSE/1.0 (mann@vivisimo.com) | |
| VSE/1.0 (myunn@vivisimo.com) | |
| VSE/1.0 (notificacionservicios@iconomultimedia.com) | |
| VSE/1.0 (nunzio.d.hayslip@lmco.com) | |
| VSE/1.0 (pagano@vivisimo.com) | |
| VSE/1.0 (turnshek@vivisimo.com) | |
| VSE/1.0 (wall@vivisimo.com) | |
| W3C-checklink/4.5 [4.154] libwww-perl/5.805 | |
| W3C-checklink/4.5 [4.154] libwww-perl/5.823 | |
| W3C-checklink/4.5 [4.160] libwww-perl/5.823 | |
| W3C-checklink/4.6 [4.194] libwww-perl/5.805 | |
| W3C-checklink/4.6 [4.194] libwww-perl/5.823 | |
| W3C-checklink/4.6 [4.194] libwww-perl/5.836 | |
| W3C-checklink/4.7 libwww-perl/5.836 | |
| W3C-checklink/4.7 libwww-perl/6.01 | |
| W3C-checklink/4.8 libwww-perl/5.836 | |
| W3C-checklink/4.8 libwww-perl/5.837 | |
| W3C-checklink/4.8 libwww-perl/6.02 | |
| W3C-checklink/4.81 libwww-perl/5.836 | |
| WaC crawler/Nutch-1.2 (Collecting linguistic data from the Web; http://www.sigwac.org.uk/) | |
| Wada.vn Vietnamese Search/2.1 | |
| WAP_Browser/5.0 (compatible; YodaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; ) | |
| waue/Nutch-1.2 (MyTest; gm1.nchc.org.tw; waue@nchc.org.tw) | |
| WBP Ninja Spider/Nutch-1.2 (http://www.webuildpages.com) | |
| WDG_SiteValidator/1.6.4 | |
| Web Spider/Nutch-0.9 | |
| webcrawler (compatible; heritrix/1.14.3 ++http://www.onb.ac.at/about/webarchivierung.htm) | |
| webcrawler (compatible; heritrix/1.14.4 ++http://www.onb.ac.at/about/webarchivierung.htm) | |
| Webcrawler/1.0 (web spider; ) | |
| Webcrawler/Nutch-1.0 (Test crawl; lucene.apache.org/nutch/; a@b.net) | |
| WebCrawler/Nutch-1.2 (WebCrawler; WebCrawler) | |
| WebImages 0.3 ( http://herbert.groot.jebbink.nl/?app=WebImages ) | |
| WebIndex (webindex.csv.lt) | |
| webmeasurement-bot, http://rvs.informatik.uni-leipzig.de/bot.php | |
| webmoney.advisor.bot/Nutch-1.0 | |
| webmoney.advisor/Nutch-1.0 | |
| WebMonitorRobots/Nutch-1.0 (Nutch_spiderman; http://www.csdl.ac.cn; zhang_zhixiong2003@yahoo.com) | |
| WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/) | |
| WebReaper v10.0 - www.webreaper.net | |
| WebReaper v9.8 - www.webreaper.net | |
| WebReaper vWebReaper v7.3 - www,otway.com/webreaper | |
| Webspider/1.0 (web spider; ) | |
| WebSurfer text/* | |
| web_bq (larbin2.6.3@unspecified.mail) | |
| WeKnow Spider/Nutch-1.0 | |
| welinktech/Nutch-1.3 | |
| weneobot/Nutch-1.2 (weneo.de bot; http://www.weneo.de/bot; bot at weneo.de) | |
| WeSEE:Search/0.1 (Alpha, http://www.wesee.com/) | |
| WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/) | |
| WeViKa | |
| WEVONZO | |
| wget-1.0 (larbin2.6.3@unspecified.mail) | |
| Wget/0.1 (linux-gnu) | |
| Wget/0.1.3 | |
| Wget/1.10 | |
| Wget/1.10.1 | |
| Wget/1.10.2 | |
| Wget/1.10.2 (Red Hat modified) | |
| Wget/1.11.1 | |
| Wget/1.11.2 | |
| Wget/1.11.3 | |
| Wget/1.11.4 | |
| Wget/1.11.4 (Red Hat modified) | |
| Wget/1.11.4 Red Hat modified | |
| Wget/1.12 (darwin10.6.0) | |
| Wget/1.12 (freebsd8.1) | |
| Wget/1.12 (linux-gnu) | |
| Wget/1.13.1 (linux-gnu) | |
| Wget/1.13.4 (darwin10.8.0) | |
| Wget/1.13.4 (darwin11.1.0) | |
| Wget/1.13.4 (darwin11.2.0) | |
| Wget/1.13.4 (darwin11.3.0) | |
| Wget/1.13.4 (linux-gnu) | |
| Wget/1.5.3 | |
| Wget/1.5.3.1 | |
| Wget/1.8.2 | |
| Wget/1.9 | |
| Wget/1.9+cvs-stable (Red Hat modified) | |
| Wget/1.9.1 | |
| whiteiexpres/Nutch-0.9 (whiteiWebBot; whiteiexpress.com; whiteye@kon-x.com) | |
| WhoYougle | |
| wiiy/Nutch-1.0 | |
| wiiy/Nutch-1.1 (Nutch Search Engineer; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) | |
| wikiwix-bot-3.0 | |
| WindowPane1.0 | |
| windows xp | |
| wirkon.com (Version: 1.06, powered by www.wirkon.com+http://www.wirkon.com) | |
| WisdomCrawler/1.0 (A crawler from Lab of Wisdom, School of Software, Dalian University of Technology, for testing purpose; http://wisdom.dlut.edu.cn/newsite/Home.aspx; liangliang0007 at 126 dot com) | |
| WMCAI-robot (http://www.topicmaster.jp/wmcai/crawler.html) | |
| woobot/1.1 | |
| Woovi/Nutch-1.0 (http://www.woovi.com/bot) | |
| WordPress/3.4-beta1; | |
| WordPress/3.4-beta2-20452; | |
| WoS | |
| Wotbox/2.0 (bot@wotbox.com; http://www.wotbox.com) | |
| Wouah/Nutch-1.0 (http://www.wouah.fr) | |
| wpss_test.pl | |
| wsrwebometer/1.0 (wsrwebometer crawler; http://www.webscienceresearch.org/crawlerInfo.html; crawladmin at webscienceresearch dot com) | |
| wttsearch/Nutch-1.0 (Crawler for SIFE (WTT); http://wtt.beecom.ch/wttsearch.html; wtt@beecom.ch) | |
| ww7bot | |
| www (wang_cyi@163.com) | |
| www.bemash.net/Nutch-1.3 (bemash; www.bemash.net; info@bemash.net) | |
| wwwster/1.4 (Beta, mailto:gue@cis.uni-muenchen.de) | |
| WyoBot 1.0 | |
| XP (XP) | |
| xp/Nutch-1.0 (Nutch spiderman; *; MyEmail) | |
| xunuu/Nutch-1.0 | |
| Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html ) | |
| Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html) | |
| Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0) | |
| Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) | |
| Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/blog-search/) | |
| Y!J-PSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) | |
| Y!J-SRD/1.0 | |
| yacy (www.yacy.net; amd64 FreeBSD 7.1-PRERELEASE; java 1.6.0_07; Europe/en) Jakarta Commons-HttpClient/3.1 | |
| yacy (www.yacy.net; amd64 FreeBSD 7.1-STABLE; java 1.6.0_07; Europe/en) Jakarta Commons-HttpClient/3.1 | |
| yacy (www.yacy.net; amd64 Linux 2.6.24-23-generic; java 1.6.0_07; Europe/de) Jakarta Commons-HttpClient/3.1 | |
| yacy (www.yacy.net; amd64 Linux 2.6.24-24-generic; java 1.6.0_07; Europe/en) Jakarta Commons-HttpClient/3.1 | |
| yacy (www.yacy.net; amd64 Linux 2.6.28-11-generic; java 1.6.0_0; Europe/de) Jakarta Commons-HttpClient/3.1 | |
| yacy (www.yacy.net; amd64 Linux 2.6.28-11-generic; java 1.6.0_0; Europe/en) Jakarta Commons-HttpClient/3.1 | |
| yacy (www.yacy.net; amd64 Linux 2.6.28-11-generic; java 1.6.0_13; Europe/en) Jakarta Commons-HttpClient/3.1 | |
| yacy (www.yacy.net; amd64 Linux 2.6.28-12-generic; java 1.6.0_0; Europe/de) Jakarta Commons-HttpClient/3.1 | |
| yacy (www.yacy.net; amd64 Windows 7 6.1; java 1.6.0_14; Europe/de) Jakarta Commons-HttpClient/3.1 | |
| yacy (www.yacy.net; amd64 Windows Vista 6.1; java 1.6.0_13; Europe/de) Jakarta Commons-HttpClient/3.1 | |
| yacy (www.yacy.net; i386 Linux 2.6.18-128.1.10.el5xen; java 1.6.0_07; Europe/de) Jakarta Commons-HttpClient/3.1 | |
| yacy (www.yacy.net; i386 Linux 2.6.18-6-686; java 1.6.0_10; Europe/de) Jakarta Commons-HttpClient/3.1 | |
| yacy (www.yacy.net; i386 Linux 2.6.23; java 1.6.0_06; Europe/en) Jakarta Commons-HttpClient/3.1 | |
| yacy (www.yacy.net; i386 Linux 2.6.24-etchnhalf.1-686; java 1.5.0_14; GMT+01:00/de) Jakarta Commons-HttpClient/3.1 | |
| yacy (www.yacy.net; i386 Linux 2.6.28-11-generic; java 1.6.0_0; Europe/en) Jakarta Commons-HttpClient/3.1 | |
| yacy (www.yacy.net; x86 Windows XP 5.1; java 1.6.0_05; Europe/de) Jakarta Commons-HttpClient/3.1 | |
| yacy (www.yacy.net; x86 Windows XP 5.1; java 1.6.0_13; Europe/de) Jakarta Commons-HttpClient/3.1 | |
| yacy/0.5910493 (www.yacy.net; x86 Windows Vista 6.0; java 1.6.0_11; Europe/de) Jakarta Commons-HttpClient/3.1 | |
| yacybot (allip/any; x86_64 Mac OS X 10.6.7; java 1.6.0_24; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 FreeBSD 7.1-PRERELEASE; java 1.6.0_07; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 FreeBSD 7.1-STABLE; java 1.6.0_07; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 FreeBSD 8.0-RELEASE-p2; java 1.6.0_07; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 FreeBSD 8.0-RELEASE-p3; java 1.6.0_07; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.16.46-0.12-smp; java 1.6.0_15; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-128.2.1.el5; java 1.6.0_14; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-128.4.1.el5; java 1.6.0_14; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-128.7.1.el5; java 1.6.0; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-164.11.1.el5; java 1.6.0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-164.15.1.el5; java 1.6.0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-164.15.1.el5xen; java 1.6.0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-164.6.1.el5; java 1.6.0; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-164.6.1.el5; java 1.6.0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-164.6.1.el5; java 1.6.0_16; Asia/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-164.9.1.el5xen; java 1.6.0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-164.el5; java 1.6.0; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-164.el5; java 1.6.0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-194.17.1.el5xen; java 1.6.0_17; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-194.3.1.el5.028stab069.6; java 1.6.0_17; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-194.3.1.el5; java 1.6.0_0; GB-Eire/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-194.3.1.el5; java 1.6.0_14; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-194.8.1.el5; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-53.el5; java 1.6.0; America/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.18-6-amd64; java 1.5.0_14; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.22-14-server; java 1.6.0_06; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.23.17-dbserv; java 1.6.0_04; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.24-16-server; java 1.6.0_07; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.24-23-generic; java 1.6.0_07; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.24-23-generic; java 1.6.0_07; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.24-23-generic; java 1.6.0_07; GMT/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.24-24-generic; java 1.6.0_07; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.24-24-generic; java 1.6.0_07; GMT/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.24-7-pve; java 1.6.0_13; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.24-etchnhalf.1-amd64; java 1.5.0_14; UTC/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.24-etchnhalf.1-amd64; java 1.6.0_07; UTC/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.24-etchnhalf.1-amd64; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.26-1-amd64; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.26-2-amd64; java 1.5.0_17; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_0; Europe/fr) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_12; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.26-2-amd64; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.26-2-xen-amd64; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.26-2-xen-amd64; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.26-2-xen-amd64; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.26-2-xen-amd64; java 1.6.0_21; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.27-11-generic; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.27-11-generic; java 1.6.0_10; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.27-11-server; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.27-14-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.27.25-170.2.72.fc10.x86_64; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.27.39-0.2-default; java 1.6.0_17; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-11-generic; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-11-generic; java 1.6.0_13; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-12-generic; java 1.6.0_13; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-12-generic; java 1.6.0_13; GMT/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-14-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-14-generic; java 1.6.0_14; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-15-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-15-generic; java 1.6.0_14; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-15-generic; java 1.6.0_14; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-15-generic; java 1.6.0_16; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-16-generic; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-16-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-16-generic; java 1.6.0_16; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-16-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-17-generic; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-17-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-17-generic; java 1.6.0_16; GMT/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-18-generic; java 1.6.0_19; GMT/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-18-generic; java 1.6.0_20; GMT/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28-hardened-r7; java 1.6.0_17; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28.3-vs2.3.0.36.4; java 1.6.0_12; Universal/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.28.3-vs2.3.0.36.4; java 1.6.0_20; Universal/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.29-ARCH; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.30-1-amd64; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.30.1; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.30.5-43.fc11.x86_64; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-11-rt; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-14-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-14-generic; java 1.6.0_15; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-15-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-15-generic; java 1.6.0_16; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-16-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-16-server; java 1.6.0_17; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-17-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-17-server; java 1.6.0_17; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-19-generic; java 1.6.0_16; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-19-server; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-20-generic; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-20-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-20-generic; java 1.6.0_0; US/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-20-generic; java 1.6.0_15; Atlantic/es) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-20-generic; java 1.6.0_15; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-20-generic; java 1.6.0_16; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-20-server; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-20-server; java 1.6.0_15; US/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-21-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-21-server; java 1.6.0_0; Asia/ja) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-21-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-22-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-22-server; java 1.6.0_18; Asia/ja) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-22-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-gentoo-r6; java 1.6.0_0; Europe/fr) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-gentoo-r6; java 1.6.0_17; Etc/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31-gentoo-r6; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31.12-0.1-desktop; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31.4-desktop-0.rc2.1mnb; java 1.6.0_0; Europe/fr) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.31.5; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32-18-server; java 1.6.0_18; America/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32-21-generic; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32-22-generic; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32-22-generic; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32-22-generic; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32-22-generic; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32-23-generic; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32-24-generic; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32-24-generic; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32-24-generic; java 1.6.0_20; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32-24-server; java 1.6.0_18; America/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32-24-server; java 1.6.0_18; Europe/cs) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32-25-generic; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32-3-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32-3-amd64; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32-gentoo; java 1.6.0_17; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32-trunk-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32.2; java 1.6.0_17; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32.2; java 1.6.0_17; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32.2; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.32.9-rscloud; java 1.6.0_20; Etc/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.33-ARCH; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.33.8-149.fc13.x86_64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.33; java 1.6.0_17; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.34-12-desktop; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.34-ARCH; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.34-rc6-git1; java 1.6.0_20; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Linux 2.6.9-023stab048.4-smp; java 1.6.0; GMT/en) http://yacy.net/bot.html | |
| yacybot (amd64 Windows 2003 5.2; java 1.6.0_19; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Windows 7 6.1; java 1.6.0_14; America/en) http://yacy.net/bot.html | |
| yacybot (amd64 Windows 7 6.1; java 1.6.0_14; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Windows 7 6.1; java 1.6.0_15; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Windows 7 6.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Windows 7 6.1; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (amd64 Windows 7 6.1; java 1.6.0_20; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Windows Server 2008 6.0; java 1.6.0_14; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Windows Server 2008 R2 6.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Windows Server 2008 R2 6.1; java 1.6.0_21; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Windows Vista 6.0; java 1.6.0_15; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Windows Vista 6.0; java 1.6.0_16; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Windows Vista 6.0; java 1.6.0_17; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Windows Vista 6.0; java 1.7.0-ea; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Windows Vista 6.1; java 1.6.0_12; Europe/de) http://yacy.net/bot.html | |
| yacybot (amd64 Windows Vista 6.1; java 1.6.0_13; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld-global; amd64 Linux 2.6.32-35-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld-global; amd64 Linux 2.6.32-36-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld-global; i386 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld-global; i386 Linux 2.6.38.8-desktop586-6.mga; java 1.6.0_22; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld-global; x86 Windows 7 6.1; java 1.6.0_24; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld-global; x86_64 Mac OS X 10.6.4; java 1.6.0_22; Europe/es) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 FreeBSD 7.1-STABLE; java 1.6.0_07; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 FreeBSD 8.2-STABLE; java 1.6.0; GMT/pl) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.16.46-0.12-smp; java 1.6.0_15; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.18-194.26.1.el5; java 1.6.0_14; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.26-2-xen-amd64; java 1.6.0_26; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.26-custom; java 1.6.0_22; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.26; java 1.6.0_22; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.28.3-vs2.3.0.36.4; java 1.6.0_20; Universal/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.31-23-server; java 1.6.0_24; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-23-server; java 1.6.0_22; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-28-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-30-generic; java 1.6.0_20; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-30-generic; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-30-server; java 1.6.0_24; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-31-generic; java 1.7.0; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-31-server; java 1.6.0_24; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-32-generic; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-32-generic; java 1.6.0_24; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-32-server; java 1.6.0_24; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-32-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-33-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-34-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-34-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-342-ec2; java 1.6.0_20; Etc/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-36-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-37-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-37-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-38-generic; java 1.7.0; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-40-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_26; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_29; US/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_22; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.7.0_147-icedtea; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-6-pve; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_24; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32.2; java 1.6.0_22; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.34.7-0.5-desktop; java 1.6.0_20; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.34.7-0.7-desktop; java 1.6.0_20; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.35-23-generic; java 1.6.0_20; Asia/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.35-24-generic; java 1.6.0_20; Asia/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.35-24-generic; java 1.6.0_22; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.35-24-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.35.6; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.36.4-vs2.3.0.36.39-netcup; java 1.6.0_18; Etc/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.37.6; java 1.6.0_25; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.37; java 1.6.0_22; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.38-10-generic; java 1.6.0_22; Asia/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.38-11-server; java 1.6.0_22; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.38-13-generic; java 1.6.0_22; America/pt) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.38-2-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.38-8-generic; java 1.6.0_22; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.38-8-generic; java 1.6.0_22; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.38-8-server; java 1.6.0_22; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.38.2-grsec-xxxx-grs-ipv6-64-filer; java 1.6.0_22; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.38.2-grsec-xxxx-grs-ipv6-64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-12-generic; java 1.6.0_26; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-12-server; java 1.6.0_23; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-12-server; java 1.6.0_26; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-13-generic; java 1.6.0_23; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-13-generic; java 1.7.0_147-icedtea; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-14-generic; java 1.6.0_23; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-14-generic; java 1.6.0_23; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-15-server; java 1.6.0_23; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-15-server; java 1.6.0_26; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-16-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-rc4-1-desktop; java 1.6.0_22; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-rc5-1-desktop; java 1.6.0_22; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.1.0-1-amd64; java 1.6.0_26; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.1.0-1.2-desktop; java 1.6.0_22; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.1.0-1.2-desktop; java 1.6.0_22; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.1.10-1.9-default; java 1.6.0_24; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.1.10-hardened; java 1.7.0_03-icedtea; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.1.1; java 1.7.0_147-icedtea; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.1.2-1.fc16.x86_64; java 1.6.0_22; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.1.2-1.fc16.x86_64; java 1.6.0_22; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.1.4-1-ARCH; java 1.7.0_147-icedtea; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.1.9-1.4-default; java 1.6.0_24; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.0-1-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.0-10-generic; java 1.7.0_147-icedtea; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.0-18-generic; java 1.7.0_03-icedtea; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.0-2-rt-amd64; java 1.6.0_24; Asia/ja) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.0-21-generic; java 1.7.0_03-icedtea; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.0; java 1.7.0_147-icedtea; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.1-2-ARCH; java 1.7.0_147-icedtea; Europe/fr) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_22; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_24; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.13-1-ARCH; java 1.7.0_03-icedtea; Europe/fr) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.2-1-ARCH; java 1.7.0_147-icedtea; Europe/fr) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.2-1.fc16.x86_64; java 1.7.0_b147-icedtea; W-SU/ru) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.3.1-1-ARCH; java 1.6.0_24; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_23; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_25; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_26; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_26; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_26; Europe/fr) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_27; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Asia/ru) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/fr) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/no) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/sv) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_01; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0_01; Europe/fr) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_03; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows Vista 6.0; java 1.6.0_23; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.18-028stab070.2; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.18-028stab070.5; java 1.6.0_22; Etc/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.18-194.26.1.el5.028stab079.2; java 1.6.0_24; Etc/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.24-1-686; java 1.6.0_0; Europe/cs) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.24-28-generic; java 1.6.0_24; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.32-24-generic; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.32-3-pve; java 1.6.0_18; Europe/cs) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.32-33-generic-pae; java 1.6.0_20; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.32-36-generic-pae; java 1.6.0_20; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.32-39-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.32-4-pve; java 1.6.0_18; Etc/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.32-5-686-bigmem; java 1.6.0_26; Etc/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.6.0_18; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.32-5-686; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.36; java 1.6.0_23; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.37-gentoo-r4-110510; java 1.6.0_22; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.37-gentoo-r4-110510; java 1.6.0_24; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.38-13-generic; java 1.6.0_20; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.38-2-686; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.38-8-generic-pae; java 1.6.0_24; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.38-8-generic; java 1.6.0_22; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.38-8-generic; java 1.6.0_22; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.38.7-smp; java 1.6.0_29; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.38; java 1.6.0_24; Europe/es) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 3.0.0-12-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 3.0.0-12-generic-pae; java 1.6.0_23; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 3.0.0-13-generic-pae; java 1.6.0_23; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 3.0.0-15-generic-pae; java 1.6.0_23; America/fr) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 3.0.0-16-generic; java 1.6.0_23; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; x86 Windows 2003 5.2; java 1.6.0_30; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; x86 Windows 7 6.1; java 1.6.0_23; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_01; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; x86 Windows XP 5.1; java 1.6.0_22; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; x86 Windows XP 5.1; java 1.6.0_23; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; x86 Windows XP 5.1; java 1.6.0_26; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; x86 Windows XP 5.1; java 1.6.0_29; Europe/es) http://yacy.net/bot.html | |
| yacybot (freeworld/global; x86 Windows XP 5.1; java 1.6.0_31; Europe/sv) http://yacy.net/bot.html | |
| yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Asia/ru) http://yacy.net/bot.html | |
| yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Europe/ru) http://yacy.net/bot.html | |
| yacybot (i386 FreeBSD 8.0-RELEASE-p2; java 1.7.0; GMT/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.4.18; java 1.5.0_15; America/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.4.35-wt1; java 1.6.0_05; CET/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.16.37-domU-v1; java 1.6.0_12; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.16.46-0.12-smp; java 1.5.0_06; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.18-028stab064.7; java 1.6.0_15; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.18-028stab069.6; java 1.6.0_20; Etc/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.18-128.1.10.el5PAE; java 1.5.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.18-128.1.10.el5xen; java 1.6.0_07; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.18-128.1.14.el5xen; java 1.6.0_07; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.18-164.2.1.el5; java 1.5.0_17; America/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.18-6-686; java 1.6.0_10; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.18-6-686; java 1.6.0_16; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.18-92.1.22.el5xen; java 1.6.0_10; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.23; java 1.6.0_06; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.23; java 1.6.0_17; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.24-1-686; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.24-10-pve; java 1.6.0_12; Etc/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.24-23-generic; java 1.6.0_07; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.24-23-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.24-23-server; java 1.6.0_17; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.24-24-generic; java 1.6.0_07; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.24-24-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.24-27-server; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.24-8-pve; java 1.6.0_12; Zulu/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.24-etchnhalf.1-686; java 1.5.0_14; GMT+01:00/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.24.6; java 1.6.0_13; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.24_UB; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.25-gentoo-r7; java 1.6.0_07; UTC/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.25.20-0.5-pae; java 1.6.0_15; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.26-2-686-bigmem; java 1.6.0_12; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.26-2-686-bigmem; java 1.6.0_14; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.26-2-686; java 1.5.0_17; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_0; America/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_12; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_12; Europe/fr) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.26-2-686; java 1.6.0_20; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.26-2-openvz-686; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.26-2-openvz-686; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.26-2-openvz-686; java 1.6.0_0; UTC/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.26-gentoo-r3-gh0st; java 1.6.0_13; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.26; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.27-11-generic; java 1.5.0_16; GMT+01:00/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.27-11-generic; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.27-9-generic; java 1.6.0_10; Australia/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.27.10-grsec-xxxx-grs-ipv4-32; java 1.6.0_07; Europe/fr) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.27.10-grsec-xxxx-grs-ipv4-32; java 1.6.0_12; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.27.10-grsec-xxxx-grs-ipv6-32; java 1.6.0_12; Europe/fr) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.27.21-0.1-default; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.27.21-0.1-pae; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-11-generic; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-11-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-11-generic; java 1.6.0_0; Europe/fr) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-11-generic; java 1.6.0_13; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-12-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-12-netbook; java 1.6.0_13; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-13-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-13-generic; java 1.6.0_13; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-13-generic; java 1.6.0_14; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-14-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-15-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-15-generic; java 1.6.0_13; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-15-server; java 1.6.0_16; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-16-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-16-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-17-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-17-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-18-generic; java 1.6.0_16; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-18-generic; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-19-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-19-server; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.28-gentoo-r5; java 1.5.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.30-2.slh.2-sidux-686; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.30-gentoo-r6-090907; java 1.6.0_17; GMT/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.30-gentoo-r6; java 1.5.0_22; Europe/el) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31-14-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31-16-generic-pae; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31-16-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31-16-generic; java 1.6.0_15; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31-17-generic-pae; java 1.6.0_17; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31-17-generic; java 1.6.0_15; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31-17-generic; java 1.6.0_15; SystemV/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31-17-generic; java 1.6.0_16; Europe/fr) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31-18-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31-19-generic-pae; java 1.6.0_15; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31-19-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31-19-generic; java 1.6.0_15; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31-20-generic-pae; java 1.6.0_0; America/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31-20-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31-20-generic; java 1.6.0_0; Europe/fr) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31-21-generic; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31-ARCH; java 1.6.0_17; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31-gentoo; java 1.6.0_17; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31.12-0.1-desktop; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31.12-mfs002-xen0-compat-i586; java 1.6.0_15; Universal/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.31.6; java 1.6.0_15; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32-21-generic-pae; java 1.6.0_18; America/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32-21-generic; java 1.6.0_18; America/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32-22-generic-pae; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32-22-generic-pae; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32-22-generic; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32-23-generic; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32-23-generic; java 1.6.0_18; Europe/es) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32-23-generic; java 1.6.0_20; Europe/es) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32-24-generic; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32-24-generic; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32-24-generic; java 1.6.0_20; America/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32-5-686; java 1.6.0_18; America/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32-5-686; java 1.6.0_18; Atlantic/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32-gentoo-r7; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32-trunk-686-bigmem; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32-trunk-686; java 1.6.0_17; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32-trunk-686; java 1.6.0_18; America/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32-trunk-686; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32.2-xxxx-std-ipv4-32; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32.2-xxxx-std-ipv4-32; java 1.6.0_17; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.32.2; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.34.6-xxxx-grs-ipv6-32; java 1.6.0_0; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.9-023stab048.6-enterprise; java 1.6.0_07; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Linux 2.6.9-5.ELsmp; java 1.6.0_12-rev; Asia/en) http://yacy.net/bot.html | |
| yacybot (i386 Mac OS X 10.5.6; java 1.5.0_16; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Mac OS X 10.5.7; java 1.5.0_16; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Mac OS X 10.5.7; java 1.5.0_19; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Mac OS X 10.5.8; java 1.5.0_19; Europe/de) http://yacy.net/bot.html | |
| yacybot (i386 Mac OS X 10.5.8; java 1.5.0_20; Europe/fr) http://yacy.net/bot.html | |
| yacybot (i386 Mac OS X 10.5.8; java 1.5.0_22; Europe/de) http://yacy.net/bot.html | |
| yacybot (ppc Mac OS X 10.5.6; java 1.5.0_13; Europe/de) http://yacy.net/bot.html | |
| yacybot (ppc Mac OS X 10.5.8; java 1.5.0_22; Europe/nl) http://yacy.net/bot.html | |
| yacybot (sparc Linux 2.6.26-2-sparc64-smp; java 1.6.0_0; America/en) http://yacy.net/bot.html | |
| yacybot (webportal-global; amd64 Linux 2.6.26-2-xen-amd64; java 1.6.0_0; Europe/en) http://yacy.net/bot.html | |
| yacybot (webportal-global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (webportal-global; amd64 Linux 3.0.0-17-generic; java 1.7.0_147-icedtea; Europe/de) http://yacy.net/bot.html | |
| yacybot (webportal-global; amd64 Linux 3.0.4-ck; java 1.6.0_22; America/en) http://yacy.net/bot.html | |
| yacybot (webportal-global; amd64 Windows 7 6.1; java 1.7.0; Europe/de) http://yacy.net/bot.html | |
| yacybot (webportal-global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/pt) http://yacy.net/bot.html | |
| yacybot (webportal/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (webportal/global; amd64 Windows 7 6.1; java 1.7.0_02; Europe/de) http://yacy.net/bot.html | |
| yacybot (webportal/global; i386 Linux 2.6.35-28-generic; java 1.6.0_20; America/en) http://yacy.net/bot.html | |
| yacybot (x86 SunOS 5.11; java 1.6.0_12; Europe/en) http://yacy.net/bot.html | |
| yacybot (x86 SunOS 5.11; java 1.6.0_13; Europe/en) http://yacy.net/bot.html | |
| yacybot (x86 SunOS 5.11; java 1.6.0_15; Europe/en) http://yacy.net/bot.html | |
| yacybot (x86 Windows 2003 5.2; java 1.6.0_14; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows 2003 5.2; java 1.6.0_16; America/en) http://yacy.net/bot.html | |
| yacybot (x86 Windows 2003 5.2; java 1.6.0_16; Europe/en) http://yacy.net/bot.html | |
| yacybot (x86 Windows 2003 5.2; java 1.6.0_17; America/en) http://yacy.net/bot.html | |
| yacybot (x86 Windows 2003 5.2; java 1.6.0_17; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows 2003 5.2; java 1.6.0_20; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows 7 6.1; java 1.6.0_16; Asia/ru) http://yacy.net/bot.html | |
| yacybot (x86 Windows 7 6.1; java 1.6.0_16; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows 7 6.1; java 1.6.0_17; America/en) http://yacy.net/bot.html | |
| yacybot (x86 Windows 7 6.1; java 1.6.0_17; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows 7 6.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows 7 6.1; java 1.6.0_18; Europe/sv) http://yacy.net/bot.html | |
| yacybot (x86 Windows 7 6.1; java 1.6.0_20; Europe/fr) http://yacy.net/bot.html | |
| yacybot (x86 Windows 7 6.1; java 1.6.0_20; Europe/sv) http://yacy.net/bot.html | |
| yacybot (x86 Windows Server 2008 6.0; java 1.6.0_16; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows Server 2008 6.0; java 1.6.0_20; Europe/nl) http://yacy.net/bot.html | |
| yacybot (x86 Windows Vista 6.0; java 1.6.0_04; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows Vista 6.0; java 1.6.0_07; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows Vista 6.0; java 1.6.0_11; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows Vista 6.0; java 1.6.0_13; Europe/en) http://yacy.net/bot.html | |
| yacybot (x86 Windows Vista 6.0; java 1.6.0_14; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows Vista 6.0; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (x86 Windows Vista 6.0; java 1.6.0_21; Europe/en) http://yacy.net/bot.html | |
| yacybot (x86 Windows Vista 6.1; java 1.5.0_13; Europe/sv) http://yacy.net/bot.html | |
| yacybot (x86 Windows Vista 6.1; java 1.6.0_11; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows Vista 6.1; java 1.6.0_12; Australia/en) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.5.0_06; Europe/sv) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_05; Asia/en) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_06; Europe/es) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_11; Europe/fr) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_12; Europe/pl) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_13; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_14; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_14; Europe/ru) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_15; America/en) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_15; America/fr) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_15; Europe/es) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_16; America/en) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_16; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_17; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_18; America/en) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_19; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_20; America/en) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_20; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.7.0-ea; Asia/zh) http://yacy.net/bot.html | |
| yacybot (x86_64 Mac OS X 10.6.2; java 1.6.0_17; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86_64 Mac OS X 10.6.3; java 1.6.0_15; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86_64 Mac OS X 10.6.3; java 1.6.0_17; America/en) http://yacy.net/bot.html | |
| yacybot (x86_64 Mac OS X 10.6.3; java 1.6.0_20; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86_64 Mac OS X 10.6.4; java 1.6.0_20; America/en) http://yacy.net/bot.html | |
| yacybot (x86_64 Mac OS X 10.6.4; java 1.6.0_20; Europe/de) http://yacy.net/bot.html | |
| yacybot (x86_64 Mac OS X 10.6.4; java 1.6.0_20; Europe/fr) http://yacy.net/bot.html | |
| yacybot (x86_64 Mac OS X 10.6.7; java 1.6.0_24; Europe/de) http://yacy.net/bot.html | |
| YaDirectBot/1.0 | |
| Yahoo Pipes 1.0 | |
| Yahoo Pipes 2.0 | |
| YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide) | |
| Yandex.Server/2009.5 | |
| Yandex/1.01.001 (compatible; Win16; H) | |
| Yandex/1.01.001 (compatible; Win16; I) | |
| Yandex/1.01.001 (compatible; Win16; I)/Nutch-1.2 | |
| Yandex/1.01.001 (compatible; Win16; m) | |
| Yandex/1.01.001 (compatible; Win16; P) | |
| Yandex/1.03.003 (compatible; Win16; D) | |
| YandexBlogs | |
| YandexSomething/1.0 | |
| Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/) | |
| yanky-nutch/Nutch-0.9 (nothing; nothing; yanky.young@gmail.com) | |
| yechizi/Nutch-1.2 | |
| yeetec_dasbot (zhaoqingfeng@yeetec.com) | |
| Yepi/1.0 (NHN Corp.; http://help.naver.com/robots/) | |
| Yeti/0.01 (nhn/1noon, yetibot@naver.com, check robots.txt daily and follows it) | |
| Yeti/1.0 (+http://help.naver.com/robots/) | |
| Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/) | |
| yolinkBot | |
| YowedoBot/Yowedo 1.0 (Search Engine crawler for yowedo.com; http://yowedo.com/en/partners.html; crawler@yowedo.com) | |
| YPBot/Raven1.1.3 (compatible; Googlebot/2.1;+http://www.yellowpages.com/about/legal/crawl) | |
| yrspider (Mozilla/5.0 (compatible; YRSpider; +http://www.yunrang.com/yrspider.html)) | |
| YUMROO/yumroo.search.bot-1.0 (search bot of http://www.yumroo.az; info at yumroo dot az) | |
| zawc/Nutch-1.0 (http://www.zawc.com) | |
| zawc/Nutch-1.2 (http://www.zawc.com) | |
| zclosure/Nutch-1.3 | |
| zclosure/Nutch-1.4 | |
| zebra457/Nutch-1.3 | |
| Zeus 76869 Webster Pro V2.9 Win32 | |
| zhuada/Nutch-1.0 (http://www.zhuada.com/spider/; spider@zhuada.com) | |
| Zhuada/Nutch-1.0 (www.zhuada.com/spider/) | |
| zibber-v0.1(www.zibb.com/crawler/) | |
| Zillabyte/Nutch-1.3 | |
| Zillsy.com Search engine /Nutch-1.4 | |
| zologo/Nutch-1.1 (zologo; http://www.zologo.com; hibo@zologo.co) | |
| zomba-bot/0.1 | |
| ZoomSpider - wrensoft.com [ZSEBOT] | |
| Zscho.de Crawler/Nutch-1.0-Zscho.de-semantic_patch (Zscho.de Crawler, collecting for machine learning; http://zscho.de/ ) | |
| Zscho.de/Nutch-0.9-semantic_patch | |
| zschobot/Nutch-0.9-semantic_patch (zschobot indexing; Zscho.de/de/bot.html) | |
| ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry) | |
| ZupeeCrawler0.1.1 | |
| ^Byte (http://CaretByte.com) | |
| _USERAGENT_ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env python | |
| # encoding: utf-8 | |
| """Parse the number of robot requests from an Apache log file | |
| The list of robot user agents comes from http://www.user-agents.org/ but has | |
| had some agents added to it since the list was last updated in 2010. | |
| """ | |
| from __future__ import division | |
| import fileinput | |
| import optparse | |
| import os | |
| import re | |
| import sys | |
| UA_RE = re.compile(r'^[^ ]+ [^ ]+ [^ ]+ \[[^\]]+\] "[^"]+" \d+ \d+ "[^"]+" "(.*?)"') | |
| def main(bot_agent_file="user-agents.txt"): | |
| all_requests = 0 | |
| bot_requests = 0 | |
| bot_count = {} | |
| bot_agents = set() | |
| with open(bot_agent_file, "r") as f: | |
| bot_agents.update(filter(None, (i.strip() for i in f))) | |
| for line in sys.stdin: | |
| all_requests += 1 | |
| m = UA_RE.match(line) | |
| ua = m.group(1) | |
| if ua == "-" or ua in bot_agents: | |
| bot_count[ua] = bot_count.get(ua, 0) + 1 | |
| bot_requests += 1 | |
| bots = bot_count.keys() | |
| bots.sort(key=lambda i: bot_count[i]) | |
| for bot in bots: | |
| print "%10d %s" % (bot_count[bot], bot) | |
| print "Bot requests: %d of %d (%0.2f%%)" % (bot_requests, all_requests, | |
| 100 * (bot_requests / all_requests)) | |
| if __name__ == "__main__": | |
| fmt = optparse.IndentedHelpFormatter(max_help_position=8) | |
| parser = optparse.OptionParser(__doc__.strip(), formatter=fmt) | |
| parser.add_option("--bot-user-agents", metavar="FILE", | |
| default=os.path.join(os.path.dirname(__file__), | |
| "user-agents.txt"), | |
| help="File containing a list of bot user-agents " | |
| "(default=%defaults)") | |
| (options, filenames) = parser.parse_args() | |
| if filenames: | |
| parser.error("Provide log data on stdin") | |
| main(bot_agent_file=options.bot_user_agents) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 'Mozilla/5.0 (compatible; BMC/1.0 (Y!J-AGENT))' | |
| ( Robots.txt Validator http://www.searchengineworld.com/cgi-bin/robotcheck.cgi ) | |
| (DreamPassport/3.0; isao/MyDiGiRabi) | |
| (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) | |
| (Privoxy/1.0) | |
| */Nutch-0.9-dev | |
| +SitiDi.net/SitiDiBot/1.0 (+Have Good Day) | |
| - | |
| - (larbin2.6.3@unspecified.mail) | |
| -DIE-KRAEHE- META-SEARCH-ENGINE/1.1 http://www.die-kraehe.de | |
| 123spider-Bot (Version: 1.02, powered by www.123spider.de | |
| 192.comAgent | |
| 1st ZipCommander (Net) - http://www.zipcommander.com/ | |
| 2Bone_LinkChecker/1.0 libwww-perl/5.64 | |
| 3485 Mozilla/5.0 (compatible; Bender; http://benderthewebrobot.tumblr.com) | |
| 4anything.com LinkChecker v2.0 | |
| 8484 Boston Project v 1.0 | |
| : Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1 | |
| :robot/1.0 (linux) ( admin e-mail: undefined http://www.neofonie.de/loesungen/search/robot.html ) | |
| A-Online Search | |
| A1 Keyword Research/1.0.2 (+http://www.micro-sys.dk/products/keyword-research/) miggibot/2007.03.27 | |
| A1 Sitemap Generator/1.0 (+http://www.micro-sys.dk/products/sitemap-generator/) miggibot/2006.01.24 | |
| aardvark-crawler | |
| AbachoBOT | |
| AbachoBOT (Mozilla compatible) | |
| ABCdatos BotLink/5.xx.xxx#BBL | |
| Aberja Checkomat | |
| abot/0.1 (abot; http://www.abot.com; abot@abot.com) | |
| Aboundex/0.2 (http://www.aboundex.com/crawler/) | |
| About/0.1libwww-perl/5.47 | |
| Abrave Spider v5.3 Robot 5 (http://robot.abrave.com) | |
| Accelatech RSSCrawler/0.4 | |
| accoona | |
| Accoona-AI-Agent/1.1.1 (crawler at accoona dot com) | |
| Accoona-AI-Agent/1.1.2 (aicrawler at accoonabot dot com) | |
| Ace Explorer | |
| Ack (http://www.ackerm.com/) | |
| AcoiRobot | |
| Acoon Robot v1.50.001 | |
| Acoon Robot v1.52 (http://www.acoon.de) | |
| Acoon v4.1.0 (www.acoon.de) | |
| Acoon v4.10.1 (www.acoon.de) | |
| Acoon v4.10.3 (www.acoon.de) | |
| Acoon v4.10.4 (www.acoon.de) | |
| Acoon v4.10.5 (www.acoon.de) | |
| Acoon v4.9.3 (www.acoon.de) | |
| Acoon v4.9.5 (www.acoon.de) | |
| Acoon-Robot 4.0.x.[xx] (http://www.acoon.de) | |
| Acoon-Robot v3.xx (http://www.acoon.de and http://www.acoon.com) | |
| AcoonBot/4.10.5 (+http://www.acoon.de) | |
| Acorn/Nutch-0.9 (Non-Profit Search Engine; acorn.isara.org; acorn at isara dot org) | |
| ActiveBookmark 1.x | |
| Activeworlds | |
| ActiveWorlds/3.xx (xxx) | |
| Ad Muncher v4.xx.x | |
| Ad Muncher v4x Build xxxxx | |
| Adaxas Spider (http://www.adaxas.net/) | |
| AddThis.com robot tech.support@clearspring.com | |
| Advanced Browser (http://www.avantbrowser.com) | |
| AESOP_com_SpiderMan | |
| agadine/1.x.x (+http://www.agada.de) | |
| Agent | |
| Agent-SharewarePlazaFileCheckBot/2.0+(+http://www.SharewarePlaza.com) | |
| AgentName/0.1 libwww-perl/5.48 | |
| Aghaven/Nutch-1.2 (www.aghaven.com) | |
| AIBOT/2.1 By +(www.21seek.com A Real artificial intelligence search engine China) | |
| AideRSS/1.0 (aiderss.com) | |
| aipbot/1.0 (aipbot; http://www.aipbot.com; aipbot@aipbot.com) | |
| aipbot/2-beta (aipbot dev; http://aipbot.com; aipbot@aipbot.com) | |
| Akregator/1.2.9; librss/remnants | |
| Aladin/3.324 | |
| Alcatel-BG3/1.0 UP.Browser/5.0.3.1.2 | |
| Aleksika Spider/1.0 (+http://www.aleksika.com/) | |
| AlertInfo 2.0 (Powered by Newsbrain) | |
| AlkalineBOT/1.3 | |
| AlkalineBOT/1.4 (1.4.0326.0 RTM) | |
| Allesklar/0.1 libwww-perl/5.46 | |
| Alligator 1.31 (www.nearsoftware.com) | |
| Allrati/1.1 (+) | |
| AltaVista Intranet V2.0 AVS EVAL search@freeit.com | |
| AltaVista Intranet V2.0 Compaq Altavista Eval sveand@altavista.net | |
| AltaVista Intranet V2.0 evreka.com crawler@evreka.com | |
| AltaVista V2.0B crawler@evreka.com | |
| amaya/x.xx libwww/x.x.x | |
| AmfibiBOT | |
| Amfibibot/0.06 (Amfibi Web Search; http://www.amfibi.com; agent@amfibi.com) | |
| Amfibibot/0.07 (Amfibi Robot; http://www.amfibi.com; agent@amfibi.com) | |
| amibot | |
| Amiga-AWeb/3.4.167SE | |
| AmigaVoyager/3.4.4 (MorphOS/PPC native) | |
| AmiTCP Miami (AmigaOS 2.04) | |
| Amoi 8512/R21.0 NF-Browser/3.3 | |
| amzn_assoc | |
| animalstxt.org/bot (http://animalstxt.org/) | |
| AnnoMille spider 0.1 alpha - http://www.annomille.it | |
| annotate_google; http://ponderer.org/download/annotate_google.user.js | |
| Anonymized by ProxyOS: http://www.megaproxy.com | |
| Anonymizer/1.1 | |
| AnswerBus (http://www.answerbus.com/) | |
| AnswerChase PROve x.0 | |
| AnswerChase x.0 | |
| ANTFresco/x.xx | |
| antibot-V1.1.5/i586-linux-2.2 | |
| AnzwersCrawl/2.0 (anzwerscrawl@anzwers.com.au;Engine) | |
| Apache-HttpClient/4.1.1 (java 1.5) | |
| Apache-HttpClient/4.1.2 (java 1.5) | |
| Apexoo Spider 1.x | |
| Aplix HTTP/1.0.1 | |
| Aplix_SANYO_browser/1.x (Japanese) | |
| Aplix_SEGASATURN_browser/1.x (Japanese) | |
| Aport | |
| appie 1.1 (www.walhello.com) | |
| Apple iPhone v1.1.4 CoreMedia v1.0.0.4A102 | |
| Apple-PubSub/65.1.1 | |
| Applied-Technologies-Inc-Spider/Nutch-1.4 | |
| ArabyBot (compatible; Mozilla/5.0; GoogleBot; FAST Crawler 6.4; http://www.araby.com;) | |
| ArachBot | |
| Arachnoidea (arachnoidea@euroseek.com) | |
| AramaM/Nutch-2.0-dev | |
| araneo | |
| aranhabot | |
| ArchitextSpider | |
| archive.org_bot | |
| Argus/1.1 (Nutch; http://www.simpy.com/bot.html; feedback at simpy dot com) | |
| Arikus_Spider | |
| Arquivo-web-crawler (compatible; heritrix/1.12.1 +http://arquivo-web.fccn.pt) | |
| Arquivo-web-crawler (compatible; heritrix/1.14.3 +http://arquivo.pt/faq-crawling) | |
| ASAHA Search Engine Turkey V.001 (http://www.asaha.com/) | |
| Asahina-Antenna/1.x | |
| Asahina-Antenna/1.x (libhina.pl/x.x ; libtime.pl/x.x) | |
| ask.24x.info | |
| AskAboutOil/0.06-rcp (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@askaboutoil.com) | |
| asked/Nutch-0.8 (web crawler; http://asked.jp; epicurus at gmail dot com) | |
| ASPSeek/1.2.5 | |
| ASPseek/1.2.9d | |
| ASPSeek/1.2.x | |
| ASPSeek/1.2.xa | |
| ASPseek/1.2.xx | |
| ASPSeek/1.2.xxpre | |
| ASPseek/1.4 | |
| ASSORT/0.10 | |
| asterias/2.0 | |
| AtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html) | |
| Atomic_Email_Hunter/4.0 | |
| Atomz/1.0 | |
| atSpider/1.0 | |
| Attentio/Nutch-0.9-dev (Attentio's beta blog crawler; www.attentio.com; info@attentio.com) | |
| AU-MIC/2.0 MMP/2.0 | |
| AUDIOVOX-SMT5600 | |
| augurfind | |
| augurnfind V-1.x | |
| autoemailspider | |
| autohttp | |
| autowebdir 1.1 (www.autowebdir.com) | |
| AV Fetch 1.0 | |
| Avant Browser (http://www.avantbrowser.com) | |
| AVSearch-1.0(peter.turney@nrc.ca) | |
| AVSearch-2.0-fusionIdx-14-CompetitorWebSites | |
| AVSearch-3.0(AltaVista/AVC) | |
| AWeb | |
| axadine/ (Axadine Crawler; http://www.axada.de/; ) | |
| AxmoRobot - Crawling your site for better indexing on www.axmo.com search engine. | |
| Azureus 2.x.x.x | |
| A_TEST_CRALWER_BY_ISSEL/Nutch-1.4 (http://issel.ee.auth.gr; kvavliak at issel dot ee dot auth dot gr) | |
| BabalooSpider/1.3 (BabalooSpider; http://www.babaloo.si; spider@babaloo.si) | |
| BaboomBot/1.x.x (+http://www.baboom.us) | |
| BacklinkCrawler (http://www.backlinktest.com/crawler.html) | |
| BackStreet Browser 3.x | |
| BaiduImagespider+(+http://www.baidu.jp/search/s308.html) | |
| BaiDuSpider | |
| Baiduspider+(+http://help.baidu.jp/system/05.html) | |
| Baiduspider+(+http://www.baidu.com/search/spider.htm) | |
| Baiduspider+(+http://www.baidu.com/search/spider_jp.html) | |
| Balihoo/Nutch-1.0-dev (Crawler for Balihoo.com search engine - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com) | |
| BanBots/1.2 (spider@banbots.com) | |
| Barca/2.0.xxxx | |
| BarcaPro/1.4.xxxx | |
| BarraHomeCrawler (albertof@barrahome.org) | |
| bCentral Billing Post-Process | |
| bdcindexer_2.6.2 (research@bdc) | |
| BDFetch | |
| BDNcentral Crawler v2.3 [en] (http://www.bdncentral.com/robot.html) (X11; I; Linux 2.0.44 i686) | |
| BeamMachine/0.5 (dead link remover of www.beammachine.net) | |
| beautybot/1.0 (+http://www.uchoose.de/crawler/beautybot/) | |
| BebopBot/2.5.1 ( crawler http://www.apassion4jazz.net/bebopbot.html ) | |
| BeebwareDirectory/v0.01 | |
| berlin-fu-cow (+http://hpsg.fu-berlin.de/cow) | |
| Big Brother (http://pauillac.inria.fr/~fpottier/) | |
| Big Fish v1.0 | |
| BigBrother/1.6e | |
| BigCliqueBOT/1.03-dev (bigclicbot; http://www.bigclique.com; bot@bigclique.com) | |
| BIGLOTRON (Beta 2;GNU/Linux) | |
| Bigsearch.ca/Nutch-x.x-dev (Bigsearch.ca Internet Spider; http://www.bigsearch.ca/; info@enhancededge.com) | |
| Bilbo/2.3b-UNIX | |
| BilgiBetaBot/0.8-dev (bilgi.com (Beta) ; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) | |
| BilgiBot/1.0(beta) (http://www.bilgi.com/; bilgi at bilgi dot com) | |
| billbot wjj@cs.cmu.edu | |
| binlar_2.6.3 (binlar2.6.3@unspecified.mail) | |
| binlar_2.6.3 (larbin2.6.3@unspecified.mail) | |
| binlar_2.6.3 (test@mgmt.mic) | |
| Bitacle bot/1.1 | |
| Bitacle Robot (V:1.0;) (http://www.bitacle.com) | |
| Biyubi/x.x (Sistema Fenix; G11; Familia Toledo; es-mx) | |
| BlackBerry7520/4.0.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/5.0.3.3 UP.Link/5.1.2.12 (Google WAP Proxy/1.0) | |
| BlackBerry9630/5.0.0.732 Profile/MIDP-2.1 Configuration/CLDC-1.1 VendorID/102 ips-agent | |
| BlackWidow | |
| Blaiz-Bee/1.0 (+http://www.blaiz.net) | |
| Blaiz-Bee/2.00.8222 (BE Internet Search Engine http://www.rawgrunt.com) | |
| Blaiz-Bee/2.00.xxxx (+http://www.blaiz.net) | |
| bldbCrawler/Nutch-1.4 | |
| BlevintronBot version 2012-04-02 contact http://en.wikipedia.org/wiki/User_talk:Blevintron | |
| BlevintronBot version 2012-04-03 contact http://en.wikipedia.org/wiki/User_talk:Blevintron | |
| BlevintronBot version 2012-04-04 contact http://en.wikipedia.org/wiki/User_talk:Blevintron | |
| BlevintronBot version 2012-04-09 contact http://en.wikipedia.org/wiki/User_talk:Blevintron | |
| BlevintronBot version 2012-05-01 contact http://en.wikipedia.org/wiki/User_talk:Blevintron | |
| BlitzBOT@tricus.net | |
| BlitzBOT@tricus.net (Mozilla compatible) | |
| BlockNote.Net | |
| BlogBot/1.x | |
| BlogBridge 2.13 (http://www.blogbridge.com/) | |
| Bloglines Title Fetch/1.0 (http://www.bloglines.com) | |
| Bloglines-Images/0.1 (http://www.bloglines.com) | |
| Bloglines/3.1 (http://www.bloglines.com) | |
| BlogMap (http://www.feedmap.net) | |
| Blogpulse (info@blogpulse.com) | |
| BlogPulseLive (support@blogpulse.com) | |
| BlogSearch/1.x +http://www.icerocket.com/ | |
| blogsearchbot-pumpkin-3 | |
| BlogsNowBot, V 2.01 (+http://www.blogsnow.com/) | |
| BlogVibeBot-v1.1 (spider@blogvibe.nl) | |
| blogWatcher_Spider/0.1 (http://www.lr.pi.titech.ac.jp/blogWatcher/) | |
| BlogzIce/1.0 (+http://icerocket.com; rhodes@icerocket.com) | |
| BlogzIce/1.0 +http://www.icerocket.com/ | |
| BloobyBot | |
| Bloodhound/Nutch-0.9 (Testing Crawler for Research - obeys robots.txt and robots meta tags ; http://balihoo.com/index.aspx; robot at balihoo dot com) | |
| BLT/1.0.4 (http://www.bltformac.com/) | |
| bluefish 0.6 HTML editor | |
| BMCLIENT | |
| BMLAUNCHER | |
| Bobby/4.0.x RPT-HTTPClient/0.3-3E | |
| boitho.com-dc/0.xx (http://www.boitho.com/dcbot.html) | |
| boitho.com-robot/1.x | |
| boitho.com-robot/1.x (http://www.boitho.com/bot.html) | |
| Bookdog/x.x | |
| Bookmark Buddy bookmark checker (http://www.bookmarkbuddy.net/) | |
| Bookmark Renewal Check Agent [http://www.bookmark.ne.jp/] | |
| Bookmark Renewal Check Agent [http://www.bookmark.ne.jp/] (Version 2.0beta) | |
| BookmarkBase(2/;http://bookmarkbase.com) | |
| Bot mailto:craftbot@yahoo.com | |
| BotLaju v2.0 (http://laju.my/) | |
| BPImageWalker/2.0 (www.bdbrandprotect.com) | |
| BravoBrian bstop.bravobrian.it | |
| BravoBrian SpiderEngine MarcoPolo | |
| BrightCrawler (http://www.brightcloud.com/brightcrawler.asp) | |
| BrokenLinkBot version 2012-03-31 contact http://en.wikipedia.org/wiki/User_talk:Blevintron | |
| BrokenLinkBot version 20120319.102558 contact http://en.wikipedia.org/wiki/User_talk:Blevintron | |
| BrokenLinkBot version 20120320.224751 contact http://en.wikipedia.org/wiki/User_talk:Blevintron | |
| BruinBot (+http://webarchive.cs.ucla.edu/bruinbot.html) | |
| BSDSeek/1.0 | |
| BStop.BravoBrian.it Agent Detector | |
| BTbot/0.x (+http://www.btbot.com/btbot.html) | |
| Btsearch/1.1 (Btsearch; http://www.test.com/test.html; test@test.com) | |
| BTWebClient/180B(9704) | |
| BuildCMS crawler (http://www.buildcms.com/crawler) | |
| Bulkfeeds/r1752 (http://bulkfeeds.net/) | |
| BullsEye | |
| bumblebee@relevare.com | |
| BunnySlippers | |
| BurstFindCrawler/1.1 (crawler.burstfind.com; http://crawler.burstfind.com; crawler@burstfind.com) | |
| buscador-age (Enterprise; T2-C9QZ24BXJWWK2; gep@ti.eptisa.com) | |
| Buscaplus Robi/1.0 (http://www.buscaplus.com/robi/) | |
| businessdbbot/v1.2 (http://www.businessdb.com/bot.php) | |
| BW-C-2.0 | |
| bwh3_user_agent | |
| Cabot/Nutch-0.9 (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; agent@amfibi.com) | |
| Cabot/Nutch-1.0-dev (Amfibi's web-crawling robot; http://www.amfibi.com/cabot/; agent@amfibi.com) | |
| CamelHttpStream/1.0 | |
| Cancer Information and Support International; | |
| carleson/1.0 | |
| Carnegie_Mellon_University_Research_WebBOT-->PLEASE READ-->http://www.andrew.cmu.edu/~brgordon/webbot/index.html http://www.andrew.cmu.edu/~brgordon/webbot/index.html | |
| Carnegie_Mellon_University_WebCrawler http://www.andrew.cmu.edu/~brgordon/webbot/index.html | |
| Catall Spider | |
| CatchBot/1.0; +http://www.catchbot.com | |
| CazoodleBot/CazoodleBot-0.1 (CazoodleBot Crawler; http://www.cazoodle.com/cazoodlebot; cazoodlebot@cazoodle.com) | |
| CCBot/1.0 (+http://www.commoncrawl.org/bot.html) | |
| ccubee/x.x | |
| CDR/1.7.1 Simulator/0.7(+http://timewe.net) Profile/MIDP-1.0 Configuration/CLDC-1.0 | |
| CE-Preload | |
| CentiverseBot | |
| CentiverseBot - investigator | |
| CentiverseBot/3.0 (http://www.centiverse-project.net) | |
| Ceramic Tile Installation Guide (http://www.floorstransformed.com) | |
| CERN-LineMode/2.15 | |
| cfetch/1.0 | |
| CFNetwork/x.x | |
| cg-eye interactive | |
| Charon/1.x (Amiga) | |
| Chat Catcher/1.0 | |
| Checkbot/1.xx LWP/5.xx | |
| CheckLinks/1.x.x | |
| CheckUrl | |
| CheckWeb | |
| Chilkat/1.0.0 (+http://www.chilkatsoft.com/ChilkatHttpUA.asp) | |
| China Local Browse 2.6 | |
| Chitika ContentHit 1.0 | |
| ChristCRAWLER 2.0 | |
| CHttpClient by Open Text Corporation | |
| cinvestav/Nutch-1.4 | |
| CipinetBot (http://www.cipinet.com/bot.html) | |
| Cityreview Robot (+http://www.cityreview.org/crawler/) | |
| CJ Spider/ | |
| CJB.NET Proxy | |
| CjxSearch/Nutch-1.4 | |
| ClariaBot/1.0 | |
| Claymont.com | |
| CloakDetect/0.9 (+http://fulltext.seznam.cz/) | |
| cloud-valley/bigdata-1.0 (http://www.cloud-valley.com/; news@mobier.mobi) | |
| CloudACL/Nutch-1.4 | |
| Clushbot/2.x (+http://www.clush.com/bot.html) | |
| Clushbot/3.x-BinaryFury (+http://www.clush.com/bot.html) | |
| Clushbot/3.xx-Ajax (+http://www.clush.com/bot.html) | |
| Clushbot/3.xx-Hector (+http://www.clush.com/bot.html) | |
| Clushbot/3.xx-Peleus (+http://www.clush.com/bot.html) | |
| COAST WebMaster Pro/4.x.x.xx (Windows NT) | |
| CoBITSProbe | |
| coccoc | |
| Cocoal.icio.us/1.0 (v36) (Mac OS X; http://www.scifihifi.com/cocoalicious) | |
| Code Search Crawler/Nutch-1.2 (Code Search Crawler; www.iai.uni-bonn.de) | |
| Cogentbot/1.X (+http://www.cogentsoftwaresolutions.com/bot.html) | |
| ColdFusion | |
| ColdFusion (BookmarkTracker.com) | |
| collage.cgi/1.xx | |
| combine/0.0 | |
| Combine/2.0 http://combine.it.lth.se/ | |
| Combine/3 http://combine.it.lth.se/ | |
| Combine/x.0 | |
| cometrics-bot, http://www.cometrics.de | |
| Commerce Browser Center | |
| Comodo-Certificates-Spider | |
| COMODOSpider/Nutch-1.2 | |
| complex_network_group/Nutch-0.9-dev (discovering the structure of the world-wide-web; http://cantor.ee.ucla.edu/~networks/crawl; nimakhaj@gmail.com) | |
| Computer_and_Automation_Research_Institute_Crawler crawler@ilab.sztaki.hu | |
| Comrite/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) | |
| Contact | |
| ContactBot/0.2 | |
| Content Crawler | |
| ContentSmartz | |
| contype | |
| Convera Internet Spider V6.x | |
| ConveraCrawler/0.2 | |
| ConveraCrawler/0.9d (+http://www.authoritativeweb.com/crawl) | |
| ConveraMultiMediaCrawler/0.1 (+http://www.authoritativeweb.com/crawl) | |
| CoolBot | |
| Cooliris/1.5 CFNetwork/459 Darwin/10.0.0d3 | |
| CoralWebPrx/0.1.1x (See http://coralcdn.org/) | |
| coruscan/Nutch-1.4 | |
| cosmos/0.8_(robot@xyleme.com) | |
| cosmos/0.9_(robot@xyleme.com) | |
| CoteoNutchCrawler/Nutch-0.9 (info [at] coteo [dot] com) | |
| CougarSearch/0.x (+http://www.cougarsearch.com/faq.shtml) | |
| Covac TexAs Arachbot | |
| CoverScout%203/3.0.1 CFNetwork/339.5 Darwin/9.5.0 (i386) (iMac5,1) | |
| Cowbot-0.1 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) | |
| Cowbot-0.1.x (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) | |
| CrawlConvera0.1 (CrawlConvera@yahoo.com) | |
| Crawler | |
| Crawler (cometsearch@cometsystems.com) | |
| Crawler admin@crawler.de | |
| Crawler V 0.2.x admin@crawler.de | |
| crawler4j (http://code.google.com/p/crawler4j/) | |
| crawler@alexa.com | |
| CrawlerBoy Pinpoint.com | |
| Crawllybot/0.1 (Crawllybot; +http://www.crawlly.com; crawler@crawlly.com) | |
| CreativeCommons/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net) | |
| Cricket-A100/1.0 UP.Browser/6.3.0.7 (GUI) MMP/2.0 | |
| CrocCrawler vx.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686) | |
| CRW1/Nutch-1.2 | |
| csci_b659/0.13 | |
| CSE HTML Validator Professional (http://www.htmlvalidator.com/) | |
| Cuam Ver0.050bx | |
| Cuasarbot/0.9b http://www.cuasar.com/spider_beta/ | |
| curl/7.10.x (i386-redhat-linux-gnu) libcurl/7.10.x OpenSSL/0.9.7a ipv6 zlib/1.1.4 | |
| curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5 | |
| curl/7.7.x (i386--freebsd4.3) libcurl 7.7.x (SSL 0.9.6) (ipv6 enabled) | |
| curl/7.8 (i686-pc-linux-gnu) libcurl 7.8 (OpenSSL 0.9.6) | |
| curl/7.9.x (win32) libcurl 7.9.x | |
| CurryGuide SiteScan 1.1 | |
| Custo x.x (www.netwu.com) | |
| Custom Spider www.bisnisseek.com /1.0 | |
| Cyberdog/2.0 (Macintosh; 68k) | |
| CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp) | |
| CyberSpyder Link Test/2.1.12 (admin@mspennyworth.com) | |
| CyberSpyder Link Test/2.2.1 | |
| CydralSpider/1.x (Cydral Web Image Search; http://www.cydral.com) | |
| CydralSpider/3.0 (Cydral Image Search; http://www.cydral.com) | |
| CydralSpider/3.2.6 (Cydral Image Search; http://www.cydral.com) | |
| DA 3.5 (www.lidan.com) | |
| DA 4.0 | |
| DA 4.0 (www.downloadaccelerator.com) | |
| DA 5.0 | |
| DA 7.0 | |
| DAP x.x | |
| Dart Communications PowerTCP | |
| DataCha0s/2.0 | |
| DataFountains/DMOZ Downloader | |
| DataFountains/Dmoz Downloader (http://ivia.ucr.edu/useragents.shtml) | |
| DataFountains/DMOZ Feature Vector Corpus Creator (http://ivia.ucr.edu/useragents.shtml) | |
| DataparkSearch/4.47 (+http://dataparksearch.org/bot) | |
| DataparkSearch/4.xx (http://www.dataparksearch.org/) | |
| DataSpear/1.0 (Spider; http://www.dataspear.com/spider.html; spider@dataspear.com) | |
| DataSpearSpiderBot/0.2 (DataSpear Spider Bot; http://dssb.dataspear.com/bot.html; dssb@dataspear.com) | |
| DatenBot( http://www.sicher-durchs-netz.de/bot.html) | |
| DaviesBot/1.7 (www.wholeweb.net) | |
| daypopbot/0.x | |
| dbDig(http://www.prairielandconsulting.com) | |
| DBrowse 1.4b | |
| DBrowse 1.4d | |
| DC-Sakura/x.xx | |
| dCSbot/1.1 | |
| DDD | |
| dds explorer v1.0 beta | |
| de.searchengine.comBot 1.2 (http://de.searchengine.com/spider) | |
| DeadLinkCheck/0.4.0 libwww-perl/5.xx | |
| Deep Link Calculator v1.0 | |
| deepak-USC/ISI | |
| DeepIndex | |
| DeepIndex ( http://www.zetbot.com ) | |
| DeepIndex (www.en.deepindex.com) | |
| DeepIndexer.ca | |
| del.icio.us-thumbnails/1.0 Mozilla/5.0 (compatible; Konqueror/3.4; FreeBSD) KHTML/3.4.2 (like Gecko) | |
| DeleGate/9.0.5-fix1 | |
| Demo Bot DOT 16b | |
| Demo Bot Z 16b | |
| Denmex websearch (http://search.denmex.com) | |
| Der große BilderSauger 2.00u | |
| dev-spider2.searchpsider.com/1.3b | |
| DevComponents.com HtmlDocument Object | |
| DiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html) | |
| Diamond/x.0 | |
| DiamondBot | |
| Digger/1.0 JDK/1.3.0rc3 | |
| DigOut4U | |
| DIIbot/1.2 | |
| Dillo/0.8.5-i18n-misc | |
| Dillo/0.x.x | |
| disastrous/1.0.5 (running with Python 2.5.1; http://www.bortzmeyer.org/disastrous.html; archangel77@del.icio.us) | |
| DISCo Pump x.x | |
| disco/Nutch-0.9 (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com) | |
| disco/Nutch-1.0-dev (experimental crawler; www.discoveryengine.com; disco-crawl@discoveryengine.com) | |
| DittoSpyder | |
| dlab spider/Nutch-1.3 | |
| dlman | |
| dloader(NaverRobot)/1.0 | |
| DNSRight.com WebBot Link Ckeck Tool. Report abuse to: dnsr@dnsright.com | |
| DoCoMo/1.0/Nxxxi/c10 | |
| DoCoMo/1.0/Nxxxi/c10/TB | |
| DoCoMo/1.0/P502i/c10 (Google CHTML Proxy/1.0) | |
| DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)''' | |
| DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/) | |
| DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/) | |
| DoCoMo/2.0 P900iV(c100;TB;W24H11) | |
| DoCoMo/2.0 SH901iS(c100;TB;W24H12),gzip(gfe) (via translate.google.com) | |
| DoCoMo/2.0 SH902i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html) | |
| DoCoMo/2.0/SO502i (compatible; Y!J-SRD/1.0; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html) | |
| DocZilla/1.0 (Windows; U; WinNT4.0; en-US; rv:1.0.0) Gecko/20020804 | |
| dodgebot/experimental | |
| DogSniffer | |
| Domnutch-Bot/Nutch-1.0 (Domnutch; http://www.Nutch.de/) | |
| DonutP; Windows98SE | |
| Doubanbot/1.0 (bot@douban.com http://www.douban.com) | |
| Download Demon/3.x.x.x | |
| Download Druid 2.x | |
| Download Express 1.0 | |
| Download Master | |
| Download Ninja 3.0 | |
| Download Wonder | |
| Download-Tipp Linkcheck (http://download-tipp.de/) | |
| Download.exe(1.1) (+http://www.sql-und-xml.de/freeware-tools/) | |
| DownloadDirect.1.0 | |
| Dr.Web (R) online scanner: http://online.drweb.com/ | |
| Dragonfly File Reader | |
| Drecombot/1.0 (http://career.drecom.jp/bot.html) | |
| DrRajendra/Nutch-0.9 (Senior Researcher; http://cse.iitkgp.ac.in/~rpr; rajendra@cse.iitkgp.ernet.in) | |
| Drupal (+http://drupal.org/) | |
| DSurf15a 01 | |
| DSurf15a 71 | |
| DSurf15a 81 | |
| DSurf15a VA | |
| DTAAgent | |
| dtSearchSpider | |
| Dual Proxy | |
| DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html) | |
| Dumbot(version 0.1 beta - dumbfind.com) | |
| Dumbot(version 0.1 beta - http://www.dumbfind.com/dumbot.html) | |
| Dumbot(version 0.1 beta) | |
| e-sense 1.0 ea(www.vigiltech.com/esensedisclaim.html) | |
| e-SocietyRobot(http://www.yama.info.waseda.ac.jp/~yamana/es/) | |
| eApolloBot/2.0 (compatible; heritrix/2.0.0-SNAPSHOT-20071024.170148 +http://www.eapollo-opto.com) | |
| EARTHCOM.info/1.x [www.earthcom.info] | |
| EARTHCOM.info/1.xbeta [www.earthcom.info] | |
| EasyDL/3.xx | |
| EasyDL/3.xx http://keywen.com/Encyclopedia/Bot | |
| EBrowse 1.4b | |
| EC2LinkFinder | |
| eCatch/3.0 | |
| EchO!/2.0 | |
| EdisterBot (http://www.edister.com/bot.html) | |
| edu.pjwstk.focusedcrawler.training-data-gatherer | |
| Educate Search VxB | |
| egothor/3.0a (+http://www.xdefine.org/robot.html) | |
| EgotoBot/4.8 (+http://www.egoto.com/about.htm) | |
| ejupiter.com | |
| EldoS TimelyWeb/3.x | |
| Element/Nutch-2.0-dev | |
| elfbot/1.0 (+http://www.uchoose.de/crawler/elfbot/) | |
| ELI/20070402:2.0 (DAUM RSS Robot, Daum Communications Corp.; +http://ws.daum.net/aboutkr.html) | |
| ELinks (0.x.x; Linux 2.4.20 i586; 132x60) | |
| ELinks/0.x.x (textmode; NetBSD 1.6.2 sparc; 132x43) | |
| EmailSiphon | |
| EmailSpider | |
| EmailWolf 1.00 | |
| emefgebot/beta (+http://emefge.de/bot.html) | |
| EmeraldShield.com WebBot | |
| EmeraldShield.com WebBot (http://www.emeraldshield.com/webbot.aspx) | |
| EMPAS_ROBOT | |
| EnaBot/1.x (http://www.enaball.com/crawler.html) | |
| endo/1.0 (Mac OS X; ppc i386; http://kula.jp/endo) | |
| Enfish Tracker | |
| Enterprise_Search/1.0 | |
| Enterprise_Search/1.0.xxx | |
| Enterprise_Search/1.00.xxx;MSSQL (http://www.innerprise.net/es-spider.asp) | |
| envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.html) | |
| envolk/1.7 (+http://www.envolk.com/envolkspiderinfo.php) | |
| envolk[ITS]spider/1.6(+http://www.envolk.com/envolkspider.html) | |
| EPiServer Link Checker | |
| EroCrawler | |
| ES.NET_Crawler/2.0 (http://search.innerprise.net/) | |
| eseek-larbin_2.6.2 (crawler@exactseek.com) | |
| ESISmartSpider | |
| eStyleSearch 4 (compatible; MSIE 6.0; Windows NT 5.0) | |
| ESurf15a 15 | |
| EuripBot/0.x (+http://www.eurip.com) GetFile | |
| EuripBot/0.x (+http://www.eurip.com) GetRobots | |
| EuripBot/0.x (+http://www.eurip.com) PreCheck | |
| Eurobot/1.0 (http://www.ayell.eu) | |
| EvaalSE - bot@evaal.com | |
| eventax/1.0 (http://www.eventax.de) | |
| eventax/1.3 (eventax; http://www.eventax.de/; info@eventax.de) | |
| EventMachine HttpClient | |
| Everest-Vulcan Inc./0.1 (R&D project; host=e-1-24; http://everest.vulcan.com/crawlerhelp) | |
| Everest-Vulcan Inc./0.1 (R&D project; http://everest.vulcan.com/crawlerhelp) | |
| Exabot-Images/1.0 | |
| Exabot-Test/1.0 | |
| Exabot/2.0 | |
| Exabot/3.0 | |
| ExactSearch | |
| ExactSeek Crawler (http://www.exactseek.com/)/Nutch-1.4 | |
| ExactSeek Crawler (nutch 1.4)/Nutch-1.4 | |
| ExactSeek Crawler/0.1 | |
| exactseek-crawler-2.63 (crawler@exactseek.com) | |
| exactseek-pagereaper-2.63 (crawler@exactseek.com) | |
| exactseek.com | |
| Exalead NG/MimeLive Client (convert/http/0.120) | |
| Excalibur Internet Spider V6.5.4 | |
| Execrawl/1.0 (Execrawl; http://www.execrawl.com/; bot@execrawl.com) | |
| exooba crawler/exooba crawler (crawler for exooba.com; http://www.exooba.com/; info at exooba dot com) | |
| exooba/exooba crawler (exooba; exooba) | |
| ExperimentalHenrytheMiragoRobot | |
| Expired Domain Sleuth | |
| Express WebPictures (www.express-soft.com) | |
| ExtractorPro | |
| Extreme Picture Finder | |
| EyeCatcher (Download-tipp.de)/1.0 | |
| Factbot 1.09 (see http://www.factbites.com/webmasters.php) | |
| factbot : http://www.factbites.com/robots | |
| FaEdit/2.0.x | |
| FairAd Client | |
| FANGCrawl/0.01 | |
| FARK.com link verifier | |
| Fast Crawler Gold Edition | |
| FAST Enterprise Crawler 6 (Experimental) | |
| FAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com/srsapp/contactus/ | |
| FAST Enterprise Crawler 6 used by Cobra Development (admin@fastsearch.com) | |
| FAST Enterprise Crawler 6 used by Comperio AS (sts@comperio.no) | |
| FAST Enterprise Crawler 6 used by FAST (FAST) | |
| FAST Enterprise Crawler 6 used by Pages Jaunes (pvincent@pagesjaunes.fr) | |
| FAST Enterprise Crawler 6 used by Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au) | |
| FAST Enterprise Crawler 6 used by Singapore Press Holdings (crawler@sphsearch.sg) | |
| FAST Enterprise Crawler 6 used by WWU (wardi@uni-muenster.de) | |
| FAST Enterprise Crawler/6 (www.fastsearch.com) | |
| FAST Enterprise Crawler/6.4 (helpdesk at fast.no) | |
| FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0) | |
| FAST MetaWeb Crawler (helpdesk at fastsearch dot com) | |
| Fast PartnerSite Crawler | |
| FAST-WebCrawler/2.2.10 (Multimedia Search) (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) | |
| FAST-WebCrawler/2.2.6 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html) | |
| FAST-WebCrawler/2.2.7 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)http://www.fast.no | |
| FAST-WebCrawler/2.2.8 (crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html)http://www.fast.no | |
| FAST-WebCrawler/3.2 test | |
| FAST-WebCrawler/3.3 (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler) | |
| FAST-WebCrawler/3.4/Nirvana (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler) | |
| FAST-WebCrawler/3.4/PartnerSite (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler) | |
| FAST-WebCrawler/3.5 (atw-crawler at fast dot no; http://fast.no/support.php?c=faqs/crawler) | |
| FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) | |
| FAST-WebCrawler/3.6/FirstPage (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler) | |
| FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) | |
| FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp) | |
| FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) | |
| FAST-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) | |
| FAST-WebCrawler/3.x Multimedia | |
| FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no) | |
| fastbot crawler beta 2.0 (+http://www.fastbot.de) | |
| FastBug http://www.ay-up.com | |
| FastCrawler 3.0.1 (crawler@1klik.dk) | |
| FastSearch Web Crawler for Verizon SuperPages (kevin.watters@fastsearch.com) | |
| Favcollector/2.0 (info@favcollector.com http://www.favcollector.com/) | |
| FavIconizer | |
| favo.eu crawler/0.6 (http://www.favo.eu) | |
| FavOrg | |
| Favorites Checking (http://campulka.net) | |
| Favorites Sweeper v.2.03 | |
| Faxobot/1.0 | |
| FDM 1.x | |
| FDM 2.x | |
| Feed Seeker Bot (RSS Feed Seeker http://www.MyNewFavoriteThing.com/fsb.php) | |
| Feed24.com | |
| Feed::Find/0.0x | |
| Feedable/0.1 (compatible; MSIE 6.0; Windows NT 5.1) | |
| FeedChecker/0.01 | |
| FeedDemon/2.7 (http://www.newsgator.com/; Microsoft Windows XP) | |
| Feedfetcher-Google-iGoogleGadgets; (+http://www.google.com/feedfetcher.html) | |
| Feedfetcher-Google; (+http://www.google.com/feedfetcher.html) | |
| FeedForAll rss2html.php v2 | |
| FeedHub FeedDiscovery/1.0 (http://www.feedhub.com) | |
| FeedHub MetaDataFetcher/1.0 (http://www.feedhub.com) | |
| Feedjit Favicon Crawler 1.0 | |
| Feedreader 3.xx (Powered by Newsbrain) | |
| Feedshow/x.0 (http://www.feedshow.com; 1 subscriber) | |
| FeedshowOnline (http://www.feedshow.com) | |
| Feedster Crawler/3.0; Feedster, Inc. | |
| FeedZcollector v1.x (Platinum) http://www.feeds4all.com/feedzcollector | |
| Felix - Mixcat Crawler (+http://mixcat.com) | |
| fetch libfetch/2.0 | |
| fetch/Nutch-1.2 (TCGfetch; http://fetch.thecyberguardian.com; TCGEmail) | |
| FFC Trap Door Spider | |
| Filangy/0.01-beta (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com) | |
| Filangy/1.0x (Filangy; http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp; filangy-agent@filangy.com) | |
| Filangy/1.0x (Filangy; http://www.nutch.org/docs/en/bot.html; filangy-agent@filangy.com) | |
| fileboost.net/1.0 (+http://www.fileboost.net) | |
| FileHound x.x | |
| Filtrbox/1.0 | |
| FindAnISP.com_ISP_Finder_v99a | |
| Findexa Crawler (http://www.findexa.no/gulesider/article26548.ece) | |
| findfiles.net/0.98 (Robot;test_robot@gmx-topmail.de) | |
| findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/2.0.4 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/2.0.5 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/2.1.5 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/2.2 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/2.5 (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| findlinks/x.xxx (+http://wortschatz.uni-leipzig.de/findlinks/) | |
| FineBot | |
| Finjan-prefetch | |
| Firefly/1.0 | |
| Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5) | |
| Firefox (kastaneta03@hotmail.com) | |
| Firefox_1.0.6 (kasparek@naparek.cz) | |
| FirstGov.gov Search - POC:firstgov.webmasters@gsa.gov | |
| firstsbot | |
| Flapbot/0.7.2 (Flaptor Crawler; http://www.flaptor.com; crawler at flaptor period com) | |
| FlashGet | |
| FLATARTS_FAVICO | |
| Flexum spider | |
| Flexum/2.0 | |
| FlickBot 2.0 RPT-HTTPClient/0.3-3 | |
| flunky | |
| fly/6.01 libwww/4.0D | |
| flyindex.net 1.0/http://www.flyindex.net | |
| FnooleBot/2.5.2 (+http://www.fnoole.com/addurl.html) | |
| FocusedSampler/1.0 | |
| Folkd.com Spider/0.1 beta 1 (www.folkd.com) | |
| FollowSite Bot ( http://www.followsite.com/bot.html ) | |
| FollowSite.com ( http://www.followsite.com/b.html ) | |
| Fooky.com/ScorpionBot/ScoutOut; http://www.fooky.com/scorpionbots | |
| Francis/1.0 (francis@neomo.de http://www.neomo.de/) | |
| Franklin Locator 1.8 | |
| free-downloads.net download-link validator /0.1 | |
| FreeFind.com-SiteSearchEngine/1.0 (http://freefind.com; spiderinfo@freefind.com) | |
| Frelicbot/1.0 +http://www.frelic.com/ | |
| FreshDownload/x.xx | |
| FreshNotes crawler< report problems to crawler-at-freshnotes-dot-com | |
| FSurf15a 01 | |
| FTB-Bot http://www.findthebest.co.uk/ | |
| Full Web Bot 0416B | |
| Full Web Bot 0516B | |
| Full Web Bot 2816B | |
| Fumae2Oo/Nutch-1.4 | |
| FuseBulb.Com | |
| FyberSpider (+http://www.fybersearch.com/fyberspider.php) | |
| FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php) | |
| Gagglebot | |
| GAIS Robot/1.0B2 | |
| Gaisbot/3.0 (indexer@gais.cs.ccu.edu.tw; http://gais.cs.ccu.edu.tw/robot.php) | |
| Gaisbot/3.0+(robot06@gais.cs.ccu.edu.tw;+http://gais.cs.ccu.edu.tw/robot.php) | |
| GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html) | |
| Gallent Search Spider v1.4 Robot 2 (http://robot.GallentSearch.com) | |
| gamekitbot/1.0 (+http://www.uchoose.de/crawler/gamekitbot/) | |
| Gamespy_Arcade | |
| GammaSpider/1.0 | |
| gazz/x.x (gazz@nttrd.com) | |
| geckobot | |
| Generic Mobile Phone (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) | |
| generic_crawler/01.0217/ | |
| GenesisBrowser (HTTP 1.1; 0.9; XP SP2; .NET CLR 2.0.50727) | |
| genieBot (http://64.5.245.11/faq/faq.html) | |
| geniebot wgao@genieknows.com | |
| GeoBot/1.0 | |
| geobucket/Nutch-1.4 | |
| GeonaBot 1.x; http://www.geona.com/ | |
| geourl/2.0b2 | |
| GeoURLBot 1.0 (http://geourl.org) | |
| GetBot | |
| GetRight/3.x.x | |
| GetRight/4.5xx | |
| GetRight/4.x | |
| GetRight/4.x[a-e] | |
| GetRight/6.1 (Pro) | |
| GetRightPro/6.0beta2 | |
| GetWeb/0.1 libwww-perl/5.16 | |
| GhostRouteHunter/20021130 (https://www.sixxs.net/tools/grh/; info@sixxs.net) | |
| gigabaz/3.1x (baz@gigabaz.com; http://gigabaz.com/gigabaz/) | |
| Gigabot/2.0 (gigablast.com) | |
| Gigabot/2.0/gigablast.com/spider.html | |
| Gigabot/2.0; http://www.gigablast.com/spider.html | |
| Gigabot/2.0att | |
| Gigabot/3.0 (http://www.gigablast.com/spider.html) | |
| Gigabot/x.0 | |
| GigabotSiteSearch/2.0 (sitesearch.gigablast.com) | |
| GNODSPIDER (www.gnod.net) | |
| Go!Zilla 3.x (www.gozilla.com) | |
| Go!Zilla/4.x.x.xx | |
| Go-Ahead-Got-It/1.1 | |
| Goblin/0.9 (http://www.goguides.org/) | |
| Goblin/0.9.x (http://www.goguides.org/goblin-info.html) | |
| GoForIt.com | |
| GOFORITBOT ( http://www.goforit.com/about/ ) | |
| GoGuides.Org Link Check | |
| GoldenFeed Spider 1.0 (http://www.goldenfeed.com) | |
| Goldfire Server | |
| gonzo1[P] +http://www.suchen.de/popups/faq.jsp | |
| gonzo2[P] +http://www.suchen.de/faq.html | |
| Goofer/0.2 | |
| Google Talk | |
| googlebot (larbin2.6.0@unspecified.mail) | |
| Googlebot-Image/1.0 | |
| Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html) | |
| Googlebot/2.1 ( http://www.google.com/bot.html) | |
| Googlebot/2.1 ( http://www.googlebot.com/bot.html) | |
| Googlebot/2.1 (+http://www.googlebot.com/bot.html) | |
| Googlebot/2.1+(+http://www.google.com/bot.html) | |
| Googlebot/Googlebot-1.0 | |
| Googlebot/Nutch-1.4 | |
| Googlebot/Test ( http://www.googlebot.com/bot.html) | |
| Gordon's Spider/Nutch-0.9 (http://www.sharethis.com; gordon@sharethis.com) | |
| GrapeFX/0.3 libwww/5.4.0 | |
| great-plains-web-spider/flatlandbot (Flatland Industries Web Spider; http://www.flatlandindustries.com/flatlandbot.php; jason@flatlandindustries.com) | |
| GreatNews/1.0 | |
| GreenBrowser | |
| gridwell (http://search.gridwell.com) | |
| GrigorBot 0.8 (http://www.grigor.biz/bot.html) | |
| Gromit/1.0 | |
| grub crawler(http://www.grub.org) | |
| grub-client | |
| gsa-crawler (Enterprise; GID-01422; jplastiras@google.com) | |
| gsa-crawler (Enterprise; GID-01742;gsatesting@rediffmail.com) | |
| gsa-crawler (Enterprise; GIX-02057; dm@enhesa.com) | |
| gsa-crawler (Enterprise; GIX-03519; cknuetter@stubhub.com) | |
| gsa-crawler (Enterprise; GIX-0xxxx; enterprise-training@google.com) | |
| gsa-crawler (Enterprise; M2-B3X6EDEBUC6JB; serveradm@dickinson.edu) | |
| gsa-crawler (Enterprise; M2-B49PXEELPJSAA; webmaster@bethel.edu) | |
| gsa-crawler (Enterprise; M2-JGHL8PEKVA2JS; webmaster@bethel.edu) | |
| gsa-crawler (Enterprise; S5-RLAPLGRTU6NAT; networkadmin@figleaf.com) | |
| gsa-crawler (Enterprise; T3-GNGZGLT73YWB2; search@byu.edu) | |
| gsa-crawler (Enterprise; T3-L39FLJP3RYWGY; dlfr-france.fr-system@atos.net) | |
| gsa-crawler (Enterprise; T3-LLNR7LKTRYSQY; dlfr-france.fr-system@atos.net) | |
| gsa-crawler (Enterprise; T3-MRHJJGX73YWBJ; estauthamer@emidconsult.com) | |
| GSiteCrawler/v1.xx rev. xxx (http://gsitecrawler.com/) | |
| GSLFbot | |
| Guestbook Auto Submitter | |
| Gulliver/1.3 | |
| Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot) | |
| Gungho/0.08004 (http://code.google.com/p/gungho-crawler/wiki/Index) | |
| GurujiBot/1.0 (+http://www.guruji.com/WebmasterFAQ.html) | |
| GurujiImageBot/1.0 (+http://www.guruji.com/en/WebmasterFAQ.html) | |
| gva/europe (Geneva Web Explorer.; http://www.genevasearch.com/; info@genevasearch.com) | |
| HablaCentro/1.0 (HablaCentro Crawler; http://hablacentro.com/; webmaster@hablacentro.com) | |
| Haier-T10C/1.0 iPanel/2.0 WAP2.0 (compatible; UP.Browser/6.2.2.4; UPG1; UP/4.0; Embedded) | |
| HappyFunBot/1.1 | |
| Harvest-NG/1.0.2 | |
| Haste/0.12 (HOME: http://haste.kytoon.com/) | |
| Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot) | |
| Hatena Antenna/0.5 (http://a.hatena.ne.jp/help) | |
| Hatena Mobile Gateway/1.0 | |
| Hatena Pagetitle Agent/1.0 | |
| Hatena RSS/0.3 (http://r.hatena.ne.jp) | |
| HatenaScreenshot/1.0 (checker) | |
| HAZY.SPIDER/Nutch-1.4 | |
| hbtronix.spider.2 -- http://hbtronix.de/spider.php | |
| HeinrichderMiragoRobot | |
| HeinrichderMiragoRobot (http://www.miragorobot.com/scripts/deinfo.asp) | |
| Helix/1.x ( http://www.sitesearch.ca/helix/) | |
| HenriLeRobotMirago (http://www.miragorobot.com/scripts/frinfo.asp) | |
| HenrytheMiragoRobot | |
| HenryTheMiragoRobot (http://www.miragorobot.com/scripts/mrinfo.asp) | |
| hgrepurl/1.0 | |
| Hi! I'm CsCrawler my homepage: http://www.kde.cs.uni-kassel.de/lehre/ss2005/googlespam/crawler.html RPT-HTTPClient/0.3-3 | |
| HiDownload | |
| Hippias/0.9 Beta | |
| HitList | |
| Hitwise Spider v1.0 http://www.hitwise.com | |
| HLoader | |
| holmes/3.11 (http://morfeo.centrum.cz/bot) | |
| holmes/3.9 (onet.pl) | |
| holmes/3.xx (OnetSzukaj/5.0; +http://szukaj.onet.pl) | |
| holmes/x.x | |
| HolmesBot (http://holmes.ge) | |
| HomePageSearch(hpsearch.uni-trier.de) | |
| Homerbot: www.homerweb.com | |
| Honda-Search/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; search@honda-search.com) | |
| HooWWWer/2.1.3 (debugging run) (+http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-info<at>hiit.fi) | |
| HooWWWer/2.1.x ( http://cosco.hiit.fi/search/hoowwwer/ | mailto:crawler-info<at>hiit.fi) | |
| HopperBot/1.1 (compatible; heritrix/3.1.0 +http://www.hopper.com) | |
| HotJava/1.0.1/JRE1.1.x | |
| Hotzonu/x.0 | |
| HPL/Nutch-0.9 - | |
| htdig/3.1.6 (http://computerorgs.com) | |
| htdig/3.1.6 (unconfigured@htdig.searchengine.maintainer) | |
| htdig/3.1.x (root@localhost) | |
| Html Link Validator (www.lithopssoft.com) | |
| HTML2JPG Blackbox, http://www.html2jpg.com | |
| HTML2JPG Enterprise | |
| HTMLParser/1.x | |
| HTTP Retriever | |
| http://Anonymouse.org/ (Unix) | |
| http://Ask.24x.Info/ (http://narres.it/) | |
| http://hilfe.acont.de/bot.html ACONTBOT | |
| http://OzySoftware.com/Index.html | |
| http://SiteIntel.net Bot | |
| http://www.almaden.ibm.com/cs/crawler | |
| http://www.almaden.ibm.com/cs/crawler [rc1.wf.ibm.com] | |
| http://www.almaden.ibm.com/cs/crawler [wf216] | |
| http://www.baidu.com/Nutch-1.2 | |
| http://www.erotis.it/sexyshop (erotis sexy shop - Migliaia di articoli nel nostro store erotico | |
| http://www.eroxxx.it/sexyshop (sexy shop - eroxxx il sex shop per i tuoi perversi desideri. Migliaia di articoli erotici, sex toys come vibratori, bambole gonfibili e tanto altro; http://www.eroxxx.it; info@eroxxx.it) | |
| http://www.ip2location.com | |
| http://www.istarthere.com_spider@istarthere.com | |
| http://www.miosexy.it/sexyshop (sexy shop - miosexy, sexshop, BDSM, vibratori, falli, vagine, lubrificanti, dvd porno, film hard, lingerie - Migliaia di articoli nel nostro sexy shop online.; http://www.miosexy.it; info@miosexy.it) | |
| http://www.monogol.de | |
| http://www.orgasmosexyshop.it/sexyshop (sexy shop - orgasmo sexy shop è l'erotic shop che cercavi, sex toys, we vibe, giochi erotici, bambole gonfibili e tanto altro; http://www.orgasmosexyshop.it; info@orgasmosexyshop.it) | |
| http://www.sexa.it/sexyshop (sexa erotic shop, sexa.it sexy shop BDSM, vibratori, falli, vagine, lubrificanti, dvd porno, film hard, lingerie - Migliaia di articoli nel nostro sexy shop online.; http://www.sexa.it; info@sexa.it) | |
| http://www.sexypepe.it/sexyshop (sexy shop - sexy toys, BDSM, vibratori, falli, vagine, lubrificanti, dvd porno, film hard, lingerie - Migliaia di articoli nel nostro sexy shop online.; http://www.sexypepe.it; info@sexypepe.it) | |
| http://www.sofiasexyshop.it/sexyshop (sexy shop - sexy toys, BDSM, vibratori, falli, vagine, lubrificanti, dvd porno, film hard, lingerie - Migliaia di articoli nel nostro sexy shop online.; http://www.sofiasexyshop.it; marketing@sofiasexyshop.it) | |
| http://www.spaziosexy.com/sexyshop (spazio sexy erotic sexy shop - Tutti i sexy toys, vibratori, bambole gonfiabili, masturbatori, tenga e tanto altro; http://www.spaziosexy.com; info@spaziosexy.com) | |
| http://www.timelyweb.com/ | |
| http://www.trendtech.dk/spider.asp) | |
| HTTP::Lite/2.x.x | |
| HTTPEyes | |
| HTTPResume v. 1.x | |
| httpunit/1.5 | |
| httpunit/1.x | |
| Huaweisymantecspider (compatible; MSIE 8.0; DSE-support@huaweisymantec.com) | |
| HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/cn/IRL/spider) | |
| HuaweiSymantecSpider/1.0+DSE-support@huaweisymantec.com+(compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR ; http://www.huaweisymantec.com/en/IRL/spider) | |
| Hybrid/1.2 [en] (OS Independent) | |
| HyperEstraier/1.x.xx | |
| i1searchbot/2.0 (i1search web crawler; http://www.i1search.com; crawler@i1search.com) | |
| IAArchiver-1.0 | |
| iaskspider | |
| iaskspider2 (iask@staff.sina.com.cn) | |
| ia_archiver | |
| ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com) | |
| ia_archiver(OS-Wayback) | |
| ia_archiver-web.archive.org | |
| ia_archiver/1.6 | |
| IBrowse/2.2 (AmigaOS 3.5) | |
| IBrowse/2.2 (Windows 3.1) | |
| iCab/2.5.2 (Macintosh; I; PPC) | |
| ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(at)ml(dot)nict(dot)go(dot)jp) | |
| ICC-Crawler(Mozilla-compatible;http://kc.nict.go.jp/icc/crawl.html;icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp) | |
| ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html) | |
| iCCrawler (http://www.iccenter.net) | |
| ICCrawler - ICjobs (http://www.icjobs.de/bot.htm) | |
| ICE Browser/5.05 (Java 1.4.0; Windows 2000 5.0 x86) | |
| ichiro/3.0 (http://help.goo.ne.jp/help/article/1142) | |
| ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html) | |
| ichiro/6.0 (http://help.goo.ne.jp/door/crawler.html) | |
| ichiro/x.0 (http://help.goo.ne.jp/door/crawler.html) | |
| ichiro/x.0 (ichiro@nttr.co.jp) | |
| IconSurf/2.0 favicon finder (see http://iconsurf.com/robot.html) | |
| IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html) | |
| ICOO Loader v.x.x.x | |
| ICRA_label_spider/x.0 | |
| icsbot-0.1 | |
| IDA | |
| ideare - SignSite/1.x | |
| iearthworm/1.0, iearthworm@yahoo.com.cn | |
| IEFav172Free | |
| iFeed.jp/2.0 (www.psychedelix.com/agents/agents.rss; 0 subscribers) | |
| igdeSpyder (compatible; igde.ru; +http://igde.ru/doc/tech.html) | |
| iGetter/1.x (Macintosh;G;PPC) | |
| iGetter/2 (Macintosh; U; PPC Mac OS X; en) | |
| IIITBOT/1.1 (Indian Language Web Search Engine; http://webkhoj.iiit.net; pvvpr at iiit dot ac dot in) | |
| ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company. For more information please visit http://www.ilial.com/crawler; http://www.ilial.com/crawler; crawl@ilial.com) | |
| ilial/Nutch-0.9-dev | |
| IlseBot/1.x | |
| IlTrovatore-Setaccio ( http://www.iltrovatore.it) | |
| Iltrovatore-Setaccio/0.3-dev (Indexing; http://www.iltrovatore.it/bot.html; info@iltrovatore.it) | |
| IlTrovatore-Setaccio/1.2 ( http://www.iltrovatore.it/aiuto/faq.html) | |
| Iltrovatore-Setaccio/1.2 (It-bot; http://www.iltrovatore.it/bot.html; info@iltrovatore.it) | |
| iltrovatore-setaccio/1.2-dev (spidering; http://www.iltrovatore.it/aiuto/.....) | |
| IlTrovatore/1.2 (IlTrovatore; http://www.iltrovatore.it/bot.html; bot@iltrovatore.it) | |
| ImageVisu/v4.x.x | |
| ImageWalker/2.0 (www.bdbrandprotect.com) | |
| Incutio HttpClient v0.x | |
| IncyWincy data gatherer(webmaster@loopimprovements.com | |
| IncyWincy page crawler(webmaster@loopimprovements.com | |
| IncyWincy(http://www.look.com) | |
| IncyWincy(http://www.loopimprovements.com/robot.html) | |
| IncyWincy/2.1(loopimprovements.com/robot.html) | |
| IndexTheWeb.com Crawler7 | |
| Industry Program 1.0.x | |
| Inet library | |
| InetURL/1.0 | |
| info@pubblisito.com- (http://www.pubblisito.com) il Sud dei Motori di Ricerca | |
| Infoaxe./Nutch-0.9 | |
| infoConveraCrawler/0.8 ( http://www.authoritativeweb.com/crawl) | |
| InfoFly/1.0 (http://www.versions-project.org/) | |
| InfoLink/1.x | |
| INFOMINE/8.0 Adders | |
| INFOMINE/8.0 RemoteServices | |
| INFOMINE/8.0 VLCrawler (http://infomine.ucr.edu/useragents) | |
| InfoNaviRobot(F107) | |
| InfoSeek Sidewinder/0.9 | |
| InfoSeek Sidewinder/1.0A | |
| InfoSeek Sidewinder/1.1A | |
| Infoseek SideWinder/1.45 (Compatible; MSIE 10.0; UNIX) | |
| Infoseek SideWinder/2.0B (Linux 2.4 i686) | |
| INGRID/3.0 MT (webcrawler@NOSPAMexperimental.net; http://webmaster.ilse.nl/jsp/webmaster.jsp) | |
| Inktomi Search | |
| InnerpriseBot/1.0 (http://www.innerprise.com/) | |
| innoventage/Nutch-1.0 (poc; www.google.com; proof of concept) | |
| innoventage/Nutch-1.0 (poc; www.innoventage.com; proof of concept) | |
| Insitor.com search and find world wide! | |
| Insitornaut | |
| InstallShield DigitalWizard | |
| integrity/1.6 | |
| Intelix/0.x (cs; http://www.microton.cz/intelix/; microton@@microton.cz) | |
| Interarchy/x.x.x (InterarchyCrawler) | |
| Internet Ninja x.0 | |
| InternetArchive/0.8-dev(Nutch;http://lucene.apache.org/nutch/bot.html;nutch-agent@lucene.apache | |
| InternetLinkAgent/3.1 | |
| InternetSeer.com | |
| intraVnews/1.x | |
| IOI/2.0 (ISC Open Index crawler; http://index.isc.org/; bot@index.isc.org) | |
| IP*Works! V5 HTTP/S Component - by /n software - www.nsoftware.com | |
| IP2MapBot/1.1 <a href=http://www.ip2map.com>http://www.ip2map.com</a> | |
| IPiumBot laurion(dot)com | |
| IpselonBot/0.xx-beta (Ipselon; http://www.ipselon.com; ipselonbot@ipselon.com) | |
| Iria/1.xxa | |
| IRLbot/1.0 ( http://irl.cs.tamu.edu/crawler) | |
| IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler/) | |
| IrssiUrlLog/0.2 | |
| Irvine/1.x.x | |
| ISC Systems iRc Search 2.1 | |
| iSiloX/4.xx Windows/32 | |
| isurf (tszhu@canada.com) | |
| itim | |
| iTunes/x.x.x | |
| IUPUI Research Bot v 1.9a | |
| iVia Page Fetcher (http://ivia.ucr.edu/useragents.shtml) | |
| iVia/4.0 CanonizeUrl (http://infomine.ucr.edu/iVia/useragents.shtml | |
| IWAgent/ 1.0 - www.brandprotect.com | |
| J-PHONE/3.0/J-SH07 | |
| Jabot/6.x (http://odin.ingrid.org/) | |
| Jabot/7.x.x (http://odin.ingrid.org/) | |
| Jabse.com Crawler v.1.0 www.jabse.com/crawler.php//imagecrawler | |
| Jack | |
| jadu-crawler (Enterprise; M2-ANLVCZEKDC2AA; admin@jadu.net) | |
| Jakarta Commons-HttpClient/2.0xxx | |
| Jakarta Commons-HttpClient/3.0-rcx | |
| Jambot/0.1.x (Jambot; http://www.jambot.com/blog; crawler@jambot.com) | |
| Jambot/0.2.1 (Jambot; http://www.jambot.com/blog/static.php?page=webmaster-robot; crawler@jambot.com) | |
| Java 1.1 | |
| Java/1.4.1_01 | |
| Java/1.6.0_07 | |
| Java/1.6.0_10 | |
| Java/1.6.0_19 | |
| Java/1.6.0_20 | |
| Java/1.6.0_24 | |
| Java/1.6.0_26 | |
| Java/1.7.0_02 | |
| Java1.0.21.0 | |
| Java1.1.xx.x | |
| Java1.3.0rc1 | |
| Java1.3.x | |
| Java1.4.0 | |
| JavaCrawler/1.1 | |
| Jayde Crawler. http://www.jayde.com | |
| JBH Agent 2.0 | |
| jBrowser/J2ME Profile/MIDP-1.0 Configuration/CLDC-1.0 (Google WAP Proxy/1.0) | |
| JCheckLinks/0.1 RPT-HTTPClient/0.3-1 | |
| jcrobot | |
| JDK/1.1 | |
| Jeode/1.x.x | |
| Jet/Nutch-1.4 | |
| Jetbot/1.0 | |
| JetBrains Omea Reader 1.0.x (http://www.jetbrains.com/omea_reader/) | |
| JetBrains Omea Reader 2.0 Release Candidate 1 (http://www.jetbrains.com/omea_reader/) | |
| JetCar | |
| Jigsaw/2.2.x W3C_CSS_Validator_JFouffa/2.0 | |
| JoBo/1.3 (http://www.matuschek.net/jobo.html) | |
| JoBo/1.x (http://www.matuschek.net/jobo.html) | |
| JoBo/@JOBO_VERSION@(http://www.matuschek.net/jobo.html) | |
| JobSpider_BA/1.1 | |
| JOC Web Spider | |
| JordoMedia/1.0 RSS File Reader (http://www.jordomedia.com) | |
| Journster [alpha] (http://journster.com/) | |
| Journster.com RSS/Atom aggregator 0.5 (http://www.journster.com/bot.phtml) | |
| JRTS Check Favorites Utility | |
| JRTwine Software Check Favorites Utility | |
| Jyxobot/x | |
| K-Meleon/0.6 (Windows; U; Windows NT 5.1; en-US; rv:0.9.5) Gecko/20011011 | |
| k2spider | |
| kacst/Nutch-1.4 | |
| KAIST AITrc Crawler | |
| KakleBot - www.kakle.com/0.1 (KakleBot - www.kakle.com; http:// www.kakle.com/bot.html; support@kakle.com) | |
| kalooga/kalooga-4.0-dev-datahouse (Kalooga; http://www.kalooga.com; info@kalooga.com) | |
| kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler; crawler@kalooga.com) | |
| Kapere (http://www.kapere.com) | |
| kapooha/Nutch-1.4 (crawler; http://www.kapooha.com; aeroganic@gmail.com) | |
| Kavande Crawler 1.0/Nutch-1.4 ( Iranian National Web Crawler ; kh3rad@gmail.com) | |
| Kavande Crawler 1.0/Nutch-1.4 (Iranian National Web Crawler) | |
| Kazehakase/0.x.x.[x] | |
| KDDI-SN22 UP.Browser/6.0.7 (GUI) MMP/1.1 (Google WAP Proxy/1.0) | |
| Kenjin Spider | |
| Kevin http://dznet.com/kevin/ | |
| Kevin http://websitealert.net/kevin/ | |
| KeywordSearchTool.co/Nutch-1.4 (http://KeywordSearchTool.co/robot) | |
| KE_1.0/2.0 libwww/5.2.8 | |
| KFSW-Bot (Version: 1.01 powered by KFSW www.kfsw.de) | |
| kinja-imagebot (http://www.kinja.com/) | |
| kinjabot (http://www.kinja.com) | |
| Kiodia Spider/Nutch-1.4 | |
| KIT-Fireball/2.0 | |
| KIT-Fireball/2.0 (compatible; Mozilla 4.0; MSIE 5.5) | |
| KiwiStatus/0.3 (NZS.com New Zealand Search; http://www.nzs.com/kiwi-status/) | |
| Klondike/1.50 (WSP Win32) (Google WAP Proxy/1.0) | |
| KnowItAll(knowitall@cs.washington.edu) | |
| Knowledge.com/0.x | |
| Kontiki Client x.xx | |
| Krugle/Krugle,Nutch/0.8+ (Krugle web crawler; http://www.krugle.com/crawler/info.html; webcrawler@krugle.com) | |
| KSbot/1.0 (KnowledgeStorm crawler; http://www.knowledgestorm.com/resources/content/crawler/index.html; crawleradmin@knowledgestorm.com) | |
| KSCrawler/Nutch-1.0 (http://www.kindsight.net/en/kscrawler; crawler@kindsight.net) | |
| kuloko-bot/0.x | |
| kulokobot www.kuloko.com kuloko@backweave.com | |
| kulturarw3/0.1 | |
| KummHttp/1.1 (compatible; KummClient; Linux rulez) | |
| KWC-KX9/1109 UP.Browser/6.2.3.9.g.1.107 (GUI) MMP/2.0 UP.Link/6.3.0.0.0 | |
| Labrador/0.2; http://ir.dcs.gla.ac.uk/labrador; craigm@dcs.gla.ac.uk | |
| Lachesis | |
| lanshanbot/1.0 | |
| lanshanbot/1.0 (+http://search.msn.com/msnbot.htm) | |
| LapozzBot/1.4 ( http://robot.lapozz.com) | |
| LapozzBot/1.5 (+http://robot.lapozz.hu) | |
| larbin (samualt9@bigfoot.com) | |
| LARBIN-EXPERIMENTAL (efp@gmx.net) | |
| larbin_2.1.1 larbin2.1.1@somewhere.com | |
| larbin_2.2.0 (crawl@compete.com) | |
| larbin_2.2.1_de_Viennot (Laurent.Viennot@inria.fr) | |
| larbin_2.2.2 (sugayama@lab7.kuis.kyoto-u.ac.jp) | |
| larbin_2.2.2_guillaume (guillaume@liafa.jussieu.fr) | |
| larbin_2.6.0 (larbin2.6.0@unspecified.mail) | |
| larbin_2.6.1 (larbin2.6.1@unspecified.mail) | |
| larbin_2.6.2 (hamasaki@grad.nii.ac.jp) | |
| larbin_2.6.2 (larbin2.6.2@unspecified.mail) | |
| larbin_2.6.2 (listonATccDOTgatechDOTedu) | |
| larbin_2.6.2 (pimenas@systems.tuc.gr) | |
| larbin_2.6.2 (tom@lemurconsulting.com) | |
| larbin_2.6.2 (vitalbox1@hotmail.com) | |
| larbin_2.6.3 (fuxiang90@gamil.com) | |
| larbin_2.6.3 (gordon0640@gmail.com) | |
| larbin_2.6.3 (hhgg20000@21cn.com) | |
| larbin_2.6.3 (hwq_super@126.com) | |
| larbin_2.6.3 (larbin2.6.3@unspecified.mail) | |
| larbin_2.6.3 (ltaa_web_crawler@groupes.epfl.ch) | |
| larbin_2.6.3 (wgao@genieknows.com) | |
| larbin_2.6.3_for_(http://cosco.hiit.fi/search/) tsilande@hiit.fi | |
| larbin_2.6_basileocaml (basile.starynkevitch@cea.fr) | |
| larbin_devel (http://pauillac.inria.fr/~ailleret/prog/larbin/) | |
| lawinfo-crawler/Nutch-0.9-dev (Crawler for lawinfo.com pages; http://www.lawinfo.com; webmaster@lawinfo.com) | |
| lb-spider/Mozilla/5.0 Gecko/20100101 Firefox/10.0.2 (lb-spider; http://www.linkbutler.de/spider; spider@linkbutler.de) | |
| lc/$ROADS::Version libwww-perl/5.00 | |
| lcabotAccept: */* | |
| LeapTag/0.8.1.beta081.r3750 (compatible; Mozilla 4.0; MSIE 5.5; robot@yoriwa.com) | |
| LECodeChecker/3.0 libgetdoc/1.0 | |
| LeechGet 200x (www.leechget.de) | |
| LEIA/2.90 | |
| LEIA/3.01pr (LEIAcrawler; [SNIP]) | |
| LetsCrawl.com/1.0 +http://letscrawl.com/ | |
| LexiBot/1.00 | |
| LG-LX260 POLARIS-LX260/2.0 MMP/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 | |
| LG/U8138/v1.0 | |
| Libby_1.1/libwww-perl/5.47 | |
| libcurl-agent/1.0 | |
| LibertyW (+http://www.lw01.com) | |
| librabot/2.0 (+http://academic.research.microsoft.com/) | |
| libWeb/clsHTTP -- hiongun@kt.co.kr | |
| libwww-perl/5.41 | |
| libwww-perl/5.45 | |
| libwww-perl/5.48 | |
| libwww-perl/5.50 | |
| libwww-perl/5.52 FP/2.1 | |
| libwww-perl/5.52 FP/4.0 | |
| libwww-perl/5.53 | |
| libwww-perl/5.63 | |
| libwww-perl/5.64 | |
| libwww-perl/5.65 | |
| libwww-perl/5.800 | |
| libwww/5.3.2 | |
| Liferea/0.x.x (Linux; en_US.UTF-8; http://liferea.sf.net/) | |
| Liferea/1.x.x (Linux; es_ES.UTF-8; http://liferea.sf.net/) | |
| LightningDownload/1.0beta2 | |
| LightningDownload/1.x.x | |
| LightningDownload/1.x.x [Accelerated x] | |
| Lijit Crawler (+http://www.lijit.com/robot/crawler) | |
| LijitSpider/Nutch-0.9 (Reports crawler; http://www.lijit.com/; info(a)lijit(d)com) | |
| Lincoln State Web Browser | |
| Link Valet Online 1.x | |
| LinkAlarm/2.x | |
| Linkbot | |
| Linkbot x.0 | |
| LinkCheck (linkcheck@inter7.com http://www.inter7.com/linkcheck) | |
| LinkChecker/3.4 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/4.9 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/5.1 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/7.0 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/7.2 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/7.3 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/7.4 (+http://linkchecker.sourceforge.net/) | |
| LinkChecker/7.5 (+http://linkchecker.sourceforge.net/) | |
| LinkLint-checkonly/2.x.x | |
| LinkLint-spider/2.x.x | |
| linknzbot | |
| LinkPimpin v1.0 | |
| LinkProver 2.1 | |
| Links (0.9x; Linux 2.4.7-10 i686) | |
| Links (0.9xpre12; Linux 2.2.14-5.0 i686; 80x24) | |
| Links (2.xpre7; Linux 2.4.18 i586; x) | |
| Links - http://gossamer-threads.com/scripts/links/ | |
| Links 2.0 (http://gossamer-threads.com/scripts/links/) | |
| Links SQL (http://gossamer-threads.com/scripts/links-sql/) | |
| Links4US-Crawler, (+http://links4us.com/) | |
| LinkScan/11.0beta2 UnixShareware robot from Elsop.com (used by Indiafocus/Indiainfo) | |
| LinkScan/9.0g Unix | |
| LinkScan/x.x Unix | |
| LinksManager.com (http://linksmanager.com/linkchecker.html) | |
| LinkSonar/1.35 | |
| LinkSweeper/1.x | |
| LinkWalker | |
| link_check3.plx libwww-perl/5.65 | |
| ListBidBot (freelance job spider http://listbid.com)<a href=http://listbid.com>Freelance</a> | |
| LitlrBot (http://litlr.me/bot.html) | |
| LiveTrans/Nutch-0.9 (maintainer: cobain at iis dot sinica dot edu dot tw; http://wkd.iis.sinica.edu.tw/LiveTrans/) | |
| Llaut/1.0 (http://mnm.uib.es/~gallir/llaut/bot.html) | |
| LMQueueBot/0.2 | |
| lmspider (lmspider@scansoft.com) | |
| lmspider/Nutch-0.9-dev (For research purposes.; www.nuance.com; lmspider@nuance.com) | |
| LNSpiderguy | |
| LocalBot/1.0 ( http://www.localbot.co.uk/) | |
| LocalcomBot/1.2.x ( http://www.local.com/bot.htm) | |
| LOCIndexer/Nutch-1.0 | |
| Lockstep Spider/1.0 | |
| Look.com | |
| Lotus-Notes/4.5 ( Windows-NT ) | |
| LotusDiscovery/x.0 (compatible; Mozilla 4.0; MSIE 4.01; Windows NT) | |
| Lovel as 1.0 ( +http://www.everatom.com) | |
| LTI/LemurProject Nutch Spider/Nutch-1.0-dev (lti crawler for CMU; http://www.lti.cs.cmu.edu; changkuk at cmu dot edu) | |
| LTI/LemurProject Nutch Spider/Nutch-1.0-dev (Research spider using Nutch; http://www.lemurproject.org; mhoy@cs.cmu.edu) | |
| luchs.at URL checker | |
| Lunascape | |
| lwp-trivial/1.32 | |
| lwp-trivial/1.34 | |
| lwp-trivial/1.35 | |
| LWP::Simple/5.22 | |
| LWP::Simple/5.36 | |
| LWP::Simple/5.48 | |
| LWP::Simple/5.50 | |
| LWP::Simple/5.51 | |
| LWP::Simple/5.53 | |
| LWP::Simple/5.63 | |
| LWP::Simple/5.803 | |
| Lycos_Spider_(modspider) | |
| Lycos_Spider_(T-Rex) | |
| Lynx/2-4-2 (Bobcat/0.5 [DOS] Jp Beta04) | |
| Lynx/2.6 libwww-FM/2.14 | |
| Lynx/2.8 (;http://seebot.org) | |
| Lynx/2.8.3dev.9 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.6 | |
| Lynx/2.8.4rel.1 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.6c (human-guided@lerly.net) | |
| Lynx/Lynx(2.0) | |
| Lynx/Lynx(2.2) (Lynx) | |
| Lynx/Lynx(8899) | |
| Lynx/Nutch-1.4 | |
| Mac Finder 1.0.xx | |
| Mackster( http://www.ukwizz.com ) | |
| Mag-Net | |
| MagicWML/1.0 (forcewml) | |
| magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net) | |
| magpie-crawler/1.1 (U; Linux i386; en-GB; +http://www.brandwatch.net) | |
| MagpieRSS/0.7x (+http://magpierss.sf.net) | |
| Mahiti.Com/Mahiti Crawler-1.0 (Mahiti.Com; http://mahiti.com ; mahiti.com) | |
| mahonie, neofonie search:robot (http://spider.neofonie.de)/Nutch-1.4 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to spider@neofonie.de; http://spider.neofonie.de) | |
| mahonie, neofonie search:robot/search:robot/0.0.1 (This is the MIA Bot - crawling for mia research project. If you feel unhappy and do not want to be visited by our crawler send an email to spider@neofonie.de; http://spider.neofonie.de; spider@neofonie.de) | |
| Mail.Ru/1.0 | |
| Mail.RU/2.0 | |
| mailto:webcraft@bea.com | |
| mammoth/1.0 ( http://www.sli-systems.com/) | |
| MANTIS/Nutch-1.4 | |
| MantraAgent | |
| MapoftheInternet.com ( http://MapoftheInternet.com) | |
| Mariner/5.1b [de] (Win95; I ;Kolibri gncwebbot) | |
| Marketwave Hit List | |
| Martini | |
| Marvin v0.3 | |
| MaSagool/1.0 (MaSagool; http://sagool.jp/; info@sagool.jp) | |
| Mass Downloader 2.x | |
| MasterSeek | |
| Mata Hari/2.00 | |
| Matrix S.p.A. - FAST Enterprise Crawler 6 (Unknown admin e-mail address) | |
| maxomobot/dev-20051201 (maxomo; http://67.102.134.34:4047/MAXOMO/MAXOMObot.html; maxomobot@maxomo.com) | |
| McBot/5.001 (windows; U; NT4.0; en-us) | |
| MDbot/1.0 (+http://www.megadownload.net/bot.html) | |
| Media Player Classic | |
| MediaCrawler-1.0 (Experimental) | |
| Mediapartners-Google/2.1 ( http://www.googlebot.com/bot.html) | |
| MediaSearch/0.1 | |
| MegaSheep v1.0 (www.searchuk.com internet sheep) | |
| Megite2.0 (http://www.megite.com) | |
| Mercator-1.x | |
| Mercator-2.0 | |
| Mercator-Scrub-1.1 | |
| merocrawl/Nutch-1.4 (merobase crawler; www.merobase.com; info@merobase.com) | |
| Metaeuro Web Crawler/0.2 (MetaEuro Web Search Clustering Engine; http://www.metaeuro.com; crawler at metaeuro dot com) | |
| MetaGer-LinkChecker | |
| MetagerBot/0.8-dev (MetagerBot; http://metager.de; ) | |
| MetaGer_PreChecker0.1 | |
| MetaProducts Download Express/1.x | |
| Metaspinner/0.01 (Metaspinner; http://www.meta-spinner.de/; support@meta-spinner.de/) | |
| metatagsdir/0.7 (+http://metatagsdir.com/directory/) | |
| MFC Foundation Class Library 4.0 | |
| MFC_Tear_Sample | |
| MFHttpScan | |
| MicroBaz | |
| Microsoft Data Access Internet Publishing Provider Cache Manager | |
| Microsoft Data Access Internet Publishing Provider DAV | |
| Microsoft Data Access Internet Publishing Provider Protocol Discovery | |
| Microsoft Log Parser 2.2 | |
| Microsoft Small Business Indexer | |
| Microsoft URL Control - 6.00.8xxx | |
| MicrosoftPrototypeCrawler (How's my crawling? mailto:newbiecrawler@hotmail.com) | |
| Microsoft_Internet_Explorer_5.00.438 (fjones@isd.net) | |
| MIIxpc/4.2 | |
| Mindjet MindManager | |
| minibot | |
| miniRank/1.6 (Website ranking; www.minirank.com; robot) | |
| MiracleAlphaTest | |
| MireoBot | |
| Missauga Locate 1.0.0 | |
| Missigua Locator 1.9 | |
| Missouri College Browse | |
| Mister Pix II 2.02a | |
| Mister PiX version.dll | |
| Misterbot-Nutch/0.7.1 (Misterbot-Nutch; http://www.misterbot.fr; admin@misterbot.fr) | |
| Miva (AlgoFeedback@miva.com) | |
| Mizzu Labs 2.2 | |
| MJ12bot/vx.x.x (http://majestic12.co.uk/bot.php?+) | |
| MJ12bot/vx.x.x (http://www.majestic12.co.uk/projects/dsearch/mj12bot.php) | |
| MJBot (SEO assessment) | |
| MLBot (www.metadatalabs.com) | |
| MLBot (www.metadatalabs.com/mlbot) | |
| MnogoSearch/3.2.xx | |
| MnoGoSearch/3.3.9 | |
| Mo College 1.9 | |
| moget/x.x (moget@goo.ne.jp) | |
| mogimogi/1.0 | |
| moiNAG 0.02 | |
| MojeekBot/0.x (archi; http://www.mojeek.com/bot.html) | |
| MOMspider/1.00 libwww-perl/0.40 | |
| monkeyagent | |
| MoonBrowser (version 0.41 Beta4) | |
| Moreoverbot/x.00 (+http://www.moreover.com) | |
| Morris - Mixcat Crawler ( http://mixcat.com) | |
| Motoricerca-Robots.txt-Checker/1.0 (http://tool.motoricerca.info/robots-checker.phtml) | |
| Motorola-V3m Obigo | |
| Mouse-House/7.4 (spider_monkey spider info at www.mobrien.com/sm.shtml) | |
| MovableType/x.x | |
| mozDex/0.xx-dev (mozDex; http://www.mozdex.com/en/bot.html; spider@mozdex.com) | |
| Mozi! | |
| Mozilla | |
| Mozilla (libwhisker/2.4) | |
| Mozilla (Mozilla@somewhere.com) | |
| Mozilla 4.0(compatible; BotSeer/1.0; +http://botseer.ist.psu.edu) | |
| Mozilla/1.1 (compatible; MSPIE 2.0; Windows CE) | |
| Mozilla/1.10 [en] (Compatible; RISC OS 3.70; Oregano 1.10) | |
| Mozilla/1.22 (compatible; MSIE 2.0; Windows 95) | |
| Mozilla/1.22 (compatible; MSIE 2.0d; Windows NT) | |
| Mozilla/1.22 (compatible; MSIE 5.01; PalmOS 3.0) EudoraWeb 2 | |
| Mozilla/2.0 | |
| Mozilla/2.0 (compatible; AOL 3.0; Mac_PowerPC) | |
| Mozilla/2.0 (Compatible; AOL-IWENG 3.0; Win16) | |
| Mozilla/2.0 (compatible; Ask Jeeves) | |
| Mozilla/2.0 (compatible; Ask Jeeves/Teoma) | |
| Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/webmasters.shtml) | |
| Mozilla/2.0 (compatible; Ask Jeeves/Teoma; http://sp.ask.com/docs/about/tech_crawling.html) | |
| Mozilla/2.0 (compatible; EZResult -- Internet Search Engine) | |
| Mozilla/2.0 (compatible; MS FrontPage x.0) | |
| Mozilla/2.0 (compatible; MSIE 2.1; Mac_PowerPC) | |
| Mozilla/2.0 (compatible; MSIE 3.02; Update a; AK; Windows NT) | |
| Mozilla/2.0 (compatible; MSIE 3.02; Update a; AOL 3.0; Windows 95) | |
| Mozilla/2.0 (compatible; MSIE 3.0; AK; Windows 95) | |
| Mozilla/2.0 (compatible; MSIE 3.0; Windows 3.1) | |
| Mozilla/2.0 (compatible; MSIE 3.0B; Win32) | |
| Mozilla/2.0 (compatible; NEWT ActiveX; Win32) | |
| Mozilla/2.0 (compatible; T-H-U-N-D-E-R-S-T-O-N-E) | |
| Mozilla/2.0 compatible; Check&Get 1.1x (Windows 98) | |
| Mozilla/2.01 (Win16; I) | |
| Mozilla/2.02Gold (Win95; I) | |
| Mozilla/3.0 (compatible) | |
| Mozilla/3.0 (compatible; AvantGo 3.2) | |
| Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com) | |
| Mozilla/3.0 (compatible; HP Web PrintSmart 04b0 1.0.1.34) | |
| Mozilla/3.0 (compatible; Indy Library) | |
| Mozilla/3.0 (compatible; Linkman) | |
| Mozilla/3.0 (compatible; MuscatFerret/1.5.4; claude@euroferret.com) | |
| Mozilla/3.0 (compatible; MuscatFerret/1.5; olly@muscat.co.uk) | |
| Mozilla/3.0 (compatible; MuscatFerret/1.6.x; claude@euroferret.com) | |
| Mozilla/3.0 (compatible; netart generator/1.0; libwww-perl/5.64) | |
| Mozilla/3.0 (compatible; NetPositive/2.2) | |
| Mozilla/3.0 (compatible; Opera/3.0; Windows 3.1) v3.1 | |
| Mozilla/3.0 (compatible; Opera/3.0; Windows 95/NT4) 3.2 | |
| Mozilla/3.0 (compatible; PerMan Surfer 3.0; Win95) | |
| Mozilla/3.0 (compatible; REL Software Web Link Validator 2.x) | |
| Mozilla/3.0 (compatible; scan4mail (advanced version) http://www.peterspages.net/?scan4mail) | |
| Mozilla/3.0 (compatible; ScollSpider; http://www.webwobot.com) | |
| Mozilla/3.0 (compatible; Web Link Validator 2.x)Web Link Validator http://www.relsoftware.com/ link validation software | |
| Mozilla/3.0 (compatible; WebCapture x.x; Auto; Windows) | |
| Mozilla/3.0 (compatible; Webinator-DEV01.home.iprospect.com/2.56) | |
| Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56) | |
| Mozilla/3.0 (compatible;) | |
| Mozilla/3.0 (Compatible;Viking/1.8) | |
| Mozilla/3.0 (DreamPassport/3.0) | |
| Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints) | |
| Mozilla/3.0 (Liberate DTV 1.1) | |
| Mozilla/3.0 (Planetweb/2.100 JS SSL US; Dreamcast US) | |
| Mozilla/3.0 (Slurp.so/Goo; slurp@inktomi.com; http://www.inktomi.com/slurp.html) | |
| Mozilla/3.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) | |
| Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) | |
| Mozilla/3.0 (Vagabondo/1.1 MT; webcrawler@NOSPAMwise-guys.nl; http://webagent.wise-guys.nl/) | |
| Mozilla/3.0 (Vagabondo/1.x MT; webagent@wise-guys.nl; http://webagent.wise-guys.nl/) | |
| Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints) | |
| Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMwise-guys.nl; http://webagent.wise-guys.nl/) | |
| Mozilla/3.0 (Win16; I) | |
| Mozilla/3.0 (Win95; I) | |
| Mozilla/3.0 (WinNT; I) | |
| Mozilla/3.0 (WorldGate Gazelle 3.5.1 build 11; FreeBSD2.2.8-STABLE) | |
| Mozilla/3.0 (X11; I; OSF1 V4.0 alpha) | |
| Mozilla/3.0 NAVIO_AOLTV (11; 13; Philips; PH200; 1; R2.0C36_AOL.0110OPTIK; R2.0.0139d_OPTIK) | |
| Mozilla/3.0 WebTV/1.2 (compatible; MSIE 2.0) | |
| Mozilla/3.01 (compatible; AmigaVoyager/2.95; AmigaOS/MC680x0) | |
| Mozilla/3.01 (Compatible; Links2Go Similarity Engine) | |
| Mozilla/3.01 (compatible; Netbox/3.5 R92; Linux 2.2) | |
| Mozilla/3.01-C-MACOS8 (Macintosh; I; PPC) | |
| Mozilla/3.01Gold (X11; I; Linux 2.0.32 i486) | |
| Mozilla/3.01Gold (X11; I; SunOS 5.5.1 sun4m) | |
| Mozilla/3.01SGoldC-SGI (X11; I; IRIX 6.3 IP32) | |
| Mozilla/3.04 (compatible; ANTFresco/2.13; RISC OS 4.02) | |
| Mozilla/3.04 (compatible; NCBrowser/2.35; ANTFresco/2.17; RISC OS-NC 5.13 Laz1UK1309) | |
| Mozilla/3.04 (compatible;QNX Voyager 2.03B ;Photon) | |
| Mozilla/3.x (I-Opener 1.1; Netpliance) | |
| Mozilla/4.0 | |
| Mozilla/4.0 (agadine3.0) www.agada.de | |
| Mozilla/4.0 (Compatible); URLBase 6 | |
| Mozilla/4.0 (compatible: AstraSpider V.2.1 : astrafind.com) | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; .NET CLR 2.0;) | |
| Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) | |
| Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) | |
| Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) | |
| Mozilla/4.0 (compatible; <a href=http://www.reget.com>ReGet Deluxe 5.1</a>; Windows NT 5.1) | |
| Mozilla/4.0 (compatible; Advanced Email Extractor v2.xx) | |
| Mozilla/4.0 (compatible; Arachmo) | |
| Mozilla/4.0 (compatible; BorderManager 3.0) | |
| Mozilla/4.0 (compatible; BOTW Spider; +http://botw.org) | |
| Mozilla/4.0 (compatible; B_L_I_T_Z_B_O_T) | |
| Mozilla/4.0 (compatible; Cerberian Drtrs Version-3.2-Build-0) | |
| Mozilla/4.0 (compatible; Check&Get 3.0; Windows NT) | |
| Mozilla/4.0 (compatible; ChristCrawler.com ChristCrawler@ChristCENTRAL.com) | |
| Mozilla/4.0 (compatible; crawlx, crawler@trd.overture.com) | |
| Mozilla/4.0 (compatible; DAUMOA-video; +http://ws.daum.net/aboutkr.html) | |
| Mozilla/4.0 (compatible; DepSpid/5.0x; +http://about.depspid.net) | |
| Mozilla/4.0 (compatible; DnloadMage 1.0) | |
| Mozilla/4.0 (compatible; FastCrawler3 support-fastcrawler3@fast.no) | |
| Mozilla/4.0 (compatible; FDSE robot) | |
| Mozilla/4.0 (compatible; GPU p2p crawler http://gpu.sourceforge.net/search_engine.php) | |
| Mozilla/4.0 (compatible; grub-client-0.2.x; Crawl your stuff with http://grub.org) | |
| Mozilla/4.0 (compatible; grub-client-0.3.x; Crawl your own stuff with http://grub.org) | |
| Mozilla/4.0 (compatible; grub-client-2.x) | |
| Mozilla/4.0 (compatible; http://search.thunderstone.com/texis/websearch/about.html) | |
| Mozilla/4.0 (compatible; ibisBrowser) | |
| Mozilla/4.0 (compatible; ICS 1.2.xxx) | |
| Mozilla/4.0 (compatible; IE-Favorites-Check-0.5) | |
| Mozilla/4.0 (compatible; Iplexx Spider/1.0 http://www.iplexx.at) | |
| Mozilla/4.0 (compatible; KeepNI web site monitor) | |
| Mozilla/4.0 (compatible; Link Utility; http://net-promoter.com) | |
| Mozilla/4.0 (compatible; Lotus-Notes/5.0; Windows-NT) | |
| Mozilla/4.0 (compatible; MSIE 4.01; AOL 4.0; Windows 98) | |
| Mozilla/4.0 (compatible; MSIE 4.01; Mac_PowerPC) | |
| Mozilla/4.0 (compatible; MSIE 4.01; MSIECrawler; Windows 95) | |
| Mozilla/4.0 (compatible; MSIE 4.01; Vonna.com b o t) | |
| Mozilla/4.0 (compatible; MSIE 4.01; Windows 95) | |
| Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; MSN Companion 2.0; 800x600; Compaq) | |
| Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320; SPV M700; OpVer 19.123.2.733) OrangeBot-Mobile 2008.0 (mobilesearch.support@orange-ftgroup.com) | |
| Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPS; 240x320) | |
| Mozilla/4.0 (compatible; MSIE 4.01; Windows NT Windows CE) | |
| Mozilla/4.0 (compatible; MSIE 4.01; Windows NT) | |
| Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 4.0 Robot) Microsoft | |
| Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 5.0 Robot) | |
| Mozilla/4.0 (compatible; MSIE 4.01; Windows NT; MS Search 6.0 Robot) | |
| Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) ACR | |
| Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Indonesia Interactive | |
| Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) WebQuest Designs | |
| Mozilla/4.0 (compatible; MSIE 5.01; Windows 95) via <B>Avirt Gateway Server</B> v4.0 | |
| Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) | |
| Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) (samualt9@bigfoot.com) | |
| Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; FeedFinder-2.0; http://bloggz.se/crawler) | |
| Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; NetCaptor 6.5.0RC1) | |
| Mozilla/4.0 (compatible; MSIE 5.01; Windows NT) | |
| Mozilla/4.0 (compatible; MSIE 5.0; AOL 5.0; Windows 95; DigExt; Gateway2000; sureseeker.com) | |
| Mozilla/4.0 (compatible; MSIE 5.0; Mac_PowerPC; AtHome021) | |
| Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience: http://www.netnose.com) | |
| Mozilla/4.0 (compatible; MSIE 5.0; Win32) via proxy gateway CERN-HTTPD/3.0 libwww/2.17 | |
| Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5 | |
| Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot BETA 1.2 (http://www.voila.com/) | |
| Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6 | |
| Mozilla/4.0 (compatible; MSIE 5.0; Windows ME) Opera 5.11 [en] | |
| Mozilla/4.0 (compatible; MSIE 5.0; Windows ME; Link Checker 2.x.xx http://www.kyosoft.com) | |
| Mozilla/4.0 (compatible; MSIE 5.0; Windows NT) | |
| Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt) | |
| Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent | |
| Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com) | |
| Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; www.psychedelix.com) | |
| Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; www.psychedelix.com/; http://www.galaxy.com/info/crawler.html) | |
| Mozilla/4.0 (compatible; MSIE 5.0; YANDEX) | |
| Mozilla/4.0 (compatible; MSIE 5.5; AOL 4.0; Windows 98; GoBeez (www.gobeez.com)) | |
| Mozilla/4.0 (compatible; MSIE 5.5; Windows 95; Transmission Segment; Hotbar 2.0) | |
| Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Crazy Browser 1.x.x) | |
| Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; KITV4.7 Wanadoo) | |
| Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; SAFEXPLORER TL) | |
| Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; SYMPA; Katiesoft 7; SimulBrowse 3.0) | |
| Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; BTinternet V8.1) | |
| Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot) | |
| Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; QXW03018) | |
| Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Active Cache Request | |
| Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Fetch API Request | |
| Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; .NET CLR 1.0.3705) | |
| Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; AIRF) | |
| Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; AspTear 1.5) | |
| Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; N_o_k_i_a) | |
| Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461) RPT-HTTPClient/0.3-3E | |
| Mozilla/4.0 (compatible; MSIE 6.0 compatible; Asterias Crawler v4; +http://www.singingfish.com/help/spider.html; webmaster@singingfish.com); SpiderThread Revision: 3.10 | |
| Mozilla/4.0 (compatible; MSIE 6.0; AOL 9.0; Windows 98; .NET CLR 1.1.4322; MEGAUPLOAD 2.0) | |
| Mozilla/4.0 (compatible; MSIE 6.0; AOL 9.0; Windows NT 5.1; SV1; HbTools 4.7.2) | |
| Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Skampy/0.9.x [en] | |
| Mozilla/4.0 (compatible; MSIE 6.0; TargetSeek/1.0; +http://www.targetgroups.net/TargetSeek.html) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Win32) WebWasher 3.0 | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows 98) REL Software Web Link Validator 2.x) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows 98) Web Link Validator 2.x) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Net M@nager V3.02 - www.vinn.com.au) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; support@illumit.com; http://www.illumit.com/Products/weblight/) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; http://www.Abolimba.de) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 1.1.4322; MEGAUPLOAD 2.0; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) (larbin2.6.3@unspecified.mail) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR 1.1.4322; Lunascape 2.1.3) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Google Wireless Transcoder;) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP entries t_st; http://tuezilla.de/t_st-odp-entries-agent.html) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ODP links test; http://tuezilla.de/test-odp-links-agent.html) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; ZoomSpider.net bot; .NET CLR 1.1.4322) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) (dns_admin@c-a-s-h.com) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 1.0.3705) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Covac UPPS Cathan 1.2.5;) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Crayon Crawler; snprtz|T04056566514940; (R1 1.5)) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Deepnet Explorer) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; FunWebProducts; SV1; .NET CLR 1.0.3705; Media Center PC 4.0; InfoPath.1; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; msn OptimizedIE8;ESMX; Alexa Toolbar; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; heritrix/1.3.0 http://www.cs.washington.edu/research/networking/websys/) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 3.0) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; iOpus-I-M) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; iRider 2.21.1108; FDM) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; KKman3.0) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MathPlayer2.0) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Maxthon) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; PeoplePal 3.0; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461; IOpener Release 1.1.04) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; QihooBot 1.0 qihoobot@qihoo.net) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SIMBAR Enabled; InfoPath.1) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; StumbleUpon.com 1.760; .NET CLR 1.1.4322) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Embedded Web Browser from: http://bsalsa.com/; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html ) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AskTbMYC-ST/5.14.1.20007; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; BTRS1088340; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; BTRS28059; GTB7.2; .NET CLR 2.0.50727; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; BTRS28059; GTB7.3; .NET CLR 2.0.50727; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; DX-Browser 5.0.0.0) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts; ezPeer+ v1.0 Beta (0.4.1.98); ezPeer+ v1.0 (0.5.0.00); .NET CLR 1.1.4322; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; IBP; .NET CLR 1.1.4322) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.1; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.2; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; AskTB5.6; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; Media Center PC 3.0; .NET CLR 1.0.3705; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MRA 4.3 (build 01218)) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MSIECrawler) | |
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot) | |
| Mozilla/4.0 (compatible; MSIE 6.1; Windows NT) | |
| Mozilla/4.0 (compatible; MSIE 6.1; Windows XP; .NET CLR 1.1.4322; .NET CLR 2.0.50727)/Nutch-1.3 | |
| Mozilla/4.0 (compatible; MSIE 7.0; Win32) Link Commander 4.0 | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows 98; Maxthon; ZangoToolbar 7.3.85; SpamBlockerUtility 7.9.95) | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322) | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 3.0.04506.30; .NET CLR 3.5.30729) | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; bgft) | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB5; User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://bsalsa.com) ; .NET CLR 2.0.50727) | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; IPMS/4669080A-14AF91D9EAE; TCO_20091110170859; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0 | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; Maxthon; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C) | |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Tablet PC 2.0) | |
| Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.1; .NET CLR 1.1.4322; InfoPath.1; .NET CLR 2.0.50727) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET CLR 1.1.4322; AskTbCLM/5.8.0.12304) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; AskTbNRO/5.14.1.20007; InfoPath.2) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; BRI/1) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS122237; SIMBAR={1B09D51F-67CD-4798-A3B7-04E7920571B8}; GTB7.2; EasyBits GO v1.0; InfoPath.1; .NET CLR 2.0.50727; .NET CLR 1.1.4322; MSN Optimized;ES; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.5; OfficeLivePatch.1.3; BRI/2; .NET4.0C; AskTB5.6) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS122288; GTB7.2; AskTbSTC-US/5.13.1.18107; FDM; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS122529; GTB7.3; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.3) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS122701; GTB7.3; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E; InfoPath.3; BRI/2) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; BTRS5841; GTB7.3; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; OfficeLiveConnector.1.3; OfficeLivePatch.0.0) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.2; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.2; .NET CLR 2.0.50727; InfoPath.2; AskTbFF/5.9.1.14019) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.3; (R1 1.6); .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.3; EasyBits GO v1.0; .NET CLR 2.0.50727) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.1; .NET CLR 2.0.50727; BRI/1) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; MRSPUTNIK 2, 4, 0, 491; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; Orange 8.0; GTB6.3; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Embedded Web Browser from: http://bsalsa.com/; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30618; OfficeLiveConnector.1.3; OfficeLivePatch.1.3) | |
| Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; .NET CLR 3.5.30729) | |
| Mozilla/4.0 (compatible; MSIE enviable; DAUMOA 2.0; DAUM Web Robot; Daum Communications Corp., Korea; +http://ws.daum.net/aboutkr.html) | |
| Mozilla/4.0 (compatible; MSIE is not me; DAUMOA/1.0.1; DAUM Web Robot; Daum Communications Corp., Korea) | |
| Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/delete_main.asp) | |
| Mozilla/4.0 (compatible; Netcraft Web Server Survey) | |
| Mozilla/4.0 (compatible; NetPromoter Spider;http://www.net-promoter.com/) | |
| Mozilla/4.0 (compatible; Opera/3.0; Windows 4.10) 3.51 [en] | |
| Mozilla/4.0 (compatible; Powermarks/3.5; Windows 95/98/2000/NT) | |
| Mozilla/4.0 (compatible; RSS Popper) | |
| Mozilla/4.0 (compatible; SiteKiosk 4.0; MSIE 5.0; Windows 98; SiteCoach 1.0) | |
| Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com) | |
| Mozilla/4.0 (compatible; SPENG) | |
| Mozilla/4.0 (compatible; SuperCleaner 2.xx; Windows 98) | |
| Mozilla/4.0 (compatible; Synapse) | |
| Mozilla/4.0 (compatible; WebCapture 3.0; Windows) | |
| Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5) | |
| Mozilla/4.0 (compatible; WSN Links) | |
| Mozilla/4.0 (compatible; www.euro-directory.com; urlchecker1.0) | |
| Mozilla/4.0 (compatible; www.galaxy.com) | |
| Mozilla/4.0 (compatible; www.linkguard.com Linkguard Online 1.0; Windows NT) | |
| Mozilla/4.0 (compatible; Y!J; for robot study; keyoshid) | |
| Mozilla/4.0 (compatible; Yahoo Japan; for robot study; kasugiya) | |
| Mozilla/4.0 (compatible;) | |
| Mozilla/4.0 (compatible;MSIE 6.0; Windows NT 5.0; H010818) | |
| Mozilla/4.0 (fantomBrowser) | |
| Mozilla/4.0 (fantomCrew Browser) | |
| Mozilla/4.0 (hhjhj@yahoo.com) | |
| Mozilla/4.0 (JemmaTheTourist;http://www.activtourist.com) | |
| Mozilla/4.0 (MobilePhone PM-8200/US/1.0) NetFront/3.x MMP/2.0 | |
| Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) | |
| Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 FAKE (compatible; Googlebot/2.1; http://www.google.com/bot.html) | |
| Mozilla/4.0 (Mozilla; http://www.mozilla.org/docs/en/bot.html; master@mozilla.com) | |
| Mozilla/4.0 (Sleek Spider/1.2) | |
| Mozilla/4.0 (Toread-Crawler/1.1; +http://news.toread.cc/crawler.php) | |
| Mozilla/4.0 compatible FurlBot/Furl Search 2.0 (FurlBot; http://www.furl.net; wn.furlbot@looksmart.net) | |
| Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com) | |
| Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com) | |
| Mozilla/4.0 compatible ZyBorg/1.0 Dead Link Checker (wn.zyborg@looksmart.net; http://www.WISEnutbot.com) | |
| Mozilla/4.0 compatible ZyBorg/1.0 for Homepage (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com) | |
| Mozilla/4.0 efp@gmx.net | |
| Mozilla/4.0 WebTV/2.6 (compatible; MSIE 4.0) | |
| Mozilla/4.0 [en] (Ask Jeeves Corporate Spider) | |
| Mozilla/4.0(compatible; Zealbot 1.0) | |
| Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1) | |
| Mozilla/4.01 (compatible; NORAD National Defence Network) | |
| Mozilla/4.01 [en](Win95;I) | |
| Mozilla/4.02 [en] (X11; I; SunOS 5.6 sun4u) | |
| Mozilla/4.03 (111111) | |
| Mozilla/4.04 (compatible; Dulance bot; +http://www.dulance.com/bot.jsp) | |
| Mozilla/4.04 [en] (X11; I; HP-UX B.10.20 9000/712) | |
| Mozilla/4.04 [en] (X11; I; IRIX 5.3 IP22) | |
| Mozilla/4.05 (Macintosh; I; 68K Nav) | |
| Mozilla/4.05 (Macintosh; I; PPC Nav) | |
| Mozilla/4.05 [en] (X11; I; SunOS 4.1.4 sun4m) | |
| Mozilla/4.08 [en] (Win98; U ;Nav) | |
| Mozilla/4.08 [en] (WinNT; U) | |
| Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_TrueRobot/1.4 libwww/5.2.8 | |
| Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_VoilaBot/1.6 libwww/5.3.2 | |
| Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98) | |
| Mozilla/4.5 (compatible; iCab 2.5.3; Macintosh; I; PPC) | |
| Mozilla/4.5 (compatible; OmniWeb/4.0.5; Mac_PowerPC) | |
| Mozilla/4.5 (compatible; OmniWeb/4.1-beta-1; Mac_PowerPC) | |
| Mozilla/4.5 RPT-HTTPClient/0.3-2 | |
| Mozilla/4.5 [en]C-CCK-MCD {RuralNet} (Win98; I) | |
| Mozilla/4.5b1 [en] (X11; I; Linux 2.0.35 i586) | |
| Mozilla/4.6 [en] (http://www.cnet.com/) | |
| Mozilla/4.61 [de] (OS/2; I) | |
| Mozilla/4.61 [en] (X11; U; ) - BrowseX (2.0.0 Windows) | |
| Mozilla/4.7 | |
| Mozilla/4.7 (compatible; http://eidetica.com/spider) | |
| Mozilla/4.7 (compatible; Intelliseek; http://www.intelliseek.com) | |
| Mozilla/4.7 (compatible; OffByOne; Windows 98) Webster Pro V3.2 | |
| Mozilla/4.7 (compatible; Whizbang) | |
| Mozilla/4.7 (compatible; WhizBang; http://www.whizbang.com/crawler) | |
| Mozilla/4.7 [en](BecomeBot@exava.com) | |
| Mozilla/4.7 [en](Exabot@exava.com) | |
| Mozilla/4.7 [en]C-CCK-MCD {Yahoo;YIP052400} (Win95; I) | |
| Mozilla/4.72 [en] (BACS http://www.ba.be) | |
| Mozilla/4.72C-CCK-MCD Caldera Systems OpenLinux [en] (X11; U; Linux 2.2.14 i686) | |
| Mozilla/4.75C-ja [ja] (X11; U; OSF1 V5.1 alpha) | |
| Mozilla/4.76 (Windows 98; U) Opera 5.12 [en] | |
| Mozilla/4.76 [en] (X11; U; FreeBSD 4.4-STABLE i386) | |
| Mozilla/4.76 [en] (X11; U; SunOS 5.7 sun4u) | |
| Mozilla/4.77C-SGI [en] (X11; U; IRIX 6.5 IP32) | |
| Mozilla/5.0 ((compatible;+Baiduspider/2.0;++http://www.baidu.com/search/spider.html)) | |
| Mozilla/5.0 (+http://www.eurekster.com/mammoth) Mammoth/0.1 | |
| Mozilla/5.0 (+http://www.sli-systems.com/) Mammoth/0.1 | |
| Mozilla/5.0 (Clustered-Search-Bot/1.0; support@clush.com; http://www.clush.com/) | |
| Mozilla/5.0 (compatible) GM RSS Panel X | |
| Mozilla/5.0 (compatible; +http://www.evri.com/evrinid) | |
| Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620 | |
| Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html;) Gecko/2008032620 | |
| Mozilla/5.0 (compatible; Abonti/0.8 - http://www.abonti.com) | |
| Mozilla/5.0 (compatible; AcoonBot/4.10.6; +http://www.acoon.de/robot.asp) | |
| Mozilla/5.0 (compatible; AcoonBot/4.10.7; +http://www.acoon.de/robot.asp) | |
| Mozilla/5.0 (compatible; AhrefsBot/1.0; +http://ahrefs.com/robot/) | |
| Mozilla/5.0 (compatible; AhrefsBot/2.0; +http://ahrefs.com/robot/) | |
| Mozilla/5.0 (compatible; AhrefsBot/3.0; +http://ahrefs.com/robot/) | |
| Mozilla/5.0 (compatible; aiHitBot/1.0; +http://www.aihit.com/) | |
| Mozilla/5.0 (compatible; aiHitBot/1.1; +http://www.aihit.com/) | |
| Mozilla/5.0 (compatible; AnsearchBot/1.x; +http://www.ansearch.com.au/) | |
| Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot) | |
| Mozilla/5.0 (compatible; archive.org_bot/1.10.0 +http://www.loc.gov/minerva/crawl.html) | |
| Mozilla/5.0 (compatible; archive.org_bot/1.13.1x http://crawler.archive.org) | |
| Mozilla/5.0 (compatible; archive.org_bot/1.5.0-200506132127 http://crawler.archive.org) Hurricane Katrina | |
| Mozilla/5.0 (compatible; archive.org_bot/3.1.1 +http://pandora.nla.gov.au/crawl.html) | |
| Mozilla/5.0 (compatible; archive.org_bot/3.1.1 +http://www.bne.es/es/LaBNE/PreservacionDominioES/AvisoWebmasters/index.html) | |
| Mozilla/5.0 (compatible; archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html) | |
| Mozilla/5.0 (compatible; Ask Jeeves/Teoma; http://about.ask.com/en/docs/about/webmasters.shtml) | |
| Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://www.ask.com/about/help/webmasters) | |
| Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) | |
| Mozilla/5.0 (compatible; BanBots/2.0b; Fetch; +http://www.banbots.com) | |
| Mozilla/5.0 (compatible; BecomeBot/1.23; http://www.become.com/webmasters.html) | |
| Mozilla/5.0 (compatible; BecomeBot/1.xx; MSIE 6.0 compatible; http://www.become.com/webmasters.html) | |
| Mozilla/5.0 (compatible; BecomeBot/2.0beta; http://www.become.com/webmasters.html) | |
| Mozilla/5.0 (compatible; BecomeBot/2.x; MSIE 6.0 compatible; http://www.become.com/site_owners.html) | |
| Mozilla/5.0 (compatible; BecomeJPBot/2.3; MSIE 6.0 compatible; +http://www.become.co.jp/site_owners.html) | |
| Mozilla/5.0 (compatible; BeetleBot; ) | |
| Mozilla/5.0 (compatible; Bender; http://benderthewebrobot.tumblr.com) | |
| Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) | |
| Mozilla/5.0 (compatible; Birubot/1.0) Gecko/2009032608 Firefox/3.0.8 | |
| Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot) | |
| Mozilla/5.0 (compatible; BlogRefsBot/0.1; http://www.blogrefs.com/about/bloggers) | |
| Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html) | |
| Mozilla/5.0 (compatible; Bot; +http://pressemitteilung.ws/spamfilter | |
| Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8 | |
| Mozilla/5.0 (compatible; BuzzRankingBot/1.0; +http://www.buzzrankingbot.com/) | |
| Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html) | |
| Mozilla/5.0 (compatible; cdlwas_bot; heritrix/1.14.1 +http://webarchives.cdlib.org/p/webmasters) | |
| Mozilla/5.0 (compatible; Charlotte/1.0b; charlotte@betaspider.com) | |
| Mozilla/5.0 (compatible; Charlotte/1.0b; http://www.searchme.com/support/) | |
| Mozilla/5.0 (compatible; crawler/3.1.0-SNAPSHOT-20110914.213704 +http://www.notconfigured.com/) | |
| Mozilla/5.0 (compatible; Crawling jpeg; http://www.yama.info.waseda.ac.jp) | |
| Mozilla/5.0 (compatible; Custo 3 (Netwu.com); Windows NT 5.1) | |
| Mozilla/5.0 (compatible; DCPbot/1.0; +http://domains.checkparams.com/) | |
| Mozilla/5.0 (compatible; DCPbot/1.1; +http://domains.checkparams.com/) | |
| Mozilla/5.0 (compatible; de/1.13.2 +http://www.de.com) | |
| Mozilla/5.0 (compatible; Diffbot/0.1; +http://www.diffbot.com) | |
| Mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html) | |
| Mozilla/5.0 (compatible; discobot/2.0; +http://discoveryengine.com/discobot.html) | |
| Mozilla/5.0 (compatible; DNS-Digger-Explorer/1.0; +http://www.dnsdigger.com) | |
| Mozilla/5.0 (compatible; DNS-Digger/1.0; +http://www.dnsdigger.com) | |
| Mozilla/5.0 (compatible; Dow Jones Searchbot) | |
| Mozilla/5.0 (compatible; EARTHCOM.info/2.01; http://www.earthcom.info) | |
| Mozilla/5.0 (compatible; EARTHCOM/2.2; +http://enter4u.eu) | |
| Mozilla/5.0 (compatible; egothor/8.0g; +http://ego.ms.mff.cuni.cz/) | |
| Mozilla/5.0 (compatible; en-US; ReverseGet/1.0; http://reverseget.com/; robot@reverseget.com) | |
| Mozilla/5.0 (compatible; en-US; RGAnalytics/1.0) | |
| Mozilla/5.0 (compatible; en-US; RGAnalytics/1.0; ) | |
| Mozilla/5.0 (compatible; EventGuruBot/1.0; +http://www.eventguru.com/spider.html) | |
| Mozilla/5.0 (compatible; Exabot Test/3.0; +http://www.exabot.com/go/robot) | |
| Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot) | |
| Mozilla/5.0 (compatible; ExaleadCloudView/5;) | |
| Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com) | |
| Mozilla/5.0 (compatible; Falconsbot; +http://ws.nju.edu.cn/falcons/) | |
| Mozilla/5.0 (compatible; FatBot 2.0; http://www.thefind.com/main/CrawlerFAQs.fhtml) | |
| Mozilla/5.0 (compatible; Galbot/1.0; +http://www.galbot.com/bot.html) | |
| mozilla/5.0 (compatible; genevabot http://www.healthdash.com) | |
| Mozilla/5.0 (compatible; Google Desktop) Paros/3.2.12 | |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | |
| Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) | |
| Mozilla/5.0 (compatible; Hailoobot/1.2; +http://www.hailoo.com/spider.html) | |
| mozilla/5.0 (compatible; heritrix/1.0.4 http://innovationblog.com) | |
| Mozilla/5.0 (compatible; heritrix/1.10.2 +http://i.stanford.edu/) | |
| Mozilla/5.0 (compatible; heritrix/1.12.1 +http://newstin.com/) | |
| Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com) | |
| Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.page-store.com) [email:paul@page-store.com] | |
| Mozilla/5.0 (compatible; heritrix/1.12.1b +http://netarkivet.dk/website/info.html) | |
| Mozilla/5.0 (compatible; heritrix/1.14.1 +http://www.nlc.gov.cn/) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.accelobot.com) | |
| Mozilla/5.0 (compatible; heritrix/1.14.3.r6601+http://www.yellowpages.com/about/legal/crawl) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://ce.yazduni.ac.ir) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://censoweb.speed.dcc.ufmg.br) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://susanszky.tatk.elte.hu) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.attrakt.com) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.test.test) | |
| Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.youscribe.com) | |
| mozilla/5.0 (compatible; heritrix/1.3.0 http://archive.crawler.org) | |
| Mozilla/5.0 (compatible; heritrix/1.4.0 +http://www.chepi.net) | |
| Mozilla/5.0 (compatible; heritrix/1.4t http://www.truveo.com/) | |
| Mozilla/5.0 (compatible; heritrix/1.5.0 http://www.l3s.de/~kohlschuetter/projects/crawling/) | |
| Mozilla/5.0 (compatible; heritrix/1.5.0-200506231921 http://pandora.nla.gov.au/crawl.html) | |
| Mozilla/5.0 (compatible; heritrix/1.6.0 http://www.worio.com/) | |
| Mozilla/5.0 (compatible; heritrix/1.7.0 +http://www.greaterera.com/) | |
| Mozilla/5.0 (compatible; heritrix/1.8.0 +http://yourwebsite.domain.com/) | |
| Mozilla/5.0 (compatible; Heritrix/1.8.0 http://www.hanzoarchives.com) | |
| Mozilla/5.0 (compatible; heritrix/1.x.x +http://www.accelobot.com) | |
| Mozilla/5.0 (compatible; heritrix/2.0.0-RC1 +http://www.aol.com) | |
| Mozilla/5.0 (compatible; heritrix/3.1.0 +http://mith.umd.edu) | |
| Mozilla/5.0 (compatible; heritrix/3.1.0 +http://sid.uncu.edu.ar/sid/cdh) | |
| Mozilla/5.0 (compatible; heritrix/3.1.0 +http://www.library.unt.edu/digitalprojects/web-archiving) | |
| Mozilla/5.0 (compatible; heritrix/3.1.0 +http://www.northgrid.co.jp) | |
| Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 +http://www.archive.org/details/archive.org_bot) | |
| Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120320.221924 +http://archive.org/details/archive.org_bot) | |
| Mozilla/5.0 (compatible; Hermit Search. Com; +http://www.hermitsearch.com) | |
| Mozilla/5.0 (compatible; HomeTags/1.0; +http://www.hometags.nl/bot) | |
| Mozilla/5.0 (compatible; http://www.IsMySiteUp.Net/bot/ ) | |
| Mozilla/5.0 (compatible; http://www.UptimeAuditor.com/bot/ ) | |
| Mozilla/5.0 (compatible; HyperixScoop/1.3; +http://www.hyperix.com) | |
| Mozilla/5.0 (compatible; iaskspider/1.0; MSIE 6.0) | |
| Mozilla/5.0 (compatible; IDBot/1.0; +http://www.id-search.org/bot.html) | |
| Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/ | |
| Mozilla/5.0 (compatible; imbot/0.1 +http://internetmemory.org/en/) | |
| Mozilla/5.0 (compatible; InterseekWeb/3.x) | |
| Mozilla/5.0 (compatible; IstellaBot/1.01.18 +http://www.tiscali.it/) | |
| Mozilla/5.0 (compatible; Jim +http://www.hanzoarchives.com) | |
| Mozilla/5.0 (compatible; KaloogaBot; http://kalooga.com/crawler) | |
| Mozilla/5.0 (compatible; KLSH-PageGet/1.03; http://www.kondo-net.gr.jp/klsh/) | |
| Mozilla/5.0 (compatible; Konqueror/2.0.1; X11); Supports MD5-Digest; Supports gzip encoding | |
| Mozilla/5.0 (compatible; Konqueror/2.1.1; X11) | |
| Mozilla/5.0 (compatible; Konqueror/2.2.2) | |
| Mozilla/5.0 (compatible; Konqueror/2.2.2; Linux 2.4.14-xfs; X11; i686) | |
| Mozilla/5.0 (compatible; Konqueror/3.2; Linux) | |
| Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails) | |
| Mozilla/5.0 (compatible; LemSpider 0.1) | |
| Mozilla/5.0 (compatible; lemurwebcrawler admin@lemurproject.org; +http://boston.lti.cs.cmu.edu/crawler_12/) | |
| Mozilla/5.0 (compatible; linkfluence/0.9; http://linkfluence.net/) | |
| Mozilla/5.0 (compatible; linkfluence; http://linkfluence.net/) | |
| Mozilla/5.0 (compatible; LinksManager.com_bot http://linksmanager.com/linkchecker.html) | |
| Mozilla/5.0 (compatible; LinkStash Bookmark Manager; http://www.xrayz.co.uk/) | |
| Mozilla/5.0 (compatible; ltbot/1.3; bot@language-tools.com) | |
| Mozilla/5.0 (compatible; LucidWorks/; +; crawler at example dot com) | |
| Mozilla/5.0 (compatible; LuminateBot/1.0; +http://www.luminate.com/bot/) | |
| Mozilla/5.0 (compatible; Mail.RU/2.0) | |
| Mozilla/5.0 (compatible; MetamojiCrawler/1.0; +http://www.metamoji.com/jp/crawler.html | |
| Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+) | |
| Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+) | |
| Mozilla/5.0 (compatible; MJ12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+) | |
| Mozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+) | |
| Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+) | |
| Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html) | |
| Mozilla/5.0 (compatible; MojeekBot/2.0; http://www.mojeek.com/bot.html) | |
| Mozilla/5.0 (compatible; MOSBookmarks/v2.6-Plus; Link Checker) | |
| Mozilla/5.0 (compatible; Mozilla/5.0 +http://www.up.ru) | |
| Mozilla/5.0 (compatible; MSIE 6.0; Podtech Network; crawler_admin@podtech.net) | |
| Mozilla/5.0 (compatible; MSIE 7.0 +http://internetmemory.org/en/); crawling on behalf of The National Archives UK Government Web Archive: http://www.nationalarchives.gov.uk/webarchive/; email: webarchive@nationalarchives.gsi.gov.uk | |
| Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; + http://tab.search.daum.net/aboutWebSearch.html) Daumoa/3.0 | |
| Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server; +http://ws.daum.net/aboutWebSearch.html) Daumoa/2.0 | |
| Mozilla/5.0 (compatible; Najdi.si/3.1) | |
| Mozilla/5.0 (compatible; Netseer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com) | |
| Mozilla/5.0 (compatible; Nigma.ru/3.0; crawler@nigma.ru) | |
| Mozilla/5.0 (compatible; oBot/2.3.1; +http://www-935.ibm.com/services/us/index.wss/detail/iss/a1029077?cntxt=a1027244) | |
| Mozilla/5.0 (compatible; OnetSzukaj/5.0; http://szukaj.onet.pl) | |
| Mozilla/5.0 (compatible; OntoPort/; +; admin@ontoport.de) | |
| Mozilla/5.0 (compatible; OpenindexDeepSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html) | |
| Mozilla/5.0 (compatible; OpenindexShallowSpider/Nutch-1.5-dev; +http://www.openindex.io/en/webmasters/spider.html) | |
| Mozilla/5.0 (compatible; PagestackerBot; http://www.pagestacker.com) | |
| Mozilla/5.0 (compatible; PalmeraBot; http://www.links24h.com/help/palmera) Version 0.001 | |
| Mozilla/5.0 (compatible; PEAR HTTP_Request class; http://feed.moo.jp/) | |
| Mozilla/5.0 (compatible; Phonifier; +http://www.phonifier.com) | |
| Mozilla/5.0 (compatible; Plukkie/1.4; http://www.botje.com/plukkie.htm) | |
| Mozilla/5.0 (compatible; pmoz.info ODP link checker; +http://pmoz.info/doc/botinfo.htm) | |
| Mozilla/5.0 (compatible; pogodak.ba/3.x) | |
| Mozilla/5.0 (compatible; Pogodak.hr/3.1) | |
| Mozilla/5.0 (compatible; Proximic crawler; +http://www.proximic.com/en/about-us/contact-us.html) | |
| Mozilla/5.0 (compatible; proximic; +http://www.proximic.com) | |
| Mozilla/5.0 (compatible; PWeBot/3.1; http://www.programacionweb.net/robot.php) | |
| Mozilla/5.0 (compatible; Quantcastbot/1.0; www.quantcast.com) | |
| Mozilla/5.0 (compatible; Robo/1.0b; http://potaru.com/robo.html)/Nutch-1.2 | |
| Mozilla/5.0 (compatible; robtexbot/1.0; http://www.robtex.com/ ) | |
| Mozilla/5.0 (compatible; RTSbot/1.0; +http://adamcaudill.com/rtsbot/) | |
| Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/) | |
| Mozilla/5.0 (compatible; ScribdReader/1.0; +http://www.float.com) | |
| Mozilla/5.0 (compatible; Scrubby/2.2; http://www.scrubtheweb.com/) | |
| Mozilla/5.0 (compatible; Semager/1.4; http://www.semager.de/blog/semager-bots/) | |
| Mozilla/5.0 (compatible; Semager/1.4c; +http://www.semager.de/blog/semager-bots/) | |
| Mozilla/5.0 (compatible; SEOkicks-Robot +http://www.seokicks.de/robot.html) | |
| Mozilla/5.0 (compatible; ShunixBot/1.x.x +http://www.shunix.com/robot.htm) | |
| Mozilla/5.0 (compatible; ShunixBot/1.x; http://www.shunix.com/bot.htm) | |
| Mozilla/5.0 (compatible; sindice-fetcher/0.1.0 +http://sindice.com/developers/bot) | |
| Mozilla/5.0 (compatible; sindice-site-manager/0.1.0 +http://sindice.com/developers/bot) | |
| Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/) | |
| Mozilla/5.0 (compatible; SkreemRBot +http://skreemr.com) | |
| Mozilla/5.0 (compatible; SnapPreviewBot; en-US; rv:1.8.0.9) Gecko/20061206 Firefox/1.5.0.9 | |
| Mozilla/5.0 (compatible; spbot/3.0; +http://www.seoprofiler.com/bot ) | |
| Mozilla/5.0 (compatible; special_archiver/1.5.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html) | |
| Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); http://nlp.fi.muni.cz/projects/biwec/) | |
| Mozilla/5.0 (compatible; SpurlBot/0.2) | |
| Mozilla/5.0 (compatible; suggybot v0.01a, http://blog.suggy.com/was-ist-suggy/suggy-webcrawler/) | |
| Mozilla/5.0 (compatible; SummizeBot +http://www.summize.com) | |
| Mozilla/5.0 (compatible; SYCLIKControl/LinkChecker;) | |
| Mozilla/5.0 (compatible; Synoobot/0.9; http://www.synoo.com/search/bot.html) | |
| Mozilla/5.0 (compatible; Theophrastus/x.x; http://users.cs.cf.ac.uk/N.A.Smith/theophrastus.php) | |
| Mozilla/5.0 (compatible; TridentSpider/3.1) | |
| Mozilla/5.0 (compatible; U; Linux i686 (x86_64); de-DE; <a href=http://wikimpress.org/>Wikimpress</a>) Wikimpress/1.0 | |
| Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) | |
| Mozilla/5.0 (compatible; Voluniabot/1.0; spider@volunia.com) | |
| Mozilla/5.0 (compatible; WASALive-Bot ; http://blog.wasalive.com/wasalive-bots/) | |
| Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html) | |
| Mozilla/5.0 (compatible; Web CEO Online robot) | |
| Mozilla/5.0 (compatible; Webduniabot/1.0; +http://search.webdunia.com/bot.aspx) | |
| Mozilla/5.0 (compatible; WebmasterCoffee/0.7; +http://webmastercoffee.com/about) | |
| Mozilla/5.0 (compatible; Windows NT 5.0; phpwebbrainBot/0.1 - http://www.monsterli.ch/phpwebbrain/) | |
| Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com) | |
| Mozilla/5.0 (compatible; woriobot +http://worio.com) | |
| Mozilla/5.0 (compatible; WoW Lemmings Kathune/2.0;http://www.wowlemmings.com/kathune.html) | |
| Mozilla/5.0 (compatible; XTbot/1.0v; +http://www.externaltest.com) | |
| Mozilla/5.0 (compatible; Yahoo! DE Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | |
| Mozilla/5.0 (compatible; Yahoo! Slurp Asia; http://help.yahoo.com/help/us/ysearch/slurp) | |
| Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html) | |
| Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) | |
| Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) | |
| Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots) | |
| Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots) | |
| Mozilla/5.0 (compatible; YesupBot/1.0; +http://www.yesup.net/bot.html) | |
| Mozilla/5.0 (compatible; Yeti/1.0; NHN Corp.; http://help.naver.com/robots/) | |
| Mozilla/5.0 (compatible; YioopBot +http://www.yioop.com/bot.php) | |
| Mozilla/5.0 (compatible; YioopBot; +http://www.yioop.com/bot.php) | |
| Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; ) | |
| Mozilla/5.0 (compatible; Yoono; http://www.yoono.com/) | |
| Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; ) | |
| Mozilla/5.0 (compatible; Zenbot/1.3; +http://zen.co.za/webmasters/) | |
| Mozilla/5.0 (compatible; zermelo +http://www.powerset.com) [email:paul@page-store.com,crawl@powerset.com] | |
| Mozilla/5.0 (compatible;archive.org_bot/1.7.1; collectionId=316; Archive-It; +http://www.archive-it.org) | |
| Mozilla/5.0 (compatible;archive.org_bot/heritrix-1.9.0-200608171144 +http://pandora.nla.gov.au/crawl.html) | |
| Mozilla/5.0 (compatible;FindITAnswersbot/1.0;+http://search.it-influentials.com/bot.htm) | |
| Mozilla/5.0 (compatible;MAINSEEK_BOT) | |
| Mozilla/5.0 (compatible;picmole/1.0 +http://www.picmole.com) | |
| Mozilla/5.0 (compatible;YodaoBot-Image/1.0;http://www.youdao.com/help/webmaster/spider/;) | |
| Mozilla/5.0 (Gecko/20070310 Mozshot/0.0.20070628; http://mozshot.nemui.org/) | |
| Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) (compatible; Yeti-Mobile/0.1; +http://help.naver.com/robots/) | |
| mozilla/5.0 (larbin2.6.3@unspecified.mail) | |
| Mozilla/5.0 (rbzt.net) | |
| Mozilla/5.0 (research@mediatrec.com) | |
| Mozilla/5.0 (Sage) | |
| Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) | |
| Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) | |
| Mozilla/5.0 (SunOS 5.8 sun4u; U) Opera 5.0 [en] | |
| Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html) | |
| Mozilla/5.0 (Version: xxxx Type:xx) | |
| Mozilla/5.0 (wgao@genieknows.com) | |
| Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.7) NimbleCrawler 1.11 obeys UserAgent NimbleCrawler For problems contact: crawler_at_dataalchemy.com | |
| Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/) | |
| Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools) | |
| Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com) | |
| Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1 (http://www.voila.com/) | |
| Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 | |
| Mozilla/5.0 (Windows; U;XMPP Tiscali Communicator v.10.0.1; Windows NT 5.1; it; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3 | |
| Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For problems contact: crawler@health | |
| Mozilla/5.0 (Windows;) NimbleCrawler 1.12 obeys UserAgent NimbleCrawler For problems contact: crawler@healthline.com | |
| Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.2.1; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/20021130 | |
| Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7 SnapPreviewBot | |
| Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19 | |
| Mozilla/5.0 gURLChecker/0.x.x (Linux) | |
| Mozilla/5.0 URL-Spider | |
| Mozilla/5.0 usww.com-Spider-for-w8.net | |
| Mozilla/5.0 wgao@genieknows.com | |
| Mozilla/5.0 whoiam [http://www.axxus.de/] | |
| Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot) | |
| Mozilla/5.0 [en] (X11; U; Linux 2.2.15 i686 +http://www.yahoo.com/index.html) | |
| Mozilla/5.0+(compatible;+CukBot;+Not+a+spammer;+++http://www.companiesintheuk.co.uk/bot.html) | |
| Mozilla/5.0/heritrix/3.1.0 (compatible;; http://vancouver.cs.washington.edu:8080/) | |
| Mozilla/5.0/Nutch-1.1 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) | |
| Mozilla/6.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | |
| Mozilla/Nutch-1.2 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.11) | |
| MQbot metaquerier.cs.uiuc.edu/crawler | |
| MQBOT/Nutch-0.9-dev (MQBOT Nutch Crawler; http://falcon.cs.uiuc.edu; mqbot@cs.uiuc.edu) | |
| MSFrontPage/4.0 | |
| MSIE 4.0 (Win95) | |
| MSIE-5.13 (larbin@unspecified.mail) | |
| msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) | |
| msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) | |
| msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm) | |
| msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm) | |
| MSNBOT/0.xx (http://search.msn.com/msnbot.htm) | |
| msnbot/2.0b (+http://search.msn.com/msnbot.htm) | |
| msnbot/2.0b (+http://search.msn.com/msnbot.htm)._ | |
| msnbot/x.xx ( http://search.msn.com/msnbot.htm) | |
| MSNBOT_Mobile MSMOBOT Mozilla/2.0 (compatible; MSIE 4.02; Windows CE; Default) | |
| MSNPTC/1.0 | |
| MSProxy/2.0 | |
| MSRBOT | |
| MSRBOT (http://research.microsoft.com/research/sv/msrbot) | |
| Mulder, VCR-1.0 | |
| multiBlocker browser | |
| multicrawler ( http://sw.deri.org/2006/04/multicrawler/robots.html) | |
| MultiText/0.1 | |
| MusicWalker2.0 ( http://www.somusical.com) | |
| MVAClient | |
| My Nutch Spider/Nutch-1.2 | |
| My Nutch Spider/Nutch-1.4 | |
| My Nutch Spider/Nutch-1.5-SNAPSHOT | |
| my nutch/Nutch-1.4 | |
| My Spider/Nutch-1.4 | |
| My WinHTTP Connection | |
| myCrawl/Nutch-1.3 | |
| myDaemon | |
| MyGetRight/1.0.0 | |
| MyGetRight/1.0b | |
| Mylinea.com Crawler 2.0 | |
| mylinkcheck/1.02 | |
| Naamah 1.0.1/Blogbot (http://blogbot.de/) | |
| Naamah 1.0a/Blogbot (http://blogbot.de/) | |
| naba/Nutch-1.4 | |
| NABOT/5.0 | |
| nabot_1.0 | |
| NameOfAgent (CMS Spider) | |
| naoFavicon4IE/1.xx | |
| NASA Search 1.0 | |
| NationalDirectory-WebSpider/1.3 | |
| NationalDirectoryAddURL/1.0 | |
| NaverBot-1.0 (NHN Corp. / +82-2-3011-1954 / nhnbot@naver.com) | |
| NaverBot_dloader/1.5 | |
| NavissoBot | |
| NavissoBot/1.7 (+http://navisso.com/) | |
| NCSA Beta 1 (http://vias.ncsa.uiuc.edu/viasarchivinginformation.html) | |
| Nebullabot/2.2 (http://bot.nebulla.info) | |
| NEC Research Agent -- compuman at research.nj.nec.com | |
| NEC-Hayek/1.0 | |
| Net-Seekr Bot/Net-Seekr Bot V1 (http://www.net-seekr.com) | |
| NetAnts/1.2x | |
| NETCOMplete/x.xx | |
| netEstate NE Crawler (+http://www.sengine.info/) | |
| NetinfoBot/1.0 (http://netinfo.bg/netinfobot.html) | |
| NetLookout/2.24 | |
| Netluchs/0.8-dev ( ; http://www.netluchs.de/; ___don't___spam_me_@netluchs.de) | |
| NetMechanic Vx.0 | |
| NetNewsWire/2.x (Mac OS X; http://ranchero.com/netnewswire/) | |
| NetNoseCrawler/v1.0 | |
| Netprospector JavaCrawler | |
| NetPumper/x.xx | |
| NetResearchServer(http://www.look.com) | |
| NetResearchServer/x.x(loopimprovements.com/robot.html) | |
| NetSeer/Nutch-0.9 (NetSeer Crawler; http://www.netseer.com; crawler@netseer.com) | |
| NetSprint -- 2.0 | |
| NetSprint -- 3.0 | |
| NetWhatCrawler/0.06-dev (NetWhatCrawler from NetWhat.com; http://www.netwhat.com; support@netwhat.com) | |
| NetZippy | |
| NeuralBot/0.2 | |
| newsearchengine (ThisUser@unspecified.mail) | |
| NewsGator FetchLinks extension/0.2.0 (http://graemef.com) | |
| NewsGatorOnline/2.0 (http://www.newsgator.com; 1 subscribers) | |
| NextGenSearchBot 1 (for information visit http://www.eliyon.com/NextGenSearchBot) | |
| NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx) | |
| NextopiaBOT (+http://www.nextopia.com) distributed crawler client beta v0.x | |
| NG-Search/0.90 (NG-SearchBot; http://www.ng-search.com; ) | |
| NG/1.0 | |
| NG/2.0 | |
| NG/4.0.1229 | |
| NgapaBot/Nutch-1.3 (NgapaBot is crawler for Indonesia Search Engine; http://www.ngapa.com; admin at ngapa dot com) | |
| nicebot | |
| NICO/1.0 | |
| Nikita the Spider (http://NikitaTheSpider.com/) | |
| NISE/Nutch-2.0-dev | |
| NITLE Blog Spider/0.01 | |
| Nitro Downloader 1.x (www.klsofttools.com) | |
| Noago Spider | |
| Nocilla/1.0 | |
| Nokia-WAPToolkit/1.2 googlebot(at)googlebot.com | |
| Nokia6300/2.0 (05.50) Profile/MIDP-2.0 Configuration/CLDC-1.1 (botmobi http://find.mobi/bot.html abuse@mtld.mobi) | |
| Nokia6610/1.0 (3.09) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html) | |
| Nokia7110/1.0 (05.01) (Google WAP Proxy/1.0) | |
| NokodoBot/1.x (+http://nokodo.com/bot.htm) | |
| Norbert the Spider(Burf.com) | |
| noxtrumbot/1.0 (crawler@noxtrum.com) | |
| noyona_0_1 | |
| NP/0.1 (NP; http://www.nameprotect.com; npbot@nameprotect.com) | |
| NPBot (http://www.nameprotect.com/botinfo.html) | |
| NPBot-1/2.0 | |
| Nsauditor/1.x | |
| NSPlayer/10.0.0.xxxx WMFSDK/10.0 | |
| nsyght.com/Nutch-1.0-dev (nsyght.com; Nsyght.com) | |
| nsyght.com/Nutch-x.x (nsyght.com; search.nsyght.com) | |
| nttdirectory_robot/0.9 (super-robot@super.navi.ocn.ne.jp) | |
| Nucleus SiteList LinkChecker/1.1 | |
| nuSearch Spider <a href='http://www.nusearch.com'>www.nusearch.com</a> (compatible; MSIE 4.01) | |
| NuSearch Spider (compatible; MSIE 6.0) | |
| NuSearch Spider www.nusearch.com | |
| Nutch | |
| Nutch crawler/Nutch-0.9 (picapage.com; admin@picapage.com) | |
| Nutch Experimental Crawler/Nutch-1.4 | |
| Nutch Spider Evaluation/Nutch-1.4 | |
| Nutch test crawler/Nutch-1.4-dev (Crawler testing; crawler/a/t/luminouslabs.com) | |
| Nutch, Leistungsschutz-Debatte Crawler/Nutch-1.4 | |
| nutch-1.4/Nutch-1.4 | |
| Nutch-1/Nutch-1.4 | |
| nutch-solr-integration-test/Nutch-1.2 (FreeCode Norway Web Crawler using Nutch 1.0; http://www.freecode.no/; reza.keshmir.90@gmail.com) | |
| Nutch/Nutch-0.9 (Eurobot; http://www.ayell.eu ) | |
| Nutch/Nutch-1.4 (A semantic crawler for a PhD thesis; http://issel.ee.auth.gr/doku.php/research; kvavliak at issel dot ee dot auth dot gr) | |
| Nutch/Nutch-1.4 (A semantic crawler for a PhD thesis; http://issel.ee.auth.gr; kvavliak at issel dot ee dot auth dot gr) | |
| NutchCVS/0.06-dev (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net) | |
| NutchCVS/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net) | |
| NutchCVS/0.7.1 (Nutch running at UW; http://www.nutch.org/docs/en/bot.html; sycrawl@cs.washington.edu) | |
| NutchEC2Test/Nutch-0.9-dev (Testing Nutch on Amazon EC2.; http://lucene.apache.org/nutch/bot.html; ec2test at lucene.com) | |
| NutchOrg/0.0x-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net) | |
| nutchsearch/Nutch-0.9 (Nutch Search 1.0; herceg_novi at yahoo dot com) | |
| NutchVinegarCrawl/Nutch-0.8.1 (Vinegar; http://www.cs.washington.edu; eytanadar at gmail dot com) | |
| Nutraspace/Nutch-1.2 (www.nutraspace.com) | |
| obidos-bot (just looking for books.) | |
| ObjectsSearch/0.01-dev (ObjectsSearch;http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com) | |
| ObjectsSearch/0.0x (ObjectsSearch; http://www.ObjectsSearch.com/bot.html; support@thesoftwareobjects.com) | |
| oBot ((compatible;Win32)) | |
| Ocelli/1.x (http://www.globalspec.com/Ocelli) | |
| Octopus | |
| Octora Beta - www.octora.com | |
| Octora Beta Bot - www.octora.com | |
| Offline Explorer 1.* | |
| OKArabia search engine/Nutch-1.4 | |
| OliverPerry | |
| OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Internet CategorizerOmniExplorer http://www.omni-explorer.com/ car & shopping search (64.62.175.xxx) | |
| OmniExplorer_Bot/1.0x (+http://www.omni-explorer.com) Job Crawler | |
| OmniExplorer_Bot/1.1x (+http://www.omni-explorer.com) Torrent Crawler | |
| OmniExplorer_Bot/x.xx (+http://www.omni-explorer.com) WorldIndexer | |
| onCHECK Linkchecker von www.scientec.de fuer www.onsinn.de | |
| onCHECK-Robot, www.onsearch.de | |
| Onet.pl SA- http://szukaj.onet.pl | |
| online link validator (http://www.dead-links.com/) | |
| Online24-Bot (Version: 1.0x, powered by www.online24.de) | |
| OntoSpider/1.0 libwww-perl/5.65 | |
| OOZBOT/0.20 ( http://www.setooz.com/oozbot.html ; agentname at setooz dot_com ) | |
| OpenAcoon v4.0.x (www.openacoon.de) | |
| OpenAcoon v4.10.5 (www.openacoon.de) | |
| Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html) | |
| OpenCalaisSemanticProxy | |
| Openfind data gatherer- Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html) | |
| Openfind Robot/1.1A2 | |
| OpenISearch/1.x (www.openisearch.com) | |
| OpenTaggerBot (http://www.opentagger.com/opentaggerbot.htm) | |
| OpenTextSiteCrawler/2.9.2 | |
| OpenWebSpider/0.x.x (http://www.openwebspider.org) | |
| OpenWebSpider/x | |
| Opera/5.0 (Linux 2.0.38 i386; U) [en] | |
| Opera/5.11 (Windows ME; U) [ru] | |
| Opera/5.12 (Windows 98; U) [en] | |
| Opera/6.01 (larbin@unspecified.mail) | |
| Opera/6.x (Linux 2.4.8-26mdk i686; U) [en] | |
| Opera/6.x (Windows NT 4.0; U) [de] | |
| Opera/7.x (Windows NT 5.1; U) [en] | |
| Opera/8.xx (Windows NT 5.1; U; en) | |
| Opera/9.0 (Windows NT 5.1; U; en) | |
| Opera/9.00 (Windows NT 5.1; U; de) | |
| Opera/9.60 (Windows NT 5.1; U; de) Presto/2.1.1 | |
| Opera/9.80 (Macintosh; Intel Mac OS X; U; en) | |
| Opera/9.80 (Windows NT 5.1; U; en) Presto/2.5.22 Version/10.50 | |
| Opera/9.80 (Windows NT 6.1; U; cs) Presto/2.10.229 Version/11.61 | |
| Opera/9.80 (Windows NT 6.1; U; en-US) Presto/2.7.62 Version/11.01 | |
| OpidooBOT (larbin2.6.3@unspecified.mail) | |
| OPWV-SDK UP.Browser/7.0.2.3.119 (GUI) MMP/2.0 Push/PO | |
| Oracle Application Server Web Cache 10g | |
| Oracle iMTCrawler | |
| Oracle Ultra Search | |
| OrangeCrawler/Nutch-1.0 (ldorange.crawler@orange-ftgroup.com) | |
| OrangeSpider | |
| Orbiter/T-2.0 (+http://www.dailyorbit.com/bot.htm) | |
| OrbitSpiderBot/1 | |
| Orca Browser (http://www.orcabrowser.com) | |
| OSSProxy 1.3.305.321 (Build 305.321 Win32 en-us)(Dec 21 2005 16:30:54) | |
| OutfoxBot/0.x (For internet experiments; http://; outfox.agent@gmail.com) | |
| OutfoxMelonBot/0.5 (for internet experiments; http://; outfoxbot@gmail.com) | |
| Overture-WebCrawler/3.8/Fresh (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) | |
| OWR_Crawler 0.1 | |
| ozelot/2.7.3 (Search engine indexer; www.flying-cat.de/ozelot; ozelot@flying-cat.de) | |
| PADLibrary Spider | |
| PageBitesHyperBot/600 (http://www.pagebites.com/) | |
| Pagebull http://www.pagebull.com/ | |
| PagePeeker.com | |
| PagePeeker.com (info: http://pagepeeker.com/robots) | |
| Pagestacker Bot | |
| page_verifier (http://www.securecomputing.com/goto/pv) | |
| PagmIEDownload | |
| Pangalactic Gargleblaster/Nutch-1.2 (Very intoxicating; http://circuitbomb.com; DigitalMail) | |
| parallelContextFocusCrawler1.1parallelContextFocusCrawler1.1 | |
| ParaSite/1.0b (http://www.ianett.com/parasite/) | |
| PArchiveCrawler/1.0 (PArchiveCrawler) | |
| Patwebbot (http://www.herz-power.de/technik.html) | |
| pavuk/0.9pl29b i686-pc-linux-gnu | |
| PBrowse 1.4b | |
| pd02_1.0.0 pd02_1.0.0@dzimi@post.sk | |
| PEAR HTTP_Request class ( http://pear.php.net/ ) | |
| PEERbot www.peerbot.com | |
| PeopleChat/Search_Engine | |
| percbotspider | |
| percbotspider <ops@percolate.com> | |
| PEval 1.4b | |
| pflab (cboc-test@lab.ntt.co.jp) | |
| PHP/3.x.xx | |
| PHP/4.0.4pl1 | |
| PHP/4.0.6 | |
| PHP/4.1.1 | |
| PHP/4.1.2 | |
| PHP/5.2.17p1 | |
| PicklBot 1.0 | |
| PicoSearch/1.0 | |
| Piffany_Web_Scraper_v0.x | |
| Piffany_Web_Spider_v0.x | |
| PigeonBot1.0 BETA | |
| PingALink Monitoring Services 1.0 | |
| PingALink Monitoring Services 1.0 (http://www.pingalink.com) | |
| Pingdom GIGRIB (http://www.pingdom.com) | |
| Pingthesemanticweb-robotvalidator/3.0 | |
| pipeLiner/0.3a (PipeLine Spider;http://www.pipeline-search.com/webmaster.html; webmaster'at'pipeline-search.com) | |
| pipeLiner/0.xx (PipeLine Spider; http://www.pipeline-search.com/webmaster.html) | |
| Pita | |
| Pixray-Seeker/1.1 (Pixray-Seeker; http://www.pixray.com/pixraybot; crawler@pixray.com) | |
| Pizilla++ ver 2.45 | |
| PJspider/3.0 (pjspider@portaljuice.com; http://www.portaljuice.com) | |
| Plagger/0.x.xx (http://plagger.org/) | |
| PlagiarBot/1.0 | |
| PlantyNet_WebRobot_V1.9 dhkang@plantynet.com | |
| plinki/0.1 (you got plinked! (thats a good thing..); http://www.plinki.com; crawl@plinki.com) | |
| PluckFeedCrawler/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://www.pluck.com; 1 subscribers) | |
| Pluggd/Nutch-0.9 (automated crawler http://www.pluggd.com;support at pluggd dot com) | |
| Pockey-GetHTML/4.11.6 (Win32; GUI; ix86) | |
| Pockey-GetHTML/4.12.0 (Win32; GUI; ix86) | |
| Pockey-GetHTML/x.xx | |
| Pockey/x.x.x | |
| Pockey7.x.x(WIN32GUI) | |
| POE-Component-Client-HTTP/0.64 (perl; N; POE; en; rv:0.640000) | |
| Poirot | |
| polybot 1.0 (http://cis.poly.edu/polybot/) | |
| Pompos/1.x http://dir.com/pompos.html | |
| Pompos/1.x pompos@iliad.fr | |
| Popdexter/1.0 | |
| Port Huron Labs | |
| PortalBSpider/2.0 (spider@portalb.com) | |
| portalmmm/2.0 S500i(c20;TB) | |
| PortalNG-Crawler/Nutch-1.4 (Crawler des Portals der Uni Ulm.; http://portal.uni-ulm.de; christian.aschoff at uni-ulm.de) | |
| PostFavorites | |
| PostPost/1.0 (+http://postpost.com/crawlers) | |
| potbot 1.0 | |
| PRCrawler/Nutch-0.9 (data mining development project; crawler@projectrialto.com) | |
| Primo Web Spider/Nutch-1.4 | |
| PrivacyFinder Cache Bot v1.0 | |
| PrivacyFinder/1.1 | |
| private test crawler curl-based | |
| Privoxy/3.0 (Anonymous) | |
| Production Bot 0116B | |
| Production Bot 2016B | |
| Production Bot DOT 3016B | |
| Program Shareware 1.0.2 | |
| Progressive Download | |
| Progressive Download HTTP check | |
| Project XP5 [2.03.07-111203] | |
| PROve AnswerBot 4.0 | |
| ProWebGuide Link Checker (http://www.prowebguide.com) | |
| psbot/0.1 (+http://www.picsearch.com/bot.html) | |
| PSurf15a 11 | |
| PSurf15a 51 | |
| PSurf15a VA | |
| psycheclone | |
| PubCrawl (pubcrawl.stanford.edu) | |
| puf/0.91beta6a (Linux 2.2.18; i686) | |
| puf/0.93.2a (Linux 2.4.18; i686) | |
| pulseBot (pulse Web Miner) | |
| PureSight | |
| PuxaRapido v1.0 | |
| PWeBot/1.2 Inspector (http://www.programacionweb.net/robot.php) | |
| PycURL | |
| PycURL/7.xx.x | |
| Python-urllib/1.17 | |
| Python-urllib/1.1x | |
| Python-urllib/2.0a1 | |
| Python-urllib/2.6 | |
| Qango.com Web Directory (http://www.qango.com/) | |
| QEAVis Agent/Nutch-0.9 (Quantitative Evaluation of Academic Websites Visibility; http://nlp.uned.es/qeavis | |
| QleeQ1/Nutch-1.4 | |
| QPCreep Test Rig ( We are not indexing- just testing ) | |
| QuepasaCreep ( crawler@quepasacorp.com ) | |
| QuepasaCreep v0.9.1x | |
| QueryN Metasearch | |
| quickobot/quickobot-1 (Quicko Labs; http://quicko.co; robot at quicko dot co) | |
| Quicksilver (Blacktree,MacOSX) | |
| QuickTime\xaa.7.0.4 (qtver=7.0.4;cpu=PPC;os=Mac 10.3.9) | |
| QweeryBot/3.01 ( http://qweerybot.qweery.nl) | |
| Qweery_robot.txt_CheckBot/3.01 (http://qweerybot.qweery.com) | |
| R6_CommentReader(www.radian6.com/crawler) | |
| R6_CommentReader_(www.radian6.com/crawler) | |
| R6_FeedFetcher_(www.radian6.com/crawler) | |
| rabaz (rabaz at gigabaz dot com) | |
| RaBot/1.0 Agent-admin/phortse@hanmail.net | |
| RADaR-Bot/Nutch-1.4 (http://radar-bot.com/) | |
| Rainbot1.1 | |
| ramBot xtreme x.x | |
| RAMPyBot - www.giveRAMP.com/0.1 (RAMPyBot - www.giveRAMP.com; http://www.giveramp.com/bot.html; support@giveRAMP.com) | |
| RAMPyBot/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) | |
| Rank Exec (rankexec.com) Reciprocal Link Manager 1.x/bot | |
| Rankivabot/3.2 (www.rankiva.com; 3.2; vzmxikn) | |
| Rational SiteCheck (Windows NT) | |
| RAYSPIDER/Nutch-0.9 | |
| ReadABlog Spider (compatible; 1.1; feed update; www.readablog.com) | |
| RealDownload/4.0.0.4x | |
| REAP-crawler Nutch/Nutch-1.0-dev (Reap Project; http://reap.cs.cmu.edu/REAP-crawler/; Reap Project) | |
| Reaper [2.03.10-031204] (http://www.sitesearch.ca/reaper/) | |
| Reaper/2.0x (+http://www.sitesearch.ca/reaper) | |
| REBOL Core 2.x.x.x.x | |
| REBOL View 1.x.x.x.x | |
| RebusnetBot (+http://www.rebusnet.biz) | |
| RebusnetPADBot/1.5x (+http://www.rebusnet.biz) | |
| reciprocal links checker (http://www.recip-links.com/) | |
| RedBot/redbot-1.0 (Rediff.com Crawler; redbot at rediff dot com) | |
| RedCarpet/1.2 (http://www.redcarpet-inc.com/robots.html) | |
| RedCell/0.1 (InfoSec Search Bot (Coming Soon); http://www.telegenetic.net/bot.html; lhall@telegenetic.net) | |
| RedCell/0.1 (RedCell; telegenetic.net/bot.html; lhall_at_telegenetic.net) | |
| RedKernel WWW-Spider 2/0 (+http://www-spider.redkernel-softwares.com/) | |
| REL Link Checker Lite x.x | |
| RepoMonkey Bait & Tackle/v1.01 | |
| Rewebber/1.2 libwww-perl/5.41 | |
| rico/0.1 | |
| RISSBot/dev (http://www.tkolb.de/rissbot.txt) | |
| RixBot (http://babelserver.org/rix) | |
| RMA/1.0 (compatible; RealMedia) | |
| rmatie.com spider/Nutch-1.4 | |
| rmatie.com/Nutch-1.4 | |
| RoboCrawl (http://www.canadiancontent.net) | |
| RoboCrawl (www.canadiancontent.net) | |
| RoboPal (http://www.findpal.com/) | |
| Robot/www.pj-search.com | |
| Robot: NutchCrawler- Owner: wdavies@acm.org | |
| Robot@SuperSnooper.Com | |
| Robozilla/1.0 | |
| Rome Client (http://tinyurl.com/64t5n) Ver: 0.9 | |
| Ronzoobot/1.5 (http://www.ronzoo.com/about/) | |
| Rotondo/3.1 libwww/5.3.1 | |
| RPT-HTTPClient/0.3-x | |
| RRC (crawler_admin@bigfoot.com) | |
| RSpider/1.0 | |
| RssBandit/1.5.0.10 (.NET CLR 1.1.4322.2407; WinNT 5.1.2600.0; http://www.rssbandit.org) (.NET CLR 1.1.4322.2407; WinNT 5.1.2600.0; ) | |
| RSSMicro.com RSS/Atom Feed Robot | |
| RSSOwl/1.2.3 2006-11-26 (Windows; U; zhtw) | |
| RSSOwl/1.2.4 Preview Release 2007-04-15 (Windows; U; zhtw) | |
| RssReader/1.0.xx.x (http://www.rssreader.com) Microsoft Windows NT 5.1.2600.0 | |
| RSurf15a 41 | |
| RSurf15a 51 | |
| RSurf15a 81 | |
| Rubbot/1.0 (+http://rubhub.com/) | |
| RufusBot (Rufus Web Miner; http://64.124.122.252/feedback.html) | |
| RufusBot (Rufus Web Miner; http://www.webaroo.com/rooSiteOwners.html) | |
| ruijie (larbin@unspecified.mail) | |
| Ruky-Roboter (Version: 1.06, powered by www.ruky.de +http://www.ruky.de/zeige/bot.html) | |
| Rumours-Agent | |
| RX Bar | |
| S&L Spider (http://search.hirners.com/) | |
| S.T.A.L.K.E.R. (http://www.seo-tools.net/en/bot.aspx) | |
| S2-Crawler/1.1 (your search engine; http://www.search2.net/; info at search2 dot net) | |
| S2Robot | |
| sadsa/Nutch-1.4 | |
| SafariBookmarkChecker (+http://www.coriolis.ch/) | |
| sait/Nutch-0.9 (SAIT Research; http://www.samsung.com) | |
| SaladSpoon/ShopSalad 1.0 (Search Engine crawler for ShopSalad.com; http://shopsalad.com/en/partners.html; crawler AT shopsalad.com) | |
| SandCrawler - Compatibility Testing | |
| SapphireWebCrawler/1.0 (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu) | |
| SapphireWebCrawler/Nutch-1.0-dev (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu) | |
| savvybot/0.2 | |
| SBIder/0.7 (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html) | |
| SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html) | |
| SBL-BOT (http://sbl.net) | |
| ScanWeb | |
| ScholarScope/Nutch-1.5 (Scholar Research Engine) | |
| ScholarUniverse/0.8 (Nutch;+http://scholaruniverse.com/bot.jsp; fetch-agent@scholaruniverse.com) | |
| schwarzmann.biz-Spider_for_paddel.org+(http://www.innerprise.net/usp-spider.asp) | |
| Science Traveller International 1X/1.0 | |
| ScollSpider/2.0 (+http://www.webwobot.com/ScollSpider.php) | |
| Scooter-3.0.EU | |
| Scooter-3.0.FS | |
| Scooter-3.0.HD | |
| Scooter-3.0.VNS | |
| Scooter-3.0QI | |
| Scooter-3.2 | |
| Scooter-3.2.BT | |
| Scooter-3.2.DIL | |
| Scooter-3.2.EX | |
| Scooter-3.2.JT | |
| Scooter-3.2.NIV | |
| Scooter-3.2.SF0 | |
| Scooter-3.2.snippet | |
| Scooter-3.3dev | |
| Scooter-ARS-1.1 | |
| Scooter-ARS-1.1-ih | |
| scooter-venus-3.0.vns | |
| Scooter-W3-1.0 | |
| Scooter-W3.1.2 | |
| Scooter/1.0 | |
| Scooter/1.0 scooter@pa.dec.com | |
| Scooter/1.1 (custom) | |
| Scooter/2.0 G.R.A.B. V1.1.0 | |
| Scooter/2.0 G.R.A.B. X2.0 | |
| Scooter/3.3 | |
| Scooter/3.3.QA.pczukor | |
| Scooter/3.3.vscooter | |
| Scooter/3.3_SF | |
| Scooter2_Mercator_x-x.0 | |
| Scooter_bh0-3.0.3 | |
| Scooter_trk3-3.0.3 | |
| Scope (Mars+) | |
| ScoutAbout | |
| ScoutAnt/0.1; +http://www.ant.com/what_is_ant.com/ | |
| scoutmaster | |
| Screaming Frog SEO Spider/1.80 | |
| Screaming Frog SEO Spider/1.90 | |
| Scrubby/2.x (http://www.scrubtheweb.com/) | |
| Scrubby/3.0 (+http://www.scrubtheweb.com/help/technology.html) | |
| Search+ | |
| Search-Engine-Studio | |
| search.ch V1.4 | |
| search.ch V1.4.2 (spiderman@search.ch; http://www.search.ch) | |
| Search/1.0 (http://www.innerprise.net/es-spider.asp) | |
| searchbot admin@google.com | |
| SearchByUsa/2 (SearchByUsa; http://www.SearchByUsa.com/bot.html; info@SearchByUsa.com) | |
| SearchdayBot | |
| SearchExpress Spider0.99 | |
| SearchGuild/DMOZ/Experiment (searchguild@gmail.com) | |
| SearchGuild_DMOZ_Experiment (chris@searchguild.com) | |
| Searchit-Now Robot/2.2 (+http://www.searchit-now.co.uk) | |
| Searchmee! Spider v0.98a | |
| SearchSight/2.0 (http://SearchSight.com/) | |
| SearchSpider.com/1.1 | |
| Searchspider/1.2 (SearchSpider; http://www.searchspider.com; webmaster@searchspider.com) | |
| SearchTone2.0 - IDEARE | |
| Seegul/Nutch-1.4 | |
| SeekBot/1.0 RobotsTxtFetcher/1.3 | |
| Seekbot/1.0 (http://www.seekbot.net/bot.html) HTTPFetcher/0.3 | |
| Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.0 (XDF) | |
| Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2 | |
| Seeker.lookseek.com | |
| semaforo.net | |
| Semager/1.1 (http://www.semager.de/blog/semager-bots/) | |
| Semager/1.x (http://www.semager.de) | |
| semanticdiscovery/0.x | |
| SemrushBot/0.91 | |
| SemrushBot/0.92 | |
| SemrushBot/Nutch-1.5-SNAPSHOT | |
| Sensis Web Crawler (search_comments\at\sensis\dot\com\dot\au) | |
| Sensis.com.au Web Crawler (search_comments\at\sensis\dot\com\dot\au) | |
| SeoBot | |
| SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm) | |
| service-public-indexeur | |
| SeznamBot/1.0 | |
| SeznamBot/1.0 (+http://fulltext.seznam.cz/) | |
| SeznamBot/2.0-test (+http://fulltext.sblog.cz/) | |
| SeznamBot/3.0 (+http://fulltext.sblog.cz/) | |
| sGroup crawler 1/Nutch-1.3 | |
| ShablastBot 1.0 | |
| Shareaza v1.x.x.xx | |
| SharewarePlazaFileCheckBot/1.0+(+http://www.SharewarePlaza.com) | |
| Shim Crawler | |
| Shim-Crawler(Mozilla-compatible; http://www.logos.ic.i.u-tokyo.ac.jp/crawler/; crawl@logos.ic.i.u-tokyo.ac.jp) | |
| ShopWiki/1.0 ( +http://www.shopwiki.com/) | |
| ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot) | |
| Shoula.com Crawler 2.0 | |
| SietsCrawler/1.1 (+http://www.siets.biz) | |
| Sigram/Nutch-1.0-dev (Test agent for Nutch development; http://www.sigram.com/bot.html; bot at sigram dot com) | |
| Siigle Orumcex v.001 Turkey (http://www.siigle.com) | |
| silk/1.0 | |
| silk/1.0 (+http://www.slider.com/silk.htm)/3.7 | |
| SimpleFavPanel/1.2 | |
| Simpy 1.x; http://www.simpy.com/ | |
| Simpy/1.x (Simpy; http://www.simpy.com/?ref=bot; feedback at simpy dot com) | |
| sindice.net sitemap fetcher | |
| Sirketcebot/v.01 (http://www.sirketce.com/bot.html) | |
| SiteBar/3.x.x (Bookmark Server; http://sitebar.org/) | |
| SiteBar/x.x | |
| SiteBar/x.x.x (Bookmark Server; http://sitebar.org/) | |
| sitecheck.internetseer.com | |
| sitecheck.internetseer.com (For more info see: http://sitecheck.internetseer.com) | |
| SiteRecon+(xx) | |
| SiteSnagger | |
| SiteSpider +(http://www.SiteSpider.com/) | |
| SiteSucker/1.x.x | |
| SiteSucker/2.2.4 | |
| SiteSucker/2.3.2 | |
| SiteSucker/2.3.3 | |
| SiteSucker/2.3.5 | |
| SiteSucker/2.3.6 | |
| SiteTaggerBot (http://www.sitetagger.com/bot.htm) | |
| SiteTruth.com site rating system | |
| SiteWinder | |
| SiteXpert | |
| Skampy/0.9.x (http://www.skaffe.com/skampy-info.html) | |
| SkimBot/1.0 (www.skimlinks.com <dev@skimlinks.com>) | |
| SkimBot/1.0 (www.skimlinks.com) | |
| Skimpy/0.x (http://www.skaffe.com/skampy-info.html) | |
| Skywalker/0.1 (Skywalker; anonymous; anonymous) | |
| Slarp/0.1 | |
| Sleipnir | |
| Sleipnir Version 1.xx | |
| Sleipnir Version2.x | |
| Sleipnir/2.xx | |
| Slider_Search_v1-de | |
| SlimBrowser | |
| Slurp/2.0 (slurp@inktomi.com; http://www.inktomi.com/slurp.html) | |
| Slurp/2.0-KiteWeekly (slurp@inktomi.com; http://www.inktomi.com/slurp.html) | |
| Slurp/si (slurp@inktomi.com; http://www.inktomi.com/slurp.html) | |
| Slurpy Verifier/1.0 | |
| SlySearch (slysearch@slysearch.com) | |
| SlySearch/1.0 http://www.plagiarism.org/crawler/robotinfo.html | |
| SlySearch/1.x http://www.slysearch.com | |
| SmartDownload/1.2.67 (Win32; Jan 12 1999) | |
| SmartDownload/1.2.77 (Win32; Feb 1 2000) | |
| SmartDownload/1.2.77 (Win32; Jun 19 2001) | |
| smartwit.com | |
| SmiffyDCMetaSpider/1.0 | |
| sna-0.0.1 (mikemuzio@msn.com) | |
| sna-0.0.1 mikeelliott@hotmail.com | |
| snap.com beta crawler v0 | |
| Snapbot/1.0 | |
| Snapbot/1.0 (Snap Shots, +http://www.snap.com) | |
| Snappy/1.1 ( http://www.urltrends.com/ ) | |
| Snarfer/0.x.x (http://www.snarfware.com/) | |
| SnoopRob/x.x | |
| Snoopy v1.xx | |
| Snoopy v1.xx- : User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MyIE2) | |
| Snoopy_v0.xx | |
| SnykeBot/0.6 (http://www.snyke.com) | |
| SocSciBot () | |
| SoftBank/1.0/812SH/SHJ001 Browser/NetFront/3.3 Profile/MIDP-2.0 Configuration/CLDC-1.1 | |
| SoftHypermarketFileCheckBot/1.0+(+http://www.softhypermaket.com) | |
| Softizerbot (http://www.softizer.com) | |
| SoftNet Nutch Spider/Nutch-1.4 | |
| sogou develop spider | |
| Sogou Orion spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) | |
| sogou spider | |
| Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07) | |
| Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) | |
| sohu agent | |
| sohu-search | |
| Sosospider+(+http://help.soso.com/webspider.htm) | |
| Space Bison/0.02 [fu] (Win67; X; SK) | |
| SpeedDownload/1.x | |
| speedfind ramBot xtreme 8.1 | |
| Speedy Spider (Beta/x.x; speedy@entireweb.com) | |
| Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/speedyspider/) | |
| Speedy_Spider (http://www.entireweb.com) | |
| Sphere Scout&v4.0 - scout at sphere dot com | |
| Sphider | |
| Spida/0.1 | |
| Spider-Sleek/2.0 (+http://search-info.com/linktous.html) | |
| spider.batsch.com | |
| Spider.TerraNautic.net - v:1.04 | |
| spider.yellopet.com - www.yellopet.com | |
| Spider/maxbot.com admin@maxbot.com | |
| spider/Nutch-1.3 (spider; http://www.xxx.com) | |
| SpiderKU/0.x | |
| SpiderMan | |
| SpiderMonkey/7.0x (SpiderMonkey.ca info at http://spidermonkey.ca/sm.shtml) | |
| Spinne/2.0 | |
| Spinne/2.0 med | |
| Spinne/2.0 med_AH | |
| Spock Crawler (http://www.spock.com/crawler) | |
| sportsuchmaschine.de-Robot (Version: 1.02- powered by www.sportsuchmaschine.de) | |
| spotinfluence/Nutch-1.2 (Spot Influence crawler; http://spotinfluence.com; hello at spotinfluence dot com) | |
| spotinfluence/Nutch-1.4 (Spot Influence crawler; http://spotinfluence.com; hello at spotinfluence dot com) | |
| sproose/0.1-alpha (sproose crawler; http://www.sproose.com/bot.html; crawler@sproose.com) | |
| SQ Webscanner | |
| Squid-Prefetch | |
| squidclam | |
| SquidClamAV_Redirector 1.x.x | |
| Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu) | |
| Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu) | |
| Sqworm/2.9.89-BETA (beta_release; 20020130-839; i686-pc-linux-gnu) | |
| SS/Nutch-1.5 (SS) | |
| ssearch_bot (sSearch Crawler; http://www.semantissimo.de) | |
| SSurf15a 11 | |
| StackRambler/x.x | |
| Stamina/1.4 | |
| Star Downloader | |
| StarDownloader/1.xx | |
| stat statcrawler@gmail.com | |
| stc-crawler/0.84 | |
| Steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) | |
| Steeler/3.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) | |
| Strategic Board Bot (+http://www.strategicboard.com) | |
| Submission Spider at surfsafely.com | |
| suchbaer.de | |
| suchbaer.de (CrawlerAgent v0.103) | |
| suchbot | |
| Suchknecht.at-Robot | |
| suchpadbot/1.0 (+http://www.suchpad.de) | |
| Summon | |
| Sunrise XP/2.x | |
| Sunrise/0.42g (Windows XP) | |
| SuperBot/x.x (Win32) | |
| SuperBot/x.x.x.xx (Windows XP) | |
| Superdownloads Spiderman | |
| SURF | |
| SurferF3 1/0 | |
| SurfMaster | |
| SurveyBot/2.2 <a href='http://www.whois.sc'>Whois Source</a> | |
| SurveyBot/2.3 (Whois Source) | |
| suzuran | |
| SWB/V1.4 (HP) | |
| swbot/0.9c libwww/5.3.1 | |
| Swooglebot/2.0. (+http://swoogle.umbc.edu/swooglebot.htm) | |
| SWSBot-Images/1.2 http://www.smartwaresoft.com/swsbot12.html | |
| SygolBot http://www.sygol.net | |
| Sylera/1.2.x | |
| SyncBot | |
| SyncIT/x.x | |
| Syndirella/0.91pre | |
| SynoBot | |
| Syntryx ANT Scout Chassis Pheromone; Mozilla/4.0 compatible crawler | |
| Szukacz/1.x | |
| Szukacz/1.x (robot; www.szukacz.pl/jakdzialarobot.html; szukacz@proszynski.pl) | |
| T-Online Browser | |
| tags2dir.com/0.8 (+http://tags2dir.com/directory/) | |
| Tagword (http://tagword.com/dmoz_survey.php) | |
| Tagyu Agent/1.0 | |
| Talkro Web-Shot/1.0 (E-mail: webshot@daumsoft.com- Home: http://222.122.15.190/webshot) | |
| TAMU_CS_IRL_CRAWLER/1.0 | |
| targetblaster.com/0.9k | |
| TargetYourNews.com bot | |
| TCDBOT/Nutch-0.8 (PhD student research;http://www.tcd.ie; mcgettrs at t c d dot IE) | |
| TE | |
| TeamSoft WinInet Component | |
| TECOMAC-Crawler/0.x | |
| Tecomi Bot (http://www.tecomi.com/bot.htm) | |
| Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://www.netseer.com/crawler.html; crawler@netseer.com) | |
| Teleport Pro/1.2x(.1xxx) | |
| Teoma MP | |
| teomaagent crawler-admin@teoma.com | |
| teomaagent1 [crawler-admin@teoma.com] | |
| teoma_agent1 | |
| Teradex Mapper; mapper@teradex.com; http://www.teradex.com | |
| TeragramCrawler | |
| terraminds-bot/1.0 (support@terraminds.de) | |
| TerrawizBot/1.0 (+http://www.terrawiz.com/bot.html) | |
| Test spider | |
| TestCrawler/Nutch-0.9 (Testing Crawler for Research ; http://balihoo.com/index.aspx; tgautier at balihoo dot com) | |
| The Expert HTML Source Viewer (http://www.expert-html.com) | |
| theasearchCrawler/Nutch-1.4 (http://www.theasearch.com/; hansod1@gmail.com) | |
| TheRarestParser/0.2a (http://therarestwords.com/) | |
| TheSuBot/0.1 (www.thesubot.de) | |
| thumbshots-de-bot (+http://www.thumbshots.de/) | |
| thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de) | |
| thumbshots-de-Bot (Version: 1.02- powered by www.thumbshots.de) | |
| TigrinyaCrawler(tirawler@gmail.com) | |
| timboBot/0.9 http://www.breakingblogs.com/timbo_bot.html | |
| TinEye/1.1 (http://tineye.com/crawler.html) | |
| tivraSpider/1.0 (crawler@tivra.com) | |
| TJG/Spider | |
| TJvMultiHttpGrabber Component | |
| Tkensaku/x.x(http://www.tkensaku.com/q.html) | |
| Tobot/Nutch-1.4 ('tobot at eel dot rdc dot toshiba dot co dot jp') | |
| toCrawl/UrlDispatcher | |
| Topodia/1.2-dev (Topodia - Crawler for HTTP content indexing; http://www.topodia.com/; support@topodia.com) | |
| TOPOS robot/1.1 (http://www.topos.com.ua/) | |
| Toutatis x-xx.x (hoppa.com) | |
| Toutatis x.x (hoppa.com) | |
| Toutatis x.x-x | |
| traazibot/testengine (+http://www.traazi.de) | |
| Trailfire-bot/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) | |
| Trailfire-bot/0.7.1 (Trailfire page content analyzer; http://trailfire.com; info@trailfire.com) | |
| Trailfire/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org) | |
| Trampelpfad-Spider | |
| Trampelpfad-Spider-v0.1 | |
| tricosMetaCheck 1.2216-08-1999 (http://www.tricos.com/metacheck) | |
| TruNorth/Nutch-1.4 | |
| TsinghuaLab | |
| TSurf15a 11 | |
| TulipChain/5.x (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://java.sun.com/) Linux/2.4.17 | |
| TulipChain/5.xx (http://ostermiller.org/tulipchain/) Java/1.x.1_0x (http://apple.com/) Mac_OS_X/10.2.8 | |
| Tumblr/1.0 RSS syndication (+http://www.tumblr.com/) (support@tumblr.com) | |
| TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html) | |
| TurnitinBot/x.x (http://www.turnitin.com/robot/crawlerinfo.html) | |
| Turnpike Emporium LinkChecker/0.1 | |
| TutorGig/1.5 (+http://www.tutorgig.com/crawler) | |
| Tutorial Crawler 1.4 (http://www.tutorgig.com/crawler) | |
| Twiceler www.cuill.com/robots.html | |
| Twiceler-0.9 http://www.cuill.com/twiceler/robot.html | |
| Twisted PageGetter | |
| Twitterbot/1.0 | |
| Twitturly / v0.x | |
| Twotrees Reactive Filter V2.0 | |
| Tycoon Agent/Nutch-1.0-dev | |
| TygoBot | |
| TygoProwler | |
| UCmore | |
| UCMore Crawler App | |
| UCWEB5.1 | |
| UDM | |
| UdmSearch/3.1.x | |
| UIowaCrawler/1.0 | |
| UKWizz/Nutch-0.8.1 (UKWizz Nutch crawler; http://www.ukwizz.com/) | |
| Ultraseek | |
| Under the Rainbow 2.2 | |
| UnisterBot (Mozilla/5.0 compatible; crawler@unister-gmbh.de) | |
| unknownbot (abc@mail.com) | |
| unknownght.com Web Server IIS vs Apache Survey. See Results at www.DNSRight.com | |
| UofTDB_experiment (leehyun@cs.toronto.edu) | |
| UP.Browser/3.01-IG01 UP.Link/3.2.3.4 | |
| updated/0.1-alpha (updated crawler; http://www.updated.com; crawler@updated.com) | |
| updated/0.1beta (updated.com; http://www.updated.com; crawler@updated.om) | |
| Updownerbot (+http://www.updowner.com/bot) | |
| UPG1 UP/4.0 (compatible; Blazer 1.0) | |
| upictoBot | |
| Uptimebot | |
| UptimeBot(www.uptimebot.com) | |
| URI::Fetch/0.06 | |
| URL Spider Pro/x.xx (innerprise.net) | |
| URLBase/6.x | |
| URLBlaze | |
| urlfan-bot/1.0; +http://www.urlfan.com/site/bot/350.html | |
| URLGetFile | |
| URL_Spider_Pro/x.x | |
| URL_Spider_Pro/x.x+(http://www.innerprise.net/usp-spider.asp) | |
| User-Agent: BoardReader Favicon Fetcher /1.0 info@boardreader.com | |
| User-Agent: BoardReader Image Fetcher /1.0 info@boardreader.com | |
| User-Agent: BoardReader-Image-Fetcher /1.0 info@boardreader.com | |
| User-Agent: LjSEEK Picture-Bot /1.0 contact@ljseek.com | |
| User-Agent: FileHeap! file downloader (http://www.fileheap.com) | |
| User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) | |
| User-Agent: Mozilla/4.0 (SKIZZLE! Distributed Internet Spider v1.0 - www.SKIZZLE.com) | |
| User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 | |
| User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729) | |
| user-agent=Mozilla/3.01Gold | |
| USyd-NLP-Spider (http://www.it.usyd.edu.au/~vinci/bot.html) | |
| UtilMind HTTPGet | |
| Utopia WebWasher 3.0 | |
| uTorrent/1500 | |
| VadixBot | |
| Vagabondo-WAP/2.0 (webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/)/1.0 Profile | |
| Vagabondo/1.x MT (webagent@wise-guys.nl) | |
| Vagabondo/2.0 MT | |
| Vagabondo/2.0 MT (webagent at wise-guys dot nl) | |
| Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl) | |
| Vagabondo/3.0 (webagent at wise-guys dot nl) | |
| Vakes/0.01 (Vakes; http://www.vakes.com/; search@vakes.com) | |
| VayalaCreep-v0.0.1 (haploid@haploid.com) | |
| Vayala|Creep-v0.0.1 (codepoet@wildties.com) | |
| vb wininet | |
| versus 0.2 (+http://versus.integis.ch) | |
| versus crawler eda.baykan@epfl.ch | |
| Verticrawlbot | |
| VeryGoodSearch.com.DaddyLongLegs | |
| verzamelgids.nl - Networking4all Bot/x.x | |
| Verzamelgids/2.2 (http://www.verzamelgids.nl) | |
| Vespa Crawler | |
| virus_detector (virus_harvester@securecomputing.com) | |
| VisBot/2.0 (Visvo.com Crawler; http://www.visvo.com/bot.html; bot@visvo.com) | |
| Visicom Toolbar | |
| Vision Research Lab image spider at vision.ece.ucsb.edu | |
| VLC media player - version 0.8.5 Janus - (c) 1996-2006 the VideoLAN team | |
| VMBot/0.x.x (VMBot; http://www.VerticalMatch.com/; vmbot@tradedot.com) | |
| Vortex/2.2 (+http://marty.anstey.ca/robots/vortex/) | |
| voyager-hc/1.0 | |
| voyager/1.0 | |
| voyager/2.0 (http://www.kosmix.com/html/crawler.html) | |
| VSE/1.0 (mann@vivisimo.com) | |
| VSE/1.0 (myunn@vivisimo.com) | |
| VSE/1.0 (testcrawler@hotmail.com) | |
| VSE/1.0 (testcrawler@vivisimo.com) | |
| vspider | |
| vspider/3.x | |
| VWBOT/Nutch-0.9-dev (VWBOT Nutch Crawler; http://vwbot.cs.uiuc.edu;+vwbot@cs.uiuc.edu | |
| W3C-checklink/3.x.x.x libwww-perl/5.xx | |
| W3C-checklink/4.81 libwww-perl/5.836 | |
| W3C-checklink/4.x [4.xx] libwww-perl/5.xxx | |
| W3C-WebCon/5.x.x libwww/5.x.x | |
| W3CLineMode/5.4.0 libwww/5.x.x | |
| W3CRobot/5.4.0 libwww/5.4.0 | |
| W3C_Validator/1.xxx libwww-perl/5.xx | |
| w3m/0.x.xx | |
| W3SiteSearch Crawler_v1.1 http://www.w3sitesearch.de | |
| Wada.vn Vietnamese Search/2.1 | |
| wadaino.jp-crawler 0.2 (http://wadaino.jp/) | |
| WannaBe (Macintosh; PPC) | |
| WapOnWindows 1.0 | |
| Watchfire WebXM 1.0 | |
| WAVcheck 1.0.x (http://www.webbanalys.se/apps/WAVcheck/) | |
| Wavefire/0.8-dev (Wavefire; http://www.wavefire.com; info@wavefire.com) | |
| Waypath development crawler - info at waypath dot com | |
| Waypath Scout v2.x - info at waypath dot com | |
| WDG_Validator/1.1 | |
| Web Image Collector | |
| Web Link Validator 1.5 | |
| Web Snooper | |
| web-bekannt (Version: 1.02, powered by www.internetservice-franken.de) | |
| web-bekannt (Version: 1.02, powered by www.web-bekannt.de) | |
| Web-Bot V1.03 | |
| Web-Robot/5.0 (en-US; web-robot.com/policy.html) Web-Robot Crawler/2.0.3 | |
| web2express.org/Nutch-0.9-dev (leveled playing field; http://web2express.org/; info at web2express.org) | |
| WebAlta Crawler/1.2.1 (http://www.webalta.ru/bot.html) | |
| WebarooBot (Webaroo Bot; http://64.124.122.252/feedback.html) | |
| WebarooBot (Webaroo Bot; http://www.webaroo.com/rooSiteOwners.html) | |
| WebAuto/3.4xxx (WinNT; I) | |
| webbandit/4.xx.0 | |
| WebBug/5.x | |
| Webclipping.com | |
| webcollage/1.xx | |
| WebCompass 2.0 | |
| WebCopier vx.x | |
| WebCopier vx.xa | |
| WebCorp/1.0 | |
| webcrawl.net | |
| webcrawler (compatible; heritrix/1.14.4 ++http://www.onb.ac.at/about/webarchivierung.htm) | |
| WebDownloader for X x.xx | |
| Webdup/0.9 | |
| WebFetch | |
| webfetch/5.x.x | |
| WebFilter Robot 1.0 | |
| WebFilter Robot 1.x | |
| WebFindBot(http://www.web-find.com) | |
| Webglimpse 2.xx.x (http://webglimpse.net) | |
| webGobbler/1.x.x | |
| webhack | |
| WebImages 0.3 ( http://herbert.groot.jebbink.nl/?app=WebImages ) | |
| WebLight/4.x.x (support@illumit.com; http://www.illumit.com/Products/weblight/) | |
| Weblink's checker/ | |
| Weblog Attitude Diffusion 1.0 | |
| webmeasurement-bot, http://rvs.informatik.uni-leipzig.de | |
| WebMiner/x.x [en] (Win98; I) | |
| WeBoX/0.xx | |
| WebPix 1.0 (www.netwu.com) | |
| WebQL | |
| WebRACE/1.1 (University of Cyprus- Distributed Crawler) | |
| WebRankSpider/1.37 (+http://ulm191.server4you.de/crawler/) | |
| WebReaper vx.x - www.webreaper.net | |
| WebReaper [info@webreaper.net] | |
| WebReaper [webreaper@webreaper.net] | |
| WebSearch.COM.AU/3.0.1 (The Australian Search Engine; http://WebSearch.COM.AU; Search@WebSearch.COM.AU) | |
| WebSearchBench WebCrawler v0.1(Experimental) | |
| WebSearchBench WebCrawler V1.0 (Beta)- Prof. Dr.-Ing. Christoph Lindemann- Universität Dortmund- cl@cs.uni-dortmund.de- http://websearchbench.cs.uni-dortmund.de/ | |
| Website Explorer/0.9.x.x | |
| Website eXtractor | |
| WebsiteWorth v1.0 | |
| Webspinne/1.0 webmaster@webspinne.de | |
| Websquash.com (Add url robot) | |
| WebStat/1.0 (Unix; beta; 20040314) | |
| Webster v0.3 ( http://webster.healeys.net/ ) | |
| webster-internet.de pad browser | |
| WebStripper/2.xx | |
| WebTrafficExpress/x.0 | |
| WebTrends/3.0 (WinNT) | |
| WebVac (webmaster@pita.stanford.edu) | |
| WebVal/1.0 | |
| Webverzeichnis.de - Telefon: 01908 / 26005 | |
| WebVulnCrawl.unknown/1.0 libwww-perl/5.803 | |
| WebWatcherMonitor/2.01 | |
| WebZIP/x.x (http://www.spidersoft.com) | |
| Wells Search II | |
| WEP Search 00 | |
| WeSEE:Search/0.1 (Alpha, http://www.wesee.com/) | |
| WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/) | |
| West Wind Internet Protocols 4.xx | |
| WeViKa | |
| WFARC | |
| Wget/1.10.2 | |
| Wget/1.10.2 (Red Hat modified) | |
| Wget/1.11.1 | |
| Wget/1.11.4 | |
| Wget/1.11.4 Red Hat modified | |
| Wget/1.12 (linux-gnu) | |
| Wget/1.13.4 (darwin10.8.0) | |
| Wget/1.13.4 (darwin11.1.0) | |
| Wget/1.13.4 (darwin11.2.0) | |
| Wget/1.13.4 (darwin11.3.0) | |
| Wget/1.13.4 (linux-gnu) | |
| Wget/1.x(.x)GNU wget http://www.gnu.org/software/wget/wget.html - file downloader | |
| Wget/1.x+cvs-stable (Red Hat modified) | |
| Wget/1.x.x+cvs | |
| Whatsup/x.x | |
| whatUseek_winona/3.0 | |
| WhizBang! Lab | |
| wikiwix-bot-3.0 | |
| Wildsoft Surfer | |
| Willow Internet Crawler by Twotrees V2.1 | |
| WinampMPEG/2.00 (larbin@unspecified.mail) | |
| WincerSong Agent v1.0 | |
| Windows-Media-Player/10.00.00.xxxx | |
| WinGet 1.1 | |
| WinHTTP Example/1.0 | |
| WinkBot/0.06 (Wink.com search engine web crawler; http://www.wink.com/Wink:WinkBot; winkbot@wink.com) | |
| WinPodder (http://winpodder.com) | |
| WinWAP/3.x (3.x.x.xx; Win32) (Google WAP Proxy/1.0) | |
| Wir sind die Borg (Version: 1.03, Sie wurden Assimiliert +http://www.yammba.com/suchmaschine/bot.html) | |
| WIRE/0.11 (Linux; i686; Bot,Robot,Spider,Crawler,aromano@cli.di.unipi.it) | |
| WIRE/0.x (Linux; i686; Bot,Robot,Spider,Crawler) | |
| wirkon.com (Version: 1.06, powered by www.wirkon.com+http://www.wirkon.com) | |
| WISEbot/1.0 (WISEbot@koreawisenut.com; http://wisebot.koreawisenut.com) | |
| WiseWire-Spider2 | |
| wish-project (http://wish.slis.tsukuba.ac.jp/) | |
| woobot/1.1 | |
| WordChampBot | |
| WordPress/3.4-beta1; | |
| WordPress/3.4-beta2-20452; | |
| WordPress/x.x.x.x PHP/4.x.xx | |
| worio heritrix bot (+http://worio.com/) | |
| woriobot ( http://www.worio.com/) | |
| WorldLight | |
| WorQmada/1.0 | |
| Wotbox/2.0 (bot@wotbox.com; http://www.wotbox.com) | |
| Wotbox/alpha0.6 (bot@wotbox.com; http://www.wotbox.com) | |
| Wotbox/alpha0.x.x (bot@wotbox.com; http://www.wotbox.com) Java/1.4.1_02 | |
| WSB WebCrawler V1.0 (Beta)- cl@cs.uni-dortmund.de | |
| WSB, http://websearchbench.cs.uni-dortmund.de | |
| wttsearch/Nutch-1.0 (Crawler for SIFE (WTT); http://wtt.beecom.ch/wttsearch.html; wtt@beecom.ch) | |
| wume_crawler/1.1 (http://wume.cse.lehigh.edu/~xiq204/crawler/) | |
| Wusage/x.0@boutell.com | |
| Wwlib/Linux | |
| WWSBOT 1.x [--- http://www.analyzer.nu ---] | |
| WWW-Mechanize/1.1x | |
| www.arianna.it | |
| www.business-socket.com registry verify/1.x | |
| www.doweb.co.uk crawler | |
| www4mail/2.x libwww-FM/2.14 (Unix; I) | |
| WWWC/1.0x | |
| WWWeasel Robot v1.00 (http://wwweasel.de) | |
| WWWOFFLE/2.x | |
| wwwster/1.x (Beta- mailto:gue@cis.uni-muenchen.de) | |
| wxDownload Fast | |
| X-Crawler | |
| Xaldon WebSpider | |
| Xenu Link Sleuth 1.xx | |
| Xenu's Link Sleuth 1.x[a-z] | |
| Xerka WebBot v1.0.0 [UPVOpenDir] | |
| xine/1.0 | |
| xirq/0.1-beta (xirq; http://www.xirq.com; xirq@xirq.com) | |
| XMLSlurp/0.1 libwww-perl/5.805 | |
| XRL/2.00b1 (Linux; i686; en-us) (+http://metamark.net/about) | |
| Xylix | |
| xyro_(xcrawler@cosmos.inria.fr) | |
| Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html ) | |
| Y!J-BRJ/YATS crawler (http://listing.yahoo.co.jp/support/faq/int/other/other_001.html) | |
| Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) | |
| Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/blog-search/) | |
| Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) | |
| Y!J-SRD/1.0 | |
| Y!J/1.0 (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) | |
| Y!OASIS/TEST no-ad Mozilla/4.08 [en] (X11; I; FreeBSD 2.2.8-STABLE i386) | |
| Y!TunnelPro | |
| yacy (www.yacy.net; v20040602; i386 Linux 2.4.26-gentoo-r13; java 1.4.2_06; MET/en) | |
| yacybot (freeworld-global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 FreeBSD 8.2-STABLE; java 1.6.0; GMT/pl) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.16.46-0.12-smp; java 1.6.0_15; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.26-2-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-31-generic; java 1.7.0; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-34-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-342-ec2; java 1.6.0_20; Etc/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-37-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-37-server; java 1.6.0_26; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-38-generic; java 1.7.0; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-40-server; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_26; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-5-amd64; java 1.6.0_29; US/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-5-openvz-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/fr) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-6-pve; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.32-custom; java 1.6.0_26; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.37.6; java 1.6.0_25; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 2.6.38-13-generic; java 1.6.0_22; America/pt) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-12-generic; java 1.6.0_26; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-12-server; java 1.6.0_23; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-12-server; java 1.6.0_26; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-14-generic; java 1.6.0_23; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-15-server; java 1.6.0_23; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-15-server; java 1.6.0_26; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.0.0-16-generic; java 1.6.0_23; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.1.0-1.2-desktop; java 1.6.0_22; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.1.0-1.2-desktop; java 1.6.0_22; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.1.10-1.9-default; java 1.6.0_24; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.1.10-hardened; java 1.7.0_03-icedtea; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.1.1; java 1.7.0_147-icedtea; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.1.2-1.fc16.x86_64; java 1.6.0_22; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.1.9-1.4-default; java 1.6.0_24; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.0-1-amd64; java 1.6.0_24; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.0-10-generic; java 1.7.0_147-icedtea; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.0-18-generic; java 1.7.0_03-icedtea; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.0-2-rt-amd64; java 1.6.0_24; Asia/ja) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.0-21-generic; java 1.7.0_03-icedtea; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.0; java 1.7.0_147-icedtea; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.1-2-ARCH; java 1.7.0_147-icedtea; Europe/fr) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_22; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.1-gentoo-r2; java 1.6.0_24; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.13-1-ARCH; java 1.7.0_03-icedtea; Europe/fr) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.2-1-ARCH; java 1.7.0_147-icedtea; Europe/fr) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.2.2-1.fc16.x86_64; java 1.7.0_b147-icedtea; W-SU/ru) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Linux 3.3.1-1-ARCH; java 1.6.0_24; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_23; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_25; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_26; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_26; Europe/fr) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.6.0_29; Europe/no) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows 7 6.1; java 1.7.0; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows Server 2008 6.0; java 1.7.0_03; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; amd64 Windows Server 2008 R2 6.1; java 1.6.0_29; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.32-39-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.32-4-pve; java 1.6.0_18; Etc/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 2.6.37-gentoo-r4-110510; java 1.6.0_22; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 3.0.0-12-generic-pae; java 1.6.0_20; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 3.0.0-12-generic-pae; java 1.6.0_23; Europe/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 3.0.0-13-generic-pae; java 1.6.0_23; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 3.0.0-15-generic-pae; java 1.6.0_23; America/fr) http://yacy.net/bot.html | |
| yacybot (freeworld/global; i386 Linux 3.0.0-16-generic; java 1.6.0_23; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; x86 Windows 2003 5.2; java 1.6.0_30; America/en) http://yacy.net/bot.html | |
| yacybot (freeworld/global; x86 Windows 7 6.1; java 1.7.0_01; Europe/de) http://yacy.net/bot.html | |
| yacybot (freeworld/global; x86 Windows XP 5.1; java 1.6.0_29; Europe/es) http://yacy.net/bot.html | |
| yacybot (freeworld/global; x86 Windows XP 5.1; java 1.6.0_31; Europe/sv) http://yacy.net/bot.html | |
| yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Asia/ru) http://yacy.net/bot.html | |
| yacybot (freeworld/global; x86_64 Mac OS X 10.6.8; java 1.6.0_29; Europe/ru) http://yacy.net/bot.html | |
| yacybot (webportal-global; amd64 Linux 2.6.32-5-xen-amd64; java 1.6.0_18; Europe/en) http://yacy.net/bot.html | |
| yacybot (webportal-global; amd64 Linux 3.0.0-17-generic; java 1.7.0_147-icedtea; Europe/de) http://yacy.net/bot.html | |
| yacybot (webportal/global; amd64 Windows 7 6.1; java 1.7.0_02; Europe/de) http://yacy.net/bot.html | |
| yacybot (webportal/global; i386 Linux 2.6.35-28-generic; java 1.6.0_20; America/en) http://yacy.net/bot.html | |
| yacybot (x86 Windows XP 5.1; java 1.5.0_06; Europe/de) yacy.net | |
| Yahoo Pipes 1.0 | |
| Yahoo! Mindset | |
| Yahoo-Blogs/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html ) | |
| Yahoo-MMAudVid/1.0 (mms dash mmaudvidcrawler dash support at yahoo dash inc dot com) | |
| Yahoo-MMAudVid/2.0(mms dash mm aud vid crawler dash support at yahoo dash inc.com ;Mozilla 4.0 compatible; MSIE 7.0;Windows NT 5.0; .NET CLR 2.0) | |
| Yahoo-MMCrawler/3.x (mm dash crawler at trd dot overture dot com) | |
| Yahoo-Test/4.0 | |
| Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler | |
| YahooFeedSeeker/2.0 (compatible; Mozilla 4.0; MSIE 5.5; http://publisher.yahoo.com/rssguide) | |
| YahooSeeker-Testing/v3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/) | |
| YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/) | |
| YahooSeeker/1.0 (compatible; Mozilla 4.0; MSIE 5.5; http://search.yahoo.com/yahooseeker.html) | |
| YahooSeeker/1.1 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/shop/merchant/) | |
| YahooSeeker/bsv3.9 (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.com/help/us/ysearch/crawling/crawling-02.html ) | |
| YahooSeeker/CafeKelsa-dev (compatible; Konqueror/3.2; FreeBSD ;cafekelsa-dev-webmaster@yahoo-inc.com ) | |
| Yandex/1.01.001 (compatible; Win16; I) | |
| YandexBlogs | |
| Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/) | |
| yarienavoir.net/0.2 | |
| Yepi/1.0 (NHN Corp.; http://help.naver.com/robots/) | |
| Yeti | |
| Yeti/0.01 (nhn/1noon, yetibot@naver.com, check robots.txt daily and follows it) | |
| Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/) | |
| yggdrasil/Nutch-0.9 (yggdrasil biorelated search engine; www dot biotec dot tu minus dresden do de slash schroeder; heiko dot dietze at biotec dot tu minus dresden dot de) | |
| YodaoBot/1.0 (http://www.yodao.com/help/webmaster/spider/; ) | |
| yolinkBot | |
| yoofind/yoofind-0.1-dev (yoono webcrawler; http://www.yoono.com ; MyEmail) | |
| yoogliFetchAgent/0.1 | |
| yoono/1.0 web-crawler/1.0 | |
| YottaCars_Bot/4.12 (+http://www.yottacars.com) Car Search Engine | |
| YottaShopping_Bot/4.12 (+http://www.yottashopping.com) Shopping Search Engine | |
| YTunnelPro | |
| Z-Add Link Checker (http://w3.z-add.co.uk/linkcheck/) | |
| Zao-Crawler | |
| Zao-Crawler 0.2b | |
| Zao/0.1 (http://www.kototoi.org/zao/) | |
| ZBot/1.00 (icaulfield@zeus.com) | |
| zclosure/Nutch-1.3 | |
| zclosure/Nutch-1.4 | |
| Zearchit | |
| ZeBot_lseek.net (bot@ze.bz) | |
| ZeBot_www.ze.bz (ze.bz@hotmail.com) | |
| zedzo.digest/0.1 (http://www.zedzo.com/) | |
| Zend_Http_Client | |
| zermelo Mozilla/5.0 compatible; heritrix/1.12.1 (+http://www.powerset.com) [email:crawl@powerset.com,email:paul@page-store.com] | |
| zerxbot/Version 0.6 libwww-perl/5.79 | |
| Zeus ThemeSite Viewer Webster Pro V2.9 Win32 | |
| Zeus xxxxx Webster Pro V2.9 Win32 | |
| Zeusbot/0.07 (Ulysseek's web-crawling robot; http://www.zeusbot.com; agent@zeusbot.com) | |
| Ziggy -- The Clown From Hell!! | |
| Zillsy.com Search engine /Nutch-1.4 | |
| ZipppBot/0.xx (ZipppBot; http://www.zippp.net; webmaster@zippp.net) | |
| ZIPPPCVS/0.xx (ZipppBot/.xx;http://www.zippp.net; webmaster@zippp.net) | |
| Zippy v2.0 - Zippyfinder.com | |
| Zoo Tycoon 2 Client -- http://www.zootycoon.com | |
| ZoomSpider - wrensoft.com | |
| ZoomSpider - wrensoft.com [ZSEBOT] | |
| Zscho.de Crawler/Nutch-1.0-Zscho.de-semantic_patch (Zscho.de Crawler | |
| zspider/0.9-dev http://feedback.redkolibri.com/ | |
| ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry) | |
| ZyBorg/1.0 (ZyBorg@WISEnut.com; http://www.WISEnut.com) | |
| _USERAGENT_ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment