Skip to content

Instantly share code, notes, and snippets.

@acka47

acka47/metadaten-community-webcrawlers_2024-01-bis-08.csv

Created August 13, 2024 08:45
Show Gist options
  • Save acka47/9aab25892728d2a3cb7e4b62e77043f9 to your computer and use it in GitHub Desktop.
Save acka47/9aab25892728d2a3cb7e4b62e77043f9 to your computer and use it in GitHub Desktop.
Download ZIP
A Discourse export of web crawler user agents requesting metadaten.community from 2024-01-01 to 2024-08-13
Raw
metadaten-community-webcrawlers_2024-01-bis-08.csv
Browserkennung Seitenaufrufe
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) 340
serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/; abuse@serpstatbot.com) 267
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) 255
Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html) 235
Mozilla/5.0 (compatible; vebidoobot/1.0; +https://blog.vebidoo.de/vebidoobot/) 232
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) 176
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 164
Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com) 161
Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/) 148
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36 144
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) 125
Monit/5.31.0 121
Mozilla/5.0 (compatible; SeznamBot/4.0; +http://napoveda.seznam.cz/seznambot-intro/) 113
curl/8.6.0 101
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) 95
CCBot/2.0 (https://commoncrawl.org/faq/) 71
Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot) 52
http.rb/5.1.1 (Mastodon/4.2.10; +https://openbiblio.social/) 51
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.126 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 45
Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler) 38
Dalvik/2.1.0 (Linux; U; Android 9.0; ZTE BA520 Build/MRA58K) 35
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) 33
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.182 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 31
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.99 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 28
Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler) 27
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.126 Mobile Safari/537.36 (compatible; GoogleOther) 26
Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com) 26
SummalyBot/2.7.0 23
SerendeputyBot/0.8.6 (http://serendeputy.com/about/serendeputy-bot) 22
Discourse Forum Onebox v3.3.0.beta5-dev 22
DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html) 17
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) 17
Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36 14
SummalyBot/5.1.0 14
Nextcloud Server Crawler 11
10
SummalyBot/5.0.3 10
Go-http-client/1.1 9
Mozilla/5.0 (compatible; ips-agent) 9
python-requests/2.32.3 9
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.182 Mobile Safari/537.36 (compatible; GoogleOther) 9
Discourse Forum Onebox v3.4.0.beta1-dev 8
Synapse (bot; +https://github.com/matrix-org/synapse) 8
AcademicBotRTU (https://academicbot.rtu.lv; mailto:caps@rtu.lv) 8
WebexTeams 7
python-requests/2.31.0 6
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com) 6
Pleroma 2.7.0-0-g711ec006e-develop; https://social.metadata.moe <pleroma@metadata.moe> 5
Iceshrimp/2023.12.8-dev-9a8b7efcd (https://fedia.social) 5
Owler (ows.eu/owler) 5
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.6422.175 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 5
SummalyBot/2.5.0 5
http.rb/5.1.1 (Mastodon/4.2.10; +https://chaos.social/) 5
python-requests/2.28.2 4
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.99 Mobile Safari/537.36 (compatible; GoogleOther) 4
http.rb/5.1.1 (Mastodon/4.2.10; +https://smnn.ch/) Bot 4
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) 4
Pleroma 2.6.3-0-gee6ba27ed-develop; https://social.metadata.moe <pleroma@metadata.moe>; Bot 4
python-requests/2.28.2,gzip(gfe) 4
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0 4
Apache-HttpClient/4.5.6 (Java/1.8.0_412) 3
Mastodon/4.3.0-nightly.2024-08-09 (http.rb/5.2.0; +https://mastodon.social/) 3
node 3
http.rb/5.1.1 (Mastodon/4.2.9; +https://m.flaem.ing/) 3
2ip bot/1.1 (+http://2ip.io) 3
Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers&#39; presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com 3
Mozilla/5.0 (compatible; CensysInspect/1.1; +https://about.censys.io/) 3
http.rb/5.1.1 (Mastodon/4.2.10; +https://det.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://fediscience.org/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://archaeo.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.1.18; +https://mastodon.cloud/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10+litera.tools; +https://literatur.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://swiss.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10-stable+ff1; +https://climatejustice.global/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://mstdn.io/) Bot 2
http.rb/5.2.0 (Mastodon/4.3.0-alpha.5+glitch; +https://neuromatch.social/) Bot 2
python-requests/2.32.2 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://framapiaf.org/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://newsie.social/) Bot 2
http.rb/5.2.0 (Mastodon/4.3.0-alpha.5+glitch; +https://eldritch.cafe/) Bot 2
Apache-HttpClient/4.5.6 (Java/1.8.0_422) 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://norden.social/) Bot 2
http.rb/5.2.0 (Mastodon/4.3.0-alpha.5+glitch; +https://infosec.exchange/) Bot 2
Mozilla/5.0 (compatible; InternetMeasurement/1.0; +https://internet-measurement.com/) 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://indieweb.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://suma-ev.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://artsandculture.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://defcon.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://mstdn.party/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://mastodon.radio/) Bot 2
ZoomBot (Linkbot 1.0 http://suite.seozoom.it/bot.html) 2
Mozilla/5.0 (compatible) facebookexternalhit/1.1 (+https://faviconkit.com/robots) 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://gilbertredman.masto.host/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://digitalcourage.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10+hometown-1.1.1; +https://digipres.club/) Bot 2
http.rb/5.1.1 (Mastodon/4.1.18; +https://noc.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://ieji.de/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://wikis.world/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://troet.cafe/) Bot 2
http.rb/5.1.1 (Mastodon/4.1.18; +https://nfdi.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://toot.wales/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://toot.io/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://mastodon.xyz/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://piaille.fr/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://mamot.fr/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://econtwitter.net/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://mastodon.nl/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://fosstodon.org/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://social.tchncs.de/) Bot 2
FreshRSS/1.24.1 (Linux; https://freshrss.org) 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://hachyderm.io/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://gruene.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://23.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://xerrem.xyz/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://social.yakshed.org/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://hessen.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://social.coop/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://mas.to/) Bot 2
http.rb/4.4.1 (Mastodon/3.2.1; +https://qoto.org/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://toot.community/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://code4lib.social/) Bot 2
http.rb/5.1.0 (Mastodon/4.0.2; +https://libraryland.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://chaos.social/) Bot 2
http.rb/5.2.0 (Mastodon/4.3.0-alpha.5+glitch; +https://octodon.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10+hometown-1.1.1; +https://digipres.club/) 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://mastodon-belgium.be/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://ecoevo.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://sueden.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://ruhr.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://fedihum.org/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10+hometown-1.1.1; +https://ausglam.space/) Bot 2
http.rb/5.2.0 (Mastodon/4.3.0-alpha.5+glitch; +https://tech.lgbt/) Bot 2
http.rb/5.0.4 (Mastodon/3.5.19; +https://ondergrond.org/) Bot 2
http.rb/5.1.1 (Mastodon/4.1.18; +https://mastodon.sdf.org/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://vorlon.space/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://techhub.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://sigmoid.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://masto.ai/) Bot 2
http.rb/5.2.0 (Mastodon/4.3.0-alpha.5; +https://mstdn.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://sfba.social/) Bot 2
http.rb/5.0.4 (Mastodon/3.5.17+hometown-1.0.8; +https://weirder.earth/) Bot 2
http.rb/5.1.0 (Mastodon/4.0.15+hometown-1.1.1; +https://merveilles.town/) Bot 2
fasthttp 2
Twitterbot/1.0 2
http.rb/5.1.1 (Mastodon/4.1.18; +https://mstdn.mx/) Bot 2
Dalvik/2.1.0 (Linux; U; Android 11; Redmi Note 8 Build/RQ3A.211001.001) 2
http.rb/5.2.0 (Mastodon/4.3.0-alpha.5+glitch; +https://strangeobject.space/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://social.cologne/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.3; +https://flashist.video/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://scicomm.xyz/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://mastodon.scot/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://social.wikimedia.de/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://kolektiva.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://zirk.us/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10+hometown-1.1.1; +https://post.lurk.org/) Bot 2
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://nrw.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://akademienl.social/) Bot 2
Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com) 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://mastodon.green/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://wandering.shop/) Bot 2
Mastodon/4.3.0-nightly.2024-08-06 (http.rb/5.2.0; +https://ohai.social/) Bot 2
GlobalWebSearchx 2
Mastodon/4.3.0-alpha.5 (http.rb/5.2.0; +https://aus.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://vis.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://mastodon.art/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://metalhead.club/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://degrowth.social/) Bot 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://c.im/) Bot 2
LivelapBot/0.2 (http://site.livelap.com/crawler) 2
http.rb/5.1.1 (Mastodon/4.2.10; +https://toad.social/) Bot 1
http.rb/5.1.1 (Mastodon/4.1.18; +https://mastodon.indie.host/) Bot 1
http.rb/5.1.1 (Mastodon/4.2.10; +https://social.bau-ha.us/) Bot 1
http.rb/5.1.1 (Mastodon/4.2.10; +https://queer.party/) Bot 1
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; FriendlyCrawler/1.0) Chrome/120.0.6099.216 Safari/605.1.15 1
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot 1
http.rb/5.1.1 (Mastodon/4.2.10; +https://zeroes.ca/) Bot 1
http.rb/5.1.1 (Mastodon/4.2.10; +https://rockosbasilisk.com/) Bot 1
http.rb/5.1.1 (Mastodon/4.2.10; +https://r.town/) Bot 1
http.rb/5.1.1 (Mastodon/4.2.10; +https://kamu.social/) Bot 1
http.rb/5.1.1 (Mastodon/4.2.10; +https://ischool.social/) Bot 1
http.rb/5.1.1 (Mastodon/4.2.10; +https://anticapitalist.party/) Bot 1
http.rb/5.1.1 (Mastodon/4.2.10; +https://ps.s10y.eu/) Bot 1
PycURL/7.45.3 libcurl/7.61.1 OpenSSL/1.1.1k zlib/1.3 brotli/1.0.6 libidn2/2.2.0 libpsl/0.20.2 (+libidn2/2.2.0) libssh/0.9.6/openssl/zlib nghttp2/1.33.0 1
http.rb/5.1.1 (Mastodon/4.2.10; +https://mindly.social/) Bot 1
http.rb/5.1.1 (Mastodon/4.2.10; +https://23.illuminati.org/) Bot 1
http.rb/5.2.0 (Mastodon/4.3.0-alpha.5+vegan; +https://veganism.social/) Bot 1
http.rb/5.1.1 (Mastodon/4.2.10; +https://awscommunity.social/) Bot 1
Mastodon/4.3.0-alpha.5+glitch+COTS (http.rb/5.2.0; +https://cupoftea.social/) Bot 1
UCWEB/2.0 (Linux; U; Adr 4.4.4; zh-CN; HUAWEI C8817E) U2/1.0.0 UCBrowser/9.4.1.362 U2/1.0.0 Mobile 1
http.rb/5.1.1 (Mastodon/4.2.10; +https://toot.berlin/) Bot 1
http.rb/5.1.1 (Mastodon/4.1.18; +https://mastodon.libre-entreprise.com/) Bot 1
http.rb/5.1.1 (Mastodon/4.2.10; +https://mastodon.mit.edu/) Bot 1
http.rb/5.1.1 (Mastodon/4.3.0-nightly.2024-02-10+glitch; +https://glitch.hanntac.net/) Bot 1
http.rb/5.1.1 (Mastodon/4.1.16; +https://geekdom.social/) Bot 1
Pleroma 2.6.3; https://fed.xnor.in <xnor-fed@xnor.in>; Bot 1
http.rb/5.1.1 (Mastodon/4.2.10; +https://mastodon.nz/) Bot 1
http.rb/5.1.1 (Mastodon/4.2.10; +https://c18.masto.host/) Bot 1
http.rb/5.1.1 (Mastodon/4.2.10; +https://lingo.lol/) Bot 1
http.rb/5.1.1 (Mastodon/4.2.10; +https://universeodon.com/) Bot 1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment