Skip to content

Instantly share code, notes, and snippets.

@grifferz

grifferz/gist:d26d9c4625846c16bd0c5951376ab49f

Created July 3, 2024 23:42
Show Gist options
  • Save grifferz/d26d9c4625846c16bd0c5951376ab49f to your computer and use it in GitHub Desktop.
Save grifferz/d26d9c4625846c16bd0c5951376ab49f to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile0.txt
1681371 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
450034 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)
30587 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; FriendlyCrawler/1.0) Chrome/120.0.6099.216 Safari/605.1.15
9100 Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
7783 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36
5716 Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)
5050 Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)
3369 Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
2997 Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.0.0 Safari/537.36
2640 Scrapy/2.11.2 (+https://scrapy.org)
1669 -
1608 Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
1329 Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
1240 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
789 Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.0) Gecko/20100101 Firefox/127.0
767 LWP::Simple/6.61 libwww-perl/6.61
737 Feedly/1.0 (+http://www.feedly.com/fetcher.html; 39 subscribers; )
728 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36
712 Mozilla/5.0 (X11; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0
641 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
620 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.6422.175 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
532 Tiny Tiny RSS/21.07-73d14338a (http://tt-rss.org/)
499 Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Mobile Safari/537.36
481 Mozilla/5.0 (iPhone; CPU iPhone OS 17_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1
476 NextCloud-News/1.0
446 Go-http-client/1.1
442 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 Edg/126.0.0.0
416 Mozilla/5.0 (compatible; Miniflux/2.1.3; +https://miniflux.app)
415 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
413 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
406 Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0
401 Tiny Tiny RSS/24.05-8b037e81 (https://tt-rss.org/)
399 Tiny Tiny RSS/24.02-fc95c988 (https://tt-rss.org/)
391 Tiny Tiny RSS/23.08-faefedb95 (Unsupported) (https://tt-rss.org/)
383 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
361 Tiny Tiny RSS/24.02-81f3139 (Unsupported) (https://tt-rss.org/)
356 Tiny Tiny RSS/1.15.3.945346c (http://tt-rss.org/)
325 Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0 (compatible; img2dataset; +https://github.com/rom1504/img2dataset)
318 Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
309 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:127.0) Gecko/20100101 Firefox/127.0
300 Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)
287 ircplanet
266 Akregator/5.24.5 (23.08.5); syndication
258 FreshRSS/1.23.1 (Linux; https://freshrss.org)
254 Custom-AsyncHttpClient
240 Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/111.0
226 Feedbin feed-id:1533256
221 UniversalFeedParser/5.2.1 +https://code.google.com/p/feedparser/
211 Tiny Tiny RSS/23.09-b15f185e3 (Unsupported) (https://tt-rss.org/)
211 Tiny Tiny RSS/23.06-dc25a9cf6 (Unsupported) (https://tt-rss.org/)
211 Tiny Tiny RSS/23.03-d210ae50a (Unsupported) (https://tt-rss.org/)
206 SpaceCowboys Android RSS Reader / 2.6.28(313)
194 Feedbin feed-id:1497364
192 LWP::Simple/6.31 libwww-perl/6.31
190 Tiny Tiny RSS/24.06-bcbbd53a8 (Unsupported) (https://tt-rss.org/)
187 Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html)
184 maubot/0.4.2 +https://github.com/maubot/rss
183 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36
174 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36
171 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15
168 Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0
157 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36
154 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/115.0
150 Mozilla/5.0 (compatible; ScrapeheroBot/1.0; +https://scrapehero.de/)
146 FreshRSS/1.21.0 (Linux; https://freshrss.org)
142 facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
141 Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com)
133 Mozilla/5.0 (X11; Linux x86_64) KIO/5.115 akregator/5.24.5 (23.08.5)
131 NetNewsWire (RSS Reader; https://netnewswire.com/)
122 WordPress/4.8-beta1-40787; https://strugglers.net/~andy/blog
121 FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)
117 Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Vivaldi/5.3.2679.68
116 okhttp/4.10.0
116 Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.66 Safari/537.36
115 Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Mobile Safari/537.36
107 curl/7.54.0
105 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36
104 Newsboat/2.36.0 (Linux x86_64)
104 Mozilla/5.0 (X11; Linux x86_64; rv:20.0; Favicon; +https://github.com/ArthurHoaro/favicon) Gecko/20100101 Firefox/78.0
104 Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko
101 Mozilla/5.0 (X11; Linux x86_64; rv:125.0) Gecko/20100101 Firefox/125.0
101 Mozilla/5.0 (Windows NT 10.0; rv:126.0) Gecko/20100101 Firefox/126.0
100 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36
100 Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/126.0.6478.153 Mobile/15E148 Safari/604.1
100 Mozilla/5.0 (compatible; Miniflux/2.1.0; +https://miniflux.app)
96 NewsBlur Feed Fetcher
96 Inoreader/1.0 (+http://www.inoreader.com/feed-fetcher; 6 subscribers; )
95 Inoreader/1.0 (+http://www.inoreader.com/feed-fetcher; 2 subscribers; )
94 Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0
93 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36
88 Mozilla/5.0 (X11; Linux x86_64; rv:6.0.1) Gecko/20110831 conkeror/0.9.3
86 Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
84 Mozilla/5.0 (Android 7.0; Mobile; rv:127.0) Gecko/127.0 Firefox/127.0
83 Mozilla/5.0 (Windows NT 10.0; rv:127.0) Gecko/20100101 Firefox/127.0
82 Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:115.0) Gecko/20100101 Thunderbird/115.12.2
80 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36
80 Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)
76 Mozilla/5.0 (compatible; SeznamBot/4.0; +http://napoveda.seznam.cz/seznambot-intro/)
75 Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36
75 Iframely/1.2.7 (+http://iframely.com/;)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment