Skip to content

Instantly share code, notes, and snippets.

@graysuit

graysuit/Bots & Crawlers User-Agent list.txt

Last active January 6, 2021 17:45
Show Gist options
  • Save graysuit/853116f0bda96e5ac18afd107f340376 to your computer and use it in GitHub Desktop.
Save graysuit/853116f0bda96e5ac18afd107f340376 to your computer and use it in GitHub Desktop.
Download ZIP
Nowadays, every professional website have their own bots and crawlers. These are used to fetch results from link/URL when a user share them. They track links to avoid phishing, scams and also for better link preview.
Raw
Bots & Crawlers User-Agent list.txt
๐†๐จ๐จ๐ ๐ฅ๐ž:
Googlebot
Googlebot/2.1
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
๐…๐š๐œ๐ž๐›๐จ๐จ๐ค & ๐ˆ๐ง๐ฌ๐ญ๐š๐ ๐ซ๐š๐ฆ & ๐Œ๐ž๐ฌ๐ฌ๐ž๐ง๐ ๐ž๐ซ:
Facebot
facebookexternalhit
facebookexternalhit/1.1
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
๐“๐ฐ๐ข๐ญ๐ญ๐ž๐ซ:
Twitterbot
Twitterbot/1.1
๐“๐ฎ๐ฆ๐›๐ฅ๐ข๐ซ:
Tumblr/14.0.835.186
๐‹๐ข๐ง๐ค๐ž๐๐ข๐ง:
LinkedInBot
LinkedInBot/1.0
LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)
๐’๐ง๐š๐ฉ๐œ๐ก๐š๐ญ:
Snapchat
Snapchat/10.12.0.2
Snapchat/10.12.0.2 (iPhone8,1; iOS 10.3.2; gzip)
๐˜๐š๐ก๐จ๐จ:
Slurp
Yahoo! Slurp
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
๐Œ๐ข๐œ๐ซ๐จ๐ฌ๐จ๐Ÿ๐ญ:
Microsoft-BotFramework
Microsoft-BotFramework/3.0
BF-DirectLine/3.0 (Microsoft-BotFramework/3.0 +https://botframework.com/ua)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment