Last active
February 7, 2021 20:56
-
-
Save antonydevanchi/6199632a64bf2ae2ffeb88c450390f46 to your computer and use it in GitHub Desktop.
Parse custom Laravel logs and get list with only unique User-Agents
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Okay. | |
| # It's just a few lines in «shitbash» language. | |
| # Shitbash is awesome! It's great tool for scripts which you run once. | |
| # So... Oh, yeah! You can go fuck yourself if you want say something about quality of this line. | |
| cat storage/logs/laravel.log | grep 'User-Agent:' | sort --unique | |
| User-Agent: node-fetch/1.0 (+https://github.com/bitinn/node-fetch) | |
| User-Agent: Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.2.15 Version/10.01 | |
| User-Agent: Pandalytics/1.0 (https://domainsbot.com/pandalytics/) | |
| User-Agent: Python/3.7 aiohttp/3.0.9 | |
| User-Agent: Python/3.7 aiohttp/3.4.4 | |
| User-Agent: Relap fetcher | |
| User-Agent: rogerbot/1.1 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+madagascarmoz.com) | |
| User-Agent: rogerbot/1.1 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+onpage@moz.com) | |
| User-Agent: rogerbot/1.1 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+vanguard@moz.com) | |
| User-Agent: rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-api-01@moz.com) | |
| User-Agent: rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-api-01@moz.com) | |
| User-Agent: rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-20@moz.com) | |
| User-Agent: rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-production-crawler-54@moz.com) | |
| User-Agent: Ruby | |
| User-Agent: Ruby | |
| User-Agent: saelmon | |
| User-Agent: SEMrushBot | |
| User-Agent: SemrushBot-CT Mozilla/5.0 (compatible; SemrushBot-CT/0.97; +http://www.semrush.com/bot.html) | |
| User-Agent: SiteSucker for macOS/3.2 | |
| User-Agent: User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:66.0) Gecko/20100101 Firefox/66.0 | |
| User-Agent: User-Agent" "Mozilla/5.0 (Windows NT 6.1;) Gecko/20100101 Firefox/13.0.1 | |
| User-Agent: WebMoney MegaStock robot | |
| User-Agent: Zadarma API | |
| awk -F"\"" '{print $6}' '/var/log/nginx/access.log' | sort --unique | |
| Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36 | |
| Mozilla/5.0 (Windows NT 6.3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 YaBrowser/18.4.1.871 Yowser/2.5 Safari/537.36 | |
| Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 | |
| Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 YaBrowser/18.4.1.871 Yowser/2.5 Safari/537.36 | |
| Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 YaBrowser/18.4.1.872 Yowser/2.5 Safari/537.36 | |
| Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36 | |
| Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon | |
| Mozilla/5.0 (iPhone; CPU iPhone OS 11_3 like Mac OS X) AppleWebKit/604.1.34 (KHTML, like Gecko) CriOS/67.0.3396.69 Mobile/15E216 Safari/604.1 | |
| Mozilla/5.0(WindowsNT6.1;rv:31.0)Gecko/20100101Firefox/31.0 | |
| Zabbix | |
| masscan/1.0 (https://github.com/robertdavidgraham/masscan) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment