Skip to content

Instantly share code, notes, and snippets.

@ethangardner
Last active June 28, 2023 19:04
Show Gist options
  • Save ethangardner/aba699e9fc1b948db975e3c9894c360f to your computer and use it in GitHub Desktop.
Save ethangardner/aba699e9fc1b948db975e3c9894c360f to your computer and use it in GitHub Desktop.
Useful awk commands for extracting data from logs

$1 is the first column. awk separates columns by spaces by default. delimiters can be specified with -F. In the examples below '[: ]' means that columns are delimited by spaces or colons.

99.56.8.181 10.0.1.239 - - [16/Nov/2018:20:45:59 +0000] "GET /app/themes/finecooking/dist/img/marketing-hero-cover.jpg HTTP/1.0" 200 38808 "https://www.finecooking.com/sw.js" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0"

Print IP and user agent for requests between the hours of 18 and 19

cat /var/log/httpd/access_log | awk -F'[: ]' '$6 >= 18 && $6 <= 19 { print }' | awk -F\" '{print $1,$6}' | more

Print entire log entry for requests between the hours of 18 and 19 and status code is 301

cat /var/log/httpd/access_log | awk -F'[: ]' '$6 >= 18 && $6 <= 19 && $13 == 301 { print }' | more

Find referrers matching a pattern

cat /var/log/httpd/access_log | awk -F'"' '$4~/(menshealth\.com|fitnessmagazine\.com|seriouseats\.com|giants\.com|soaphub\.com|tmz\.com|bleacherreport\.com)/ {print $1,$4}' | more

Find requests matching a pattern

cat /var/log/httpd/access_log | awk -F'"' '$2~/\/search\?q=/ {print}' | more

More links

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment