11/9/15 DOS Attack access.log investigation

ddos_11_9_15_notes.md

number of lines in logfile (aka num requests)

$ wc -l access.log 376359 access.log

first line of logfile

$ head -n1 access.log 172.31.16.226 - - [07/Nov/2016:00:00:02 +0000] "GET /d3bt6306j428ad.cloudfront.net/assets/embedded-ac9659c5db13c873c87adade48e8ce4ef71dd7532f860dec198757b3f8622d7e.css HTTP/1.1" 301 0 "https://screendoor.dobt.co/d3bt6306j428ad.cloudfront.net/assets" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

last line of logfile

$ tail -n1 access.log 172.31.16.226 - - [07/Nov/2016:17:18:25 +0000] "GET /api/form_renderer/load?v=0&project_id=1610 HTTP/1.1" 200 13464 "http://thoughtcatalog.com/submissions/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"

first 10 timestamps extracted

$ head access.log | grep -o "[.*]" [07/Nov/2016:00:00:02 +0000] [07/Nov/2016:00:00:02 +0000] [07/Nov/2016:00:00:02 +0000] [07/Nov/2016:00:00:02 +0000] [07/Nov/2016:00:00:03 +0000] [07/Nov/2016:00:00:03 +0000] [07/Nov/2016:00:00:03 +0000] [07/Nov/2016:00:00:03 +0000] [07/Nov/2016:00:00:03 +0000] [07/Nov/2016:00:00:04 +0000]

all timestamps extracted and written to timestamps.txt

$ grep -o "[.*+0000]" access.log > timestamps.txt

first ip address (remote_addr of request) extracted

note: these are the aws load balancers

$ head -n1 access.log | cut -d "-" -f1 172.31.16.226

all ip addresses written to ip.txt

$ cut -d "-" -f1 access.log > ip.txt

ip addresses combined with their timestamps written to file

note: not useful, these are aws load balancers

$ paste -d, ip.txt timestamps.txt > ip_with_ts.csv

user agents extracted from access logs

$ grep -o ""[^\"]*"$" access.log > agents.txt

user agents along with occurrence counts, sorted

$ sort agents.txt | uniq -c | sort -n > req_by_agent.txt