Checking for 404s in Apache log

404.md

Get a list of 404s in the accesslogs:

sudo -s
grep "HTTP/1.1\" 404" /var/log/httpd/access_log | awk '{print $7 } ' | sort | uniq -c | sort -n > 404s.txt

another way

cut -d'"' -f2,3 /var/log/httpd/access_log | awk '$4=404{print $4" "$2}' | sort | uniq -c | sort -rg > 404s.txt

Checking status of links for 404s:

while read LINE; do
  curl -o /dev/null --silent --head --write-out '%{http_code}' "$LINE"
  echo " $LINE"
done < 404s.txt

A better way:

#!/bin/bash
filename='404s.txt'
filelines=`cat $filename`
echo Start
for line in $filelines ; do
    curl -o /dev/null --silent --head --write-out '%{http_code}' "$line"
    echo ",$line"
done