Script to test for 404s

404-check.sh

#!/bin/sh

# 404-check.sh - Script to test for 404s
# author: lkptrzk

# Usage:
#  Assuming a file named 'input' with one URL per line
#  the following command will output which files were 404s:
#     cat input | xargs 404-check.sh

# Notes:
#  curl -I = only get headers
#  2>/dev/null = prevent stderr from going through the pipe
#                (causes problems when piping curl)
#  head -1 = HTTP response code should be first line of output from curl

for arg in $@ ; do

    currUrl=$arg

    curl -I 2>/dev/null $currUrl | head -1 | grep 404 >/dev/null

    if [ $? == 0 ] ; then
        echo $currUrl
    fi

done

Just a note: because of the head -1, this script doesn't currently support redirects.

Perhaps you could try: curl -I 2>/dev/null $currUrl | grep "HTTP/" | tail -1 | grep 404 >/dev/null instead?