Check multiple URLs from file using bash

check-urls-from-file.bash

#!/bin/bash

#
# How to check status of URLs from text file
# https://stackoverflow.com/a/25136723
#
# This script reads a file with URLs (line-by-line) and checks their http status code
# The results will be written to a temporary file locaed in ~/tmp
#

# takes first argument for file with URLs (line-by-line)
URLS_FILE=$1
# you might want to adjust the filepath
RESULTS_FILE=~/tmp/$(date +%s%N)

if [[ -z ${URLS_FILE} ]] ; then
    echo "Please provide a file with URLs (line-by-line)!"
    exit
fi

if [[ ! -r ${URLS_FILE} ]] ; then
    echo "File \"${URLS_FILE}\" does not exist or is not readable!"
    exit
fi

while read url
do
    urlstatus=$(curl -o /dev/null --silent --head --write-out '%{http_code}' "${url}")
    # use tabulator as delimiter
    echo -e "$url\t$urlstatus" >> ${RESULTS_FILE}
done < $URLS_FILE

echo "Results written to ${RESULTS_FILE}"

Before you use the script, you should apply permission to execute the file.

chmod +x check-urls-from-file.bash

Check URLs from file

bash check-urls-from-file.bash urls.txt

Work with results

returns only URLs

cat ~/tmp/123456789 | cut -d$'\t' -f1

returns only HTTP status codes

cat ~/tmp/123456789 | cut -d$'\t' -f2

Show all lines NOT containing HTTP status code "200"

cat ~/tmp/123456789 | grep -vP "\t200"

Show only URLs NOT containing HTTP status code "200"

cat ~/tmp/123456789 | grep -vP "\t200" | cut -d$'\t' -f1