Check multiple URLs from file using bash

check-urls-from-file.bash

#!/bin/bash

#
# URL Status Checker
#
# This script reads a file containing URLs (one per line) and checks their HTTP status codes.
# The results are written to a temporary file located in the ~/tmp directory.
# It utilizes curl to perform HTTP HEAD requests and captures the status codes.
#

# Takes the first argument as the file containing URLs (line-by-line)
URLS_FILE=$1
# The filepath for the temporary results file
RESULTS_FILE=$(mktemp ~/tmp/results.XXXXXXXXXX)

# Check if a file containing URLs is provided
if [[ -z ${URLS_FILE} ]] ; then
    echo "Please provide a file with URLs (one per line)!"
    exit
fi

# Check if the provided URLs file exists and is readable
if [[ ! -r ${URLS_FILE} ]] ; then
    echo "The file \"${URLS_FILE}\" does not exist or is not readable!"
    exit
fi

# Apply dos2unix to clean line breaks in the URLs file
dos2unix --keepdate --quiet ${URLS_FILE}

# Iterate over each URL in the file
while read url
do
    # Perform an HTTP HEAD request to get the status code
    urlstatus=$(curl -o /dev/null --silent --head --write-out '%{http_code}' "${url}")
    # Use tabulator as delimiter for readability
    echo -e "$url\t$urlstatus" >> ${RESULTS_FILE}
done < $URLS_FILE

echo "Results written to ${RESULTS_FILE}"

Before you use the script, you should apply permission to execute the file.

chmod +x check-urls-from-file.bash

Check URLs from file

bash check-urls-from-file.bash urls.txt

Work with results

returns only URLs

cat ~/tmp/123456789 | cut -d$'\t' -f1

returns only HTTP status codes

cat ~/tmp/123456789 | cut -d$'\t' -f2

Show all lines NOT containing HTTP status code "200"

cat ~/tmp/123456789 | grep -vP "\t200"

Show only URLs NOT containing HTTP status code "200"

cat ~/tmp/123456789 | grep -vP "\t200" | cut -d$'\t' -f1