Skip to content

Instantly share code, notes, and snippets.

@shun91
Created September 13, 2015 09:55
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save shun91/c4dc835ae6c423926706 to your computer and use it in GitHub Desktop.
Save shun91/c4dc835ae6c423926706 to your computer and use it in GitHub Desktop.
URLリストを読み込み,存在しないURLのみファイルに追記書き出し.
#!/bin/sh
# URLリストを読み込み,存在しないURLのみファイルに追記書き出し.
FILE_PATH=urls.txt
S3_PATH=$1
DL_PATH=${S3_PATH}
cat ${FILE_PATH} | while read line
do
#echo ${line} >> notExist.txt
wget --spider ${line} 2>&1 | grep -Ei -B 3 "(unable to resolve host address|404 not found|302 found)" & >> notExist.txt &
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment