Created
September 13, 2015 09:55
-
-
Save shun91/c4dc835ae6c423926706 to your computer and use it in GitHub Desktop.
URLリストを読み込み,存在しないURLのみファイルに追記書き出し.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/sh | |
# URLリストを読み込み,存在しないURLのみファイルに追記書き出し. | |
FILE_PATH=urls.txt | |
S3_PATH=$1 | |
DL_PATH=${S3_PATH} | |
cat ${FILE_PATH} | while read line | |
do | |
#echo ${line} >> notExist.txt | |
wget --spider ${line} 2>&1 | grep -Ei -B 3 "(unable to resolve host address|404 not found|302 found)" & >> notExist.txt & | |
done |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment