Skip to content

Instantly share code, notes, and snippets.

@Miopas

Miopas/spider.sh

Last active May 15, 2018 02:17
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Miopas/ca0ca463a04be8bc6a33e38a82266e50 to your computer and use it in GitHub Desktop.
Save Miopas/ca0ca463a04be8bc6a33e38a82266e50 to your computer and use it in GitHub Desktop.
Download ZIP
把一堆url放在文件里用curl爬取网页的超简单脚本。（以及 shell 脚本读文件模板）
Raw
spider.sh
set -e -x
while read line; do
read var1 var2 <<< $line # $line is spilited by '\t'
curl $var1 > tmp
iconv -f gbk -t utf8 tmp > pages/$var2 # conv encoding
done < $1
rm tmp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment