Skip to content

Instantly share code, notes, and snippets.

@paulswartz
Created July 15, 2013 14:44
Show Gist options
  • Save paulswartz/6000514 to your computer and use it in GitHub Desktop.
Save paulswartz/6000514 to your computer and use it in GitHub Desktop.
run all scrapy spiders
#!/bin/sh
pushd `dirname $0`/pluckr > /dev/null
OUTPUT_DIR=$1
for spider in `scrapy list`
do
echo Scraping $spider...
rm -r $spider.jl
scrapy crawl $spider --output=$spider.jl --loglevel=INFO
mv $spider.jl "$OUTPUT_DIR"
done
popd > /dev/null
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment