Skip to content

Instantly share code, notes, and snippets.

@dmugtasimov

dmugtasimov/run.sh

Last active January 5, 2017 19:01
Show Gist options
  • Save dmugtasimov/61f127134afccf9c89bac82323180194 to your computer and use it in GitHub Desktop.
Save dmugtasimov/61f127134afccf9c89bac82323180194 to your computer and use it in GitHub Desktop.
Download ZIP
Running spiders and parsers
Raw
run.sh
# Before any run
python manage.py register_spiders
python manage.py register_parsers
# Run every day
python manage.py run_spider trounceflow.spiders.impl.countries.hungary.spider.HungarySpider
python manage.py run_spider trounceflow.spiders.impl.countries.indonesia.spider.IndonesiaSpider
python manage.py run_spider trounceflow.spiders.impl.countries.mexico.spider.MexicoSpider
# Run twice a week: Tuesday and Friday
python manage.py run_spider trounceflow.spiders.impl.countries.turkey.spider.TurkeySpider
# Run every week
python manage.py run_spiders -t country -f
# After spiders run
python manage.py update_file_parse_queue -t country
python manage.py run_queued_parsers -t country
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment