Skip to content

Instantly share code, notes, and snippets.

@Markbnj

Markbnj/debug-scrapyd-ipdb.py

Last active August 29, 2015 14:27
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save Markbnj/c4ecefbc5a6f248a192b to your computer and use it in GitHub Desktop.
Save Markbnj/c4ecefbc5a6f248a192b to your computer and use it in GitHub Desktop.
Download ZIP
User ipdb to debug a running spider in scrapyd
Raw
debug-scrapyd-ipdb.py
# The scrapyd daemon adds a lot of power and flexibility to scrapy, but it also adds a layer of indirection
# (another process) that can make debugging a running spider troublesome. Use this code to launch scrapyd
# in ipdb (or pdb). You'll be able to breakpoint and catch exceptions inside the spiders it launches.
import scrapyd.runner
import ipdb
ipdb.runcall(scrapyd.runner.main)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment