Skip to content

Instantly share code, notes, and snippets.

@8maki

8maki/file0.py

Created March 21, 2014 08:39
Show Gist options
  • Save 8maki/9682074 to your computer and use it in GitHub Desktop.
Save 8maki/9682074 to your computer and use it in GitHub Desktop.
Download ZIP
ScrapyをScriptから実行する際のスクリプト ref: http://qiita.com/8maki/items/62b39130c116d3844c1c
Raw
file0.py
spider = FollowAllSpider(domain='scrapinghub.com')
settings = get_project_settings()
crawler = Crawler(settings)
crawler.signals.connect(reactor.stop, signal=signals.spider_closed)
crawler.configure()
crawler.crawl(spider)
crawler.start()
log.start()
reactor.run()
Raw
file1.py
spider = FollowAllSpider(domain='scrapinghub.com')
settings = get_project_settings()
crawler = Crawler(settings)
crawler.signals.connect(reactor.stop, signal=signals.spider_closed)
log.start_from_crawler(crawler)
crawler.configure()
crawler.crawl(spider)
crawler.start()
reactor.run()
Raw
file2.py
def start_from_crawler(crawler):
return start_from_settings(crawler.settings, crawler)
Raw
log.py
def start_from_crawler(crawler):
return start_from_settings(crawler.settings, crawler)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment