Skip to content

Instantly share code, notes, and snippets.

@petrushev

petrushev/scrapy_large_start_urls.py

Created January 3, 2016 15:56
Show Gist options
  • Save petrushev/9284eb2308ae068c27e9 to your computer and use it in GitHub Desktop.
Save petrushev/9284eb2308ae068c27e9 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
scrapy_large_start_urls.py
with open('init_links.csv', 'r') as f:
init_links = f.read().split('\n')
class LargeSpider(scrapy.Spider):
start_urls = init_links
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment