Skip to content

Instantly share code, notes, and snippets.

@nyov

nyov/requests_from_generator.py

Created February 28, 2014 06:38
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save nyov/9266443 to your computer and use it in GitHub Desktop.
Save nyov/9266443 to your computer and use it in GitHub Desktop.
Download ZIP
scrapy ranged start_requests from generator
Raw
requests_from_generator.py
class MySpider(Spider):
# [...]
# start requests from generator
def start_requests(self):
url = 'http://some.page.tld/%s/category'
for page in xrange(1, 247):
link = url % page
yield Request(url=link)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment