Skip to content

Instantly share code, notes, and snippets.

@eupendra
Created July 10, 2020 03:04
Show Gist options
  • Save eupendra/02d882ed950c77de17e76c12405a8fce to your computer and use it in GitHub Desktop.
Save eupendra/02d882ed950c77de17e76c12405a8fce to your computer and use it in GitHub Desktop.
import scrapy
base = 'http://quotes.toscrape.com/api/quotes?page={}'
class ScrollSpider(scrapy.Spider):
name = 'scroll'
start_urls = [base.format(1)]
def parse(self, response):
data = response.json() #scrapy 2.2
for quote in data["quotes"]:
yield {
'Author': quote["author"]["name"],
'Quote' : quote["text"]
}
current_page = data["page"]
if data["has_next"]:
next_page_url = base.format(current_page+1)
yield scrapy.Request(next_page_url)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment