Skip to content

Instantly share code, notes, and snippets.

@AnderRV

AnderRV/quotestoscrape.py Secret

Created August 20, 2021 10:19
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save AnderRV/08feeafc01861b10a414b0e4675a17cf to your computer and use it in GitHub Desktop.
Save AnderRV/08feeafc01861b10a414b0e4675a17cf to your computer and use it in GitHub Desktop.
Download ZIP
Raw
quotestoscrape.py
def extract_content(url, soup):
return [{
'quote': product.find(class_='text').text,
'author': product.find(class_='author').text
} for product in soup.select('.quote')]
def store_content(url, content):
for item in content:
if item['quote'] and item['author']:
list_key = f"crawling:quote:{item['author']}"
repo.add_to_list(list_key, item['quote'])
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment