Skip to content

Instantly share code, notes, and snippets.

@AlexDel

AlexDel/scraping_full.py

Created May 8, 2019 10:14
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save AlexDel/cba94ead93b12455ebce2906cd473de8 to your computer and use it in GitHub Desktop.
Save AlexDel/cba94ead93b12455ebce2906cd473de8 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
scraping_full.py
import requests
from bs4 import BeautifulSoup
from googlesearch import search
def find_on_amazon(item_name):
return [url for url in search(f'site:www.amazon.com {item_name}', stop=20)]
res_urls = find_on_amazon("Bearington Lil' Grizby Small Plush Stuffed Animal Brown Grizzly Bear, 7 inches.")
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36'}
response = requests.get(res_urls[0], headers=headers)
soup = BeautifulSoup(response.text, 'html5lib')
breadcrumbs_node = soup.find('div', id='wayfinding-breadcrumbs_feature_div')
categories = [a.get_text().strip() for a in breadcrumbs_node.find_all('a')]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment