Skip to content

Instantly share code, notes, and snippets.

@rsarai
Created August 26, 2017 23:58
Show Gist options
  • Save rsarai/b8a7db3289827b5f1f2e43cf6502c7a8 to your computer and use it in GitHub Desktop.
Save rsarai/b8a7db3289827b5f1f2e43cf6502c7a8 to your computer and use it in GitHub Desktop.
Just a quick example of how to use Beautiful Soup
import requests
from bs4 import BeautifulSoup
url = 'http://www.someurl.com.br/products'
headers = {
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36'
}
r = requests.get(url, headers=headers, timeout=5)
# make sure that the page exist
if r.status_code == 200:
html = r.text
soup = BeautifulSoup(html, 'lxml')
# get all products
products_list_item = soup.find('div', {'class': 'row', 'id': 'products'})
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment