Skip to content

Instantly share code, notes, and snippets.

@renfredxh
renfredxh / soupURL.py
Last active December 21, 2015 15:59 — forked from simonw/gist:104413
Turn a BeautifulSoup form in to url encoded fields and default values - useful for screen scraping forms and then resubmitting them
from urllib.parse import urlencode
def extract_form_fields(self, soup):
"Turn a BeautifulSoup form into url encoded fields and default values"
fields = ""
for input in soup.findAll('input'):
# ignore submit/image with no name attribute
if input['type'] in ('submit', 'image') and not input.has_attr('name'):
continue
@renfredxh
renfredxh / parser.py
Created August 18, 2013 18:44
Scrape and print a site's HTML with python 3
from urllib.request import urlopen
from bs4 import BeautifulSoup
webpage = urlopen('https://www.google.com')
soup = BeautifulSoup(webpage)
print(soup.prettify())