Skip to content

Instantly share code, notes, and snippets.

@Lukasa
Created March 31, 2013 16:28
Show Gist options
  • Star 13 You must be signed in to star a gist
  • Fork 3 You must be signed in to fork a gist
  • Save Lukasa/5281176 to your computer and use it in GitHub Desktop.
Save Lukasa/5281176 to your computer and use it in GitHub Desktop.
Requests and Beautiful Soup example, following the form of http://bpaste.net/show/kMetvCdrfnzh5RgiUKU4/
from BeautifulSoup import BeautifulSoup
import requests
import urlparse
URL = 'example.com'
s = requests.Session()
def fetch(url, data=None):
if data is None:
return s.get(url).content
else:
return s.post(url, data=data).content
soup = BeautifulSoup(fetch(URL))
form = soup.find('form')
fields = form.findAll('input')
formdata = dict( (field.get('name'), field.get('value')) for field in fields)
formdata['username'] = u'username'
formdata['password'] = u'password'
print formdata
posturl = urlparse.urljoin(URL, form['action'])
print posturl
r = s.post(posturl, data=formdata)
print r.text
print s.get(URL).text
@JulianaGuama
Copy link

Olá!
Recentemente testei seu código para um projeto. Gostaria de aproveitar a oportunidade e sugerir modificações em função do uso de Python3.

`#Python3
from bs4 import BeautifulSoup
import requests
from urllib.parse import urljoin

URL = 'example.com'
s = requests.Session()

def fetch(url, data=None):
if data is None:
return s.get(url).content
else:
return s.post(url, data=data).content

soup = BeautifulSoup(fetch(URL))
form = soup.find('form')
fields = form.findAll('input')

formdata = dict( (field.get('name'), field.get('value')) for field in fields)

formdata['username'] = u'username'
formdata['password'] = u'password'

print (formdata)
posturl = urljoin(URL, form['action'])
print (posturl)

r = s.post(posturl, data=formdata)
print (r.text)

print (s.get(URL).text())`

@david-drinn
Copy link

This is a good snippet. Can it be updated to capture other types of inputs, such as "select" inputs?

@rgov
Copy link

rgov commented Feb 11, 2020

@IMMACORP
Copy link

IMMACORP commented Jul 4, 2020

Thanks! This is helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment