Skip to content
Create a gist now

Instantly share code, notes, and snippets.

urllib2 vs requests
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import urllib2
gh_url = ''
req = urllib2.Request(gh_url)
password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, gh_url, 'user', 'pass')
auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_manager)
handler = urllib2.urlopen(req)
print handler.getcode()
print handler.headers.getheader('content-type')
# ------
# 200
# 'application/json'
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import requests
r = requests.get('', auth=('user', 'pass'))
print r.status_code
print r.headers['content-type']
# ------
# 200
# 'application/json'

looks short as shit. now do py3


And you could use httplib2 which has python 3 support

import httplib2

h = httplib2.Http(".cache")
h.add_credentials('user', 'pass')
r, content = h.request("", "GET")

print r['status']
print r['content-type']

Python3 support is planned.

I'm rather certain that httplib2 doesn't support multipart file uploads, and requires you to urlencode your POST data yourself. It doesn't support parameters at all. Or Unicode. Or PUT Requests.

Don't get me wrong, httplib2 is awesome. It does have advanced caching support, which is extremely useful sometimes. I don't want to cache. I want to make requests.

Most importantly, it also requires more verbosity, which is the whole reason requests exists. Why do I have to create an Http object? When will I ever not use Http???


This isn't exactly fair -- you're using temporary variables (gh_user, gh_pass) in the urllib2 version but not in the requests version.

Also, your one variable in the requests version is a single character, compared to (for example) password_manager.


+1 about the temporary variables.


it's still massively shorter than httplib2, no matter how you name it


This example isn't a SLOC competition. It's intended to show the best possible way to make the same request with both libraries.


@justquick you meant massively shorter than urllib2, because it is the same size for httplib2


I don't think that it's misleading, but I've changed the urllib2 example anyway.


Damn. Requests has a way better API design. I love that .get() is a method, auth is an optional parameter, and status is a field, not a method. So many little design decisions done right. Can we please get this into the standard/default python distribution? Thankssssss!

Seriously, I always put off doing HTTP requests in python because the API is such a pain in the ass. I end up running wget at the command line and copying and pasting html into files and loading the files in python because open() is an easier API. But with request I'd just do it all in python! What a freaking improvement!


In httplb2 if you reuse the same http object it will reuse the connection so you don't have the overhead of building up and tearing down connections. How do you reuse a connection with Request?


@espeed: requests doesn't have keep-alive support at this time, but it should be added in soon.


@kennethreitz I like your example, thanks!


I think that the urllib2 example is a straw man. Here is a code, that I use in my code to access github using urllib2:

import urllib2
from base64 import encodestring

request = urllib2.Request('')
base64string = encodestring('%s:%s' % ('user', 'pass')).replace('\n', '')
request.add_header('Authorization', 'Basic %s' % base64string)
r = urllib2.urlopen(request)

print r.getcode()
print r.headers["content-type"]
print r.headers["X-RateLimit-Limit"]

Here is the same code using requests:

import requests

r = requests.get('', auth=('user', 'pass'))

print r.status_code
print r.headers['content-type']
print r.headers['X-RateLimit-Limit']

Both print (make sure you change your username and password):


While the requests code is much simpler, the urllib2 code is much better than your original example: you just need to specify the url once (not twice), as well as you access the headers in the same way as in requests. And it's 4 lines (to open the url), not 8 lines as in your original example. So one should be fair.


And it looks like that you can simplify it even further:

import urllib2
from base64 import b64encode

request = urllib2.Request('')
request.add_header('Authorization', 'Basic ' + b64encode('user' + ':' + 'pass'))
r = urllib2.urlopen(request)

print r.getcode()
print r.headers["content-type"]
print r.headers["X-RateLimit-Limit"]

So it's only 3 lines and it's compatible with the usual urllib2 call. So while the API of urllib2 really isn't good, it isn't nearly as bad as your original example. So I think that you should use this 3 lines version instead to be fair.

EDIT: Make sure you use the b64encode function above. Then you don't need to remove the trailing '\n' from the string.

EDIT 2: Simplified a little more. I think it's as simple as it can get now.


@certik Very clever!
But for the purpose of this post, don't you think we should be comparing the common case? Even though you can make the call shorter by skipping use of the urllib password and auth manager and just adding a header manually with a base64 encode... you're basically NOT using the urllib api anymore.

So this isn't so much a comparison of urllib to requests, as a comparison of clever raw header hacking + urllib to requests.


@toomim: I think you are right. But I think that the original example installs some stuff into urllib2 with my passwords (!?), which I really don't feel comfortable with. So I would say that urllib2 is missing this functionality completely, and one has to simply include the header by hand. But as you can see, it's really simple (in this case).


How do you post xml data? I used, data=xml_string), but that didn't work.


httplib2 does indeed support PUT requests.


This is very nice indeed. It can be very useful for small scripts that need "advanced" features of urllib2 (can someone please stop with the libname2 stuff already?)

Does it handle all authentication types? What about NTLM?


@certik: it doesn't actually "install" anything - install_opener is just a method to add an opener callback to the pre-request stack of urllib2:

Ambiguously named things like that are another reason to use Requests!


@justquick done!


@bradleywright: It does install the opener for all future requests going through urllib2, which is bad idea for authentication.
The no-side-effects (also shorter!) way is:

handler =

Anyway, Requests API is much cleaner.


@kennethreitz Hi Kenneth - Did requests get keepalive (


@kennethreitz Cool. Then I'm going to look at switching Bulbs ( over to it, which currently uses httplib2. Let me run some performance tests and I'll let you know!


Another reason to use Requests: this urllib2 Gist will only work if the host returns HTTP error code 401 and a "WWW-Authenticate" header. Of course the base64 encoded method also works.


Congratulations with keep-alive support. Kind of "must have" feature. Could you elaborate on usage of it in docs? Unfortunately in documentation i see only how to disable keep alive. Do you need to create session object like Http object in httplib2?


Well done. I don't agree with the httplib2 argument, I think they just don't get the point. I do, it's lazier to use Requests.


Regardless of how excessive or standard the urllib2 example is, API usage and flow is everything. Do you want to spend brain cycles on deciphering your libraries or spend them on solving application problems?

Requests does not require as many brain cycles.


The problem with urllib2 is not that it's impossible to write a short script to do something, it's that it's impossible to do so from memory.

Having to google and copy-paste-modify an example every time I want to do something more complicated than a GET request with no parameters or authentication or cookies is not acceptable.


I hate external dependencies but for a lib like requests...i just cant even remotely bring myself to hate it...not even a little bit


You won me over with r.encoding and offering a unicode object. Handling that in urllib2 is a pain in the ass -- you have to regex out the response code from the headers and use that to encode the response to unicode.

thank you.


Oh how we all love requests!


Move over guys, I need room in this circlejerk.


you can sit next to me mr garamonde


@Quest79: Nice Cyan!


I am a freshman, at the first look, I like the stytle of requests..
ps: this is my first time to read python opensource project


One thing that everone misses from above arguments is that urllib2 does not handle SSL connections in a sane way at all. This makes https pointless. requests make this trivially easy and robust. I used it to create a custom transport for SUDS so I could do https, because SUDS uses urllib2. My custom SUDS transport is 1 file (170 lines). The standard SUDS transport is 472 lines and takes a long time to get your head around in addition to it's other deficiencies.



So simple,So elegent! i like requests


I was just bitten by httplib not being thread safe - that is why I use requests.


requests does HTTPs validation, and urllib does not. Right? So these actually are not equivalent?


I wonder why I always get error on the "response" command ? well newbee to Phyton, hope for answer :)

Script to fetch SPOT url and: 1) save lat,lon coordinates to a file, 2) save all spot messages as JSON, 3) and xml as well.
Files get re-written on every run, if there's new data. Oh, it writes to stderr if batteryState isn't GOOD - so if you run from cron, you'll get email when your spot batteries need to be changed.
import urllib2
import json
import sys

""" Edit these items: """
#spot_id = "0Vn4kA4MiPgNSYD52NgPjuVJDpUCSUlGW"
spot_id = "0GletCbxEpVK72XjjWVRP4CTNLF6cbOz"
last_latlon_cache = "/home/gary/lastspotlocation.txt"
json_cache = "/home/gary/spotlocations.json"
xml_cache = "/home/gary/spotlocations.xml"

url = "" % spot_id

response = urllib2.urlopen(url)
data = json.load(response)
xml_response = urllib2.urlopen(url + ".xml").read()
except Exception, err:

the API isn't always reliable.. exit silently

sys.stdout.write("ERROR retreiving URL: %s" % err)

if 'errors' in data.get('response', {}):
data = data['response']['feedMessageResponse']
except KeyError:
sys.stderr.write("ERROR: JSON received from URL contains no feedMessageResponse,"
" but response->errors wasn't populated. WAT.")

count = int(data.get('count', 0))
if count == 0:
sys.stdout.write("0 locations found?!\n%s" % data)

last_message = data.get('messages', {}).get('message', {})[0]

write to stderr (so you get cron mail) if batteryState is not GOOD.

if 'GOOD' not in last_message.get('batteryState', 'GOOD'):
sys.stderr.write("WARNING: spot battery is in state: %s" % last_message.get('batteryState'))

write the last lat,lon to a file:

fh = open(last_latlon_cache, 'w')
fh.write("%s,%s" % (last_message.get('latitude', 0), last_message.get('longitude', 0)))

write all messages return from spot API as json:

fh = open(json_cache, 'w')
fh.write(str(data.get('messages', {}).get('message', {})))

write all messages return from spot API as XML:

fh = open(xml_cache, 'w')


Great job!




nice work.




Requests is wonderful!!!


Thanks to requests module. I just switched to python because requests made it much easier to do some API calls.


like @pauldraper said. The https certificate validation is the most important thing +1


Nice work!
Well done!


request is really simple


Ah, you guys suck. Code in requests:

>>> import requests
>>> r = requests.get('http://localhost:8080')
>>> r.status_code
>>> r.content # you can also use r.text. r.text will return the same thing, but as a unicode string.

4 lines in requests. Now let's see urllib2

>>> import urllib2 as url
>>> request = url.Request('http://localhost:8080')
>>> handler = url.urlopen(request)
>>> handler.code

5 lines in urllib2. They add up. Another disadvantage with urllib2, is if you are looking to get the content, then you need to have to save it into another variable. urllib2 is a lot like a file object (which I find disadvantageous), in that you need to call the .read() method in order to find the data. You can only call it once, exactly like a file object.

Requests is altogether a much cleaner, simpler api to follow, which is why it's no surprise that everyone uses it.


so we are always use requests Module, the urllib2 is so complex....right?


I am using kerberos authenticated login for sharepoint. Please help me to port to requests lib for the below code

opener = urllib2.build_opener()
site = SharePointSite(url, opener=opener)
size_list = site.lists["02D82CEF-7A32-4DF0-AF30-F082F7B7345"]


Unfortunately, code to format an url with requests is longer and more bloated than urllib:

# requests
from requests import Request
url = Request(None, '', params={'Data1': 'data'}).prepare().url

# urllib
import urllib
url = '' + urllib.parse.urlencode({'Data1': 'data'})

amazing job!


Ehm.... Just installed Requests in Python 2.7.11
but the original example
r = requests.get('', auth=('user', 'pass'))
returns r.status_code = 401 (instead of 200)
and r.text = {"message":"Bad credentials","documentation_url":""}


@seraf69 you have to provide your own credentials :)


I have been reading about you for quite long and it actually inspired me to work out on Python. I strongly feel that "requests" is one of the most amazing things a single person could initiate! Thanks a tonne - it actually makes several possibilities easier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.