Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
urllib2 vs requests
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import urllib2
gh_url = ''
req = urllib2.Request(gh_url)
password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, gh_url, 'user', 'pass')
auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_manager)
handler = urllib2.urlopen(req)
print handler.getcode()
print handler.headers.getheader('content-type')
# ------
# 200
# 'application/json'
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import requests
r = requests.get('', auth=('user', 'pass'))
print r.status_code
print r.headers['content-type']
# ------
# 200
# 'application/json'

looks short as shit. now do py3

karlcow commented May 31, 2011

And you could use httplib2 which has python 3 support

import httplib2

h = httplib2.Http(".cache")
h.add_credentials('user', 'pass')
r, content = h.request("", "GET")

print r['status']
print r['content-type']

kennethreitz commented May 31, 2011

Python3 support is planned.

I'm rather certain that httplib2 doesn't support multipart file uploads, and requires you to urlencode your POST data yourself. It doesn't support parameters at all. Or Unicode. Or PUT Requests.

Don't get me wrong, httplib2 is awesome. It does have advanced caching support, which is extremely useful sometimes. I don't want to cache. I want to make requests.

Most importantly, it also requires more verbosity, which is the whole reason requests exists. Why do I have to create an Http object? When will I ever not use Http???

This isn't exactly fair -- you're using temporary variables (gh_user, gh_pass) in the urllib2 version but not in the requests version.

Also, your one variable in the requests version is a single character, compared to (for example) password_manager.

+1 about the temporary variables.

it's still massively shorter than httplib2, no matter how you name it


kennethreitz commented Aug 14, 2011

This example isn't a SLOC competition. It's intended to show the best possible way to make the same request with both libraries.

karlcow commented Aug 14, 2011

@justquick you meant massively shorter than urllib2, because it is the same size for httplib2


kennethreitz commented Aug 14, 2011

I don't think that it's misleading, but I've changed the urllib2 example anyway.

gorlum0 commented Aug 15, 2011

Why not a bit shorter?

import urllib2

gh_url = ''

auth_handler = urllib2.HTTPBasicAuthHandler()
auth_handler.add_password(None, gh_url, 'user', 'passwd')

opener = urllib2.build_opener(auth_handler)
handler = urllib2.urlopen(gh_url)

print handler.getcode()
print handler.headers.getheader('content-type')

toomim commented Aug 15, 2011

Damn. Requests has a way better API design. I love that .get() is a method, auth is an optional parameter, and status is a field, not a method. So many little design decisions done right. Can we please get this into the standard/default python distribution? Thankssssss!

Seriously, I always put off doing HTTP requests in python because the API is such a pain in the ass. I end up running wget at the command line and copying and pasting html into files and loading the files in python because open() is an easier API. But with request I'd just do it all in python! What a freaking improvement!


kennethreitz commented Aug 15, 2011

@espeed: requests doesn't have keep-alive support at this time, but it should be added in soon.

@kennethreitz I like your example, thanks!

certik commented Oct 13, 2011

I think that the urllib2 example is a straw man. Here is a code, that I use in my code to access github using urllib2:

import urllib2
from base64 import encodestring

request = urllib2.Request('')
base64string = encodestring('%s:%s' % ('user', 'pass')).replace('\n', '')
request.add_header('Authorization', 'Basic %s' % base64string)
r = urllib2.urlopen(request)

print r.getcode()
print r.headers["content-type"]
print r.headers["X-RateLimit-Limit"]

Here is the same code using requests:

import requests

r = requests.get('', auth=('user', 'pass'))

print r.status_code
print r.headers['content-type']
print r.headers['X-RateLimit-Limit']

Both print (make sure you change your username and password):


While the requests code is much simpler, the urllib2 code is much better than your original example: you just need to specify the url once (not twice), as well as you access the headers in the same way as in requests. And it's 4 lines (to open the url), not 8 lines as in your original example. So one should be fair.

certik commented Oct 13, 2011

And it looks like that you can simplify it even further:

import urllib2
from base64 import b64encode

request = urllib2.Request('')
request.add_header('Authorization', 'Basic ' + b64encode('user' + ':' + 'pass'))
r = urllib2.urlopen(request)

print r.getcode()
print r.headers["content-type"]
print r.headers["X-RateLimit-Limit"]

So it's only 3 lines and it's compatible with the usual urllib2 call. So while the API of urllib2 really isn't good, it isn't nearly as bad as your original example. So I think that you should use this 3 lines version instead to be fair.

EDIT: Make sure you use the b64encode function above. Then you don't need to remove the trailing '\n' from the string.

EDIT 2: Simplified a little more. I think it's as simple as it can get now.

toomim commented Oct 13, 2011

@certik Very clever!
But for the purpose of this post, don't you think we should be comparing the common case? Even though you can make the call shorter by skipping use of the urllib password and auth manager and just adding a header manually with a base64 encode... you're basically NOT using the urllib api anymore.

So this isn't so much a comparison of urllib to requests, as a comparison of clever raw header hacking + urllib to requests.

certik commented Oct 13, 2011

@toomim: I think you are right. But I think that the original example installs some stuff into urllib2 with my passwords (!?), which I really don't feel comfortable with. So I would say that urllib2 is missing this functionality completely, and one has to simply include the header by hand. But as you can see, it's really simple (in this case).


kennethreitz commented Oct 13, 2011

@toomim 👍

minhl commented Oct 27, 2011

How do you post xml data? I used, data=xml_string), but that didn't work.

httplib2 does indeed support PUT requests.

myoffe commented Dec 11, 2011

This is very nice indeed. It can be very useful for small scripts that need "advanced" features of urllib2 (can someone please stop with the libname2 stuff already?)

Does it handle all authentication types? What about NTLM?

@certik: it doesn't actually "install" anything - install_opener is just a method to add an opener callback to the pre-request stack of urllib2:

Ambiguously named things like that are another reason to use Requests!


kennethreitz commented Jan 23, 2012

@justquick done!

cben commented Feb 6, 2012

@bradleywright: It does install the opener for all future requests going through urllib2, which is bad idea for authentication.
The no-side-effects (also shorter!) way is:

handler =

Anyway, Requests API is much cleaner.

Another reason to use Requests: this urllib2 Gist will only work if the host returns HTTP error code 401 and a "WWW-Authenticate" header. Of course the base64 encoded method also works.

temoto commented Apr 3, 2012

Congratulations with keep-alive support. Kind of "must have" feature. Could you elaborate on usage of it in docs? Unfortunately in documentation i see only how to disable keep alive. Do you need to create session object like Http object in httplib2?

Well done. I don't agree with the httplib2 argument, I think they just don't get the point. I do, it's lazier to use Requests.

Regardless of how excessive or standard the urllib2 example is, API usage and flow is everything. Do you want to spend brain cycles on deciphering your libraries or spend them on solving application problems?

Requests does not require as many brain cycles.

The problem with urllib2 is not that it's impossible to write a short script to do something, it's that it's impossible to do so from memory.

Having to google and copy-paste-modify an example every time I want to do something more complicated than a GET request with no parameters or authentication or cookies is not acceptable.

I hate external dependencies but for a lib like requests...i just cant even remotely bring myself to hate it...not even a little bit

You won me over with r.encoding and offering a unicode object. Handling that in urllib2 is a pain in the ass -- you have to regex out the response code from the headers and use that to encode the response to unicode.

thank you.

Oh how we all love requests!

Quest79 commented Jul 19, 2013

Move over guys, I need room in this circlejerk.

you can sit next to me mr garamonde

@Quest79: Nice Cyan!

I am a freshman, at the first look, I like the stytle of requests..
ps: this is my first time to read python opensource project

ldrumm commented Dec 8, 2013

One thing that everone misses from above arguments is that urllib2 does not handle SSL connections in a sane way at all. This makes https pointless. requests make this trivially easy and robust. I used it to create a custom transport for SUDS so I could do https, because SUDS uses urllib2. My custom SUDS transport is 1 file (170 lines). The standard SUDS transport is 472 lines and takes a long time to get your head around in addition to it's other deficiencies.


cxiaoer commented Jul 5, 2014

So simple,So elegent! i like requests

I was just bitten by httplib not being thread safe - that is why I use requests.

requests does HTTPs validation, and urllib does not. Right? So these actually are not equivalent?

MrWIFI commented Aug 30, 2014

I wonder why I always get error on the "response" command ? well newbee to Phyton, hope for answer :)


Script to fetch SPOT url and: 1) save lat,lon coordinates to a file, 2) save all spot messages as JSON, 3) and xml as well.
Files get re-written on every run, if there's new data. Oh, it writes to stderr if batteryState isn't GOOD - so if you run from cron, you'll get email when your spot batteries need to be changed.
import urllib2
import json
import sys

""" Edit these items: """

spot_id = "0Vn4kA4MiPgNSYD52NgPjuVJDpUCSUlGW"

spot_id = "0GletCbxEpVK72XjjWVRP4CTNLF6cbOz"
last_latlon_cache = "/home/gary/lastspotlocation.txt"
json_cache = "/home/gary/spotlocations.json"
xml_cache = "/home/gary/spotlocations.xml"

url = "" % spot_id

response = urllib2.urlopen(url)
data = json.load(response)
xml_response = urllib2.urlopen(url + ".xml").read()
except Exception, err:

the API isn't always reliable.. exit silently

sys.stdout.write("ERROR retreiving URL: %s" % err)

if 'errors' in data.get('response', {}):
data = data['response']['feedMessageResponse']
except KeyError:
sys.stderr.write("ERROR: JSON received from URL contains no feedMessageResponse,"
" but response->errors wasn't populated. WAT.")

count = int(data.get('count', 0))
if count == 0:
sys.stdout.write("0 locations found?!\n%s" % data)

last_message = data.get('messages', {}).get('message', {})[0]

write to stderr (so you get cron mail) if batteryState is not GOOD.

if 'GOOD' not in last_message.get('batteryState', 'GOOD'):
sys.stderr.write("WARNING: spot battery is in state: %s" % last_message.get('batteryState'))

write the last lat,lon to a file:

fh = open(last_latlon_cache, 'w')
fh.write("%s,%s" % (last_message.get('latitude', 0), last_message.get('longitude', 0)))

write all messages return from spot API as json:

fh = open(json_cache, 'w')
fh.write(str(data.get('messages', {}).get('message', {})))

write all messages return from spot API as XML:

fh = open(xml_cache, 'w')

Great job!

hodwik commented Oct 14, 2014


joelmwas commented Dec 7, 2014

nice work.


texom512 commented Mar 1, 2015

Requests is wonderful!!!

gansai commented Apr 4, 2015

Thanks to requests module. I just switched to python because requests made it much easier to do some API calls.

like @pauldraper said. The https certificate validation is the most important thing +1

vbs commented May 12, 2015

Nice work!
Well done!

request is really simple

Ah, you guys suck. Code in requests:

>>> import requests
>>> r = requests.get('http://localhost:8080')
>>> r.status_code
>>> r.content # you can also use r.text. r.text will return the same thing, but as a unicode string.

4 lines in requests. Now let's see urllib2

>>> import urllib2 as url
>>> request = url.Request('http://localhost:8080')
>>> handler = url.urlopen(request)
>>> handler.code

5 lines in urllib2. They add up. Another disadvantage with urllib2, is if you are looking to get the content, then you need to have to save it into another variable. urllib2 is a lot like a file object (which I find disadvantageous), in that you need to call the .read() method in order to find the data. You can only call it once, exactly like a file object.

Requests is altogether a much cleaner, simpler api to follow, which is why it's no surprise that everyone uses it.

so we are always use requests Module, the urllib2 is so complex....right?

I am using kerberos authenticated login for sharepoint. Please help me to port to requests lib for the below code

opener = urllib2.build_opener()
site = SharePointSite(url, opener=opener)
size_list = site.lists["02D82CEF-7A32-4DF0-AF30-F082F7B7345"]

Unfortunately, code to format an url with requests is longer and more bloated than urllib:

# requests
from requests import Request
url = Request(None, '', params={'Data1': 'data'}).prepare().url

# urllib
import urllib
url = '' + urllib.parse.urlencode({'Data1': 'data'})

amazing job!

Seraf69 commented Jan 9, 2016

Ehm.... Just installed Requests in Python 2.7.11
but the original example
r = requests.get('', auth=('user', 'pass'))
returns r.status_code = 401 (instead of 200)
and r.text = {"message":"Bad credentials","documentation_url":""}


kennethreitz commented Jan 16, 2016

@Seraf69 you have to provide your own credentials :)

techite commented Jan 27, 2016

I have been reading about you for quite long and it actually inspired me to work out on Python. I strongly feel that "requests" is one of the most amazing things a single person could initiate! Thanks a tonne - it actually makes several possibilities easier.

Any help, i have been trying to use this header file: from urllib2.requests import urlopen anytime i use it i do get error message saying: No module named requests

MoroJr commented Feb 23, 2016

@richoco196, first of, requests is a module, urllib2 is another module ; based on the received error, you don't have the module requests installed.

hxzqlh commented May 10, 2016

So what? You want to make a big news?Too simple,sometimes naive.As an older, I need to give you some personal experiences,:Stay young, stay simple,and you will make a big money!

vog commented Aug 24, 2016 edited

The comment by hxzqlh has exactly nothing to do with the topic, and doesn't make sense at all. Is hxzqlh the account of a spam bot?

FaizalReza007 commented Sep 15, 2016 edited

Hi @kennethreitz

Windows : 10
python: 3.5

I am using the below code and I want to retrieve the list of emails count to 2. I am getting the error 401.

import urllib
import requests
from requests.auth import HTTPBasicAuth
resp = requests.get('', headers={"content-type":"application/json"}, auth=HTTPBasicAuth(site, user, password))
if resp.status_code !=200:


I also want to make some post requests but I am not quite sure on how to add parameters in python with OAUTH2.
take a look at link below Endpoints/Emails/post-assets-email.htm%3FTocPath%3D%2520REST%2520API%7C1.0%2520Endpoints%7CEmails%7C_____1

I like the crisp syntax of requests but this article here builds a strong case for httplib2 -

This might help decide.

anybody please tell how to add these request scripts to becase for latest version of kali some tools required to add requset-python to

jpyper commented Nov 23, 2016

That's what's nice about Python and open languages and libraries. It's really not a matter of which library is best for the task at hand, but which one works the way you need it to. As I read through this whole thread, I kept thinking to myself this isn't ... :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment