Instantly share code, notes, and snippets.

Embed
What would you like to do?
urllib2 vs requests
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import urllib2
gh_url = 'https://api.github.com'
req = urllib2.Request(gh_url)
password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, gh_url, 'user', 'pass')
auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_manager)
urllib2.install_opener(opener)
handler = urllib2.urlopen(req)
print handler.getcode()
print handler.headers.getheader('content-type')
# ------
# 200
# 'application/json'
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import requests
r = requests.get('https://api.github.com', auth=('user', 'pass'))
print r.status_code
print r.headers['content-type']
# ------
# 200
# 'application/json'
@justquick

This comment has been minimized.

Show comment
Hide comment
@justquick

justquick May 16, 2011

looks short as shit. now do py3

justquick commented May 16, 2011

looks short as shit. now do py3

@karlcow

This comment has been minimized.

Show comment
Hide comment
@karlcow

karlcow May 31, 2011

And you could use httplib2 which has python 3 support

import httplib2

h = httplib2.Http(".cache")
h.add_credentials('user', 'pass')
r, content = h.request("https://api.github.com", "GET")

print r['status']
print r['content-type']

karlcow commented May 31, 2011

And you could use httplib2 which has python 3 support

import httplib2

h = httplib2.Http(".cache")
h.add_credentials('user', 'pass')
r, content = h.request("https://api.github.com", "GET")

print r['status']
print r['content-type']
@kennethreitz

This comment has been minimized.

Show comment
Hide comment
@kennethreitz

kennethreitz May 31, 2011

Python3 support is planned.

I'm rather certain that httplib2 doesn't support multipart file uploads, and requires you to urlencode your POST data yourself. It doesn't support parameters at all. Or Unicode. Or PUT Requests.

Don't get me wrong, httplib2 is awesome. It does have advanced caching support, which is extremely useful sometimes. I don't want to cache. I want to make requests.

Most importantly, it also requires more verbosity, which is the whole reason requests exists. Why do I have to create an Http object? When will I ever not use Http???

Owner

kennethreitz commented May 31, 2011

Python3 support is planned.

I'm rather certain that httplib2 doesn't support multipart file uploads, and requires you to urlencode your POST data yourself. It doesn't support parameters at all. Or Unicode. Or PUT Requests.

Don't get me wrong, httplib2 is awesome. It does have advanced caching support, which is extremely useful sometimes. I don't want to cache. I want to make requests.

Most importantly, it also requires more verbosity, which is the whole reason requests exists. Why do I have to create an Http object? When will I ever not use Http???

@mattparlane

This comment has been minimized.

Show comment
Hide comment
@mattparlane

mattparlane Aug 14, 2011

This isn't exactly fair -- you're using temporary variables (gh_user, gh_pass) in the urllib2 version but not in the requests version.

Also, your one variable in the requests version is a single character, compared to (for example) password_manager.

mattparlane commented Aug 14, 2011

This isn't exactly fair -- you're using temporary variables (gh_user, gh_pass) in the urllib2 version but not in the requests version.

Also, your one variable in the requests version is a single character, compared to (for example) password_manager.

@jrhorn424

This comment has been minimized.

Show comment
Hide comment
@jrhorn424

jrhorn424 Aug 14, 2011

+1 about the temporary variables.

jrhorn424 commented Aug 14, 2011

+1 about the temporary variables.

@justquick

This comment has been minimized.

Show comment
Hide comment
@justquick

justquick Aug 14, 2011

it's still massively shorter than httplib2, no matter how you name it

justquick commented Aug 14, 2011

it's still massively shorter than httplib2, no matter how you name it

@kennethreitz

This comment has been minimized.

Show comment
Hide comment
@kennethreitz

kennethreitz Aug 14, 2011

This example isn't a SLOC competition. It's intended to show the best possible way to make the same request with both libraries.

Owner

kennethreitz commented Aug 14, 2011

This example isn't a SLOC competition. It's intended to show the best possible way to make the same request with both libraries.

@karlcow

This comment has been minimized.

Show comment
Hide comment
@karlcow

karlcow Aug 14, 2011

@justquick you meant massively shorter than urllib2, because it is the same size for httplib2

karlcow commented Aug 14, 2011

@justquick you meant massively shorter than urllib2, because it is the same size for httplib2

@kennethreitz

This comment has been minimized.

Show comment
Hide comment
@kennethreitz

kennethreitz Aug 14, 2011

I don't think that it's misleading, but I've changed the urllib2 example anyway.

Owner

kennethreitz commented Aug 14, 2011

I don't think that it's misleading, but I've changed the urllib2 example anyway.

@gorlum0

This comment has been minimized.

Show comment
Hide comment
@gorlum0

gorlum0 Aug 15, 2011

Why not a bit shorter?

import urllib2

gh_url = 'https://github.com'

auth_handler = urllib2.HTTPBasicAuthHandler()
auth_handler.add_password(None, gh_url, 'user', 'passwd')

opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
handler = urllib2.urlopen(gh_url)

print handler.getcode()
print handler.headers.getheader('content-type')

gorlum0 commented Aug 15, 2011

Why not a bit shorter?

import urllib2

gh_url = 'https://github.com'

auth_handler = urllib2.HTTPBasicAuthHandler()
auth_handler.add_password(None, gh_url, 'user', 'passwd')

opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
handler = urllib2.urlopen(gh_url)

print handler.getcode()
print handler.headers.getheader('content-type')
@toomim

This comment has been minimized.

Show comment
Hide comment
@toomim

toomim Aug 15, 2011

Damn. Requests has a way better API design. I love that .get() is a method, auth is an optional parameter, and status is a field, not a method. So many little design decisions done right. Can we please get this into the standard/default python distribution? Thankssssss!

Seriously, I always put off doing HTTP requests in python because the API is such a pain in the ass. I end up running wget at the command line and copying and pasting html into files and loading the files in python because open() is an easier API. But with request I'd just do it all in python! What a freaking improvement!

toomim commented Aug 15, 2011

Damn. Requests has a way better API design. I love that .get() is a method, auth is an optional parameter, and status is a field, not a method. So many little design decisions done right. Can we please get this into the standard/default python distribution? Thankssssss!

Seriously, I always put off doing HTTP requests in python because the API is such a pain in the ass. I end up running wget at the command line and copying and pasting html into files and loading the files in python because open() is an easier API. But with request I'd just do it all in python! What a freaking improvement!

@espeed

This comment has been minimized.

Show comment
Hide comment
@espeed

espeed Aug 15, 2011

In httplb2 if you reuse the same http object it will reuse the connection so you don't have the overhead of building up and tearing down connections. How do you reuse a connection with Request?

espeed commented Aug 15, 2011

In httplb2 if you reuse the same http object it will reuse the connection so you don't have the overhead of building up and tearing down connections. How do you reuse a connection with Request?

@kennethreitz

This comment has been minimized.

Show comment
Hide comment
@kennethreitz

kennethreitz Aug 15, 2011

@espeed: requests doesn't have keep-alive support at this time, but it should be added in soon.

Owner

kennethreitz commented Aug 15, 2011

@espeed: requests doesn't have keep-alive support at this time, but it should be added in soon.

@wimfeijen

This comment has been minimized.

Show comment
Hide comment
@wimfeijen

wimfeijen Aug 25, 2011

@kennethreitz I like your example, thanks!

wimfeijen commented Aug 25, 2011

@kennethreitz I like your example, thanks!

@certik

This comment has been minimized.

Show comment
Hide comment
@certik

certik Oct 13, 2011

I think that the urllib2 example is a straw man. Here is a code, that I use in my code to access github using urllib2:

import urllib2
from base64 import encodestring

request = urllib2.Request('https://api.github.com/user')
base64string = encodestring('%s:%s' % ('user', 'pass')).replace('\n', '')
request.add_header('Authorization', 'Basic %s' % base64string)
r = urllib2.urlopen(request)

print r.getcode()
print r.headers["content-type"]
print r.headers["X-RateLimit-Limit"]

Here is the same code using requests:

import requests

r = requests.get('https://api.github.com/user', auth=('user', 'pass'))

print r.status_code
print r.headers['content-type']
print r.headers['X-RateLimit-Limit']

Both print (make sure you change your username and password):

200
application/json
5000

While the requests code is much simpler, the urllib2 code is much better than your original example: you just need to specify the url once (not twice), as well as you access the headers in the same way as in requests. And it's 4 lines (to open the url), not 8 lines as in your original example. So one should be fair.

certik commented Oct 13, 2011

I think that the urllib2 example is a straw man. Here is a code, that I use in my code to access github using urllib2:

import urllib2
from base64 import encodestring

request = urllib2.Request('https://api.github.com/user')
base64string = encodestring('%s:%s' % ('user', 'pass')).replace('\n', '')
request.add_header('Authorization', 'Basic %s' % base64string)
r = urllib2.urlopen(request)

print r.getcode()
print r.headers["content-type"]
print r.headers["X-RateLimit-Limit"]

Here is the same code using requests:

import requests

r = requests.get('https://api.github.com/user', auth=('user', 'pass'))

print r.status_code
print r.headers['content-type']
print r.headers['X-RateLimit-Limit']

Both print (make sure you change your username and password):

200
application/json
5000

While the requests code is much simpler, the urllib2 code is much better than your original example: you just need to specify the url once (not twice), as well as you access the headers in the same way as in requests. And it's 4 lines (to open the url), not 8 lines as in your original example. So one should be fair.

@certik

This comment has been minimized.

Show comment
Hide comment
@certik

certik Oct 13, 2011

And it looks like that you can simplify it even further:

import urllib2
from base64 import b64encode

request = urllib2.Request('https://api.github.com/user')
request.add_header('Authorization', 'Basic ' + b64encode('user' + ':' + 'pass'))
r = urllib2.urlopen(request)

print r.getcode()
print r.headers["content-type"]
print r.headers["X-RateLimit-Limit"]

So it's only 3 lines and it's compatible with the usual urllib2 call. So while the API of urllib2 really isn't good, it isn't nearly as bad as your original example. So I think that you should use this 3 lines version instead to be fair.

EDIT: Make sure you use the b64encode function above. Then you don't need to remove the trailing '\n' from the string.

EDIT 2: Simplified a little more. I think it's as simple as it can get now.

certik commented Oct 13, 2011

And it looks like that you can simplify it even further:

import urllib2
from base64 import b64encode

request = urllib2.Request('https://api.github.com/user')
request.add_header('Authorization', 'Basic ' + b64encode('user' + ':' + 'pass'))
r = urllib2.urlopen(request)

print r.getcode()
print r.headers["content-type"]
print r.headers["X-RateLimit-Limit"]

So it's only 3 lines and it's compatible with the usual urllib2 call. So while the API of urllib2 really isn't good, it isn't nearly as bad as your original example. So I think that you should use this 3 lines version instead to be fair.

EDIT: Make sure you use the b64encode function above. Then you don't need to remove the trailing '\n' from the string.

EDIT 2: Simplified a little more. I think it's as simple as it can get now.

@toomim

This comment has been minimized.

Show comment
Hide comment
@toomim

toomim Oct 13, 2011

@certik Very clever!
But for the purpose of this post, don't you think we should be comparing the common case? Even though you can make the call shorter by skipping use of the urllib password and auth manager and just adding a header manually with a base64 encode... you're basically NOT using the urllib api anymore.

So this isn't so much a comparison of urllib to requests, as a comparison of clever raw header hacking + urllib to requests.

toomim commented Oct 13, 2011

@certik Very clever!
But for the purpose of this post, don't you think we should be comparing the common case? Even though you can make the call shorter by skipping use of the urllib password and auth manager and just adding a header manually with a base64 encode... you're basically NOT using the urllib api anymore.

So this isn't so much a comparison of urllib to requests, as a comparison of clever raw header hacking + urllib to requests.

@certik

This comment has been minimized.

Show comment
Hide comment
@certik

certik Oct 13, 2011

@toomim: I think you are right. But I think that the original example installs some stuff into urllib2 with my passwords (!?), which I really don't feel comfortable with. So I would say that urllib2 is missing this functionality completely, and one has to simply include the header by hand. But as you can see, it's really simple (in this case).

certik commented Oct 13, 2011

@toomim: I think you are right. But I think that the original example installs some stuff into urllib2 with my passwords (!?), which I really don't feel comfortable with. So I would say that urllib2 is missing this functionality completely, and one has to simply include the header by hand. But as you can see, it's really simple (in this case).

@kennethreitz

This comment has been minimized.

Show comment
Hide comment
@kennethreitz
Owner

kennethreitz commented Oct 13, 2011

@toomim 👍

@minhl

This comment has been minimized.

Show comment
Hide comment
@minhl

minhl Oct 27, 2011

How do you post xml data? I used session.post(url, data=xml_string), but that didn't work.

minhl commented Oct 27, 2011

How do you post xml data? I used session.post(url, data=xml_string), but that didn't work.

@mattnworb

This comment has been minimized.

Show comment
Hide comment
@mattnworb

mattnworb Nov 29, 2011

httplib2 does indeed support PUT requests.

mattnworb commented Nov 29, 2011

httplib2 does indeed support PUT requests.

@myoffe

This comment has been minimized.

Show comment
Hide comment
@myoffe

myoffe Dec 11, 2011

This is very nice indeed. It can be very useful for small scripts that need "advanced" features of urllib2 (can someone please stop with the libname2 stuff already?)

Does it handle all authentication types? What about NTLM?

myoffe commented Dec 11, 2011

This is very nice indeed. It can be very useful for small scripts that need "advanced" features of urllib2 (can someone please stop with the libname2 stuff already?)

Does it handle all authentication types? What about NTLM?

@bradwright

This comment has been minimized.

Show comment
Hide comment
@bradwright

bradwright Jan 23, 2012

@certik: it doesn't actually "install" anything - install_opener is just a method to add an opener callback to the pre-request stack of urllib2:

http://docs.python.org/library/urllib2.html#urllib2.install_opener

Ambiguously named things like that are another reason to use Requests!

bradwright commented Jan 23, 2012

@certik: it doesn't actually "install" anything - install_opener is just a method to add an opener callback to the pre-request stack of urllib2:

http://docs.python.org/library/urllib2.html#urllib2.install_opener

Ambiguously named things like that are another reason to use Requests!

@kennethreitz

This comment has been minimized.

Show comment
Hide comment
@kennethreitz
Owner

kennethreitz commented Jan 23, 2012

@justquick done!

@cben

This comment has been minimized.

Show comment
Hide comment
@cben

cben Feb 6, 2012

@bradleywright: It does install the opener for all future requests going through urllib2, which is bad idea for authentication.
The no-side-effects (also shorter!) way is:

handler = opener.open(req)

Anyway, Requests API is much cleaner.

cben commented Feb 6, 2012

@bradleywright: It does install the opener for all future requests going through urllib2, which is bad idea for authentication.
The no-side-effects (also shorter!) way is:

handler = opener.open(req)

Anyway, Requests API is much cleaner.

@espeed

This comment has been minimized.

Show comment
Hide comment
@espeed

espeed commented Feb 21, 2012

@kennethreitz Hi Kenneth - Did requests get keepalive (https://gist.github.com/973705#gistcomment-45426)?

@kennethreitz

This comment has been minimized.

Show comment
Hide comment
@espeed

This comment has been minimized.

Show comment
Hide comment
@espeed

espeed Feb 21, 2012

@kennethreitz Cool. Then I'm going to look at switching Bulbs (https://github.com/espeed/bulbs) over to it, which currently uses httplib2. Let me run some performance tests and I'll let you know!

espeed commented Feb 21, 2012

@kennethreitz Cool. Then I'm going to look at switching Bulbs (https://github.com/espeed/bulbs) over to it, which currently uses httplib2. Let me run some performance tests and I'll let you know!

@mikofski

This comment has been minimized.

Show comment
Hide comment
@mikofski

mikofski Mar 14, 2012

Another reason to use Requests: this urllib2 Gist will only work if the host returns HTTP error code 401 and a "WWW-Authenticate" header. Of course the base64 encoded method also works.

mikofski commented Mar 14, 2012

Another reason to use Requests: this urllib2 Gist will only work if the host returns HTTP error code 401 and a "WWW-Authenticate" header. Of course the base64 encoded method also works.

@temoto

This comment has been minimized.

Show comment
Hide comment
@temoto

temoto Apr 3, 2012

Congratulations with keep-alive support. Kind of "must have" feature. Could you elaborate on usage of it in docs? Unfortunately in documentation i see only how to disable keep alive. Do you need to create session object like Http object in httplib2?

temoto commented Apr 3, 2012

Congratulations with keep-alive support. Kind of "must have" feature. Could you elaborate on usage of it in docs? Unfortunately in documentation i see only how to disable keep alive. Do you need to create session object like Http object in httplib2?

@colwilson

This comment has been minimized.

Show comment
Hide comment
@colwilson

colwilson Jul 7, 2012

Well done. I don't agree with the httplib2 argument, I think they just don't get the point. I do, it's lazier to use Requests.

colwilson commented Jul 7, 2012

Well done. I don't agree with the httplib2 argument, I think they just don't get the point. I do, it's lazier to use Requests.

@petermelias

This comment has been minimized.

Show comment
Hide comment
@petermelias

petermelias Dec 14, 2012

Regardless of how excessive or standard the urllib2 example is, API usage and flow is everything. Do you want to spend brain cycles on deciphering your libraries or spend them on solving application problems?

Requests does not require as many brain cycles.

petermelias commented Dec 14, 2012

Regardless of how excessive or standard the urllib2 example is, API usage and flow is everything. Do you want to spend brain cycles on deciphering your libraries or spend them on solving application problems?

Requests does not require as many brain cycles.

@johntyree

This comment has been minimized.

Show comment
Hide comment
@johntyree

johntyree Feb 17, 2013

The problem with urllib2 is not that it's impossible to write a short script to do something, it's that it's impossible to do so from memory.

Having to google and copy-paste-modify an example every time I want to do something more complicated than a GET request with no parameters or authentication or cookies is not acceptable.

johntyree commented Feb 17, 2013

The problem with urllib2 is not that it's impossible to write a short script to do something, it's that it's impossible to do so from memory.

Having to google and copy-paste-modify an example every time I want to do something more complicated than a GET request with no parameters or authentication or cookies is not acceptable.

@mouseroot

This comment has been minimized.

Show comment
Hide comment
@mouseroot

mouseroot Feb 26, 2013

I hate external dependencies but for a lib like requests...i just cant even remotely bring myself to hate it...not even a little bit

mouseroot commented Feb 26, 2013

I hate external dependencies but for a lib like requests...i just cant even remotely bring myself to hate it...not even a little bit

@jvanasco

This comment has been minimized.

Show comment
Hide comment
@jvanasco

jvanasco May 20, 2013

You won me over with r.encoding and offering a unicode object. Handling that in urllib2 is a pain in the ass -- you have to regex out the response code from the headers and use that to encode the response to unicode.

thank you.

jvanasco commented May 20, 2013

You won me over with r.encoding and offering a unicode object. Handling that in urllib2 is a pain in the ass -- you have to regex out the response code from the headers and use that to encode the response to unicode.

thank you.

@mwatts272

This comment has been minimized.

Show comment
Hide comment
@mwatts272

mwatts272 Jul 9, 2013

Oh how we all love requests!

mwatts272 commented Jul 9, 2013

Oh how we all love requests!

@Quest79

This comment has been minimized.

Show comment
Hide comment
@Quest79

Quest79 Jul 19, 2013

Move over guys, I need room in this circlejerk.

Quest79 commented Jul 19, 2013

Move over guys, I need room in this circlejerk.

@intangir

This comment has been minimized.

Show comment
Hide comment
@intangir

intangir Oct 10, 2013

you can sit next to me mr garamonde

intangir commented Oct 10, 2013

you can sit next to me mr garamonde

@MestreLion

This comment has been minimized.

Show comment
Hide comment
@MestreLion

MestreLion commented Oct 22, 2013

@Quest79: Nice Cyan!

@huatangzhi

This comment has been minimized.

Show comment
Hide comment
@huatangzhi

huatangzhi Nov 20, 2013

I am a freshman, at the first look, I like the stytle of requests..
ps: this is my first time to read python opensource project

huatangzhi commented Nov 20, 2013

I am a freshman, at the first look, I like the stytle of requests..
ps: this is my first time to read python opensource project

@ldrumm

This comment has been minimized.

Show comment
Hide comment
@ldrumm

ldrumm Dec 8, 2013

One thing that everone misses from above arguments is that urllib2 does not handle SSL connections in a sane way at all. This makes https pointless. requests make this trivially easy and robust. I used it to create a custom transport for SUDS so I could do https, because SUDS uses urllib2. My custom SUDS transport is 1 file (170 lines). The standard SUDS transport is 472 lines and takes a long time to get your head around in addition to it's other deficiencies.

requests++

ldrumm commented Dec 8, 2013

One thing that everone misses from above arguments is that urllib2 does not handle SSL connections in a sane way at all. This makes https pointless. requests make this trivially easy and robust. I used it to create a custom transport for SUDS so I could do https, because SUDS uses urllib2. My custom SUDS transport is 1 file (170 lines). The standard SUDS transport is 472 lines and takes a long time to get your head around in addition to it's other deficiencies.

requests++

@cxiaoer

This comment has been minimized.

Show comment
Hide comment
@cxiaoer

cxiaoer Jul 5, 2014

So simple,So elegent! i like requests

cxiaoer commented Jul 5, 2014

So simple,So elegent! i like requests

@joakim-hove

This comment has been minimized.

Show comment
Hide comment
@joakim-hove

joakim-hove Jul 18, 2014

I was just bitten by httplib not being thread safe - that is why I use requests.

joakim-hove commented Jul 18, 2014

I was just bitten by httplib not being thread safe - that is why I use requests.

@pauldraper

This comment has been minimized.

Show comment
Hide comment
@pauldraper

pauldraper Aug 10, 2014

requests does HTTPs validation, and urllib does not. Right? So these actually are not equivalent?

pauldraper commented Aug 10, 2014

requests does HTTPs validation, and urllib does not. Right? So these actually are not equivalent?

@MrWIFI

This comment has been minimized.

Show comment
Hide comment
@MrWIFI

MrWIFI Aug 30, 2014

I wonder why I always get error on the "response" command ? well newbee to Phyton, hope for answer :)

!/usr/bin/python

"""
Script to fetch SPOT url and: 1) save lat,lon coordinates to a file, 2) save all spot messages as JSON, 3) and xml as well.
Files get re-written on every run, if there's new data. Oh, it writes to stderr if batteryState isn't GOOD - so if you run from cron, you'll get email when your spot batteries need to be changed.
"""
import urllib2
import json
import sys

""" Edit these items: """

spot_id = "0Vn4kA4MiPgNSYD52NgPjuVJDpUCSUlGW"

spot_id = "0GletCbxEpVK72XjjWVRP4CTNLF6cbOz"
last_latlon_cache = "/home/gary/lastspotlocation.txt"
json_cache = "/home/gary/spotlocations.json"
xml_cache = "/home/gary/spotlocations.xml"

url = "https://api.findmespot.com/spot-main-web/consumer/rest-api/2.0/public/feed/%s/message" % spot_id

try:
response = urllib2.urlopen(url)
data = json.load(response)
xml_response = urllib2.urlopen(url + ".xml").read()
except Exception, err:

the API isn't always reliable.. exit silently

sys.stdout.write("ERROR retreiving URL: %s" % err)
sys.exit(1)

if 'errors' in data.get('response', {}):
sys.stdout.write(str(data['response']['errors']))
sys.exit(0)
else:
try:
data = data['response']['feedMessageResponse']
except KeyError:
sys.stderr.write("ERROR: JSON received from URL contains no feedMessageResponse,"
" but response->errors wasn't populated. WAT.")
sys.exit(0)

count = int(data.get('count', 0))
if count == 0:
sys.stdout.write("0 locations found?!\n%s" % data)
sys.exit(0)

last_message = data.get('messages', {}).get('message', {})[0]

write to stderr (so you get cron mail) if batteryState is not GOOD.

if 'GOOD' not in last_message.get('batteryState', 'GOOD'):
sys.stderr.write("WARNING: spot battery is in state: %s" % last_message.get('batteryState'))

write the last lat,lon to a file:

fh = open(last_latlon_cache, 'w')
fh.write("%s,%s" % (last_message.get('latitude', 0), last_message.get('longitude', 0)))
fh.close()

write all messages return from spot API as json:

fh = open(json_cache, 'w')
fh.write(str(data.get('messages', {}).get('message', {})))
fh.close()

write all messages return from spot API as XML:

fh = open(xml_cache, 'w')
fh.write(str(xml_response))
fh.close()

MrWIFI commented Aug 30, 2014

I wonder why I always get error on the "response" command ? well newbee to Phyton, hope for answer :)

!/usr/bin/python

"""
Script to fetch SPOT url and: 1) save lat,lon coordinates to a file, 2) save all spot messages as JSON, 3) and xml as well.
Files get re-written on every run, if there's new data. Oh, it writes to stderr if batteryState isn't GOOD - so if you run from cron, you'll get email when your spot batteries need to be changed.
"""
import urllib2
import json
import sys

""" Edit these items: """

spot_id = "0Vn4kA4MiPgNSYD52NgPjuVJDpUCSUlGW"

spot_id = "0GletCbxEpVK72XjjWVRP4CTNLF6cbOz"
last_latlon_cache = "/home/gary/lastspotlocation.txt"
json_cache = "/home/gary/spotlocations.json"
xml_cache = "/home/gary/spotlocations.xml"

url = "https://api.findmespot.com/spot-main-web/consumer/rest-api/2.0/public/feed/%s/message" % spot_id

try:
response = urllib2.urlopen(url)
data = json.load(response)
xml_response = urllib2.urlopen(url + ".xml").read()
except Exception, err:

the API isn't always reliable.. exit silently

sys.stdout.write("ERROR retreiving URL: %s" % err)
sys.exit(1)

if 'errors' in data.get('response', {}):
sys.stdout.write(str(data['response']['errors']))
sys.exit(0)
else:
try:
data = data['response']['feedMessageResponse']
except KeyError:
sys.stderr.write("ERROR: JSON received from URL contains no feedMessageResponse,"
" but response->errors wasn't populated. WAT.")
sys.exit(0)

count = int(data.get('count', 0))
if count == 0:
sys.stdout.write("0 locations found?!\n%s" % data)
sys.exit(0)

last_message = data.get('messages', {}).get('message', {})[0]

write to stderr (so you get cron mail) if batteryState is not GOOD.

if 'GOOD' not in last_message.get('batteryState', 'GOOD'):
sys.stderr.write("WARNING: spot battery is in state: %s" % last_message.get('batteryState'))

write the last lat,lon to a file:

fh = open(last_latlon_cache, 'w')
fh.write("%s,%s" % (last_message.get('latitude', 0), last_message.get('longitude', 0)))
fh.close()

write all messages return from spot API as json:

fh = open(json_cache, 'w')
fh.write(str(data.get('messages', {}).get('message', {})))
fh.close()

write all messages return from spot API as XML:

fh = open(xml_cache, 'w')
fh.write(str(xml_response))
fh.close()

@tolerious

This comment has been minimized.

Show comment
Hide comment
@tolerious

tolerious Oct 3, 2014

Great job!

tolerious commented Oct 3, 2014

Great job!

@richardlaborde

This comment has been minimized.

Show comment
Hide comment
@richardlaborde

richardlaborde commented Oct 14, 2014

beautiful.

@joelmwas

This comment has been minimized.

Show comment
Hide comment
@joelmwas

joelmwas Dec 7, 2014

nice work.

joelmwas commented Dec 7, 2014

nice work.

@shantanuo

This comment has been minimized.

Show comment
Hide comment
@shantanuo

shantanuo Jan 26, 2015

Interesting!

shantanuo commented Jan 26, 2015

Interesting!

@authmane512

This comment has been minimized.

Show comment
Hide comment
@authmane512

authmane512 Mar 1, 2015

Requests is wonderful!!!

authmane512 commented Mar 1, 2015

Requests is wonderful!!!

@gansai

This comment has been minimized.

Show comment
Hide comment
@gansai

gansai Apr 4, 2015

Thanks to requests module. I just switched to python because requests made it much easier to do some API calls.

gansai commented Apr 4, 2015

Thanks to requests module. I just switched to python because requests made it much easier to do some API calls.

@cornelinux

This comment has been minimized.

Show comment
Hide comment
@cornelinux

cornelinux May 5, 2015

like @pauldraper said. The https certificate validation is the most important thing +1

cornelinux commented May 5, 2015

like @pauldraper said. The https certificate validation is the most important thing +1

@vbs

This comment has been minimized.

Show comment
Hide comment
@vbs

vbs May 12, 2015

Nice work!
Well done!

vbs commented May 12, 2015

Nice work!
Well done!

@wodeqiangne

This comment has been minimized.

Show comment
Hide comment
@wodeqiangne

wodeqiangne May 21, 2015

request is really simple

wodeqiangne commented May 21, 2015

request is really simple

@Zizouz212

This comment has been minimized.

Show comment
Hide comment
@Zizouz212

Zizouz212 Aug 27, 2015

Ah, you guys suck. Code in requests:

>>> import requests
>>> r = requests.get('http://localhost:8080')
>>> r.status_code
200
>>> r.content # you can also use r.text. r.text will return the same thing, but as a unicode string.
'100'

4 lines in requests. Now let's see urllib2

>>> import urllib2 as url
>>> request = url.Request('http://localhost:8080')
>>> handler = url.urlopen(request)
>>> handler.code
'200'
>>> handler.read()
'100'

5 lines in urllib2. They add up. Another disadvantage with urllib2, is if you are looking to get the content, then you need to have to save it into another variable. urllib2 is a lot like a file object (which I find disadvantageous), in that you need to call the .read() method in order to find the data. You can only call it once, exactly like a file object.

Requests is altogether a much cleaner, simpler api to follow, which is why it's no surprise that everyone uses it.

Zizouz212 commented Aug 27, 2015

Ah, you guys suck. Code in requests:

>>> import requests
>>> r = requests.get('http://localhost:8080')
>>> r.status_code
200
>>> r.content # you can also use r.text. r.text will return the same thing, but as a unicode string.
'100'

4 lines in requests. Now let's see urllib2

>>> import urllib2 as url
>>> request = url.Request('http://localhost:8080')
>>> handler = url.urlopen(request)
>>> handler.code
'200'
>>> handler.read()
'100'

5 lines in urllib2. They add up. Another disadvantage with urllib2, is if you are looking to get the content, then you need to have to save it into another variable. urllib2 is a lot like a file object (which I find disadvantageous), in that you need to call the .read() method in order to find the data. You can only call it once, exactly like a file object.

Requests is altogether a much cleaner, simpler api to follow, which is why it's no surprise that everyone uses it.

@wanggalex

This comment has been minimized.

Show comment
Hide comment
@wanggalex

wanggalex Oct 14, 2015

so we are always use requests Module, the urllib2 is so complex....right?

wanggalex commented Oct 14, 2015

so we are always use requests Module, the urllib2 is so complex....right?

@prakashr85

This comment has been minimized.

Show comment
Hide comment
@prakashr85

prakashr85 Oct 22, 2015

I am using kerberos authenticated login for sharepoint. Please help me to port to requests lib for the below code

opener = urllib2.build_opener()
opener.add_handler(HTTPKerberosAuthHandler())
site = SharePointSite(url, opener=opener)
size_list = site.lists["02D82CEF-7A32-4DF0-AF30-F082F7B7345"]

prakashr85 commented Oct 22, 2015

I am using kerberos authenticated login for sharepoint. Please help me to port to requests lib for the below code

opener = urllib2.build_opener()
opener.add_handler(HTTPKerberosAuthHandler())
site = SharePointSite(url, opener=opener)
size_list = site.lists["02D82CEF-7A32-4DF0-AF30-F082F7B7345"]

@source-creator

This comment has been minimized.

Show comment
Hide comment
@source-creator

source-creator Nov 20, 2015

Unfortunately, code to format an url with requests is longer and more bloated than urllib:

# requests
from requests import Request
url = Request(None, 'http://example.com/?', params={'Data1': 'data'}).prepare().url

# urllib
import urllib
url = 'http://example.com/?' + urllib.parse.urlencode({'Data1': 'data'})

source-creator commented Nov 20, 2015

Unfortunately, code to format an url with requests is longer and more bloated than urllib:

# requests
from requests import Request
url = Request(None, 'http://example.com/?', params={'Data1': 'data'}).prepare().url

# urllib
import urllib
url = 'http://example.com/?' + urllib.parse.urlencode({'Data1': 'data'})
@SunnyMarkLiu

This comment has been minimized.

Show comment
Hide comment
@SunnyMarkLiu

SunnyMarkLiu Jan 3, 2016

amazing job!

SunnyMarkLiu commented Jan 3, 2016

amazing job!

@Seraf69

This comment has been minimized.

Show comment
Hide comment
@Seraf69

Seraf69 Jan 9, 2016

Ehm.... Just installed Requests in Python 2.7.11
but the original example 1_requests.py:
r = requests.get('https://api.github.com', auth=('user', 'pass'))
returns r.status_code = 401 (instead of 200)
and r.text = {"message":"Bad credentials","documentation_url":"https://developer.github.com/v3"}
Suggestions?

Seraf69 commented Jan 9, 2016

Ehm.... Just installed Requests in Python 2.7.11
but the original example 1_requests.py:
r = requests.get('https://api.github.com', auth=('user', 'pass'))
returns r.status_code = 401 (instead of 200)
and r.text = {"message":"Bad credentials","documentation_url":"https://developer.github.com/v3"}
Suggestions?

@kennethreitz

This comment has been minimized.

Show comment
Hide comment
@kennethreitz

kennethreitz Jan 16, 2016

@Seraf69 you have to provide your own credentials :)

Owner

kennethreitz commented Jan 16, 2016

@Seraf69 you have to provide your own credentials :)

@techite

This comment has been minimized.

Show comment
Hide comment
@techite

techite Jan 27, 2016

I have been reading about you for quite long and it actually inspired me to work out on Python. I strongly feel that "requests" is one of the most amazing things a single person could initiate! Thanks a tonne - it actually makes several possibilities easier.

techite commented Jan 27, 2016

I have been reading about you for quite long and it actually inspired me to work out on Python. I strongly feel that "requests" is one of the most amazing things a single person could initiate! Thanks a tonne - it actually makes several possibilities easier.

@richoco196

This comment has been minimized.

Show comment
Hide comment
@richoco196

richoco196 Feb 20, 2016

Any help, i have been trying to use this header file: from urllib2.requests import urlopen anytime i use it i do get error message saying: No module named requests

richoco196 commented Feb 20, 2016

Any help, i have been trying to use this header file: from urllib2.requests import urlopen anytime i use it i do get error message saying: No module named requests

@MoroJr

This comment has been minimized.

Show comment
Hide comment
@MoroJr

MoroJr Feb 23, 2016

@richoco196, first of, requests is a module, urllib2 is another module ; based on the received error, you don't have the module requests installed.

MoroJr commented Feb 23, 2016

@richoco196, first of, requests is a module, urllib2 is another module ; based on the received error, you don't have the module requests installed.

@hxzqlh

This comment has been minimized.

Show comment
Hide comment
@hxzqlh

hxzqlh May 10, 2016

So what? You want to make a big news?Too simple,sometimes naive.As an older, I need to give you some personal experiences,:Stay young, stay simple,and you will make a big money!

hxzqlh commented May 10, 2016

So what? You want to make a big news?Too simple,sometimes naive.As an older, I need to give you some personal experiences,:Stay young, stay simple,and you will make a big money!

@vog

This comment has been minimized.

Show comment
Hide comment
@vog

vog Aug 24, 2016

The comment by hxzqlh has exactly nothing to do with the topic, and doesn't make sense at all. Is hxzqlh the account of a spam bot?

vog commented Aug 24, 2016

The comment by hxzqlh has exactly nothing to do with the topic, and doesn't make sense at all. Is hxzqlh the account of a spam bot?

@FaizalReza007

This comment has been minimized.

Show comment
Hide comment
@FaizalReza007

FaizalReza007 Sep 15, 2016

Hi @kennethreitz

Windows : 10
python: 3.5

I am using the below code and I want to retrieve the list of emails count to 2. I am getting the error 401.

import urllib
import requests
from requests.auth import HTTPBasicAuth
resp = requests.get('https://secure.p03.eloqua.com/API/REST/1.0/assets/emails', headers={"content-type":"application/json"}, auth=HTTPBasicAuth(site, user, password))
if resp.status_code !=200:
print(resp.status_code)

response_parameters

I also want to make some post requests but I am not quite sure on how to add parameters in python with OAUTH2.
take a look at link below
http://docs.oracle.com/cloud/latest/marketingcs_gs/OMCAB/index.html#Developers/RESTAPI/1.0 Endpoints/Emails/post-assets-email.htm%3FTocPath%3D%2520REST%2520API%7C1.0%2520Endpoints%7CEmails%7C_____1

FaizalReza007 commented Sep 15, 2016

Hi @kennethreitz

Windows : 10
python: 3.5

I am using the below code and I want to retrieve the list of emails count to 2. I am getting the error 401.

import urllib
import requests
from requests.auth import HTTPBasicAuth
resp = requests.get('https://secure.p03.eloqua.com/API/REST/1.0/assets/emails', headers={"content-type":"application/json"}, auth=HTTPBasicAuth(site, user, password))
if resp.status_code !=200:
print(resp.status_code)

response_parameters

I also want to make some post requests but I am not quite sure on how to add parameters in python with OAUTH2.
take a look at link below
http://docs.oracle.com/cloud/latest/marketingcs_gs/OMCAB/index.html#Developers/RESTAPI/1.0 Endpoints/Emails/post-assets-email.htm%3FTocPath%3D%2520REST%2520API%7C1.0%2520Endpoints%7CEmails%7C_____1

@mayankrungta

This comment has been minimized.

Show comment
Hide comment
@mayankrungta

mayankrungta Nov 4, 2016

I like the crisp syntax of requests but this article here builds a strong case for httplib2 -

http://www.diveintopython3.net/http-web-services.html

This might help decide.

mayankrungta commented Nov 4, 2016

I like the crisp syntax of requests but this article here builds a strong case for httplib2 -

http://www.diveintopython3.net/http-web-services.html

This might help decide.

@hackrade

This comment has been minimized.

Show comment
Hide comment
@hackrade

hackrade Nov 20, 2016

anybody please tell how to add these request scripts to setup.py becase for latest version of kali some tools required to add requset-python to stepup.py

hackrade commented Nov 20, 2016

anybody please tell how to add these request scripts to setup.py becase for latest version of kali some tools required to add requset-python to stepup.py

@jpyper

This comment has been minimized.

Show comment
Hide comment
@jpyper

jpyper Nov 23, 2016

That's what's nice about Python and open languages and libraries. It's really not a matter of which library is best for the task at hand, but which one works the way you need it to. As I read through this whole thread, I kept thinking to myself this isn't https://codegolf.stackexchange.com/ ... :-)

jpyper commented Nov 23, 2016

That's what's nice about Python and open languages and libraries. It's really not a matter of which library is best for the task at hand, but which one works the way you need it to. As I read through this whole thread, I kept thinking to myself this isn't https://codegolf.stackexchange.com/ ... :-)

@rohan-ramesh

This comment has been minimized.

Show comment
Hide comment
@rohan-ramesh

rohan-ramesh Aug 19, 2017

Great gist, but I find your comparison unfair. The urllib2 version uses verbose variables in high quantity, whereas the requests version uses one single-character variable. This only makes a slight difference, though, and this can also be attributed to traditional coding style. Doesn't change the fact that requests is overall simpler to grasp and more often than not shorter.

rohan-ramesh commented Aug 19, 2017

Great gist, but I find your comparison unfair. The urllib2 version uses verbose variables in high quantity, whereas the requests version uses one single-character variable. This only makes a slight difference, though, and this can also be attributed to traditional coding style. Doesn't change the fact that requests is overall simpler to grasp and more often than not shorter.

@jemshit

This comment has been minimized.

Show comment
Hide comment
@jemshit

jemshit commented Dec 26, 2017

I missed Retrofit :/

@mohamuud

This comment has been minimized.

Show comment
Hide comment
@mohamuud

mohamuud Sep 2, 2018

Lol requests (pow + simple)^2

mohamuud commented Sep 2, 2018

Lol requests (pow + simple)^2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment