Skip to content

Instantly share code, notes, and snippets.

@kennethreitz
Created May 16, 2011 00:17
Star You must be signed in to star a gist
Save kennethreitz/973705 to your computer and use it in GitHub Desktop.
urllib2 vs requests
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import urllib2
gh_url = 'https://api.github.com'
req = urllib2.Request(gh_url)
password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, gh_url, 'user', 'pass')
auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_manager)
urllib2.install_opener(opener)
handler = urllib2.urlopen(req)
print handler.getcode()
print handler.headers.getheader('content-type')
# ------
# 200
# 'application/json'
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import requests
r = requests.get('https://api.github.com', auth=('user', 'pass'))
print r.status_code
print r.headers['content-type']
# ------
# 200
# 'application/json'
@SunnyMarkLiu
Copy link

amazing job!

@S-Alvanides
Copy link

Ehm.... Just installed Requests in Python 2.7.11
but the original example 1_requests.py:
r = requests.get('https://api.github.com', auth=('user', 'pass'))
returns r.status_code = 401 (instead of 200)
and r.text = {"message":"Bad credentials","documentation_url":"https://developer.github.com/v3"}
Suggestions?

@kennethreitz
Copy link
Author

@seraf69 you have to provide your own credentials :)

@techite
Copy link

techite commented Jan 27, 2016

I have been reading about you for quite long and it actually inspired me to work out on Python. I strongly feel that "requests" is one of the most amazing things a single person could initiate! Thanks a tonne - it actually makes several possibilities easier.

@richoco196
Copy link

Any help, i have been trying to use this header file: from urllib2.requests import urlopen anytime i use it i do get error message saying: No module named requests

@MoroJr
Copy link

MoroJr commented Feb 23, 2016

@richoco196, first of, requests is a module, urllib2 is another module ; based on the received error, you don't have the module requests installed.

@hxzqlh
Copy link

hxzqlh commented May 10, 2016

So what? You want to make a big news?Too simple,sometimes naive.As an older, I need to give you some personal experiences,:Stay young, stay simple,and you will make a big money!

@vog
Copy link

vog commented Aug 24, 2016

The comment by hxzqlh has exactly nothing to do with the topic, and doesn't make sense at all. Is hxzqlh the account of a spam bot?

@FaizalReza007
Copy link

FaizalReza007 commented Sep 15, 2016

Hi @kennethreitz

Windows : 10
python: 3.5

I am using the below code and I want to retrieve the list of emails count to 2. I am getting the error 401.

import urllib
import requests
from requests.auth import HTTPBasicAuth
resp = requests.get('https://secure.p03.eloqua.com/API/REST/1.0/assets/emails', headers={"content-type":"application/json"}, auth=HTTPBasicAuth(site, user, password))
if resp.status_code !=200:
print(resp.status_code)

response_parameters

I also want to make some post requests but I am not quite sure on how to add parameters in python with OAUTH2.
take a look at link below
http://docs.oracle.com/cloud/latest/marketingcs_gs/OMCAB/index.html#Developers/RESTAPI/1.0 Endpoints/Emails/post-assets-email.htm%3FTocPath%3D%2520REST%2520API%7C1.0%2520Endpoints%7CEmails%7C_____1

@mayankrungta
Copy link

I like the crisp syntax of requests but this article here builds a strong case for httplib2 -

http://www.diveintopython3.net/http-web-services.html

This might help decide.

@hackrade
Copy link

anybody please tell how to add these request scripts to setup.py becase for latest version of kali some tools required to add requset-python to stepup.py

@jpyper
Copy link

jpyper commented Nov 23, 2016

That's what's nice about Python and open languages and libraries. It's really not a matter of which library is best for the task at hand, but which one works the way you need it to. As I read through this whole thread, I kept thinking to myself this isn't https://codegolf.stackexchange.com/ ... :-)

@rohan-ramesh
Copy link

Great gist, but I find your comparison unfair. The urllib2 version uses verbose variables in high quantity, whereas the requests version uses one single-character variable. This only makes a slight difference, though, and this can also be attributed to traditional coding style. Doesn't change the fact that requests is overall simpler to grasp and more often than not shorter.

@jemshit
Copy link

jemshit commented Dec 26, 2017

I missed Retrofit :/

@mohamuud
Copy link

mohamuud commented Sep 2, 2018

Lol requests (pow + simple)^2

@martinmogusu
Copy link

Simple. Clear. Elegant... This is definitely my role model when it comes to building a python library...

Copy link

ghost commented Feb 18, 2019

Requests is simply wonderful!
Although a little unfair.
@kennethreitz

@HeyITGuyFixIt
Copy link

Now how does urllib3 compare?

@debambi
Copy link

debambi commented Jan 3, 2020

Now how does urllib3 compare?

urllib2 vs urllib3 vs requests

@debambi
Copy link

debambi commented Jan 3, 2020

@trami996
Copy link

trami996 commented Jun 8, 2021

Và có vẻ như bạn có thể đơn giản hóa nó hơn nữa:

nhập  urllib2 
từ  base64  nhập  b64encode

yêu cầu  =  urllib2 . Yêu cầu ( 'https://api.github.com/user' )  <meta property="fb:pages" content="112008217311363" /> _header ( 'Ủy quyền' , 'Cơ bản'  +  b64encode ( 'user'  +  ':'  +  'pass' ))
 r  =  urllib2 . urlopen ( yêu cầu )

in  r . getcode ()
 print  r . headers [ "content-type" ]
 print  r . tiêu đề [ "X-RateLimit-Limit" ]

Vì vậy, nó chỉ có 3 dòng và nó tương thích với urllib2cuộc gọi thông thường . Vì vậy, mặc dù API của urllib2thực sự không tốt, nhưng nó không tệ như ví dụ ban đầu của bạn. Vì vậy, tôi nghĩ rằng bạn nên sử dụng phiên bản 3 dòng này để thay thế cho công bằng.

CHỈNH SỬA: Đảm bảo rằng bạn sử dụng b64encodechức năng trên. Sau đó, bạn không cần xóa dấu '\ n' theo sau khỏi chuỗi.

CHỈNH SỬA 2: Đơn giản hơn một chút. Tôi nghĩ nó đơn giản như bây giờ.

@mangalan516
Copy link

It's look interesting guys keep going.

@chajun2106
Copy link

chajun2106 commented Jun 23, 2021

Já utilizaram request no Plone versão 4.3.18? Com os mesmo códigos sugeridos, tenho no Plone resposta de "Privilegios Insuficientes."

Have you already used request in Plone version 4.3.18? With the same suggested codes, I have "Insufficient Privileges" in Plone.

@itczl22
Copy link

itczl22 commented Nov 10, 2021

nice

@abdallaabdalrhman
Copy link

import requests
response = requests.get("https://en.wikipedia.org/wiki/Python")
print(response)
print(response.status_code)

@cyrusDev1
Copy link

Really simple

@HGStyle
Copy link

HGStyle commented Dec 31, 2022

URLLIB (Python3)

from urllib import urlopen
r = urlopen('https://google.com/')
print(r.read().decode())
print(r.code)

Requests (Python3)

from requests import get
r = get('https://google.com/')
print(r.text)
print(r.status_code)

4 lines in each code !
But yes, when (like in the example) you want to login to a website, thats annother thing...
URLLIB is great for little things if you cant download Requests.
Requests is really better because Python code with Requests its a lot more human-readable.

@Farzin300
Copy link

0_urllib2.py
#!/usr/bin/env python

-- coding: utf-8 --

import urllib2

gh_url = 'https://api.github.com'

req = urllib2.Request(gh_url)

password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_manager.add_password(None, gh_url, 'user', 'pass')

auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)
opener = urllib2.build_opener(auth_manager)

urllib2.install_opener(opener)

handler = urllib2.urlopen(req)

print handler.getcode()
print handler.headers.getheader('content-type')

------

200

'application/json'``

[[](https://perfectbt. com/crash/php)

@lsloan
Copy link

lsloan commented Mar 14, 2024

@ HGStyle wrote…

4 lines in each code ! But yes, when (like in the example) you want to login to a website, thats annother thing... URLLIB is great for little things if you cant download Requests. Requests is really better because Python code with Requests its a lot more human-readable.

Yes, that's the point of @kennethreitz's examples. Operations that are complicated with urllib are simple with requests.

@HGStyle
Copy link

HGStyle commented Mar 14, 2024

@source-creator wrote...

Unfortunately, code to format an url with requests is longer and more bloated than urllib:

# requests
from requests import Request
url = Request(None, 'http://example.com/?', params={'Data1': 'data'}).prepare().url

# urllib
import urllib
url = 'http://example.com/?' + urllib.parse.urlencode({'Data1': 'data'})

Just read this: https://stackoverflow.com/a/46783596
You can just use requests.utils (and other submodules, I've seen annother one for functions included in urllib.parse but I don't remember it) instead of urllib.parse in most cases I think.
(and it does not requires requests.utils to be imported because importing requests also imports requests.utils)
NOTE: The functions are comming right from the URLLIB package, so the actual documentation is in URLLIB for some (not all) of the function of this submodule of requests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment