Skip to content

Instantly share code, notes, and snippets.

@manugarri
Last active January 13, 2016 16:52
Show Gist options
  • Save manugarri/f560146408e8659328d9 to your computer and use it in GitHub Desktop.
Save manugarri/f560146408e8659328d9 to your computer and use it in GitHub Desktop.
Crawlera Post Form Data with Requests
import os
from requests.auth import HTTPProxyAuth
import requests
CRAWLERA_API_KEY = os.environ.get('CRAWLERA_API_KEY')
CRAWLERA_HOST = "proxy.crawlera.com"
CRAWLERA_PORT = "8010"
#Download the crawlera cert from http://doc.scrapinghub.com/_downloads/crawlera-ca.crt
CRAWLERA_CERT_PATH = 'path/to/crawlera/cert'
proxy_auth = HTTPProxyAuth(CRAWLERA_API_KEY, "")
proxies = {"https": "https://{}:{}/".format(CRAWLERA_HOST, CRAWLERA_PORT)}
url = 'URL YOU WANT TO POST TO'
form_data = {
'key1':'value1',
'key2':'value2'
}
resp = requests.post(
url,
proxies=proxies,
auth=proxy_auth,
verify=CRAWLERA_CERT_PATH,
data=form_data
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment