Skip to content

Instantly share code, notes, and snippets.

@rafikahmed
Last active May 26, 2020 19:32
Show Gist options
  • Save rafikahmed/c6e6f9fe2e521ad3d6b46cccc8887ac3 to your computer and use it in GitHub Desktop.
Save rafikahmed/c6e6f9fe2e521ad3d6b46cccc8887ac3 to your computer and use it in GitHub Desktop.
# Make sure to add the proxy address, username and passowrd
# Make sure to append the ProxyMiddleware class to the DOWNLOADER_MIDDLEWARES dict in the settings.py file
from w3lib.http import basic_auth_header
class ProxyMiddleware:
def process_request(self, request, spider):
request.meta['proxy'] = "COMPANY PROXY URL OR ADDRESS"
request.headers['Proxy-Authorization'] = basic_auth_header('USERNAME', 'PASSWORD')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment