Skip to content

Instantly share code, notes, and snippets.

@saasindustries
Created January 12, 2021 16:36
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save saasindustries/f296c18aab65fd0a9ab0497d40816e81 to your computer and use it in GitHub Desktop.
Save saasindustries/f296c18aab65fd0a9ab0497d40816e81 to your computer and use it in GitHub Desktop.
def data_scraper(request_method, url, **kwargs):
while true:
try:
proxy = proxy_generator()
print("Proxy currently being used: {}".format(proxy))
response = requests.request(request_method, url, proxies=proxy, timeout=7, **kwargs)
break
# if the request is successful, no exception is raised
except:
print("Connection error, looking for another proxy")
pass
return response
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment