Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
def data_scraper(request_method, url, **kwargs):
while true:
try:
proxy = proxy_generator()
print("Proxy currently being used: {}".format(proxy))
response = requests.request(request_method, url, proxies=proxy, timeout=7, **kwargs)
break
# if the request is successful, no exception is raised
except:
print("Connection error, looking for another proxy")
pass
return response
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment