Created
January 12, 2021 16:36
-
-
Save saasindustries/f296c18aab65fd0a9ab0497d40816e81 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def data_scraper(request_method, url, **kwargs): | |
while true: | |
try: | |
proxy = proxy_generator() | |
print("Proxy currently being used: {}".format(proxy)) | |
response = requests.request(request_method, url, proxies=proxy, timeout=7, **kwargs) | |
break | |
# if the request is successful, no exception is raised | |
except: | |
print("Connection error, looking for another proxy") | |
pass | |
return response |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment