Skip to content

Instantly share code, notes, and snippets.

@tzvc
Created October 25, 2022 11:41
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save tzvc/f23d2e840f1147530f57673cd9ac6c64 to your computer and use it in GitHub Desktop.
Save tzvc/f23d2e840f1147530f57673cd9ac6c64 to your computer and use it in GitHub Desktop.

As @Grubshka mentionned, it's always a good practice to make your request look like it's coming from a web browser by setting "common" headers. More infos here.

In your case though, it looks like the website you are targeting is rate-limiting based on IP address, and that 25 requests coming from the same IP in 10 minutes is exceeding their limit.

The easiest way to get around this would be to use a residential proxy, which would give you a different IP address everytime you make a request and help you avoid getting rate-limited.

Here's an example code using both anti detection headers and a proxy server:

import requests

s = requests.Session()

url = "htttps://thesiteyouraretargeting.com"

# Add common browser's header
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-US,en;q=0.5",
}

# configure a proxy server
http_proxy = "http://0.0.0.0.0:0000"
proxies = {
    "http": http_proxy,
}

r = s.get(url, headers=headers, proxies=proxies)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment