Skip to content

Instantly share code, notes, and snippets.

@dougg0k
Last active August 21, 2021 23:56
Show Gist options
  • Save dougg0k/beff6d2b6107bfc7cee718b47f6782b3 to your computer and use it in GitHub Desktop.
Save dougg0k/beff6d2b6107bfc7cee718b47f6782b3 to your computer and use it in GitHub Desktop.
Bypass Rate Limiter by IP by making requests through a proxy list.
import got from "got";
import tunnel from "tunnel";
const PROXY_LIST = [
{ ip: "", port: 80 },
{ ip: "", port: 80 },
{ ip: "", port: 80 },
{ ip: "", port: 80 },
];
export async function getRequestThroughProxy(
url: string,
additionalHeaders?: Record<string, any>,
proxyCount = 0
) {
const response = await got.get(url, {
/*
Make sure you know what you need, if http to https or vice versa, or both of the same.
https://github.com/koichik/node-tunnel
*/
agent: {
http: tunnel.httpOverHttp({
proxy: {
host: PROXY_LIST[proxyCount].ip,
port: PROXY_LIST[proxyCount].port,
},
}),
},
throwHttpErrors: false,
cache: false,
responseType: "json",
headers: {
"Content-Type": "application/json",
...additionalHeaders,
},
});
if (response.statusCode === 429) {
const updatedCount = proxyCount + 1;
if (updatedCount > PROXY_LIST.length) {
throw Error("Proxies no longer working");
}
return await getRequestThroughProxy(url, additionalHeaders, updatedCount);
}
return response;
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment