Skip to content

Instantly share code, notes, and snippets.

@dinigo
Created November 17, 2017 14:32
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save dinigo/0d064221e9ef51e13d98a52a4df8e5b4 to your computer and use it in GitHub Desktop.
Save dinigo/0d064221e9ef51e13d98a52a4df8e5b4 to your computer and use it in GitHub Desktop.
Request rate limiter mok implementation.
// libs
const request = require('request');
const {RateLimiter} = require('limiter');
// config
const pool = {maxSockets: 100};
const url = 'http://google.com';
const numRequests = 10000;
const requestLimit = 5000;
const limiter = new RateLimiter(requestLimit, 'hour');
let completed = 0;
// request function
function doRequest(){
return new Promise((good, bad) => {
request({url , pool}, (err, res) => err? bad(err) : good(res));
};
}
// limited request function
function doLimitedRequest(){
limiter.removeTokens(1, function(err, remaining) {
const timestamp = (new Date()).toTimeString().split(' ')[0];
doRequest().then(() => {
completed++;
if(completed % 100 === 0) console.log(timestamp + ' completed: ' + ~~completed);
});
});
}
// run the requests
for(let i=0; i<numRequests; i++){
doLimitedRequest();
}
@dinigo
Copy link
Author

dinigo commented Nov 17, 2017

Install dependencies with yarn add request limiter and run the script with node requester.js

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment