Skip to content

Instantly share code, notes, and snippets.

@irzhywau
Last active June 21, 2019 16:21
Show Gist options
  • Save irzhywau/7ed794c5c3e0a130a232a13b82f84077 to your computer and use it in GitHub Desktop.
Save irzhywau/7ed794c5c3e0a130a232a13b82f84077 to your computer and use it in GitHub Desktop.
Fetch complete dataset
parallelLimit(_map(offsets, o => (callback: Function) => {
if (cancelRequest) {
callback(new Error(`Request canceled by client`), null);
}
// we implemented retry pattern which focus on RateLimitError only
retry({
times: retries
, interval: function (retryCount) {
console.log(`retry... #${retryCount}, last client: ${podio.clientId}`);
if (retryCount > 0) {
podio.switchClient((err: Error | null, client: any) => {
if (!err) {
podio = client;
}
}, { grant_type: "app", app_id: app.app_id, app_token: app.token })
}
return 10;
}
, errorFilter: (err: Error): boolean => {
return err.name && err.name === "PodioRateLimitError"
}
}, (doRetry: Function) => {
// this part is responsible of retrieving data from Podio
// request is rate-limited here
PodioItem.filter(app_id, doRetry, Object.assign({ filters: filters }, { limit: step, offset: o }), options, podio);
}, (err: Error | null, requestResponse: any) => {
callback(err, (requestResponse || {}).items);
})
}), 20, (error: Error | null, response: Array<any>) => {
if (error) {
return afterItemGetNewOffset(error, app, []);
}
// allItems has been inited before all this statement
// will not be used in case of error
response.forEach(items => {
allItems = allItems.concat(items)
})
afterItemGetNewOffset(error, app, allItems);
});
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment