Skip to content

Instantly share code, notes, and snippets.

@kirill-konshin
Created May 7, 2020 18:31
Show Gist options
  • Save kirill-konshin/2d6b42121bf16b343fe17a388340bd78 to your computer and use it in GitHub Desktop.
Save kirill-konshin/2d6b42121bf16b343fe17a388340bd78 to your computer and use it in GitHub Desktop.
Sequential promise-based batches for JavaScript or TypeScript
/**
* Task is just an async function that accepts no arguments and returns a promise with result
* Example: `async () => new Promise(...)`
*/
export const sequentialBatches = async <T>(tasks: (() => Promise<T>)[], maxConcurrency = 100): Promise<T[]> => {
const batches = tasks.reduce((res, task, index) => {
const batchIndex = Math.floor(index / maxConcurrency);
res[batchIndex] = res[batchIndex] || [];
res[batchIndex].push(task);
return res;
}, []);
let res = [];
console.log('Batch processing of', tasks.length, 'tasks');
for (const batch of batches) {
console.log('Batch', batches.indexOf(batch) + 1, 'of', batches.length, 'this batch length', batch.length);
res = res.concat(await Promise.all(batch.map(task => task())));
}
console.log('Batch processing done', res.length, 'tasks finished');
return res;
};
// usage
const ids = [1,2,3,4,5,6,7,8,9,10];
const getSomething = sequentialBatches(ids.map(id => async () => fetch(`http://example.com/${id})), 5);
//TODO Make an NPM lib
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment