Skip to content

Instantly share code, notes, and snippets.

@onmax
Created January 26, 2024 04:14
Show Gist options
  • Save onmax/3550417272e8c7d0cabb984ca2d90474 to your computer and use it in GitHub Desktop.
Save onmax/3550417272e8c7d0cabb984ca2d90474 to your computer and use it in GitHub Desktop.
Effective approach to processing items in batches asynchronously in JavaScript. This function processByBatches allows parallel processing of items, with a customizable batch size. There's room for enhancements, like incorporating Promise.allSettled or adding more options. Feedback and improvements are welcome!
interface ProcessByBatchesOptions {
/**
* The number of items to process in parallel.
* @default 10
*/
batchSize?: number
}
async function processByBatches<ItemType, ResultType>(
items: ItemType[],
callback: (item: ItemType) => Promise<ResultType>,
options: ProcessByBatchesOptions = {},
): Promise<ResultType[]> {
const { batchSize = 10 } = options
const batches = []
for (let i = 0; i < items.length; i += batchSize)
batches.push(items.slice(i, i + batchSize))
const maybePromise: Promise<ResultType[]>[] = []
for (const batch of batches)
maybePromise.push(Promise.all(batch.map(callback)))
return Promise.all(maybePromise).then(results => results.flat())
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment