Skip to content

Instantly share code, notes, and snippets.

@ycmjason
Created July 6, 2024 11:30
Show Gist options
  • Save ycmjason/de703c549db23110080b8f82f40206d3 to your computer and use it in GitHub Desktop.
Save ycmjason/de703c549db23110080b8f82f40206d3 to your computer and use it in GitHub Desktop.
async queue with async generator

with this async generator queue, you could make sure the tasks given are run with at a maximum concurrency

async function* queue(tasks, maxConcurrency = 5) {
const runningPromises = [];
const resultPromises = tasks.map(async (task) => {
while (runningPromises.length > maxConcurrency)
await Promise.race(runningPromises);
const promise = task();
runningPromises.push(promise);
const result = await promise;
runningPromises.splice(runningPromises.indexOf(promise), 1);
return result;
});
for (const p of resultPromises) {
yield await p;
}
}
const tasks = Array.from({ length: 50 }, (_, i) => async () => {
console.log(`start task ${i}`);
await new Promise((res) => setTimeout(res, 5000 * Math.random()));
console.log(`end task ${i}`);
return i;
});
for await (const result of queue(tasks)) console.log({ result });
/*
Example result:start task 0
start task 1
start task 2
start task 3
start task 4
start task 5
end task 0
start task 6
{result: 0}
end task 3
start task 7
end task 2
start task 8
end task 6
start task 9
end task 5
start task 10
end task 4
start task 11
end task 1
start task 12
{result: 1}
{result: 2}
{result: 3}
{result: 4}
{result: 5}
{result: 6}
end task 9
start task 13
end task 7
start task 14
{result: 7}
end task 10
start task 15
end task 8
start task 16
{result: 8}
{result: 9}
{result: 10}
end task 13
start task 17
end task 15
start task 18
end task 12
start task 19
end task 14
start task 20
end task 11
start task 21
{result: 11}
{result: 12}
{result: 13}
{result: 14}
{result: 15}
end task 20
start task 22
end task 18
start task 23
end task 22
start task 24
end task 21
start task 25
end task 25
start task 26
end task 16
start task 27
{result: 16}
end task 17
start task 28
{result: 17}
{result: 18}
end task 27
start task 29
end task 24
start task 30
end task 30
start task 31
end task 19
start task 32
{result: 19}
{result: 20}
{result: 21}
{result: 22}
end task 28
start task 33
end task 23
start task 34
{result: 23}
{result: 24}
{result: 25}
end task 29
start task 35
end task 26
start task 36
{result: 26}
{result: 27}
{result: 28}
{result: 29}
{result: 30}
end task 32
start task 37
end task 33
start task 38
end task 34
start task 39
end task 36
start task 40
end task 35
start task 41
end task 31
start task 42
{result: 31}
{result: 32}
{result: 33}
{result: 34}
{result: 35}
{result: 36}
end task 38
start task 43
end task 43
start task 44
end task 40
start task 45
end task 37
start task 46
{result: 37}
{result: 38}
end task 39
start task 47
{result: 39}
{result: 40}
end task 42
start task 48
end task 46
start task 49
end task 41
{result: 41}
{result: 42}
{result: 43}
end task 44
{result: 44}
end task 49
end task 45
{result: 45}
{result: 46}
end task 47
{result: 47}
end task 48
{result: 48}
{result: 49}
*/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment