Skip to content

Instantly share code, notes, and snippets.

@rafaelhenrique

rafaelhenrique/generate_chunks.py

Last active August 9, 2022 19:18
Show Gist options
  • Save rafaelhenrique/d984f44a9ced20b76350b35048abd826 to your computer and use it in GitHub Desktop.
Save rafaelhenrique/d984f44a9ced20b76350b35048abd826 to your computer and use it in GitHub Desktop.
Download ZIP
Generate chunks
Raw
generate_chunks.py
import itertools
def generate_chunks(data, chunk_size: int):
iterable = iter(data)
while chunk := tuple(itertools.islice(iterable, chunk_size)):
yield chunk
@rafaelhenrique
Copy link
Author

A cool use of this...

for jobs_to_run in generate_chunks(jobs, settings.BIGBOOST_REQUEST_CONCURRENCY):
    await asyncio.gather(*jobs_to_run)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment