Skip to content

Instantly share code, notes, and snippets.

@osule
Created May 28, 2020 09:35
Show Gist options
  • Save osule/3806b6c590a7c5c5c65eb363cd42f540 to your computer and use it in GitHub Desktop.
Save osule/3806b6c590a7c5c5c65eb363cd42f540 to your computer and use it in GitHub Desktop.
Apply function to sequence in chunks
#!/bin/env python
import time
from itertools import islice
def process_sequence_in_chunks(fn, seq, chunk_size=100):
"""
Apply function to sequence in chunks.
Params
------
fn: callable
Function to apply.
seq: iterable
Sequence to apply function on.
chunk_size: int
Chunk size to break sequence into for function to be applied.
"""
it = iter(seq)
chunk = list(islice(it, chunk_size))
while chunk:
fn(chunk)
time.sleep(2)
chunk = list(islice(it, chunk_size))
if __name__ == '__main__':
process_sequence_in_chunks(print, range(200), chunk_size=60)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment