Skip to content

Instantly share code, notes, and snippets.

@sjmcgrath
Last active August 24, 2016 18:22
Show Gist options
  • Save sjmcgrath/2d80f6263d119b4e2cd0828c798e1759 to your computer and use it in GitHub Desktop.
Save sjmcgrath/2d80f6263d119b4e2cd0828c798e1759 to your computer and use it in GitHub Desktop.
Generator that breaks a sequence up into a given size for batch processing items in the sequence
def batches(sequence, batch_size):
"""Iterate over a sequence in batches
Example:
for batch in batches([1, 2, 3, 4, 5, 6, 7], 3):
process(batch)
"""
try:
sequence_length
except NameError:
sequence_length = len(sequence)
for start_index in range(0, sequence_length, batch_size):
end_index = min(start_index + batch_size, sequence_length)
yield sequence[start_index:end_index]
@sjmcgrath
Copy link
Author

I've seen similar examples here and there that always seem to restrict the sequence length to a multiple of the batch size. Why? I have no idea.

This behaves more sensibly: the last batch will contain however many items are remaining even if it's less than the full batch size.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment