Last active
August 24, 2016 18:22
-
-
Save sjmcgrath/2d80f6263d119b4e2cd0828c798e1759 to your computer and use it in GitHub Desktop.
Generator that breaks a sequence up into a given size for batch processing items in the sequence
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def batches(sequence, batch_size): | |
"""Iterate over a sequence in batches | |
Example: | |
for batch in batches([1, 2, 3, 4, 5, 6, 7], 3): | |
process(batch) | |
""" | |
try: | |
sequence_length | |
except NameError: | |
sequence_length = len(sequence) | |
for start_index in range(0, sequence_length, batch_size): | |
end_index = min(start_index + batch_size, sequence_length) | |
yield sequence[start_index:end_index] |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I've seen similar examples here and there that always seem to restrict the sequence length to a multiple of the batch size. Why? I have no idea.
This behaves more sensibly: the last batch will contain however many items are remaining even if it's less than the full batch size.