Skip to content

Instantly share code, notes, and snippets.

Created January 1, 2017 11:28
Show Gist options
  • Save anonymous/9897cd3c38490a7f68f4996f91367a86 to your computer and use it in GitHub Desktop.
Save anonymous/9897cd3c38490a7f68f4996f91367a86 to your computer and use it in GitHub Desktop.
Split lines from file
#!/usr/bin/env python
def split_into_groups(iterable, group_size):
"""Split an iterable collection into groups with fixed size.
Yield
-----
list[any]
Groups of elements.
"""
buf = []
it = iter(iterable)
try:
while True:
for i in range(group_size):
buf.append(next(it))
yield buf
del buf[:]
except StopIteration:
pass
if buf:
yield buf
with open('input.txt', 'rb') as fin:
for i, lines in enumerate(split_into_groups(fin, group_size=10000)):
with open('output%d.txt' % i, 'wb') as fout:
fout.write(b''.join(lines))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment