Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save 0x24bin/c0e3d15e83181a4a0c066da8d4b1e95e to your computer and use it in GitHub Desktop.
Save 0x24bin/c0e3d15e83181a4a0c066da8d4b1e95e to your computer and use it in GitHub Desktop.
Split lines from file
#!/usr/bin/env python
def split_into_groups(iterable, group_size):
"""Split an iterable collection into groups with fixed size.
Yield
-----
list[any]
Groups of elements.
"""
buf = []
it = iter(iterable)
try:
while True:
for i in range(group_size):
buf.append(next(it))
yield buf
del buf[:]
except StopIteration:
pass
if buf:
yield buf
with open('input.txt', 'rb') as fin:
for i, lines in enumerate(split_into_groups(fin, group_size=10000)):
with open('output%d.txt' % i, 'wb') as fout:
fout.write(b''.join(lines))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment