Skip to content

Instantly share code, notes, and snippets.

@highoncarbs
Last active March 10, 2018 11:50
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save highoncarbs/e43c3a92e3f584e9199650e246b42535 to your computer and use it in GitHub Desktop.
Save highoncarbs/e43c3a92e3f584e9199650e246b42535 to your computer and use it in GitHub Desktop.
#!/usr/bin/python
import sys
def breakdown(filename, chunk):
lines = []
line_counter = 0
file_counter = 0
with open(filename) as file:
for line in file:
line_counter += 1
lines.append(line)
if line_counter == int(chunk):
file_counter += 1
writeSmall(lines, line_counter, file_counter)
lines = []
line_counter = 0
if line_counter:
writeSmall(lines, line_counter, file_counter)
def writeSmall(lines, line_counter, file_counter):
with open('file_{0}_part{1}.txt'.format(line_counter, file_counter), 'w') as smallfile:
print("writing {0} lines to file_{1}_part_{2}.txt".format(
line_counter, line_counter, file_counter))
smallfile.write(''.join(lines))
if __name__ == "__main__":
if len(sys.argv) > 1 and sys.argv[1] is not None or sys.argv[2] is not None:
filename = sys.argv[1]
chunk = sys.argv[2]
breakdown(filename, chunk)
else:
print("Please enter filename and chunk size")
@highoncarbs
Copy link
Author

Running the program:

  1. chmod +x split.py

  2. ./split.py <file_name> <chunk_size>
    eg. ./split.py book.txt 2000

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment