Skip to content

Instantly share code, notes, and snippets.

@JesseAldridge
Created July 3, 2021 20:43
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save JesseAldridge/086ac4cddc5c440a422cf831056b90b7 to your computer and use it in GitHub Desktop.
Save JesseAldridge/086ac4cddc5c440a422cf831056b90b7 to your computer and use it in GitHub Desktop.
break up a large csv into pieces
import csv
def break_up(in_path):
with open(in_path) as f_in:
reader = csv.reader(f_in)
chunk = 0
while True:
with open(f'{chunk}.csv', 'w') as f_out:
writer = csv.writer(f_out)
for i, row in enumerate(reader):
writer.writerow(row)
if i > 100:
break
else:
return
chunk += 1
break_up("big.csv")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment