Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@g33klord
Created May 3, 2016 11:07
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save g33klord/1efe551ef896ded01f74b2e3f8e9ef4b to your computer and use it in GitHub Desktop.
Save g33klord/1efe551ef896ded01f74b2e3f8e9ef4b to your computer and use it in GitHub Desktop.
rows = []
with open('large.csv', 'rb') as f:
csvrows = csv.DictReader(f)
for line in csvrows:
rows.append(line)
chunks = [rows[x:x+45000] for x in xrange(0, len(rows), 45000) ]
for i, chunk in enumerate(chunks):
print 'Chunk: {}/{}'.format(i, len(chunks))
with open(str(i)+'.csv', 'wb') as f:
dictwriter = csv.DictWriter(f, fieldnames=chunk[0].keys())
dictwriter.writeheader()
for c in chunk:
dictwriter.writerow(c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment