Skip to content

Instantly share code, notes, and snippets.

@mrizwan47
Created January 26, 2023 11:30
Show Gist options
  • Save mrizwan47/9dde4fd4a6315d34b4659f61e6c98fd5 to your computer and use it in GitHub Desktop.
Save mrizwan47/9dde4fd4a6315d34b4659f61e6c98fd5 to your computer and use it in GitHub Desktop.
Save Pandas dataframe to csv in chunks
import pandas as pd
df = pd.read_csv('large_file.csv')
n = 49000 # Max number of rows you want each chunk to have
chunks = [df[i:i+n].copy() for i in range(0,df.shape[0],n)]
k = 1
for chunk in chunks:
chunk.to_csv('data/chunk_{}.csv'.format(k), index=False)
k=k+1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment