Skip to content

Instantly share code, notes, and snippets.

@mrizwan47

mrizwan47/pandas_df_to_csv_in_chunks.py

Created January 26, 2023 11:30
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mrizwan47/9dde4fd4a6315d34b4659f61e6c98fd5 to your computer and use it in GitHub Desktop.
Save mrizwan47/9dde4fd4a6315d34b4659f61e6c98fd5 to your computer and use it in GitHub Desktop.
Download ZIP
Save Pandas dataframe to csv in chunks
Raw
pandas_df_to_csv_in_chunks.py
import pandas as pd
df = pd.read_csv('large_file.csv')
n = 49000 # Max number of rows you want each chunk to have
chunks = [df[i:i+n].copy() for i in range(0,df.shape[0],n)]
k = 1
for chunk in chunks:
chunk.to_csv('data/chunk_{}.csv'.format(k), index=False)
k=k+1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment