Skip to content

Instantly share code, notes, and snippets.

@fernandorovai
Created July 15, 2019 15:41
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save fernandorovai/580bd4b272b7306f871de44e1bc06659 to your computer and use it in GitHub Desktop.
Save fernandorovai/580bd4b272b7306f871de44e1bc06659 to your computer and use it in GitHub Desktop.
Merge CSVs in Python
# Merge CSV files from a folder and save a new merged csv file
import os # Handle directories
import pandas as pd # Handle dataframes
import time # Get datetime
csvsDirectory = 'csvs'
outputMergedDirectory = 'csvs'
mergedFilename = 'merged.csv'
fileList = [csvFile for csvFile in os.listdir(csvsDirectory) if csvFile.endswith(".csv")]
# merge all csv files
rows = []
for csv in fileList:
if csv == 'merged.csv':
continue
df = pd.read_csv(os.path.join(csvsDirectory, csv), index_col=None, header=0, encoding='latin1')
rows.append(df)
os.remove(os.path.join(csvsDirectory, csv))
bulkDF = pd.read_csv(os.path.join(csvsDirectory, mergedFilename), encoding='latin1')
merged = pd.concat(rows, axis=0, ignore_index=True)
merged = pd.concat([bulkDF, merged])
merged.to_csv(os.path.join(outputMergedDirectory, mergedFilename), index=False)
@sireenx
Copy link

sireenx commented Mar 28, 2020

can I merge several CSV files without using glob or pandas libraries ?
only using os

@fitoprincipe
Copy link

fitoprincipe commented Feb 8, 2021

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment