combine all csv files in a folder

combineCSVfiles.R

## assuming you have a folder full of .csv's to merge
## csv's must all have identical column names.

folder <- "./path/to/csv/files"
filenames <- list.files(folder)

all_files <- Reduce(rbind, lapply(filenames, read.csv))

I have a folder full of .csv's to merge, but they have different column names and may have different order of columns. Also, they sum up to 33.5gb so I can't load all of them in the memory. Do you know any way I can combine all the csv files in the folder?

If it’s that big I think you need to use the right tool for the job such as BigQuery to load the files into a database. If not that then I would use command line tools such as awk and head to manipulate the files directly.

…

________________________________ From: Ramom Ferreira <notifications@github.com> Sent: Tuesday, July 30, 2019 5:15 PM To: MarkEdmondson1234 Cc: Mark; Author Subject: Re: MarkEdmondson1234/combineCSVfiles.R I have a folder full of .csv't to merge, but they have different column names and may have different order of columns. Also, they sum up to 33.5gb so I can't load all of them in the memory. Do you know any way I can combine all the csv files in the folder? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://gist.github.com/c119bfe81af5d5ab81c8?email_source=notifications&email_token=AAYCPLAFVAETXFFBKVHKTWTQCBLJ3A5CNFSM4IH5PDL2YY3PNVWWK3TUL52HS4DFVNDWS43UINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAFWGI4#gistcomment-2985102>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAYCPLA2344C62ZSL53HKADQCBLJ3ANCNFSM4IH5PDLQ>.