Skip to content

Instantly share code, notes, and snippets.

@DASpringate
Last active December 19, 2015 15:28
Show Gist options
  • Save DASpringate/5976235 to your computer and use it in GitHub Desktop.
Save DASpringate/5976235 to your computer and use it in GitHub Desktop.
A faster, multicore rbind function. Still way slower than rbindlist but has all of the checks in rbind and returns a data frame, not data.table
fast_rbind <- function(to_bind, cores = 12, splits = 10){
# to_bind : list of dataframes to bind
require(multicore)
myseq <- c(seq(from = 1,to = length(dat2),
by = floor(length(dat2) / max(cores, splits))),
length(dat2))
intermediates <- mclapply(1:length(myseq),
function(x) {
if(x != length(myseq)){
do.call(`rbind`, dat2[myseq[x]:(myseq[x+1] -1)])
}
}, mc.cores = cores)
do.call(`rbind`, intermediates)
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment