Instantly share code, notes, and snippets.

# bobthecat/bigcorPar.r Last active Nov 27, 2019

 bigcorPar <- function(x, nblocks = 10, verbose = TRUE, ncore="all", ...){ library(ff, quietly = TRUE) require(doMC) if(ncore=="all"){ ncore = multicore:::detectCores() registerDoMC(cores = ncore) } else{ registerDoMC(cores = ncore) } NCOL <- ncol(x) ## test if ncol(x) %% nblocks gives remainder 0 if (NCOL %% nblocks != 0){stop("Choose different 'nblocks' so that ncol(x) %% nblocks = 0!")} ## preallocate square matrix of dimension ## ncol(x) in 'ff' single format corMAT <- ff(vmode = "single", dim = c(NCOL, NCOL)) ## split column numbers into 'nblocks' groups SPLIT <- split(1:NCOL, rep(1:nblocks, each = NCOL/nblocks)) ## create all unique combinations of blocks COMBS <- expand.grid(1:length(SPLIT), 1:length(SPLIT)) COMBS <- t(apply(COMBS, 1, sort)) COMBS <- unique(COMBS) ## iterate through each block combination, calculate correlation matrix ## between blocks and store them in the preallocated matrix on both ## symmetric sides of the diagonal results <- foreach(i = 1:nrow(COMBS)) %dopar% { COMB <- COMBS[i, ] G1 <- SPLIT[[COMB]] G2 <- SPLIT[[COMB]] if (verbose) cat("Block", COMB, "with Block", COMB, "\n") flush.console() COR <- cor(x[, G1], x[, G2], ...) corMAT[G1, G2] <- COR corMAT[G2, G1] <- t(COR) COR <- NULL } gc() return(corMAT) }

### bkutlu commented May 9, 2015

 Replace MAT with x in line 35

### slukowski commented Sep 3, 2015

 'multicore' has been deprecated. Replaced with parallel

### KhanIrfanEusysbio commented Oct 20, 2016

 I have the following dimensions in the data. Kindly let me know how many cores should I assign for the calculation as I repeatedly gives me the error Choose different 'nblocks' so that ncol(x) %% nblocks = 0! dim(data)  514 26346

### melissamlwong commented Jul 19, 2018 • edited

 Thanks Bob for the code. The only problem is the size limitation of ff matrix is about 45,000. In addition, converting ff matrix to ffdf and then writing to file takes a long time. I made a fork to modify the code to handle ~120,000 columns (unlimited number in theory) and print flatten correlation to a file. I run the script for 24 chromosomes separately and it took about 15 hours to complete using 128 CPUs and 4Gb of memory.