Download reference genome:
wget ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/001/405/GCA_000001405.15_GRCh38/seqs_for_alignment_pipelines.ucsc_ids/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.gz
gunzip GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.gz
mv GCA_000001405.15_GRCh38_no_alt_analysis_set.fna GRCh38.fa
Index reference genome:
bwa index GRCh38.fa
Download sample data:
wget ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/phase3/data/NA12878/sequence_read/ERR001268_1.filt.fastq.gz
gunzip ERR001268_1.filt.fastq.gz
mv ERR001268_1.filt.fastq ERR001268_1.fastq
wget ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/phase3/data/NA12878/sequence_read/ERR001268_2.filt.fastq.gz
gunzip ERR001268_2.filt.fastq.gz
mv ERR001268_2.filt.fastq ERR001268_2.fastq
On this system (64GB memory) when we tried to run 24 instances of BWA to utilise maximum resources, the BWA instances gave us [bwa_idx2mem] Failed to allocate N bytes at bwa.c: Cannot allocate memory error. So we reduced the half of the resources to compare the reults.
System information, used in this test:
$lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 24
On-line CPU(s) list: 0-23
Thread(s) per core: 1
Core(s) per socket: 12
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
Stepping: 2
CPU MHz: 1200.024
CPU max MHz: 3500.0000
CPU min MHz: 1200.0000
BogoMIPS: 5187.72
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 30720K
NUMA node0 CPU(s): 0-11
NUMA node1 CPU(s): 12-23
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
$lsmem
RANGE SIZE STATE REMOVABLE BLOCK
0x0000000000000000-0x000000007fffffff 2G online no 0
0x0000000100000000-0x000000107fffffff 62G online no 2-32
Memory block size: 2G
Total online memory: 64G
Total offline memory: 0B
Run BWA multi-threaded
bwa mem -t 12 GRCh38.fa ERR001268_1.fastq ERR001268_2.fastq > out.sam
Output log:
[main] CMD: bwa mem -t 12 GRCh38.fa ERR001268_1.fastq ERR001268_2.fastq
[main] Real time: 551.804 sec; CPU: 6329.594 sec
Create FASTQ chunks
seqkit split2 --threads=<8> -1 ERR001268_1.fastq -2 ERR001268_2.fastq -p <12> -O parts -f
To run multiple BWA instances in paralell on a single machine
import threading
import subprocess
import time
import sys
import os
import glob
def run_BWA(i):
## importing socket module
import socket
## getting the hostname by socket.gethostname() method
hostname = socket.gethostname()
out = 'out_'+str(i)+'.sam'
cmd='/home/tahmad/tahmad/BWA/org/bwa/bwa mem -t 1 /scratch-shared/tahmad/bio_data/GRCh38/GRCh38.fa ' + fqfile1[i] + ' ' + fqfile2[i] + ' -o '+ out
os.system(cmd)
return hostname
fqfile1 =[]
fqfile2 =[]
if __name__ == '__main__':
start = time.time()
fqfile1 = sorted(glob.glob('parts/ERR001268_1.part_'+ '*.fastq'))
fqfile2 = sorted(glob.glob('parts/ERR001268_2.part_'+ '*.fastq'))
p = ThreadPool(12)
p.map(run_BWA, range(12))
stop = time.time()
print('BWA took {} seconds.'
.format(stop - start))
Output log:
BWA took 1437.45400715 seconds.
On this system (96GB memory) we are able to run maximum 16 instances of BWA to utilise maximum resources.
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 62
Model name: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
Stepping: 4
CPU MHz: 2147.827
CPU max MHz: 2500.0000
CPU min MHz: 1200.0000
BogoMIPS: 4999.55
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 20480K
NUMA node0 CPU(s): 0-7
NUMA node1 CPU(s): 8-15
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm epb ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
$ lsmem
RANGE SIZE STATE REMOVABLE BLOCK
0x0000000000000000-0x000000007fffffff 2G online no 0
0x0000000100000000-0x000000187fffffff 94G online no 2-48
Memory block size: 2G
Total online memory: 96G
Total offline memory: 0B
Run BWA multi-threaded
[main] Real time: 696.684 sec; CPU: 10032.395 sec
Run BWA multiple-instances
BWA took 1831.63775206 seconds.