Skip to content

Instantly share code, notes, and snippets.

@binarybana
Last active December 11, 2015 07:48
Show Gist options
  • Save binarybana/4568246 to your computer and use it in GitHub Desktop.
Save binarybana/4568246 to your computer and use it in GitHub Desktop.
A set of timings for parallel compression utilities in Linux

Baseline

cat test.fastq 2>&3 0.00s user 0.17s system 98% cpu 0.171 total
cat test.fastq 2>&3 0.00s user 0.16s system 97% cpu 0.164 total

Compressing

gzip -c test.fastq 2>&3 84.22s user 0.19s system 99% cpu 1:24.59 total
gzip -c test.fastq 2>&3 96.61s user 0.96s system 94% cpu 1:43.75 total
pigz -c test.fastq 2>&3 88.85s user 0.34s system 1181% cpu 7.546 total
lbzip2 -c test.fastq 2>&3 91.08s user 3.01s system 1142% cpu 8.238 total
bzip2 -c test.fastq 2>&3 93.33s user 0.22s system 99% cpu 1:33.77 total

Decompressing

zcat test.fastq.gz 2>&3 8.77s user 0.09s system 99% cpu 8.880 total
zcat test.fastq.gz 2>&3 8.48s user 0.12s system 99% cpu 8.620 total
unpigz -c test.fastq.gz 2>&3 7.70s user 0.38s system 153% cpu 5.265 total
lbzcat test.bz2 2>&3 62.22s user 2.00s system 1174% cpu 5.468 total
bzcat test.bz2 2>&3 46.50s user 0.15s system 99% cpu 46.752 total

Sizes

751M test.fastq
344M test.fastq.gz (0.458 of original)
283M test.fastq.bz2 (0.376 of original)

Command Line

{time bzcat test.bz2 2>&3} 3>&2 > /dev/null 2>> times

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment