Skip to content

Instantly share code, notes, and snippets.

@rhpvorderman
Last active July 14, 2020 09:43
Show Gist options
  • Save rhpvorderman/4f1201c3f39518ff28dde45409eb696b to your computer and use it in GitHub Desktop.
Save rhpvorderman/4f1201c3f39518ff28dde45409eb696b to your computer and use it in GitHub Desktop.
Compression benchmarks igzip and pigz

size

The original file is 1.6G

compresslevel pigz igzip
0 1,6G 712M
1 658M 620M
2 640M 618M
3 623M 622M
4 610M -
5 596M -
6 582M -
7 573M -
8 569M -
9 568M -
11 531M -

pigz level 0 does not compress while igzip level 0 does.

Speed

igzip

$ for clevel in {0..3}; do hyperfine -w 3 -r 10 -P threads 1 3 "igzip -$clevel -T {threads} -f -o ramdisk/big2.fastq.gz big2.fastq" && du -h ramdisk/big2.fastq.gz ; done
Benchmark #1: igzip -0 -T 1 -f -o ramdisk/big2.fastq.gz big2.fastq
  Time (mean ± σ):      4.380 s ±  0.021 s    [User: 4.022 s, System: 0.357 s]
  Range (min … max):    4.349 s …  4.406 s    10 runs
 
Benchmark #2: igzip -0 -T 2 -f -o ramdisk/big2.fastq.gz big2.fastq
  Time (mean ± σ):      4.383 s ±  0.023 s    [User: 4.001 s, System: 0.381 s]
  Range (min … max):    4.350 s …  4.417 s    10 runs
 
Benchmark #3: igzip -0 -T 3 -f -o ramdisk/big2.fastq.gz big2.fastq
  Time (mean ± σ):      4.387 s ±  0.026 s    [User: 4.025 s, System: 0.361 s]
  Range (min … max):    4.355 s …  4.425 s    10 runs
 
Summary
  'igzip -0 -T 1 -f -o ramdisk/big2.fastq.gz big2.fastq' ran
    1.00 ± 0.01 times faster than 'igzip -0 -T 2 -f -o ramdisk/big2.fastq.gz big2.fastq'
    1.00 ± 0.01 times faster than 'igzip -0 -T 3 -f -o ramdisk/big2.fastq.gz big2.fastq'
712M	ramdisk/big2.fastq.gz
Benchmark #1: igzip -1 -T 1 -f -o ramdisk/big2.fastq.gz big2.fastq
  Time (mean ± σ):      4.616 s ±  0.025 s    [User: 4.276 s, System: 0.338 s]
  Range (min … max):    4.582 s …  4.646 s    10 runs
 
Benchmark #2: igzip -1 -T 2 -f -o ramdisk/big2.fastq.gz big2.fastq
  Time (mean ± σ):      4.614 s ±  0.027 s    [User: 4.265 s, System: 0.347 s]
  Range (min … max):    4.578 s …  4.657 s    10 runs
 
Benchmark #3: igzip -1 -T 3 -f -o ramdisk/big2.fastq.gz big2.fastq
  Time (mean ± σ):      4.604 s ±  0.025 s    [User: 4.258 s, System: 0.345 s]
  Range (min … max):    4.581 s …  4.647 s    10 runs
 
Summary
  'igzip -1 -T 3 -f -o ramdisk/big2.fastq.gz big2.fastq' ran
    1.00 ± 0.01 times faster than 'igzip -1 -T 2 -f -o ramdisk/big2.fastq.gz big2.fastq'
    1.00 ± 0.01 times faster than 'igzip -1 -T 1 -f -o ramdisk/big2.fastq.gz big2.fastq'
620M	ramdisk/big2.fastq.gz
Benchmark #1: igzip -2 -T 1 -f -o ramdisk/big2.fastq.gz big2.fastq
  Time (mean ± σ):      4.865 s ±  0.033 s    [User: 4.490 s, System: 0.373 s]
  Range (min … max):    4.828 s …  4.909 s    10 runs
 
Benchmark #2: igzip -2 -T 2 -f -o ramdisk/big2.fastq.gz big2.fastq
  Time (mean ± σ):      4.874 s ±  0.027 s    [User: 4.507 s, System: 0.365 s]
  Range (min … max):    4.832 s …  4.906 s    10 runs
 
Benchmark #3: igzip -2 -T 3 -f -o ramdisk/big2.fastq.gz big2.fastq
  Time (mean ± σ):      4.864 s ±  0.023 s    [User: 4.525 s, System: 0.337 s]
  Range (min … max):    4.831 s …  4.904 s    10 runs
 
Summary
  'igzip -2 -T 3 -f -o ramdisk/big2.fastq.gz big2.fastq' ran
    1.00 ± 0.01 times faster than 'igzip -2 -T 1 -f -o ramdisk/big2.fastq.gz big2.fastq'
    1.00 ± 0.01 times faster than 'igzip -2 -T 2 -f -o ramdisk/big2.fastq.gz big2.fastq'
618M	ramdisk/big2.fastq.gz
Benchmark #1: igzip -3 -T 1 -f -o ramdisk/big2.fastq.gz big2.fastq
  Time (mean ± σ):      8.355 s ±  0.061 s    [User: 8.004 s, System: 0.349 s]
  Range (min … max):    8.264 s …  8.427 s    10 runs
 
Benchmark #2: igzip -3 -T 2 -f -o ramdisk/big2.fastq.gz big2.fastq
  Time (mean ± σ):      8.331 s ±  0.051 s    [User: 7.970 s, System: 0.358 s]
  Range (min … max):    8.273 s …  8.416 s    10 runs
 
Benchmark #3: igzip -3 -T 3 -f -o ramdisk/big2.fastq.gz big2.fastq
  Time (mean ± σ):      8.331 s ±  0.066 s    [User: 7.979 s, System: 0.350 s]
  Range (min … max):    8.263 s …  8.439 s    10 runs
 
Summary
  'igzip -3 -T 2 -f -o ramdisk/big2.fastq.gz big2.fastq' ran
    1.00 ± 0.01 times faster than 'igzip -3 -T 3 -f -o ramdisk/big2.fastq.gz big2.fastq'
    1.00 ± 0.01 times faster than 'igzip -3 -T 1 -f -o ramdisk/big2.fastq.gz big2.fastq'
622M	ramdisk/big2.fastq.gz

pigz

Benchmark #1: pigz -0 -p 1 -c big2.fastq > ramdisk/big2.fastq.gz
  Time (mean ± σ):      1.615 s ±  0.007 s    [User: 1.141 s, System: 0.473 s]
  Range (min … max):    1.603 s …  1.625 s    10 runs
 
Benchmark #2: pigz -0 -p 2 -c big2.fastq > ramdisk/big2.fastq.gz
  Time (mean ± σ):      1.106 s ±  0.044 s    [User: 2.600 s, System: 0.908 s]
  Range (min … max):    1.027 s …  1.149 s    10 runs
 
Benchmark #3: pigz -0 -p 3 -c big2.fastq > ramdisk/big2.fastq.gz
  Time (mean ± σ):     989.4 ms ±  23.7 ms    [User: 2.739 s, System: 0.937 s]
  Range (min … max):   962.4 ms … 1040.2 ms    10 runs
 
Summary
  'pigz -0 -p 3 -c big2.fastq > ramdisk/big2.fastq.gz' ran
    1.12 ± 0.05 times faster than 'pigz -0 -p 2 -c big2.fastq > ramdisk/big2.fastq.gz'
    1.63 ± 0.04 times faster than 'pigz -0 -p 1 -c big2.fastq > ramdisk/big2.fastq.gz'
1,6G	ramdisk/big2.fastq.gz
Benchmark #1: pigz -1 -p 1 -c big2.fastq > ramdisk/big2.fastq.gz
  Time (mean ± σ):     22.207 s ±  0.451 s    [User: 21.896 s, System: 0.307 s]
  Range (min … max):   21.814 s … 23.405 s    10 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Benchmark #2: pigz -1 -p 2 -c big2.fastq > ramdisk/big2.fastq.gz
  Time (mean ± σ):     11.519 s ±  0.129 s    [User: 23.991 s, System: 0.772 s]
  Range (min … max):   11.405 s … 11.849 s    10 runs
 
Benchmark #3: pigz -1 -p 3 -c big2.fastq > ramdisk/big2.fastq.gz
  Time (mean ± σ):      7.977 s ±  0.048 s    [User: 24.800 s, System: 0.765 s]
  Range (min … max):    7.893 s …  8.052 s    10 runs
 
Summary
  'pigz -1 -p 3 -c big2.fastq > ramdisk/big2.fastq.gz' ran
    1.44 ± 0.02 times faster than 'pigz -1 -p 2 -c big2.fastq > ramdisk/big2.fastq.gz'
    2.78 ± 0.06 times faster than 'pigz -1 -p 1 -c big2.fastq > ramdisk/big2.fastq.gz'
658M	ramdisk/big2.fastq.gz
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment