compression-test-solbuild-images.md

Is (p)lzip the best fit for compressing solbuild images?

Hypothesis

Based on the available literature, (p)lzip might be a better solution than both xz -9 and zstd -19 (optionally with --long).

Technical details

Experiments were conducted on an AMD Ryzen 7 2700X w/32GiB DDR4-3000 RAM, using the same solbuild image.

Output images were written to /tmp to remove harddisk performance as a factor.

Results

xz -9 multithreaded compression

ermo@solbox:/tmp
$ time xz -v -9 -T0 -c /var/lib/solbuild/images/unstable-x86_64.img > unstable-x86_64.img.xz
xz: Reduced the number of threads from 16 to 6 to not exceed the memory usage limit of 8.004 MiB
/var/lib/solbuild/images/unstable-x86_64.img (1/1)
100 %     268,5 MiB / 9.536,7 MiB = 0,028    73 MiB/s       2:10             

real    2m10,832s
user    9m38,586s
sys     0m2,926s

xz single-threaded decompression

ermo@solbox:/tmp
$ for i in $(seq 1 5); do echo "Run $i:"; time xz -v --decompress --stdout --keep unstable-x86_64.img.xz > /dev/null; done
Run 1:
unstable-x86_64.img.xz (1/1)
100 %     268,5 MiB / 9.536,7 MiB = 0,028   318 MiB/s       0:29             

real    0m29,988s
user    0m29,557s
sys     0m0,350s
Run 2:
unstable-x86_64.img.xz (1/1)
100 %     268,5 MiB / 9.536,7 MiB = 0,028   319 MiB/s       0:29             

real    0m29,922s
user    0m29,480s
sys     0m0,361s
Run 3:
unstable-x86_64.img.xz (1/1)
100 %     268,5 MiB / 9.536,7 MiB = 0,028   320 MiB/s       0:29             

real    0m29,827s
user    0m29,428s
sys     0m0,320s
Run 4:
unstable-x86_64.img.xz (1/1)
100 %     268,5 MiB / 9.536,7 MiB = 0,028   324 MiB/s       0:29             

real    0m29,435s
user    0m29,005s
sys     0m0,354s
Run 5:
unstable-x86_64.img.xz (1/1)
100 %     268,5 MiB / 9.536,7 MiB = 0,028   323 MiB/s       0:29             

real    0m29,551s
user    0m29,141s
sys     0m0,333s
ermo@solbox:/tmp

xz multi-threaded decompression

ermo@solbox:/tmp
$ for i in $(seq 1 5); do echo "Run $i:"; time xz -T0 -v --decompress --stdout --keep unstable-x86_64.img.xz > /dev/null; done
Run 1:
unstable-x86_64.img.xz (1/1)
100 %     268,5 MiB / 9.536,7 MiB = 0,028   2,2 GiB/s       0:04             

real    0m4,179s
user    0m38,022s
sys     0m3,233s
Run 2:
unstable-x86_64.img.xz (1/1)
100 %     268,5 MiB / 9.536,7 MiB = 0,028   2,2 GiB/s       0:04             

real    0m4,328s
user    0m38,069s
sys     0m3,081s
Run 3:
unstable-x86_64.img.xz (1/1)
100 %     268,5 MiB / 9.536,7 MiB = 0,028   2,2 GiB/s       0:04             

real    0m4,182s
user    0m37,947s
sys     0m3,102s
Run 4:
unstable-x86_64.img.xz (1/1)
100 %     268,5 MiB / 9.536,7 MiB = 0,028   2,2 GiB/s       0:04             

real    0m4,201s
user    0m38,220s
sys     0m3,216s
Run 5:
unstable-x86_64.img.xz (1/1)
100 %     268,5 MiB / 9.536,7 MiB = 0,028   2,2 GiB/s       0:04             

real    0m4,230s
user    0m37,937s
sys     0m3,088s

Default plzip -9 compression

ermo@solbox:/tmp
$ time plzip -n16 -v -9 -o unstable-x86_64.img.lz9 /var/lib/solbuild/images/unstable-x86_64.img 
/var/lib/solbuild/images/unstable-x86_64.img: 33.143:1,  3.02% ratio, 96.98% saved, 10000000000 in, 301723365 out.

real    3m22,910s
user    46m39,320s
sys     0m5,197s

Tweaked plzip compression (equivalent to xz -9)

According to the lzip author, -m64 -s64MiB is equivalent to the default xz -9 options:

ermo@solbox:/tmp
$ time plzip -n16 -v -m64 -s64MiB -o unstable-x86_64.img.lz /var/lib/solbuild/images/unstable-x86_64.img 
/var/lib/solbuild/images/unstable-x86_64.img: 36.448:1,  2.74% ratio, 97.26% saved, 10000000000 in, 274363515 out.

real    2m5,818s
user    22m25,898s
sys     0m9,837s

plzip single-threaded decompression

ermo@solbox:/tmp
$ for i in $(seq 1 5); do echo "Run $i:"; time lzip -v -k -c -d unstable-x86_64.img.lz > /dev/null; done
Run 1:
unstable-x86_64.img.lz: done

real    0m49,850s
user    0m49,283s
sys     0m0,441s
Run 2:
unstable-x86_64.img.lz: done

real    0m50,121s
user    0m49,504s
sys     0m0,486s
Run 3:
unstable-x86_64.img.lz: done

real    0m49,556s
user    0m48,999s
sys     0m0,433s
Run 4:
unstable-x86_64.img.lz: done

real    0m49,761s
user    0m49,198s
sys     0m0,440s
Run 5:
unstable-x86_64.img.lz: done

real    0m49,735s
user    0m49,165s
sys     0m0,446s

plzip multi-threaded decompression

ermo@solbox:/tmp
$ for i in $(seq 1 5); do echo "Run $i:"; time plzip -v -n16 -k -c -d unstable-x86_64.img.lz > /dev/null; done
Run 1:
unstable-x86_64.img.lz: done

real    0m11,940s
user    0m59,342s
sys     0m2,315s
Run 2:
unstable-x86_64.img.lz: done

real    0m11,883s
user    0m59,935s
sys     0m2,019s
Run 3:
unstable-x86_64.img.lz: done

real    0m11,895s
user    0m59,886s
sys     0m2,057s
Run 4:
unstable-x86_64.img.lz: done

real    0m11,868s
user    1m0,101s
sys     0m1,946s
Run 5:
unstable-x86_64.img.lz: done

real    0m11,804s
user    0m59,672s
sys     0m1,932s

zstd -19

ermo@solbox:/tmp
$ time zstd -T0 -19 -o unstable-x86_64.img.zst /var/lib/solbuild/images/unstable-x86_64.img 
/var/lib/solbuild/images/unstable-x86_64.img :  3.41%   (  9.31 GiB =>    325 MiB, unstable-x86_64.img.zst) 

real    1m29,773s
user    8m15,946s
sys     0m2,856s

zstd -19 w/long range matching

ermo@solbox:/tmp
$ time zstd -T0 -19 --long -o unstable-x86_64.img.zstlong /var/lib/solbuild/images/unstable-x86_64.img 
/var/lib/solbuild/images/unstable-x86_64.img :  2.95%   (  9.31 GiB =>    282 MiB, unstable-x86_64.img.zstlong) 

real    2m5,012s
user    8m25,644s
sys     0m2,447s

zstd single-threaded decompression

ermo@solbox:/tmp
$ for i in $(seq 1 5); do echo "Run $i:"; time zstd --single-thread -v -k -c -d unstable-x86_64.img.zstlong > /dev/null; done
Run 1:
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***
unstable-x86_64.img.zstlong: 10000000000 bytes                                 

real    0m2,979s
user    0m3,025s
sys     0m0,659s
Run 2:
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***
unstable-x86_64.img.zstlong: 10000000000 bytes                                 

real    0m3,000s
user    0m3,081s
sys     0m0,620s
Run 3:
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***
unstable-x86_64.img.zstlong: 10000000000 bytes                                 

real    0m2,996s
user    0m3,050s
sys     0m0,654s
Run 4:
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***
unstable-x86_64.img.zstlong: 10000000000 bytes                                 

real    0m2,972s
user    0m3,014s
sys     0m0,620s
Run 5:
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***
unstable-x86_64.img.zstlong: 10000000000 bytes                                 

real    0m2,995s
user    0m3,060s
sys     0m0,646s

zstd multi-threaded decompression

ermo@solbox:/tmp
$ for i in $(seq 1 5); do echo "Run $i:"; time zstd -T0 -v -k -c -d unstable-x86_64.img.zstlong > /dev/null; done
Run 1:
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***
Note: 8 physical core(s) detected 
unstable-x86_64.img.zstlong: 10000000000 bytes                                 

real    0m2,924s
user    0m2,972s
sys     0m0,584s
Run 2:
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***
Note: 8 physical core(s) detected 
unstable-x86_64.img.zstlong: 10000000000 bytes                                 

real    0m2,998s
user    0m3,039s
sys     0m0,670s
Run 3:
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***
Note: 8 physical core(s) detected 
unstable-x86_64.img.zstlong: 10000000000 bytes                                 

real    0m2,998s
user    0m3,020s
sys     0m0,676s
Run 4:
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***
Note: 8 physical core(s) detected 
unstable-x86_64.img.zstlong: 10000000000 bytes                                 

real    0m2,984s
user    0m3,034s
sys     0m0,652s
Run 5:
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***
Note: 8 physical core(s) detected 
unstable-x86_64.img.zstlong: 10000000000 bytes                                 

real    0m2,980s
user    0m3,058s
sys     0m0,635s

Summarized results

Compression sizes vs. Decompression time

ermo@solbox:/tmp
$ ls -lh unstable-x86_64.img.*
-rw-r--r-- 1 ermo ermo 262M Feb 16 23:00 unstable-x86_64.img.lz      | ST=50s, MT=12s
-rw-r--r-- 1 ermo ermo 288M Feb 16 23:00 unstable-x86_64.img.lz9     | (no data)
-rw-rw-r-- 1 ermo ermo 269M Feb 17 10:38 unstable-x86_64.img.xz      | ST=30s, MT=4s
-rw-r--r-- 1 ermo ermo 325M Feb 16 23:00 unstable-x86_64.img.zst     | (no data)
-rw-r--r-- 1 ermo ermo 282M Feb 16 23:00 unstable-x86_64.img.zstlong | ST=3s, MT=3s

Conclusion:

All three options take the same time to compress the image in terms of wall clock time.

For this particular test, plzip wins in terms of compression size (as hypothesized), but is also the slowest to decompress.

Surprisingly, it turns out that xz -T0 might be the best overall solution, provided the tooling we use can effectively utilise both multi-thread compression and decompression.

zstd is hands down the quickest decompression solution, at the cost of a slight increase in download size.