knoxite compression test

00-Test methodology.md

I ran the below script to backup one folder with mixed data. This is "production" export that I'm interested in backing up. It contains text and binary files of different sizes. Your test contents might drastically change the numbers presented.

The test was performed just once on a personal machine which wasn't used for the test duration, but might introduce a lot of variability into the equation. I'll be re-running these tests on a dedicated machine.

For my dataset zlib, gzip and flate are recommendable, while zstd and especially lzma should be avoided.

01-compression_test.sh

#!/usr/bin/env bash
set -euo pipefail

base_dir="${HOME}/.tmp/backups" # dir in home, since /tmp is in RAM
backup_content=( # List of dirs to make the snapshots of
  "${HOME}/projects/gits/knoxite"
)
log_file="log_compression"
stdout_log="${log_file}.stdout"

knoxite() {
  command knoxite "${@}" >"${stdout_log}"
}
export KNOXITE_PASSWORD=1234

for algo in none flate gzip lzma zlib zstd; do
  printf '##### %s #####\n' "${algo^^}"
  export KNOXITE_REPOSITORY="${base_dir}/${algo}"
  mkdir -p "${KNOXITE_REPOSITORY}"
  knoxite repo init
  knoxite volume init "My ${algo^^} Volume"
  start_date="$(date +%s)"
  for content in "${backup_content[@]}"; do
    #knoxite store latest "${content}" --compression "${algo}"
    /usr/bin/time knoxite store latest "${content}" --compression "${algo}" > "${stdout_log}"
  done
  end_date="$(date +%s)"
  backup_size="$(du -sh "${KNOXITE_REPOSITORY}" | cut -f 1)"
  printf 'Test for %s algorithm took %i seconds and takes %s of disk space\n' "${algo}" "$((end_date - start_date))" "${backup_size}"
done

02-stdout.raw

##### NONE #####
540.64user 40.76system 3:05.19elapsed 313%CPU (0avgtext+0avgdata 483884maxresident)k
21063176inputs+21354232outputs (7major+113392minor)pagefaults 0swaps
Test for none algorithm took 185 seconds and takes 11G of disk space
##### FLATE #####
1215.41user 45.42system 6:14.21elapsed 336%CPU (0avgtext+0avgdata 543248maxresident)k
21017848inputs+16004824outputs (6major+155906minor)pagefaults 0swaps
Test for flate algorithm took 374 seconds and takes 7.8G of disk space
##### GZIP #####
1191.05user 45.06system 6:26.02elapsed 320%CPU (0avgtext+0avgdata 530640maxresident)k
21013464inputs+16008008outputs (27major+142418minor)pagefaults 0swaps
Test for gzip algorithm took 386 seconds and takes 7.8G of disk space
##### LZMA #####
9795.31user 104.66system 41:53.14elapsed 393%CPU (0avgtext+0avgdata 988320maxresident)k
21017968inputs+15984560outputs (40major+353981minor)pagefaults 0swaps
Test for lzma algorithm took 2513 seconds and takes 7.8G of disk space
##### ZLIB #####
1203.18user 45.28system 5:55.04elapsed 351%CPU (0avgtext+0avgdata 532768maxresident)k
21011632inputs+16005744outputs (2major+136404minor)pagefaults 0swaps
Test for zlib algorithm took 355 seconds and takes 7.8G of disk space
##### ZSTD #####
4011.71user 77.33system 16:55.78elapsed 402%CPU (0avgtext+0avgdata 919160maxresident)k
21012080inputs+16057048outputs (7major+258047minor)pagefaults 0swaps
Test for zstd algorithm took 1015 seconds and takes 7.8G of disk space

03-Summary

Time (minutes:seconds):
none: 3:05
zlib: 5:55
flate: 6:14
gzip: 6:26
zstd: 16:56
lzma: 41:53

Memory (max resident):
none: 472.54M
gzip: 518.20M
zlib: 520.28M
flate: 530.52M
zstd: 897.62M
lzma: 965.16M

CPU:
none: 313%
gzip: 320%
flate: 336%
zlib: 351%
lzma: 393%
zstd: 402%