Skip to content

Instantly share code, notes, and snippets.

@prideout

prideout/compression_experiment.md

Last active March 29, 2022 20:53
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save prideout/4b4042ff8d629f81f33c73bacec245c4 to your computer and use it in GitHub Desktop.
Save prideout/4b4042ff8d629f81f33c73bacec245c4 to your computer and use it in GitHub Desktop.
Download ZIP
Ubershader Archive Compression
Raw
compression_experiment.md

zstd is a C library that features very fast decompression (similar to snappy) and dictionary-based compression.

Asking zstd to perform best possible compression is slow, but decompressing the result is still fast.

It also has command tool:

brew install zstd

GL-only release build

(all sizes are decimal)

  • Simply gzipping all the gltfio filamat files results in a 808K archive.
  • zstd results in 516K file, but with a 112K dictionary, together these gzip down to 520K.

To repro, set your CMake options to exclude Vk and Mtl, make a release build, then do:

cd out/cmake-release/libs/gltfio
du -ch *.filamat                       # Lists the uncompressed files = 3.3M
zstd --train *.filamat -o uberdict     # Produces the dictionary = 112K
zstd --ultra -22 -D uberdict *.filamat # Produces zst files = 516K
zip -9 with_zstd.zip *.zst uberdict    # gzips the dict + zst files = 520K
zip -9 without_zstd.zip *.filamat      # Simple gzip = 808K

GL+Mtl+Vk release build

  • Uncompressed gltfio filamat files total to 10M (!)
  • Simply gzipping the filamat files results in 2.7M.
  • The zstd+gzip procedure results in 1.8M.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment