Skip to content

Instantly share code, notes, and snippets.

@klauspost
Created November 20, 2019 15:44
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save klauspost/8597feb49515b3811da17bee399b0e18 to your computer and use it in GitHub Desktop.

Congratulations on the Badger 2.0 release!

Being 'interested' in data compression I have a little feedback:

It uses the  zstd  compression algorithm when Badger is built with  Cgo  enabled. When built without Cgo enabled, it uses the  snappy  algorithm.

This seems like an odd choice. IMO compatibility shouldn't be determined by compilation settings. I would humbly like to point out that there is pure Go zstd implementation. While performance is only close to the cgo version, if you go for the fastest setting there shouldn't be much of a difference.

I don't know the reason for choosing Snappy, but LZ4 typically outperforms Snappy.

If you are willing to go with a "non-standard" scheme I have written S2 which is a Snappy extension that compresses better than Snappy and typically decompresses faster. You can see direct comparisons in the Block compression section. S2 can decompress Snappy blocks but not the other way around.

I assume you have verification of blocks since snappy blocks have no integrity check and datadog zstd doesn't read or write CRC info.

If you have some representative 'blocks' I can do a comparison between the different schemes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment