Skip to content

Instantly share code, notes, and snippets.

@klauspost

klauspost/post.md Secret

Created November 20, 2019 15:44
Show Gist options
  • Save klauspost/8597feb49515b3811da17bee399b0e18 to your computer and use it in GitHub Desktop.
Save klauspost/8597feb49515b3811da17bee399b0e18 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
post.md

Congratulations on the Badger 2.0 release!

Being 'interested' in data compression I have a little feedback:

It uses the  zstd  compression algorithm when Badger is built with  Cgo  enabled. When built without Cgo enabled, it uses the  snappy  algorithm.

This seems like an odd choice. IMO compatibility shouldn't be determined by compilation settings. I would humbly like to point out that there is pure Go zstd implementation. While performance is only close to the cgo version, if you go for the fastest setting there shouldn't be much of a difference.

I don't know the reason for choosing Snappy, but LZ4 typically outperforms Snappy.

If you are willing to go with a "non-standard" scheme I have written S2 which is a Snappy extension that compresses better than Snappy and typically decompresses faster. You can see direct comparisons in the Block compression section. S2 can decompress Snappy blocks but not the other way around.

I assume you have verification of blocks since snappy blocks have no integrity check and datadog zstd doesn't read or write CRC info.

If you have some representative 'blocks' I can do a comparison between the different schemes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment