Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Compression on large JSON file

The Data

Compression

  • Original: 708MB
  • xz -3: 70MB, 5:41.02 (2.07 mb/sec - 10:1 ratio)
  • bzip2 -3: 74MB, 4:39.11 (2.53 mb/sec - 9.5:1 ratio)
  • gzip -3: 103MB: 0:15.15 (46.73 mb/sec - 6.87:1 ratio)
  • lzop -3: 146MB, 0:06.53 (108.42 mb/sec - 4.85:1 ratio)

Decompression

(value is seconds, lower is faster)

  • bzip2 - 39.4
  • xz - 14.97
  • gzip - 7.98
  • lzop - 2.37

The Test

echo bzip2 gzip xz lzop | xargs -tn1 sh -c 'time $1 -3c data.json > data.json.$1' -

produces 'data.json.{compression_type}' files

@mdumitrean

This comment has been minimized.

Copy link

@mdumitrean mdumitrean commented Aug 8, 2017

How about brotli?

@muthukumarse

This comment has been minimized.

Copy link

@muthukumarse muthukumarse commented Nov 13, 2018

xz is best from my usage

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment