Skip to content

Instantly share code, notes, and snippets.

@DigitalDwagon

DigitalDwagon/gist:580b9182ad076e61c0f0c3505d6bbdf9 Secret

Last active December 16, 2024 05:49
Show Gist options
  • Save DigitalDwagon/580b9182ad076e61c0f0c3505d6bbdf9 to your computer and use it in GitHub Desktop.
Save DigitalDwagon/580b9182ad076e61c0f0c3505d6bbdf9 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. DigitalDwagon renamed this gist Dec 16, 2024. 1 changed file with 0 additions and 0 deletions.
    0 gistfile1.txt → gistfile1.md
    View file
    File renamed without changes.
  2. DigitalDwagon created this gist Dec 16, 2024.
    50 changes: 50 additions & 0 deletions gistfile1.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,50 @@
    # ZSTD compression benchmarks
    File: close to 5gb of urls extracted from CommonCrawl data, after `sort -u`

    ```
    +------------+------------+--------+------+--------------+
    | ZSTD level | Time | Ratio | Size | Size (bytes) |
    +============+============+========+======+==============+
    | 1 | 0m16.462s | 16.25% | 832M | 872196390 |
    +------------+------------+--------+------+--------------+
    | 2 | 0m19.339s | 16.02% | 821M | 860134887 |
    +------------+------------+--------+------+--------------+
    | 3 | 0m25.771s | 15.64% | 801M | 839619119 |
    +------------+------------+--------+------+--------------+
    | 4 | 0m30.111s | 15.63% | 801M | 839266269 |
    +------------+------------+--------+------+--------------+
    | 5 | 1m0.817s | 14.85% | 761M | 797393562 |
    +------------+------------+--------+------+--------------+
    | 6 | 1m22.100s | 14.69% | 753M | 788561778 |
    +------------+------------+--------+------+--------------+
    | 7 | 1m29.658s | 14.07% | 721M | 755186767 |
    +------------+------------+--------+------+--------------+
    | 8 | 1m49.934s | 13.81% | 707M | 741232051 |
    +------------+------------+--------+------+--------------+
    | 9 | 2m29.740s | 13.74% | 704M | 737584455 |
    +------------+------------+--------+------+--------------+
    | 10 | 3m49.963s | 13.63% | 698M | 731525591 |
    +------------+------------+--------+------+--------------+
    | 11 | 4m53.604s | 13.57% | 695M | 728624733 |
    +------------+------------+--------+------+--------------+
    | 12 | 6m31.217s | 13.52% | 693M | 725747036 |
    +------------+------------+--------+------+--------------+
    | 13 | 6m37.774s | 13.48% | 691M | 723767315 |
    +------------+------------+--------+------+--------------+
    | 14 | 7m49.687s | 13.40% | 687M | 719550316 |
    +------------+------------+--------+------+--------------+
    | 15 | 10m47.738s | 13.32% | 682M | 715082801 |
    +------------+------------+--------+------+--------------+
    | 16 | 16m10.087s | 13.07% | 670M | 701771738 |
    +------------+------------+--------+------+--------------+
    | 17 | 24m32.162s | 12.77% | 654M | 685535421 |
    +------------+------------+--------+------+--------------+
    | 18 | 35m 3.523s | 12.68% | 650M | 680886789 |
    +------------+------------+--------+------+--------------+
    | 19 | 60m16.104s | 12.38% | 635M | 664886399 |
    +------------+------------+--------+------+--------------+
    | 20 --ultra | 68m50.976s | 12.10% | 620M | 649851015 |
    +------------+------------+--------+------+--------------+
    | 21 --ultra | 78m20.160s | 11.93% | 611M | 640601969 |
    +------------+------------+--------+------+--------------+
    ```