zstd urls performance comparison

gistfile1.md

ZSTD compression benchmarks

File: close to 5gb of urls extracted from CommonCrawl data, after sort -u

+------------+------------+--------+------+--------------+
| ZSTD level | Time       | Ratio  | Size | Size (bytes) |
+============+============+========+======+==============+
| 1          |  0m16.462s | 16.25% | 832M | 872196390    |
+------------+------------+--------+------+--------------+
| 2          |  0m19.339s | 16.02% | 821M | 860134887    |
+------------+------------+--------+------+--------------+
| 3          |  0m25.771s | 15.64% | 801M | 839619119    |
+------------+------------+--------+------+--------------+
| 4          |  0m30.111s | 15.63% | 801M | 839266269    |
+------------+------------+--------+------+--------------+
| 5          |  1m0.817s  | 14.85% | 761M | 797393562    |
+------------+------------+--------+------+--------------+
| 6          |  1m22.100s | 14.69% | 753M | 788561778    |
+------------+------------+--------+------+--------------+
| 7          |  1m29.658s | 14.07% | 721M | 755186767    |
+------------+------------+--------+------+--------------+
| 8          |  1m49.934s | 13.81% | 707M | 741232051    |
+------------+------------+--------+------+--------------+
| 9          |  2m29.740s | 13.74% | 704M | 737584455    |
+------------+------------+--------+------+--------------+
| 10         |  3m49.963s | 13.63% | 698M | 731525591    |
+------------+------------+--------+------+--------------+
| 11         |  4m53.604s | 13.57% | 695M | 728624733    |
+------------+------------+--------+------+--------------+
| 12         |  6m31.217s | 13.52% | 693M | 725747036    |
+------------+------------+--------+------+--------------+
| 13         |  6m37.774s | 13.48% | 691M | 723767315    |
+------------+------------+--------+------+--------------+
| 14         |  7m49.687s | 13.40% | 687M | 719550316    |
+------------+------------+--------+------+--------------+
| 15         | 10m47.738s | 13.32% | 682M | 715082801    |
+------------+------------+--------+------+--------------+
| 16         | 16m10.087s | 13.07% | 670M | 701771738    |
+------------+------------+--------+------+--------------+
| 17         | 24m32.162s | 12.77% | 654M | 685535421    |
+------------+------------+--------+------+--------------+
| 18         | 35m 3.523s | 12.68% | 650M | 680886789    |
+------------+------------+--------+------+--------------+
| 19         | 60m16.104s | 12.38% | 635M | 664886399    |
+------------+------------+--------+------+--------------+
| 20 --ultra | 68m50.976s | 12.10% | 620M | 649851015    |
+------------+------------+--------+------+--------------+
| 21 --ultra | 78m20.160s | 11.93% | 611M | 640601969    |
+------------+------------+--------+------+--------------+