Skip to content

Instantly share code, notes, and snippets.

@koraa
Created February 16, 2020 09:01
Show Gist options
  • Save koraa/d0622d1abbc8b428fb15ed2036425dc1 to your computer and use it in GitHub Desktop.
Save koraa/d0622d1abbc8b428fb15ed2036425dc1 to your computer and use it in GitHub Desktop.
huniq2 uniq optimization pass one
repetitions implemetation seconds memory/kb
5 0original 2.88 29484
5 1anyhow 2.91 29544
5 2iolock 2.97 29544
5 3noalloc 2.75 29520
5 4singlehash 2.51 29628
5 5ahash 2.06 29608
5 6fxhash 1.84 29536
5 7copyelision 1.51 29736
5 8cleanup 1.87 29468
5 9seeded-fxhas 1.71 29540
5 10cache-seed 1.48 29524
5 11xxh3 1.24 29492
5 huniq1 6.09 26128
5 quniq 1.75 111316
5 datamash 9.20 9956
5 awk 7.59 321940
5 shell 42.33 10172
10 0original 6.91 29532
10 1anyhow 5.98 29640
10 2iolock 6.64 29612
10 3noalloc 5.58 29520
10 4singlehash 4.66 29560
10 5ahash 4.37 29540
10 6fxhash 4.00 29596
10 7copyelision 3.46 29556
10 8cleanup 4.05 29732
10 9seeded-fxhas 3.59 29616
10 10cache-seed 3.51 29600
10 11xxh3 3.04 29552
10 huniq1 11.81 26132
10 quniq 3.67 111292
10 datamash 21.44 10036
10 awk 14.01 321960
10 shell 88.10 10252
50 0original 31.43 29532
50 1anyhow 31.48 29632
50 2iolock 31.08 29692
50 3noalloc 26.51 29620
50 4singlehash 24.12 29568
50 5ahash 21.26 29556
50 6fxhash 20.35 29736
50 7copyelision 16.06 29620
50 8cleanup 23.99 29520
50 9seeded-fxhas 22.01 29616
50 10cache-seed 18.05 29624
50 11xxh3 14.91 29544
50 huniq1 55.85 26176
50 quniq 13.31 111164
50 datamash 111.66 10892
50 awk 69.43 322044
50 shell 367.12 11020
5 0original 2.88 29544
5 1anyhow 2.93 29616
5 2iolock 3.06 29556
5 3noalloc 2.80 29672
5 4singlehash 2.28 29524
5 5ahash 1.99 29552
5 6fxhash 1.71 29516
5 7copyelision 1.48 29548
5 8cleanup 1.82 29528
5 9seeded-fxhas 1.66 29520
5 10cache-seed 1.42 29652
5 11xxh3 1.22 29492
5 huniq1 5.69 26176
5 quniq 1.41 111176
5 datamash 8.95 9964
5 awk 6.45 321952
5 shell 38.94 10040
10 0original 5.83 29668
10 1anyhow 5.75 29560
10 2iolock 5.87 29752
10 3noalloc 5.09 29620
10 4singlehash 4.39 29764
10 5ahash 4.41 29540
10 6fxhash 3.57 29496
10 7copyelision 2.84 29608
10 8cleanup 4.01 29516
10 9seeded-fxhas 3.53 29668
10 10cache-seed 2.90 29744
10 11xxh3 2.56 29600
10 huniq1 11.16 26108
10 quniq 2.55 111312
10 datamash 17.55 10096
10 awk 13.70 321936
10 shell 77.56 10056
50 0original 29.91 29484
50 1anyhow 29.65 29564
50 2iolock 30.76 29664
50 3noalloc 26.10 29544
50 4singlehash 23.80 29632
50 5ahash 21.35 29612
50 6fxhash 17.96 29556
50 7copyelision 15.03 29548
50 8cleanup 20.13 29668
50 9seeded-fxhas 17.17 29664
50 10cache-seed 14.42 29608
50 11xxh3 12.46 29548
50 huniq1 54.48 26136
50 quniq 11.45 111184
50 datamash 95.37 10896
50 awk 62.73 322052
50 shell 374.30 11180
5 0original 3.43 29644
5 1anyhow 3.19 29516
5 2iolock 3.16 29620
5 3noalloc 2.80 29564
5 4singlehash 2.45 29648
5 5ahash 2.18 29572
5 6fxhash 1.89 29608
5 7copyelision 1.66 29604
5 8cleanup 2.03 29544
5 9seeded-fxhas 1.83 29752
5 10cache-seed 1.79 29500
5 11xxh3 1.49 29528
5 huniq1 6.32 26112
5 quniq 1.96 111276
5 datamash 9.22 10000
5 awk 8.20 322048
5 shell 41.25 10144
10 0original 6.40 29608
10 1anyhow 6.49 29576
10 2iolock 6.90 29600
10 3noalloc 5.90 29704
10 4singlehash 5.01 29524
10 5ahash 4.49 29620
10 6fxhash 3.86 29592
10 7copyelision 3.34 29528
10 8cleanup 4.45 29536
10 9seeded-fxhas 3.70 29772
10 10cache-seed 3.28 29500
10 11xxh3 2.88 29532
10 huniq1 12.67 26164
10 quniq 3.51 111292
10 datamash 17.98 10000
10 awk 13.06 321928
10 shell 82.32 10048
50 0original 32.58 29488
50 1anyhow 35.97 29548
50 2iolock 31.15 29512
50 3noalloc 28.20 29620
50 4singlehash 26.81 29548
50 5ahash 24.10 29616
50 6fxhash 21.32 29652
50 7copyelision 17.27 29620
50 8cleanup 22.59 29584
50 9seeded-fxhas 19.67 29512
50 10cache-seed 17.08 29608
50 11xxh3 15.09 29592
50 huniq1 55.60 26176
50 quniq 12.39 111276
50 datamash 98.65 10892
50 awk 66.22 322052
50 shell 403.50 11120
5 0original 2.99 29528
5 1anyhow 2.88 29676
5 2iolock 2.85 29504
5 3noalloc 2.69 29624
5 4singlehash 2.13 29524
5 5ahash 1.96 29760
5 6fxhash 1.65 29608
5 7copyelision 1.63 29560
5 8cleanup 1.83 29744
5 9seeded-fxhas 1.61 29556
5 10cache-seed 1.42 29544
5 11xxh3 1.26 29604
5 huniq1 6.15 26172
5 quniq 1.36 111212
5 datamash 8.90 10060
5 awk 7.11 322040
5 shell 39.55 9944
10 0original 5.87 29548
10 1anyhow 5.76 29628
10 2iolock 5.70 29604
10 3noalloc 5.33 29616
10 4singlehash 4.45 29676
10 5ahash 4.14 29668
10 6fxhash 3.28 29696
10 7copyelision 2.99 29476
10 8cleanup 3.80 29536
10 9seeded-fxhas 3.32 29512
10 10cache-seed 2.93 29556
10 11xxh3 2.44 29492
10 huniq1 12.34 26092
10 quniq 2.71 111164
10 datamash 19.04 9936
10 awk 12.98 322008
10 shell 76.28 10148
50 0original 30.39 29576
50 1anyhow 29.46 29520
50 2iolock 30.67 29496
50 3noalloc 26.27 29560
50 4singlehash 23.21 29524
50 5ahash 20.86 29624
50 6fxhash 17.65 29544
50 7copyelision 14.93 29500
50 8cleanup 19.12 29604
50 9seeded-fxhas 16.58 29524
50 10cache-seed 14.71 29524
50 11xxh3 13.10 29540
50 huniq1 55.01 26172
50 quniq 11.53 111280
50 datamash 115.15 10836
50 awk 74.74 322004
50 shell 399.18 11016
5 0original 3.14 29532
5 1anyhow 3.15 29696
5 2iolock 2.90 29612
5 3noalloc 2.52 29628
5 4singlehash 2.27 29764
5 5ahash 2.05 29516
5 6fxhash 1.69 29552
5 7copyelision 1.45 29524
5 8cleanup 1.89 29544
5 9seeded-fxhas 1.81 29692
5 10cache-seed 1.60 29496
5 11xxh3 1.51 29552
5 huniq1 8.56 26088
5 quniq 1.83 111184
5 datamash 13.15 9948
5 awk 8.63 321980
5 shell 40.66 10000
10 0original 6.51 29536
10 1anyhow 6.39 29620
10 2iolock 6.96 29588
10 3noalloc 5.65 29540
10 4singlehash 5.34 29628
10 5ahash 5.20 29620
10 6fxhash 3.82 29608
10 7copyelision 3.01 29608
10 8cleanup 3.81 29516
10 9seeded-fxhas 3.55 29552
10 10cache-seed 3.39 29600
10 11xxh3 3.27 29488
10 huniq1 13.59 26176
10 quniq 3.06 111168
10 datamash 22.39 10032
10 awk 15.05 321936
10 shell 90.57 10152
50 0original 34.12 29536
50 1anyhow 34.02 29676
50 2iolock 34.50 29616
50 3noalloc 30.25 29540
50 4singlehash 26.77 29628
50 5ahash 23.47 29756
50 6fxhash 20.30 29496
50 7copyelision 17.71 29616
50 8cleanup 21.89 29536
50 9seeded-fxhas 19.45 29508
50 10cache-seed 17.58 29532
50 11xxh3 15.21 29548
50 huniq1 59.62 26084
50 quniq 13.86 111216
50 datamash 111.62 10836
50 awk 72.52 322020
50 shell 486.72 10956
5 0original 3.63 29592
5 1anyhow 3.87 29700
5 2iolock 3.80 29668
5 3noalloc 3.07 29628
5 4singlehash 2.72 29764
5 5ahash 2.40 29512
5 6fxhash 2.15 29648
5 7copyelision 1.80 29496
5 8cleanup 2.24 29672
5 9seeded-fxhas 2.00 29632
5 10cache-seed 1.80 29608
5 11xxh3 1.61 29496
5 huniq1 7.49 26112
5 quniq 1.72 111108
5 datamash 10.58 9992
5 awk 7.85 322048
5 shell 48.56 9992
10 0original 6.54 29492
10 1anyhow 6.41 29640
10 2iolock 6.33 29540
10 3noalloc 5.37 29704
10 4singlehash 4.70 29636
10 5ahash 4.42 29620
10 6fxhash 3.80 29528
10 7copyelision 3.03 29556
10 8cleanup 4.25 29520
10 9seeded-fxhas 3.62 29508
10 10cache-seed 3.27 29736
10 11xxh3 2.66 29520
10 huniq1 12.16 26128
10 quniq 2.65 111252
10 datamash 18.60 10000
10 awk 13.59 322004
10 shell 95.87 10148
50 0original 34.57 29728
50 1anyhow 32.63 29620
50 2iolock 34.25 29664
50 3noalloc 28.64 29548
50 4singlehash 24.15 29708
50 5ahash 23.81 29572
50 6fxhash 19.15 29544
50 7copyelision 16.44 29608
50 8cleanup 24.71 29520
50 9seeded-fxhas 18.62 29688
50 10cache-seed 15.56 29620
50 11xxh3 14.26 29644
50 huniq1 57.22 26136
50 quniq 11.67 111220
50 datamash 100.18 10844
50 awk 71.96 321956
50 shell 408.44 10960
5 0original 3.14 29728
5 1anyhow 3.09 29560
5 2iolock 3.23 29572
5 3noalloc 2.88 29580
5 4singlehash 2.27 29528
5 5ahash 2.01 29512
5 6fxhash 1.72 29496
5 7copyelision 1.41 29556
5 8cleanup 1.94 29604
5 9seeded-fxhas 1.79 29552
5 10cache-seed 1.61 29596
5 11xxh3 1.31 29616
5 huniq1 7.00 26176
5 quniq 1.53 111160
5 datamash 9.47 10096
5 awk 8.52 321980
5 shell 42.64 9988
10 0original 5.80 29604
10 1anyhow 6.14 29544
10 2iolock 7.63 29508
10 3noalloc 6.57 29628
10 4singlehash 5.44 29560
10 5ahash 5.75 29564
10 6fxhash 5.61 29492
10 7copyelision 4.24 29500
10 8cleanup 4.49 29488
10 9seeded-fxhas 3.56 29748
10 10cache-seed 3.40 29496
10 11xxh3 2.94 29736
10 huniq1 14.69 26180
10 quniq 3.35 111216
10 datamash 20.79 10056
10 awk 15.54 322040
10 shell 94.25 10252
50 0original 39.11 29520
50 1anyhow 38.15 29524
50 2iolock 39.13 29512
50 3noalloc 34.34 29620
50 4singlehash 30.30 29764
50 5ahash 23.59 29672
50 6fxhash 20.46 29604
50 7copyelision 16.88 29604
50 8cleanup 23.06 29528
50 9seeded-fxhas 19.15 29548
50 10cache-seed 18.62 29680
50 11xxh3 14.19 29492
50 huniq1 62.12 26112
50 quniq 12.93 111164
50 datamash 113.02 10896
50 awk 86.06 322028
50 shell 419.46 10944
5 0original 3.13 29592
5 1anyhow 3.11 29544
5 2iolock 3.72 29508
5 3noalloc 2.73 29576
5 4singlehash 2.33 29524
5 5ahash 2.13 29560
5 6fxhash 1.81 29556
5 7copyelision 1.57 29556
5 8cleanup 1.95 29668
5 9seeded-fxhas 1.74 29548
5 10cache-seed 1.60 29532
5 11xxh3 1.34 29740
5 huniq1 6.88 26080
5 quniq 1.60 111276
5 datamash 9.06 9956
5 awk 7.30 322012
5 shell 42.89 10048
10 0original 6.24 29488
10 1anyhow 6.18 29564
10 2iolock 6.22 29508
10 3noalloc 5.42 29548
10 4singlehash 4.81 29580
10 5ahash 4.39 29636
10 6fxhash 3.70 29604
10 7copyelision 3.05 29620
10 8cleanup 3.94 29684
10 9seeded-fxhas 3.83 29568
10 10cache-seed 3.40 29500
10 11xxh3 2.87 29536
10 huniq1 12.65 26164
10 quniq 2.70 111176
10 datamash 20.39 10096
10 awk 15.51 321940
10 shell 87.80 10056
50 0original 32.58 29484
50 1anyhow 33.86 29640
50 2iolock 33.46 29600
50 3noalloc 27.88 29620
50 4singlehash 25.09 29584
50 5ahash 21.35 29532
50 6fxhash 18.63 29532
50 7copyelision 15.58 29500
50 8cleanup 20.89 29640
50 9seeded-fxhas 17.91 29660
50 10cache-seed 16.16 29500
50 11xxh3 14.00 29520
50 huniq1 57.20 26172
50 quniq 13.10 111160
50 datamash 109.55 10996
50 awk 67.33 322052
50 shell 369.63 11152
5 0original 2.90 29488
5 1anyhow 2.80 29520
5 2iolock 3.00 29696
5 3noalloc 2.54 29628
5 4singlehash 2.16 29632
5 5ahash 1.90 29556
5 6fxhash 1.60 29552
5 7copyelision 1.39 29524
5 8cleanup 1.94 29532
5 9seeded-fxhas 1.60 29664
5 10cache-seed 1.47 29524
5 11xxh3 1.24 29556
5 huniq1 6.09 26092
5 quniq 1.45 111108
5 datamash 9.79 9992
5 awk 6.82 321960
5 shell 37.74 10144
10 0original 5.99 29588
10 1anyhow 6.01 29516
10 2iolock 6.41 29632
10 3noalloc 5.33 29672
10 4singlehash 4.66 29648
10 5ahash 4.09 29672
10 6fxhash 3.56 29600
10 7copyelision 3.04 29536
10 8cleanup 4.20 29588
10 9seeded-fxhas 3.33 29748
10 10cache-seed 2.98 29496
10 11xxh3 2.55 29524
10 huniq1 11.22 26108
10 quniq 2.59 111172
10 datamash 18.14 9928
10 awk 13.78 322004
10 shell 77.44 10060
50 0original 30.09 29512
50 1anyhow 32.12 29672
50 2iolock 36.64 29612
50 3noalloc 33.45 29628
50 4singlehash 25.57 29528
50 5ahash 21.57 29516
50 6fxhash 17.98 29544
50 7copyelision 15.48 29608
50 8cleanup 23.84 29544
50 9seeded-fxhas 21.29 29612
50 10cache-seed 17.26 29552
50 11xxh3 14.47 29592
50 huniq1 57.91 26176
50 quniq 13.13 111248
50 datamash 112.31 10896
50 awk 68.22 322044
50 shell 445.40 11048
5 0original 3.34 29508
5 1anyhow 3.44 29620
5 2iolock 3.50 29620
5 3noalloc 2.85 29628
5 4singlehash 2.51 29520
5 5ahash 2.41 29520
5 6fxhash 1.85 29540
5 7copyelision 1.65 29612
5 8cleanup 2.32 29472
5 9seeded-fxhas 1.89 29620
5 10cache-seed 1.67 29600
5 11xxh3 1.55 29492
5 huniq1 7.53 26196
5 quniq 1.71 111164
5 datamash 10.03 9904
5 awk 8.12 321940
5 shell 44.84 10052
10 0original 6.79 29488
10 1anyhow 6.32 29632
10 2iolock 6.94 29548
10 3noalloc 5.69 29524
10 4singlehash 4.83 29524
10 5ahash 4.72 29696
10 6fxhash 3.89 29500
10 7copyelision 3.29 29500
10 8cleanup 4.14 29724
10 9seeded-fxhas 3.62 29620
10 10cache-seed 3.30 29496
10 11xxh3 2.92 29600
10 huniq1 12.31 26132
10 quniq 3.28 111296
10 datamash 20.99 10036
10 awk 15.80 321956
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment