Create a gist now

Instantly share code, notes, and snippets.

Specter 0.13.0 benchmark
Benchmark code at: https://github.com/nathanmarz/specter/blob/master/scripts/benchmarks.clj
Run against Clojure 1.7.0 and Java 1.7.0 on Mac OSX 10.11.6
Benchmark: get value in nested map (2500000 iterations)
Avg(ms) vs best Code
53.528 1.00 (-> data (get :a) (get :b) (get :c))
54.708 1.02 (-> data :a :b :c)
103.03 1.92 (compiled-select-any p data)
109.92 2.05 (select-any [:a :b :c] data)
111.81 2.09 (select-any [(keypath :a) (keypath :b) (keypath :c)] data)
158.11 2.95 (specter-dynamic-nested-get data :a :b :c)
170.28 3.18 (get-in data [:a :b :c])
********************************
Benchmark: update value in nested map (500000 iterations)
Avg(ms) vs best Code
83.648 1.00 (manual-transform data inc)
86.441 1.03 (transform [:a :b :c] inc data)
529.97 6.34 (update-in data [:a :b :c] inc)
********************************
Benchmark: transform values of a small map (500000 iterations)
Avg(ms) vs best Code
45.241 1.00 (transform MAP-VALS inc data)
74.505 1.65 (reduce-kv (fn [m k v] (assoc m k (inc v))) {} data)
93.615 2.07 (reduce-kv (fn [m k v] (assoc m k (inc v))) (empty data) data)
102.49 2.27 (persistent! (reduce-kv (fn [m k v] (assoc! m k (inc v))) (transient {}) data))
170.22 3.76 (transform [ALL LAST] inc data)
424.46 9.38 (into {} (for [[k v] data] [k (inc v)]))
476.91 10.5 (zipmap (keys data) (map inc (vals data)))
491.83 10.9 (into {} (map (fn [e] [(key e) (inc (val e))]) data))
********************************
Benchmark: transform values of large map (600 iterations)
Avg(ms) vs best Code
89.321 1.00 (persistent! (reduce-kv (fn [m k v] (assoc! m k (inc v))) (transient clojure.lang.PersistentHashMap/EMPTY) data))
94.117 1.05 (persistent! (reduce-kv (fn [m k v] (assoc! m k (inc v))) (transient {}) data))
94.839 1.06 (transform MAP-VALS inc data)
115.98 1.30 (reduce-kv (fn [m k v] (assoc m k (inc v))) {} data)
120.07 1.34 (reduce-kv (fn [m k v] (assoc m k (inc v))) (empty data) data)
132.00 1.48 (transform [ALL LAST] inc data)
188.62 2.11 (into {} (for [[k v] data] [k (inc v)]))
196.81 2.20 (into {} (map (fn [e] [(key e) (inc (val e))]) data))
258.00 2.89 (zipmap (keys data) (map inc (vals data)))
********************************
Benchmark: map a function over a vector (1000000 iterations)
Avg(ms) vs best Code
165.45 1.00 (into [] (map inc) data)
169.01 1.02 (mapv inc data)
188.44 1.14 (transform ALL inc data)
442.41 2.67 (vec (map inc data))
********************************
Benchmark: filter a sequence (500000 iterations)
Avg(ms) vs best Code
110.79 1.00 (filterv even? data)
129.44 1.17 (select [ALL even?] data)
167.40 1.51 (into [] (filter even?) data)
204.44 1.85 (doall (filter even? data))
246.84 2.23 (select-any (filterer even?) data)
********************************
Benchmark: even :a values from sequence of maps (500000 iterations)
Avg(ms) vs best Code
83.723 1.00 (into [] xf data)
128.32 1.53 (select [ALL :a even?] data)
148.40 1.77 (into [] (comp (map :a) (filter even?)) data)
312.08 3.73 (->> data (mapv :a) (filter even?) doall)
********************************
Benchmark: update every value in a tree (represented with vectors) (50000 iterations)
Avg(ms) vs best Code
81.533 1.00 (tree-value-transform (fn [e] (if (even? e) (inc e) e)) data)
88.786 1.09 (transform [TreeValuesProt even?] inc data)
94.852 1.16 (transform [TreeValues even?] inc data)
324.52 3.98 (walk/postwalk (fn [e] (if (and (number? e) (even? e)) (inc e) e)) data)
339.70 4.17 (transform [(walker number?) even?] inc data)
********************************
Benchmark: Traverse into a set (5000 iterations)
Avg(ms) vs best Code
815.69 1.00 (into #{} (traverse ALL data))
824.53 1.01 (set data)
883.21 1.08 (persistent! (reduce conj! (transient #{}) (traverse ALL data)))
997.47 1.22 (set (select ALL data))
1258.4 1.54 (reduce conj #{} (traverse ALL data))
Benchmark code at: https://github.com/nathanmarz/specter/blob/master/scripts/benchmarks.clj
Run against Clojure 1.8.0 and Java 1.7.0 on Mac OSX 10.11.6
Benchmark: get value in nested map (2500000 iterations)
Avg(ms) vs best Code
56.953 1.00 (-> data :a :b :c)
62.537 1.10 (-> data (get :a) (get :b) (get :c))
106.82 1.88 (compiled-select-any p data)
113.30 1.99 (select-any [(keypath :a) (keypath :b) (keypath :c)] data)
118.95 2.09 (select-any [:a :b :c] data)
136.59 2.40 (select-one [:a :b :c] data)
139.85 2.46 (select-first [:a :b :c] data)
147.64 2.59 (select-one! [:a :b :c] data)
165.78 2.91 (specter-dynamic-nested-get data :a :b :c)
175.39 3.08 (get-in data [:a :b :c])
********************************
Benchmark: update value in nested map (500000 iterations)
Avg(ms) vs best Code
88.003 1.00 (transform [:a :b :c] inc data)
100.64 1.14 (manual-transform data inc)
509.91 5.79 (update-in data [:a :b :c] inc)
********************************
Benchmark: transform values of a small map (500000 iterations)
Avg(ms) vs best Code
52.867 1.00 (transform MAP-VALS inc data)
78.891 1.49 (map-vals-map-iterable data inc)
124.30 2.35 (map-vals-map-iterable-transient data inc)
153.92 2.91 (persistent! (reduce-kv (fn [m k v] (assoc! m k (inc v))) (transient {}) data))
154.18 2.92 (reduce-kv (fn [m k v] (assoc m k (inc v))) (empty data) data)
158.01 2.99 (reduce-kv (fn [m k v] (assoc m k (inc v))) {} data)
177.76 3.36 (transform [ALL LAST] inc data)
414.94 7.85 (zipmap (keys data) (map inc (vals data)))
471.38 8.92 (into {} (for [[k v] data] [k (inc v)]))
471.92 8.93 (into {} (map (fn [e] [(key e) (inc (val e))]) data))
********************************
Benchmark: transform values of large map (600 iterations)
Avg(ms) vs best Code
115.11 1.00 (persistent! (reduce-kv (fn [m k v] (assoc! m k (inc v))) (transient {}) data))
119.70 1.04 (transform MAP-VALS inc data)
120.55 1.05 (persistent! (reduce-kv (fn [m k v] (assoc! m k (inc v))) (transient clojure.lang.PersistentHashMap/EMPTY) data))
135.85 1.18 (reduce-kv (fn [m k v] (assoc m k (inc v))) (empty data) data)
138.02 1.20 (map-vals-map-iterable-transient data inc)
142.85 1.24 (reduce-kv (fn [m k v] (assoc m k (inc v))) {} data)
160.36 1.39 (transform [ALL LAST] inc data)
162.97 1.42 (map-vals-map-iterable data inc)
208.67 1.81 (into {} (for [[k v] data] [k (inc v)]))
211.33 1.84 (into {} (map (fn [e] [(key e) (inc (val e))]) data))
236.55 2.06 (zipmap (keys data) (map inc (vals data)))
********************************
Benchmark: map a function over a vector (1000000 iterations)
Avg(ms) vs best Code
164.71 1.00 (into [] (map inc) data)
166.34 1.01 (mapv inc data)
198.93 1.21 (transform ALL inc data)
438.70 2.66 (vec (map inc data))
********************************
Benchmark: filter a sequence (500000 iterations)
Avg(ms) vs best Code
94.955 1.00 (filterv even? data)
121.26 1.28 (select [ALL even?] data)
160.50 1.69 (doall (filter even? data))
163.26 1.72 (into [] (filter even?) data)
229.03 2.41 (select-any (filterer even?) data)
********************************
Benchmark: even :a values from sequence of maps (500000 iterations)
Avg(ms) vs best Code
80.767 1.00 (into [] xf data)
125.40 1.55 (select [ALL :a even?] data)
148.34 1.84 (into [] (comp (map :a) (filter even?)) data)
257.12 3.18 (->> data (mapv :a) (filter even?) doall)
********************************
Benchmark: update every value in a tree (represented with vectors) (50000 iterations)
Avg(ms) vs best Code
72.504 1.00 (tree-value-transform (fn [e] (if (even? e) (inc e) e)) data)
76.809 1.06 (transform [TreeValuesProt even?] inc data)
89.608 1.24 (transform [TreeValues even?] inc data)
247.96 3.42 (transform [(walker number?) even?] inc data)
289.12 3.99 (walk/postwalk (fn [e] (if (and (number? e) (even? e)) (inc e) e)) data)
********************************
Benchmark: Traverse into a set (5000 iterations)
Avg(ms) vs best Code
788.05 1.00 (set data)
846.60 1.07 (into #{} (traverse ALL data))
879.60 1.12 (persistent! (reduce conj! (transient #{}) (traverse ALL data)))
962.73 1.22 (set (select ALL data))
1376.1 1.75 (reduce conj #{} (traverse ALL data))
@greywolve

This is useful, but is there any reason you didn't use something like Criterium to benchmark? I'm really weary of micro benchmark results. Thanks for sharing though. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment