Tuple performance testing with core.matrix

gistfile1.clj

;; Using criterium to test performance of Tuples
;;
;; Benchmark using parts of the core.matrix implementation test suite
;;
;; ===========================================================
;; Common setup:

(use '[criterium.core :as c])
(require 'clojure.core.matrix.compliance-tester)

;; ===========================================================
;; clojure-1.7.0 (latest release)

=> (c/quick-bench  (clojure.core.matrix.compliance-tester/test-implementation :persistent-vector))
WARNING: Final GC required 19.63663961511742 % of runtime
Evaluation count : 174384 in 6 samples of 29064 calls.
             Execution time mean : 3.451892 µs
    Execution time std-deviation : 7.879428 ns
   Execution time lower quantile : 3.443191 µs ( 2.5%)
   Execution time upper quantile : 3.460931 µs (97.5%)
                   Overhead used : 2.386776 ns

=> (c/quick-bench  (clojure.core.matrix.compliance-tester/instance-test [1 2 3 4]))
WARNING: Final GC required 43.40813590012227 % of runtime
Evaluation count : 654 in 6 samples of 109 calls.
             Execution time mean : 912.850700 µs                *;; really?*
    Execution time std-deviation : 8.026725 µs
   Execution time lower quantile : 902.902495 µs ( 2.5%)
   Execution time upper quantile : 921.503866 µs (97.5%)
                   Overhead used : 2.386776 ns

;; ===========================================================
;; clojure-1.8.0-alpha2 (current master)

=> (c/quick-bench  (clojure.core.matrix.compliance-tester/test-implementation :persistent-vector))
WARNING: Final GC required 16.4858267835708 % of runtime
Evaluation count : 198138 in 6 samples of 33023 calls.
             Execution time mean : 3.014784 µs
    Execution time std-deviation : 4.422516 ns
   Execution time lower quantile : 3.010180 µs ( 2.5%)
   Execution time upper quantile : 3.021322 µs (97.5%)
                   Overhead used : 2.668436 ns
                   
=> (c/quick-bench  (clojure.core.matrix.compliance-tester/instance-test [1 2 3 4]))
WARNING: Final GC required 37.97769588255513 % of runtime
Evaluation count : 492 in 6 samples of 82 calls.
             Execution time mean : 1.222167 ms
    Execution time std-deviation : 8.185579 µs
   Execution time lower quantile : 1.209295 ms ( 2.5%)
   Execution time upper quantile : 1.229899 ms (97.5%)
                   Overhead used : 2.668436 ns

;; ===========================================================
;; clojure-1.8.0-temp (Mike's implementation using Zach Tellman's CLJ-1517 unrolled vectors)
;; 
;; see: https://github.com/mikera/clojure/tree/clj-1517
;; Specific commit: https://github.com/mikera/clojure/commit/39f5a0be18811a0883fcb6789d1e30a256032d94

=> (c/quick-bench  (clojure.core.matrix.compliance-tester/test-implementation :persistent-vector))
WARNING: Final GC required 15.0502480080872 % of runtime
Evaluation count : 207318 in 6 samples of 34553 calls.
             Execution time mean : 2.893920 µs
    Execution time std-deviation : 2.357140 ns
   Execution time lower quantile : 2.891510 µs ( 2.5%)
   Execution time upper quantile : 2.897401 µs (97.5%)
                   Overhead used : 2.371984 ns
                   
=> (c/quick-bench  (clojure.core.matrix.compliance-tester/instance-test [1 2 3 4]))
WARNING: Final GC required 22.18890187636415 % of runtime
Evaluation count : 588 in 6 samples of 98 calls.
             Execution time mean : 1.023060 ms
    Execution time std-deviation : 6.188940 µs
   Execution time lower quantile : 1.019909 ms ( 2.5%)
   Execution time upper quantile : 1.033670 ms (97.5%)
                   Overhead used : 2.401067 ns

;; =======================
;; Conclusion
;;
;; Zach's CLJ-1517 approach seems to have a slight edge over 1.8.0-alpha2 for these test cases:
;;    ~5% faster for the :persistent-vector implementation test
;;    ~20% faster for testing a specific tuple instance (4 elements)
;;
;; Interestingly, Clojure 1.7.0 seems marginally faster than either for the instance test
;;
;; Need more investigation as to why this is... but I think it might be because the instances 
;; are getting converted into persistent vectors at some point