Experiences using Midje & what we can learn from it

midje-learnings.md

I'd like to briefly present some ideas on:

• What Midje has to offer, both the good and the bad
• How to take all the Midje goodness over to clojure.test and clojurescript

What is Midje and why am I talking about it?

Midje started back in 2010, and was the vision of someone with decades worth of experience when it comes to software testing.

Up through 2013 it accumulated a bunch of useful features:

• mocking: mocks based on argument-predicate matches and assert call-counts
• test runner with watching / auto-test
• checkers: base values, sequences, and maps
• meta-constants

My organization, which has lots of Clojure code, has been using Midje for the last ~5 years. I joined 2 years ago and took up maintanence of the project after the creator moved on to other projects and communities.

Some pain points

• runs tests at namespace load time: difficult to integrate with in-editor and standalone test runners
• DSL evolution happened organically. For example, provided, background, against-background all do similar or identical things
• non-lispy syntax of DSL is at odds with the rest of the ecosystem
• changes and fixes are non-trivial given codebase size and the DSL's expressiveness

Porting Midje's ideas to a framework-agnostic setup

Break-up the features in Midje into a series of smaller repositories that can be used with different test frameworks and with both Clojure and Clojurescript:

• matcher-combinators, like Midje checkers but tends to be less verbose and include more diffing information
• mockfn is a rewrite of the most useful Midje mocking features
• A test runner. TBD, I haven't had a chance to seriously try any out.

All of these tools can be layered on top of clojure.test to get a very similar experience to Midje.

At Nubank we also use Midje to run our integration tests via selvage. I've extended this to work with clojure.test as well.

Migrating unit tests

tabular

clojure.test has are, which is pretty much midje's tabular. For example:

(facts "on prometheus metrics"
  (tabular
    (GET :text "/prometheus/metrics" 200) => (contains ?metrics)
    ?metrics
    "TYPE services_http_requests_total counter"
    "TYPE services_http_errors_total counter"
    "TYPE services_http_request_latency_seconds histogram"))

becomes

(deftest prometheus "on prometheus metrics"
  (are [metric]
       (is (clojure.string/includes? (GET :text "/prometheus/metrics" 200) metric))
       "TYPE services_http_requests_total counter"
       "TYPE services_http_errors_total counter"
       "TYPE services_http_request_latency_seconds histogram"))

mocks and metaconstants via mockfn

mockfn is very similar to midje's provided, you can assert call counts and do redefs for specific invocations by matching arguments (even using matcher-combinators). It is also more expressive, because you can distinguish wether you should redef to a base value or a function (see calling here).

metaconstants

While you won't be albe to do fancy metaconstant map mocking via =contains=>, you can mimic basic midje metaconstant functionality:

(ns controller-test
  (:require [midje.sweet :refer :all]))

(fact "should return dataset if it exists"
  (controller/fetch-dataset-row ..sl.. "foo" "bar") => :uhu
  (provided
    (serving-layer/fetch-one ..sl.. "foo" "bar") => :uhu))

becomes

(ns controller-test
  (:require [clojure.test :refer :all]
            [mockfn.macros :refer [providing]]))

(deftest dataset-fetching
  (testing "should return dataset if it exists"
    (providing [(serving-layer/fetch-one '..sl.. "foo" "bar") :uhu]
      (is (= :uhu
             (controller/fetch-dataset-row '..sl.. "foo" "bar"))))))

throw mocking

=throws=> can be mimicked in mockfn with calling

So before it was

...
(fact "if a datomic connection fails, the lock will be released"
  (#'controllers.lock/lock-dbs! locks-atom
                                (:config base-system)
                                (:discovery base-system)
                                (:prometheus base-system)
                                (:zookeeper base-system))
  => irrelevant
  (provided
    (api/connect (:url wololo-db)) =throws=> (Exception.)
    (api/connect (:url double-entry-db)) => ..double-entry-conn..)

and now it is

(ns ..
  (:require [mockfn.macros :refer [calling providing]]))

(deftest foo
  (testing "if a datomic connection fails, the lock will be released"
    (providing [(api/connect (:url wololo-db)) (calling (fn [& args] (throw (Exception.))))
                (api/connect (:url double-entry-db)) '..double-entry-conn..]
      (controllers.lock/lock-dbs! locks-atom config discovery prometheus zookeeper))))

throws checking

(fact "the validator throws a helpful exception if the input dataset does not conform"
  (controller/validate-archived-dataset! (dissoc metapod-archived-dataset :path))
  => (throws Exception))

becomes

(ns ...
  (:require [clojure.test :refer :all]))

(deftest foo
  (testing "the validator throws a helpful exception if the input dataset does not conform"
    (is (thrown?
         ExceptionInfo
         (controller/validate-archived-dataset! (dissoc metapod-archived-dataset :path))))))

Migrating integration tests

(ns integration.a-flow
  (:require [midje.sweet :refer :all]
            [selvage.midje.flow :refer [*world* defflow]]
            [matcher-combinators.midje :refer [match]]
            [matcher-combinators.matchers :as m]))
(flow "process transaction"
  transition-step-1
  transition-step-2
  ...
  (fact "check for datasets served returns no missing partitions"
    (:check-datasets-served *world*)
    => (match {:transaction-id tx-id
               :target-date target-date
               :all-served true
               :missing-partitions nil}))
  ...)

becomes

(ns integration.a-flow
  (:require [clojure.test :refer :all]
            ...
            [matcher-combinators.test]
            [matcher-combinators.matchers :as m]
            [selvage.test.flow :refer [*world* defflow]]))
(defflow process-transaction
  transition-step-1
  transition-step-2
  ...
  (testing "check for datasets served returns no missing partitions"
    (is (match? {:transaction-id tx-id
                 :target-date target-date
                 :all-served true
                 :missing-partitions nil}
                (:check-datasets-served *world*))))
  ...)