Instantly share code, notes, and snippets.

@oakes /boot_deps.md
Last active Nov 21, 2018

Embed
What would you like to do?

Boot and deps.edn

At some point, we all decided that scripting builds with executable code was a bad idea, so we tossed out makefiles in favor of specifying our builds in a data file (package.json, build.xml, project.clj, etc) and having a build tool do the heavy lifting. These data files could be leveraged by IDEs and other tools. When projects inevitably need to do things their build tool couldn't anticipate, they use plugins or bash scripts in order to...script their build with executable code. Oops.

The truth is, we never stopped scripting builds, we just made it more annoying. In Clojure land, the things Leiningen can't do just end up in a plugin or an ad hoc bash script. To make matters worse, the project.clj format isn't even technically static data, because it is very willing to execute code. It is the worst of both worlds: more restrictive and idiosyncratic than normal code, yet still impossible for external tools to reliably parse. That sucks!

Perhaps in response to this, Boot came along and offered a way to just script our builds directly using Clojure. Suddenly, it was trivial to do things that were painful when our builds were hidden behind the brittle abstraction of a "data" file. But as the author of the Cursive IDE pointed out, it is even harder for tools to parse a build.boot than a project.clj. That means it will be difficult and error-prone for Cursive to provide code completion to Boot projects. It also means that cool things like Github's security alerts will likely never be possible for Boot projects. That sucks!

It's pretty clear that both declarative and imperative build systems have flaws. What if there was a way to get the benefits of both? I want to specify dependencies declaratively, and script my builds imperatively. Trying to do these wildly different things the same way will always lead to problems.

When Clojure 1.9 came out, a lot of people were confused about the new CLI tool that came with it. Although it seemed to overlap with Leiningen and Boot, it's focused on just the first "stage" of building -- getting a basic classpath set up -- not on building jar files. You specify dependencies in a deps.edn file (which, unlike project.clj, is pure data). The tool relies on a library that anyone can use to work with this new file.

Let's say you have a project with a build.boot like this:

(set-env!
  :source-paths #{"src"}
  :resource-paths #{"resources"}
  :dependencies '[[adzerk/boot-cljs "2.1.4" :scope "test"]
                  [adzerk/boot-reload "0.5.2" :scope "test"]
                  [org.clojure/clojure "1.9.0"]
                  [ring "1.6.3"]
                  [org.clojure/clojurescript "1.9.946" :scope "test"]
                  [reagent "0.8.0-alpha2" :scope "test"]])

(require
  '[adzerk.boot-cljs :refer [cljs]]
  '[adzerk.boot-reload :refer [reload]]
  '[my-full-stack-app.core :refer [-main]])

(deftask run []
  (comp
    (with-pass-thru _
      (-main))
    (watch)
    (reload :asset-path "public")
    (cljs :optimizations :none)
    (target)))

(deftask build []
  (comp
    (cljs :optimizations :advanced)
    (aot) (pom) (uber) (jar) (target)))

We can move the paths and project deps to deps.edn like this:

{:paths ["src" "resources"]
 :deps {org.clojure/clojure {:mvn/version "1.9.0"}
        ring {:mvn/version "1.6.3"}
        org.clojure/clojurescript {:mvn/version "1.9.946"
                                   :scope "test"}
        reagent {:mvn/version "0.8.0-alpha2"
                 :scope "test"}}

Now the question is, how do we get Boot to read it? We don't want to duplicate our dependencies. One possibility is to use a task called boot-tools-deps. With Boot, however, it's fairly easy to roll your own solution without the help of any external libraries. Here's an example:

(defn read-deps-edn [aliases-to-include]
  (let [{:keys [paths deps aliases]} (-> "deps.edn" slurp clojure.edn/read-string)
        deps (->> (select-keys aliases aliases-to-include)
                  vals
                  (mapcat :extra-deps)
                  (into deps)
                  (reduce
                    (fn [deps [artifact info]]
                      (if-let [version (:mvn/version info)]
                        (conj deps
                          (transduce cat conj [artifact version]
                            (select-keys info [:scope :exclusions])))
                        deps))
                    []))]
    {:dependencies deps
     :source-paths (set paths)
     :resource-paths (set paths)}))

(let [{:keys [source-paths resource-paths dependencies]} (read-deps-edn [])]
  (set-env!
    :source-paths source-paths
    :resource-paths resource-paths
    :dependencies (into '[[adzerk/boot-cljs "2.1.4" :scope "test"]
                          [adzerk/boot-reload "0.5.2" :scope "test"]]
                        dependencies))

(require
  '[adzerk.boot-cljs :refer [cljs]]
  '[adzerk.boot-reload :refer [reload]]
  '[my-full-stack-app.core :refer [-main]])

(deftask run []
  (comp
    (with-pass-thru _
      (-main))
    (watch)
    (reload :asset-path "public")
    (cljs :optimizations :none)
    (target)))

(deftask build []
  (comp
    (cljs :optimizations :advanced)
    (aot) (pom) (uber) (jar) (target)))

To be honest, the benefits of this setup are mostly speculative. There aren't any tools making use of deps.edn yet. For now, the main benefit is that I can now use the new clj tool to fire up a quick REPL in my projects. Beyond that, I guess the benefits remain to be seen. That didn't stop me from converting all of my projects anyway...I had a boring Christmas. You can see some examples here:

@puredanger

This comment has been minimized.

puredanger commented Dec 26, 2017

"I want to specify dependencies declaratively, and script my builds imperatively." +100

Note that in your deps.edn files, the :scope attribute has no meaning. Scopes are a very clumsy tool for choosing the portions of a dependency graph to use when building classpaths in Maven (particularly they use the same words for both setting a "scope" at definition time which implies something about which transitive dependencies are included at classpath creation time based on your stated "context"). In deps.edn, we just state dependencies. If you want to modify your classpath for different classpath contexts, use aliases.

So rather than having :scope "test", create a :test alias to add deps. https://github.com/oakes/Nightcode/blob/master/deps.edn rewritten:

{:paths ["src/clj" "src/cljs" "resources"]
 :deps {org.clojure/clojure {:mvn/version "1.9.0"}
        leiningen {:mvn/version "2.8.1" :exclusions [leiningen.search]}
        ring {:mvn/version "1.6.1"}
        hawk {:mvn/version "0.2.11"}
        play-cljs/lein-template {:mvn/version "0.11.2.1"}
        eval-soup {:mvn/version "1.2.3" :exclusions [org.clojure/core.async]}
        org.eclipse.jgit/org.eclipse.jgit {:mvn/version "4.6.0.201612231935-r"}}

 :aliases {
   :test {
     :extra-deps
     {org.clojure/clojurescript {:mvn/version "1.9.946"}
      paren-soup {:mvn/version "2.9.3"}
      mistakes-were-made {:mvn/version "1.7.4"}
      javax.xml.bind/jaxb-api {:mvn/version "2.3.0"} ; necessary for Java 9 compatibility
      cljsjs/codemirror {:mvn/version "5.24.0-1"}}
 }
}

Then your default classpath is just your runtime dependencies and you can use -R:test when you want the extra test deps added to your classpath (I confess to not knowing exactly how to do that through the Boot plugin).

@oakes

This comment has been minimized.

Owner

oakes commented Dec 26, 2017

@puredanger Awesome info Alex, thank you! This was definitely the bit that I didn't understand. It looks like I can pass the alias with (deps :aliases [:test]). However, if I use this approach, I believe the build task will still include those :extra-deps in the final uberjar, because it is running both the ClojureScript compiler (which will need those deps) as well as building the uberjar. Not sure how to deal with that.

@martinklepsch

This comment has been minimized.

martinklepsch commented Dec 28, 2017

@oakes That uberjar aspect is interesting. The uber task currently uses information directly from the Boot environment but I think it should be possible to read the dependencies from the pom.xml as well. With that kind of addition you could use the pom task with a custom set of dependencies and they would be used to build the uberjar.

(deftask build []
  (comp (deps) (aot) (pom :dependencies {,,,}) (uber :exclude jar-exclusions) (jar) (sift) (target)))
@oakes

This comment has been minimized.

Owner

oakes commented Dec 29, 2017

@martinklepsch Is there anything special I need to do in the uber task to get it to read from the pom.xml? I tried setting the dependencies in the pom task but it doesn't appear to affect what is put in the jar.

edit: This issue seems to be related boot-clj/boot#509

@theronic

This comment has been minimized.

theronic commented Jan 30, 2018

@oakes have you seen boot-tools-deps? It now supports Git checkouts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment