cljs_modules.clj

{:optimizations :advanced
 :output-dir "./target/client/production/"
 :cache-analysis true
 :output-modules {
  {:id :common
   :out "./resources/assets/js/common.js"  
   :entries '#{com.foo.common}}
  {:id :landing
   :out "./resources/assets/js/landing.js" 
   :entries '#{com.foo.landing}
   :deps #{:common}
  {:id :editor
   :out "./resources/assets/js/editor.js"
   :entries '#{com.foo.editor}
   :deps #{:common}}}}}

Google Closure Modules allow breaking up advanced builds into multiple files so your entire production build doesn't need to be loaded all at once.

:output-modules replaces :output-to. :output-modules is a map from output module to a set of module definitions. output modules definitions should define :id a keyword, :out a file to output on disk, :entries all namespaces to be placed into this module, :deps module dependencies.

One tricky edge case is foreign JS libraries. For example the com.foo.editor namespace might rely on CodeMirror. When http://dev.clojure.org/jira/browse/CLJS-965 is resolved we could easily compute and emit a editor.deps.js file.

Could the split points be defined in code? Similar to that used with webpack? https://github.com/petehunt/webpack-howto#9-async-loading
This approach allows code to surround the async loading of the module like showing/hiding a loader in the application.

@karlmikko not going to support that

How would preambles be handled? Only prepend them to the common.js module?

@mdhaney that sounds reasonable to me

So conceptually, it seems to me like you will always need some core/common build with cljs, closure, preambles, and foreign libs. You may optionally have additional modules defined that include specific namespaces. Any other namespaces you don't specify are dumped into the common build.

Even better would be for the compiler to move certain common namespaces, foreign libs, etc. into one of the optional modules if it determines that module is the only user. Otherwise, the common module could easily end up containing 80% of the code (just from 3rd party libs alone) which would negate some of the benefits.

If it's easier, I think it would still be beneficial to implement the naive approach (just dumping stuff in common) first, and then work on the optimization in a future release.

What you have there pretty much matches (conceptually) what I'm doing with shadow-build here:

https://github.com/robert-stuttaford/stuttaford.me/blob/master/project.clj#L41-L56

• :core-libs lists all the namespaces to explicitly include, which will bring in all dependent namespaces. Assumption baked in that these all end up in core.js.
• :modules lists individual .js files each with their entry points. Assumption baked into Shadow/Cljs/GClosure somewhere that only stuff not already in core.js goes into individual module .js files.

I'm using shadow primarily for its module support. I wouldn't have to if its supported by Cljs directly and we could just pass options from lein-cljsbuild or boot-cljs. Highly interested in seeing this feature realised :-)

My feedback based on experience with shadow-build:

• How do you define module dependencies? eg. module-c depends on module-b depends on module-a, module-d depends on module-a (but not the rest). That should be computable somehow but I had problems with that. Especially if that graph had "duplicate" files. (eg. module-c depends on something, module-d depends on something). Somehow something needs to be moved to module-a. Instead of making module-d depend on module-c.
• I don't think "matchers" are a good idea. The dependency graph will usually address this by itself. (eg. cljs.core.async will bring the impl namespaces) but if I don't use cljs.core.async in the "common" module it should not be in there just because I said '#{cljs.core}. While :main namespaces might not be the best of names they capture the semantics very well (Closure calls them Entry Points). The resulting dependency graph is "optimal" (ie. stuff that doesn't need to be in "common" isn't).
• All of this should work with :none. I think it is completely unacceptable that dev-builds require different HTML than production builds but that might be my heavily biased opinion. (related http://dev.clojure.org/jira/browse/CLJS-851)

I do a few other things in shadow-build to make life easier (eg. just name one output-dir and all modules go into that dir, instead of passing a full path for each module) but that again is my opinion and not necessarily the "best".

@robert-stuttaford: Open a shadow-build issue if it is missing something boot or cljsbuild have. Been working on it the last few days. ;)

@theller

• module dependencies should be automatically computed
• while most applications will want "common" to be computed, this won't always be the case. "matchers" provides an acceptable amount of control in these cases without imposing full enumeration. A good example of why this is useful is wanting to partition the ClojureScript standard library for JavaScript consumers like Mori does. Perhaps "matchers" should be explicit - cljs.core.*?
• Making this work with :none is not under discussion

@mdhaney the goal here is to just get it right, optimizations and all the first time around. In the cases where common is automatically computed common will be the intersection of shared dependencies across all modules and modules will always include exactly the dependencies they need. "matchers" are provided specifically because we don't want to make the system any smarter than it needs to be. For example out of module A, B, C only A and B need core.match and core.async.

{:id "production"
 :source-paths ["src/cljs"]
 :compiler
 {:optimizations :advanced
  :pretty-print false
  :output-modules {
     "../../assets/js/shared.js"  :common
     "../../assets/js/comm.js"    '#{cljs.core.async.* cljs.core.match.*}
     "../../assets/js/moduleA.js" '#{com.foo.moduleA.*}
     "../../assets/js/moduleB.js" '#{com.foo.moduleB.*}
     "../../assets/js/moduleC.js" '#{com.foo.moduleC.*}}}}

Module dependencies will be inferred automatically.

I went down the path you are on right now, in Theory it is fine. It just didn't work out so well in practice.

• Computing dependencies works until you run into the edge case I described
• "Matchers" (I used regexp) didn't offer much value over explicit namespaces. Say module-b uses cljs.core.async, why do I need to add a "matcher" for that if I can derive this information (dependency graph) by looking at the :main (entry point) of module-b. In practice "matchers" just were way too explicit.

I don't think mori is a good example use-case for modules since it is far too simple.
Not sure if this is what you had in mind, but when I run it through shadow-build I end up with:

  12K mori.chain.js
 189K mori.js
 1.3K mori.mutable.js
 7.9K mori.reducers.js
 6.2K mori.zip.js

With gzip

 2.3K mori.chain.js.gz
  41K mori.js.gz
 644B mori.mutable.js.gz
 1.7K mori.reducers.js.gz
 1.6K mori.zip.js.gz

https://github.com/thheller/mori/tree/shadow-build
run lein run -m build/release

Given these tiny sizes for "addon" modules it is probably more overhead to have an extra request fetching these than just serving them at all times. But I'm not familar with mori, maybe I'm missing something.

https://github.com/thheller/mori/blob/shadow-build/dev/build.clj#L22-L26
Note that I only entered application specific information (ie. mori.*), implementation details (cljs.core) ended up in the correct places.

https://github.com/thheller/mori/blob/shadow-build/release/manifest.json
The manifest contains an overview which sources ended up in which module.

I'm not going to say that my solution is perfect or better in any way, just trying to make it clear which problems I faced so you might skip over them. You probably have a different view on things, which is good. Definitely going to watch what you come up with.

@theller the Mori metrics feedback is great thanks! My intent was to remove everything from Mori except for the collection functions. But I suppose since all modules are compiled together this won't prevent the shared module from being quite large. But this isn't a problem with "matchers" per se it's a problem with having a monolithic core. Still I see your point. We'll drop matchers for now.

UPDATE: It appears Google Closure supports code motion between modules under advanced optimizations. Still, I think dropping "matchers" is fine.

RE: computing module dependencies I suppose if Google & you have gone down this path you're probably aware of the pitfalls.

Updating the proposal.

Quick question on the current proposal - are entry points purely for the purpose of computing dependencies (ie entry points for the dependency graph, not for execution)?

Besides that, it looks good to me.