Transitive Foreign Dependencies

transitive_foreign.md

NOTE: This proposal is dead. A simpler solution was arrived at via the new `:bundle` target.

ClojureScript Support for Foreign Libraries via NPM

Problem Description

The current Webpack solution works reasonable well for application oriented development, but does not address the wider issue of developing ClojureScript libraries that depend on the wider JavaScript ecosystem where the artifacts do not reside in Java-centric distrubtion like Maven (i.e. NPM). Currently a ClojureScript library cannot express these kinds of foreign library dependencies at all. Thus users must constantly redefine basic dependencies from NPM such as React, boilerplate Webpack configuration, boilerplate import/export files, and thus cannot create and share reusable ClojureScript components that depend on this vast ecosystem.

ClojureScript actually has nearly all of the pieces to solve this problem today due to prior work around :npm-deps, as well as :foreign-libs changes to support community distribution efforts like CLJSJS.

Proposal

Allow users to create deps.cljs file in their artifact that expresses which Node dependencies they need. Previously :npm-deps was only for describing which NPM artifacts to install and which libraries to index ultimately for Closure compiler. We can extend this to allow :npm-deps to be combined with :foreign-libs. This can be done simply by extending the way we handle the :foreign-libs :file key. Instead of supplying a file, users can specify a special keyword - :bundle. This can communicate that this file doesn't yet exist and will be constructed by some other tool.

We will also extend :global-exports to support a more expressive syntax.

{cljs-react "react"}
{cljs-foo {* "foo"}}
{cljs-bar {[Foo Baz] "bar"}}

This extension allows us to generate the required index.js files for Webpack and provides enough flexiblity for users to reshape existing libraries into a more idiomatic shape for ClojureScript consumption.

Finally we will supply a new compiler option :bundle-fn a symbol that will take the the deep merged :file :bundle foreign lib entries. This deep merged foreign lib entry is exactly what users are doing manually by hand today.

We will supply a default :bundle-fn, cljs.webpack/bundle which will take this foreign lib entry and automatically generate the index.js files as well as the webpack.config.js file and then invoke webpack. We could supply :bundle-args to supply other useful flags (i.e. development, production).

Note this design isn't actually tied in anyway to webpack. If some other popular tool comes along, users can easily accomodate that by supplying their own :bundle-fn. At the same this proposal makes it easy to handle other kinds of assets like image and stylesheets via Webpack in a reusable way.

I know you’re looking for direct feedback on the idea itself but I am also a little confused so I hope I can ask some questions about it first.

I understand this is to solve the problem: As a developer, when I install a CLJS library that depends on a node module, I want that node module to be installed automatically and bundled for me for use in my CLJS application.

---

Under the proposed solution, as a library developer I would specify a deps.cljs like so:

{:npm-deps {:react "16.8.1" :react-dom "16.8.1"} ;; so the libs will be installed
 :foreign-libs [{:file :bundle
                 :provides ["react" "react-dom"]
                 :global-exports {react "react"
                                  react-dom "react-dom"}}]}

And use it like so in my library code:

(ns my-lib.core
   (:require [react :as r]
             [react-dom :as rdom]))

---

As an application developer, I would put my-lib in my deps.edn file and write some code that uses it:

(ns app.core
   (:require [my-lib.core :as mlc]))

And a build.edn that looks like:

{:main app.core
 :output-to "out/main.js"
 :output-dir "out"
 :infer-externs true
 :npm-deps :install-deps}

The ClojureScript compiler would then analyze this, detect that my-lib.core has a deps.cljs with the foreign-libs configuration, and then execute the default :bundle-fn cljs.webpack/bundle that would create 3 files:

1. An entry point index.js with the correct imports of libraries for my-lib.core as specified in the deps.cljs
2. A webpack.config.js with the appropriate configuration for processing the libraries specified by my-lib.core
3. Using the above two, generate a bundle that can then be included in the final ClojureScript bundle

---

Assuming I understand that correctly, then that leads me to ask:

1. I'm not sure if the :npm-deps thing is a total distraction. It seems like it would still be quite useful to install dependencies for consumers by having it in their classpath / including it in their app. You also mentioned how sometimes it might be good to specify both GCC to process the npm-libs and to bundle them 😵. Willing to ignore it to talk about the rest of this.

2. Could we remove the need for explicitly specifying foreign-libs completely by analyzing the ns requires? I think this may have been what Thomas was talking about in Slack. It sounds like there might be some complicated cases that the simple case (:require ["react" :as react]) doesn't cover, and might mean inventing new syntax to handle them. Is that why we want to be more explicit by specifying the import pattern and the other foreign-deps information in config?

3. Would there be a way for libraries to specify other bundling-related configuration options e.g. assets and additional loaders? I think that would go a long way in taking a load off of library users to have to create their own custom webpack stuff for each project and maintaining encapsulation of each library. webpack's config might not be composable in that way :(.

4. Currently it's pretty difficult to get code splitting and tree-shaking to work correctly in webpack and then have it interop with GCC. I trust that Thomas went to a lot of effort trying to get it to work before he went to the current solution in shadow-cljs where the processing and wrapping in IIFEs is done in-house. It sounds like it would be a good idea to explore this again with the latest webpack versions, but if it's not possible, are there other alternatives that would enable better tree shaking foreign code (at least at the module level)?

I hope these questions are in line with what's being discussed and I'm not totally off track and I'm not speaking out of turn by asking them. Honestly, just writing out the above (before my questions) helped clarify a lot so I hope I captured it accurately. Let me know if I can correct or update anything.

Thanks!

1. Nothing is changing about :npm-deps.
2. We're not going to change the ns form. However we need the ability to reshape JS libs to be more idiomatic namespaces. JS patterns around default, modules as functions etc. are not idiomatic. Note that reshaping is optional. There are cases where you need to both bundle a build and get libraries via require via npm. Being explicit is just simpler from a compiler effort perspective and it's really not much of a burden for the primary beneficiaries of the proposal - downstream users.
3. This is trivial to add to the current proposal simply by adding some custom keys to the foreign entry that :bundle-fn will understand. You could imagine :webpack-assets and :webpack-plugins.
4. Tree-shaking is still an unsolved problem in JavaScript, there's really nothing to talk about here and shadow doesn't offer any improvement to the problem as far as I know, so there's no inspiration to take there. There is no need for Webpack to interact with GCC. That was true before and still true under this proposal. Far as I know Thomas did not look at the current state of Webpack so we cannot draw any conclusions from his previous efforts. Looking around it seems to work reasonably well and was changed to address previous shortcomings.

Hope this answers your questions.

I created an example repo showing a scenario that provided the most difficulty when trying to combine CLJS code-splitting with webpack. Note that this example is deliberately simple with simple npm packages that don't involve webpack-style tree shaking. If someone wants to figure out a webpack config that achieves an acceptable split this might serve as a good example starter project. Can always make it more complex later.

So has this been implemented? I'm kind of confused at what is today's best practice for packaging a ClojureScript library that depends on some NPM dependencies which are not compatible with Google Closure.

No, this problem was solved in a simpler way with the :bundle target.