(Rough Notes) Importing a javascript module into ClojureScript

gistfile1.md

Importing a JavaScript Module into ClojureScript

Clojurescript.org Documentation

Documentation for interop is spread all over the place. Under the "Reference" section:

1. "Dependencies": mostly about how to consume javascript libraries
2. clojurescript.org: "Packaging Foreign Dependenciees": mostly about how to provide javascript libraries for others to consume
3. "Advanced Compilation": mostly duplicative, but also describes how to access cljs vars from javascript
4. "JS Module Support": describes an alpha-quality feature from GCC to allow optimized imports of GCC compatible libraries without using goog.provide

In addition, under "Guides", there is:

1. "Externs (Alpha)": describes an alpha-quality feature that apparently infers "externs", or, in other words, infers which symbols GCC should not rename. This document is super confusing. I can't figure out what it actually does or how I'm supposed to use it. There's a variable called *warn-on-infer* that really means "warn on failure to infer" I think.
2. "T. Heller's Improved Externs Inference": this is a bit easier to understand and points out that specific "typing" isn't necessary. Not that I understand what "typing" even means in this context.

1. clojurescript.org: "Dependencies"

The primary example is a library that simply assigns a function to a global:

yayQuery = function() {
  var yay = {};
  yay.sayHello = function(message) {
    console.log(message);
  };
  yay.getMessage = function() {
    return 'Hello, world!';
  };
  return yay;
};

The structure of this page was not initially obvious. It is divided into "external" code (i.e. code that is loaded using a <script> tag) and "bundled" code (i.e. code that the cljs compiler will pass along to GCC to bundle into the javascript target)

"External" code

• The steps are:
  1. Inline library in a <script> tag
  2. Pass :externs to cljs compiler to tame GCC
  3. In your code, access the global variable (e.g. js/yayQuery)

"Bundled" code

1. Write GCC compliant code and import using the :libs compiler option
2. Use the :foreign-libs compiler option:
   1. Put the library in a file (or, presumably, copy a UMD build from an npm download)
   2. Pass this option to the compiler :foreign-libs [{:file "mylib.js" :provides ["arbitrary-ns"]}]
   3. Pass :externs to cljs compiler to tame GCC
   4. In your code, do a :require [arbitrary-ns], access the global variable (e.g. js/yayQuery). Obviously you should use yayQuery instead of arbitrary-ns but I just did that to show that the library dictates how you use it whereas the :foreign-libs compiler option dictates how you include it in the target bundle.
3. Use cljsjs:
   1. Either download the jar directly or reference it in your build tool so it downloads it for you (the compiler doesn't need any build options--it just need to be in the classpath presumably)
   2. Cljsjs jars will contain a deps.js file. This file doesn't appear to be documented anywhere except by example in this page, at least, not that google knows about. It appears to be a map that can contain only one key: the foreign-libs key. (Note, the documentation suggests that you provide a :file and a :file-min here. The docs here explain that :file-min is used with advanced compilations. The point here is that the cljsjs jar contains a pre-built javascript target that presumably sets some global variables, correct externs, and instructions to the compiler so it can find all this stuff.

Note that the only difference between option 2 of the "bundled" options and the "external" version is that you pass the compiler :foreign-libs, which tells it where the library is and tells it how to know what cljs code is importing it. You don't need the provides statement in the external version because you will just access a global variable at runtime.

Further note that in option 2, you never actually use the arbitrary-ns outside of the require. One wonders why this is even needed--if you provided the foreign-libs in your build system presumably you want it bundled. I assume this is to minimize what gets included in the bundle.

2. clojurescript.org: "Packaging Foreign Dependenciees"

This page is almost completely duplicative of option 3 of the above "Bundled" code options. Some odd unexplained thing here:

1. The page mentions "your JAR" out of the blue like I know what that is. Aren't we in a javascript environment? I guess cljs uses jars instead of zipfiles. If you are going to write a separate page on packaging foreign dependencies, should you tell me how to do that? Or maybe link to it? I'm sure this is obvious to cljs library providers.
2. Again, we are told we should use :file-min but not told what really happens with it.
3. Most confusingly, we are told we must provide transitive dependences in a :requires vector. It doesn't actually explain, but from the example it appears that that contents of the :requires vectors are other synthetic namespaces, just like the arbitrary-ns from the "Bundled" code example. Given that the examples so far have made themselves available with global variables, I'm not sure what difference it makes to cljs to have this dependency graph. One possibility: cljs doesn't automatically include the :files listed inteh :foreign-libs vector. Instead, perhaps cljs follows the dependency graph starting from the require statement in the core code all the way through all foreign libs. That would make sense if the goal was to eliminate unused javascript libraries.
4. If my supposition in 3 is correct, then maybe the right thing to do is to omit the :requires in the case of dependencies that are going to be provided via other mechanisms, such as how react is often supplied by reagent or rum.
5. How are you supposed to manage versions with this system? If you have several react libraries, you don't want each of them importing their own copy.

clojurescript.org: "Advanced Compilation"

This page isn't about advanced compilation so much as it is about the problems that GCC's name munging will cause with javascript interop.

1. Provide an externs file. This file only affects optimized cljs code, not javascript library code. When using an externs file, the library code will be left alone. (Of course, if you write GCC compatible javascript with a goog.provide, then everything will be munged. But in that case you won't be using an externs file.)
2. To access cljs vars from javascript, annotate with ^:export metadata.

clojurescript.org: "Javascript Module Support (Alpha)"

GCC can apparently "convert JavaScript modules into Google Closure modules," meaning you can feed certain javascript libraries and it will somehow optimize the code.

You do this by providing the :module-type key to the :foreign-libs compiler option. Its values can be :commonjs, :amd, and :es6.

The example only shows a commonjs example. To use the exports object from the library, you just :require the synthetic namespace (e.g. the arbitrary-ns from above). The properties of the exports object will be available as vars within the synthetic namespace. (E.g.: (calc/add 4 5) to access module.exports.add.)

• How do you obtain default ES6 exports? Do you just treat the namespace as a var? I assume named es6 exports work as if you did a import * as MyLib from 'mylib.js'?

• This apparently will break sometimes with node modules because GCC doesn't implement node's complicate name resolution scheme completely. To investigate what exactly will break it? Is this a practical concern? Can it be fixed?

• The page explains that the code must be GCC compliant. So really, all this does is deal with the lack of goog.provide. Otherwise it is like the :libs option of importing real GCC modules.

• To investigate This is apparently "alpha" quality, but I have no idea what happens if something breaks. Compiler error? Subtle error during optimized builds? How do I debug it?

Short Diversion: Dependency Management and Library Versioning

You must keep note of the fact that there are two kinds of dependency analyses going on in a normal ClojureScript project: one performed by the ClojureScript compiler and one performed by lein/boot.

• The ClojureScript compiler doesn't understand library versioning. The compiler doesn't read the version string in the :dependencies vector in your project.clj file. The only dependency information passed to the compiler is through (1) :libs and :foreign-libs compiler options, (2) deps.clj files inside foreign lib jars, (3) :require expressions in cljs code. None of these directives contain version information.
• Note that each of these methods tell the compiler where the relevant source code is on disk. In the case of ClojureScript, the namespace dictates its location on disk. In the case of JavaScript, the location is spelled out in the :foreign-libs directive.
• I think this means that the compiler traverses a dependency graph that begins at the entry point of the program, and uses the dependency information to figure out what to include in the final target.
• I think this means that you can only have one version of a library installed at a time. This seems supported by the fact that Leiningen provides a :excludes option that you can tack on to the :dependencies vector if your dependencies require conflicting versions of one library. You can globally exclude a library and manually include the right version in your dependencies vector. See this stackoverflow answer.
• This is different from npm's "maximally flat" version tree, which will automatically install multiple versions of the same dependency if needed.

Example: One Failure and One Success to Use react-dnd in a ClojureScript Project

Preliminaries

React-Dnd needs two node modules to work: react-dnd and react-dnd-html5-backend, but I'm only going to try to get react-dnd imported for starters. Both of them come what I believe are Webpack UMD distros:

$ head ReactDnD.js
(function webpackUniversalModuleDefinition(root, factory) {
	if(typeof exports === 'object' && typeof module === 'object')
		module.exports = factory(require("react"));
	else if(typeof define === 'function' && define.amd)
		define(["react"], factory);
	else if(typeof exports === 'object')
		exports["ReactDnD"] = factory(require("react"));
	else
		root["ReactDnD"] = factory(root["React"]);
})(this, function(__WEBPACK_EXTERNAL_MODULE_2__) {

I'm not an expert on UMD, but my understanding is that they allow you to stick them in a <script> tag and they will load themselves into a global (basically the last else clause above). More specifically, here, loading the script in a browser environment will set the ReactDnD global, just like the yayQuery example in the ClojureScript documentation above. That seems promising.

I note that the UMD module requires "React". In the global version, it will simply read the "React" global variable and use that. In the AMD and CommonJS variants, it will use the return value of require("react").

Try #1 Let's Use :npm-deps

I add the following compiler options in my project.clj:

:npm-deps {:react-dnd "2.4.0"}
:install-deps true

And I added (require [react-dnd]) to my core.cljs.

Doesn't work:

Compiling "target/cljsbuild/public/js/app.js" from ["src/cljs" "src/cljc" "env/dev/cljs"]...
events.js:182
      throw er; // Unhandled 'error' event
      ^

Error: Can't resolve 'react' in '/Users/me/testproject/client-cljs/node_modules/react-dnd/lib'
    at onError (/Users/me/testproject/client-cljs/node_modules/enhanced-resolve/lib/Resolver.js:61:15)
    at loggingCallbackWrapper (/Users/me/testproject/client-cljs/node_modules/enhanced-resolve/lib/createInnerCallback.js:31:19)
    at runAfter (/Users/me/testproject/client-cljs/node_modules/enhanced-resolve/lib/Resolver.js:158:4)
    at innerCallback (/Users/me/testproject/client-cljs/node_modules/enhanced-resolve/lib/Resolver.js:146:3)
    at loggingCallbackWrapper (/Users/me/testproject/client-cljs/node_modules/enhanced-resolve/lib/createInnerCallback.js:31:19)
    at next (/Users/me/testproject/client-cljs/node_modules/tapable/lib/Tapable.js:252:11)
    at innerCallback (/Users/me/testproject/client-cljs/node_modules/enhanced-resolve/lib/Resolver.js:144:11)
    at loggingCallbackWrapper (/Users/me/testproject/client-cljs/node_modules/enhanced-resolve/lib/createInnerCallback.js:31:19)
    at next (/Users/me/testproject/client-cljs/node_modules/tapable/lib/Tapable.js:249:35)
    at resolver.doResolve.createInnerCallback (/Users/me/testproject/client-cljs/node_modules/enhanced-resolve/lib/DescriptionFilePlugin.js:44:6)

WARNING: uri? already refers to: cljs.core/uri? being replaced by: cognitect.transit/uri? at line 332 target/cljsbuild/public/js/cognitect/transit.cljs
Jan 16, 2018 3:48:56 PM com.google.javascript.jscomp.LoggerErrorManager println
SEVERE: /Users/justinlee/seekeasy/client-cljs/target/cljsbuild/public/js/seekeasy/core.js:9: ERROR - required "react_dnd" namespace never provided
goog.require('react_dnd');
^^^^^^^^^^^^^^^^^^^^^^^^^

Jan 16, 2018 3:48:56 PM com.google.javascript.jscomp.LoggerErrorManager printSummary
WARNING: 1 error(s), 0 warning(s)
ERROR: JSC_MISSING_PROVIDE_ERROR. required "react_dnd" namespace never provided at /Users/justinlee/seekeasy/client-cljs/target/cljsbuild/public/js/seekeasy/core.js line 9 : 0
Successfully compiled ["target/cljsbuild/public/js/app.js"] in 31.225 seconds.

Remember how the documentation said node modules won't always work?

The Node.js module specification varies slightly from the CommonJS specification in that the module identifier that is passed to require() doesn’t always need to be an absolute or relative path. This makes it difficult for the Google Closure compiler to resolve the dependencies of a node module since th compiler was implemented following the standard CommonJS specification. Therefore, it might not be possible for a node module to be converted to a Google Closure module.

I think this is what it was talking about. That require("react") in the UMD header is possibly what's causing this problem.

Some question and points about the above:

1. Everything from "SEVERE" on down gets eaten by figwheel, so you need to run lein cljsbuild once to see it.
2. If you run lein cljsbuild once again, it doesn't do anything!
3. Why in the hell did compilation succeed?
4. Why is the compiler trying to load the UMD dependencies during compilation? If it is going to do this, shouldn't it be reading the package.json? This must be the difference between CommmonJS and node modules (?). Confusing since this feature is actually called npm-deps.
5. If I'm right, it would be handy to have an option to ignore it since I'm going to provide the react library myself anyway.

Okay I'm out of ideas. On to the next thing.

Try #2: Less Fancy: Let's Use :foreign-libs

I've copied the UMD builds into a lib directory and added this:

:foreign-libs [{:file "resources/lib/ReactDnD.js"
                :file-min "lib/ReactDnD.min.js"
                :provides ["react-dnd"]
                :requries ["react"]}]

Same problem. That's weird. I changed :file it to point to a nonexistent file. Same problem. Okay the node_modules are left over. I delete them and the package.json and try again.

Much better! Good news: I get a sane error message:

Caused by: clojure.lang.ExceptionInfo: No such namespace: react-dnd, could not locate react_dnd.cljs, react_dnd.cljc, or JavaScript source providing "react-dnd" in file src/cljs/seekeasy/core.cljs {:tag :cljs/analysis-error}

Bad news: nothing I do to the :foreign-libs directive actually does anything. I feel like I have two potential problems:

1. I'm wondering if I've got a pathing problem here. Since I get the same error even if I point :foreign-libs to a non-existent file, it's hard to say.
2. I'm wondering if I've specified the :foreign-libs correctly in project.clj. Hard to say!

Turns out to be neither. The template I'm using has two builds configured and lein cljsbuild once either builds both or only builds the :min build. I had only added the :foreign-libs diretive to one of the two. For the record, you get a good error message if you specify a nonexistent file in :foreign-libs. Whew.

I still get a runtime console error:

base.js:1357 Uncaught Error: Undefined nameToPath for react

Per this stackoverflow post, I change it to cljsjs/react.

That's it! I can access js/ReactDnD now.

One other point: completely eliminating the :requires key also works. This confirms my understanding that the compiler require-tree is only needed to ensure everything ends up in the final bundle. Since I'm including rum and rum includes react, its presense in the :foreign-libs entry for react-dnd was technically redundant (though obviously I'll keep it in).