Skip to content

Instantly share code, notes, and snippets.

@mattdesl
Last active September 13, 2017 16:03
Show Gist options
  • Star 13 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save mattdesl/aaf759da84cc44c22305 to your computer and use it in GitHub Desktop.
Save mattdesl/aaf759da84cc44c22305 to your computer and use it in GitHub Desktop.

agnostic modules

Most of the modules I write are "agnostic" in that they should work in Node, browserify, webpack, Rollup, jspm... hell, even Unreal.js. It's just ES5, CommonJS and a few quirks like process.nextTick or require('path') (which browserify and webpack will shim).

Other modules are a bit trickier, and need to include a static asset like HTML, GLSL or another file.

In Node you might see this:

var fs = require('fs')
var path = require('path')
var html = fs.readFileSync(path.join(__dirname, 'index.html'), 'utf8')

If I specify brfs in my package.json config, the above code will work fine when the end-user installs and uses my module with browserify (even if it's deep in their dependency tree). The asset gets statically inlined during the bundle step.

This is great since the module has a single entry point for both node and the browser, and I can test/develop with node, nodemon and tape.This approach can even handle large base64/binary files as a Buffer; while it may not be great for a web application, it can be great for quick prototyping in the browser.

However, this approach is incompatible with webpack and other bundlers. Examples of problematic modules which depend on a browserify transform:

solutions

transform-loader

Webpack has a thing called transform-loader. This is more of a hack, and it forces configuration on the end-user of your module. It's not automatically resolved, see #378 – whereas browserify transforms are resolved per module. This is especially annoying with dependencies-of-dependencies.

Further, it assumes that your end-user is either using browserify or webpack. Other bundlers like jspm and Rollup will not work.

Update Jan 2016

ify-loader can now be used in Webpack configs to resolve browserify transforms automatically across your node_modules, as browserify intends it to be. This fixes the automatic transform resolution, but does not solve the end-user configuration problem.

bundling

You can run your module through webpack or browserify, but this will lead to duplication in the end-user's final bundle, since some dependencies may be getting bundled twice and won't be de-duplicated by browserify/webpack.

It also makes your published code pretty ugly to look at, and you'll want to add source maps.

Update Dec 2015

With webpack, you can use external to avoid bundling a module and preserve de-duplication. See the discussion thread below.

babel plugin

I think it would be great if somebody developed a Babel plugin/transform that supports browserify transform streams listed in package.json "browserify" field. This would allow you to transpile the source rather than bundling it, fixing the dedupe issue.

I'm not sure how complex this is, and whether it carries other problems.

Update Jan 2016

babel-plugin-webpack-loaders can now be used which may help build "agnostic modules" by transpiling webpack loaders on pre-publish (like CSS or HTML inlining). However, it is a bit klunky since Babel is not designed to run asynchronous source transforms.

simple transpiler

Another solution is if somebody build a source transpiler that doesn't try to bundle code (like browserify/webpack) but instead just tries to support browserify transforms.

This way you can publish an npm dist that looks exactly like the source entry point, but with static files inlined so that it works in Node, browserify, webpack, etc.

Update Jan 2016

With transpilify (work in progress) you can transpile a source file with simple browserify transforms, but without adding the overhead of the bundler itself.


Sadly all of these solutions require more configuration and setup in one way or another, adding some overhead for module authors. :)

This is closely related with some of the things webpack tries to solve, like handling image and SVG assets. I think it would be novel if there was a standardized solution that supported Node and all modern bundlers... But I think a more realistic goal (for now) is just to get ASCII assets (HTML/GLSL/text/etc) working.

@mjackson
Copy link

it forces configuration on the end-user of your module

That's not true. For example, if I write my code in ES2015 and use Babel in my build, I don't assume that someone who is consuming my module needs to know about it. It's all part of my build step. How is brfs different? Isn't that just another step in my build?

@mjackson
Copy link

Also, I published something about using pixi.js with webpack https://gist.github.com/mjackson/ecd3914ebee934f4daf4

@KyleAMathews
Copy link

Yeah, honestly I blame Browserify for being too magical and then module writers unwittingly relying on the magic. The browser is a very different world than Node and I'd prefer module writers (who know their code better than anyone) to build to a universal ES5/ES3 base.

@mattdesl
Copy link
Author

It's all part of my build step. How is brfs different? Isn't that just another step in my build?

Try using any module with a transform step (like soundcloud-badge), and you will get an error like this:

Module not found: Error: Cannot resolve module 'fs' in /webpack-example/node_modules/soundcloud-badge

This isn't just limited to browserify transforms. Say your dependency foo suddenly chooses to use Webpack's raw-loader. If you haven't set that up in your webpack.config.js, your app will not work.

On the other hand, browserify users don't need to worry about this problem. During bundling, the transforms are resolved per-module, so even if soundcloud-badge is deep in the dependency tree, the static asset will be inlined without any configuration for the end-user.

I write my code in ES2015 and use Babel in my build, I don't assume that someone who is consuming my module needs to know about it.

Off-topic, but this has bit me in the ass before. You start depending on ES2015 modules, and eventually authors will commit some code that relies on Array.prototype.find or another feature that Babel doesn't transpile. Suddenly, the root app is breaking on legacy browsers because a module deep in the dependency tree was authored in ES2015 and has a peer dependency on babel-polyfill. This is especially frustrating when the app you are writing isn't even being authored in ES2015.

@mattdesl
Copy link
Author

@KyleAMathews I agree browserify is easy to blame here, especially since the package.json "browserify" config basically guarantees vendor-lock in by its name alone.

Inlining the files during pre-publish is the right way to go, but it's another thing module authors need to deal with, and can complicate tooling (e.g. developing multiple modules at once with npm link). I haven't found a good solution yet, so my modules that need GLSL etc. still tend to be browserify-focused. 😞

@AriaMinaei
Copy link

You can run your module through webpack or browserify, but this will lead to duplication in the end-user's final bundle, since some dependencies may be getting bundled twice and won't be de-duplicated by browserify/webpack.

I use webpack's externals option to bundle my npm package without its external dependencies. That way, I'm free to use any loader like transform?brfs inside the package, yet allow the consumer of the package to use it with any other bundler or none at all. And since external dependencies are left unbundled, the config wouldn't mess up de-duplication either. Example here.

@mattdesl
Copy link
Author

That's a pretty nice way to avoid the duplication. 😁 You still end up with some pretty ugly distribution code, and source maps on shapely do not seem to work:

screen shot 2015-12-29 at 9 40 37 am

@AriaMinaei
Copy link

You still end up with some pretty ugly distribution code

It is ugly indeed :D But would that matter much if the source map actually worked? I mean, babel's output, even unbundled, doesn't always look nice. And the more abstraction we add to the toolchain, the uglier the output code will get. Personally, I'd rather look at the original code via sourcemaps than look at the output code.

and source maps on shapely do not seem to work

Hey thanks for the heads up :) It used to work, so maybe I've broken it or something :D I'll look into it.

@mattdesl
Copy link
Author

Yeah, in theory source maps would solve everything. In practice, I find myself often needing to look at the original source for one reason or another. e.g. Debugging in legacy browsers, reading and tweaking a dependency in my node_modules/ to test a bug fix, looking through Promise stack traces which have destroyed source maps, or determining why Babel has introduced performance regressions in a hot code path.

To add to this, dependencies using a towering stack of language features tend to scare me off contributing. i.e. I probably won't submit PRs for a CoffeeScript-authored module since the source is mostly foreign to me. The same is becoming true with some of the experimental Babel features and plugins.

This is why I'd prefer to find a solution that doesn't involve "ugly code" for the end-user. 😄

@AriaMinaei
Copy link

Yeah, in theory source maps would solve everything. In practice, I find myself often needing to look at the original source for one reason or another. e.g. Debugging in legacy browsers, reading and tweaking a dependency in my node_modules/ to test a bug fix, looking through Promise stack traces which have destroyed source maps, or determining why Babel has introduced performance regressions in a hot code path.

Agreed. Been there in all cases. Output code must get better. But I think the long term solution is to just make source maps work better in practice. (Btw, promise stack traces can be patched to use sourcemaps).

To add to this, dependencies using a towering stack of language features tend to scare me off contributing. i.e. I probably won't submit PRs for a CoffeeScript-authored module since the source is mostly foreign to me. The same is becoming true with some of the experimental Babel features and plugins.

I see. I felt that way when trying to read react's source code. The bundled file was just too big to read, and the original source code seemed to have too much magic in it. Files require()ed packages that didn't actually exist (they were aliases for internal modules apparently). To me, that was just confusing.

Though generally I'm okay with different build systems and languages (they're all usually either JavaScript-ish or CoffeeScript-ish), as long as the build config is easy to understand, easy to set up, and there is no 'magic' in it.

All and all, I agree with you. Making bundled code look nicer will make everything easier.

@ahdinosaur
Copy link

@mattdesl: i reckon @mmckegg and i are keen for your simple transpiler idea. we were also thinking about a bundler for node using browserify, so require wouldn't have to synchronously look up the filesystem (sync require is slow!), but you're right that it's probably best to split those into separate modules.

@mattdesl
Copy link
Author

mattdesl commented Jan 3, 2016

Two new things:

@mattdesl
Copy link
Author

mattdesl commented Jan 4, 2016

@ahdinosaur
Copy link

@mattdesl 😻

@tbroyer
Copy link

tbroyer commented Feb 10, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment