Skip to content

Instantly share code, notes, and snippets.

@mattdesl
Last active September 13, 2017 16:03
Show Gist options
  • Star 13 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save mattdesl/aaf759da84cc44c22305 to your computer and use it in GitHub Desktop.
Save mattdesl/aaf759da84cc44c22305 to your computer and use it in GitHub Desktop.

agnostic modules

Most of the modules I write are "agnostic" in that they should work in Node, browserify, webpack, Rollup, jspm... hell, even Unreal.js. It's just ES5, CommonJS and a few quirks like process.nextTick or require('path') (which browserify and webpack will shim).

Other modules are a bit trickier, and need to include a static asset like HTML, GLSL or another file.

In Node you might see this:

var fs = require('fs')
var path = require('path')
var html = fs.readFileSync(path.join(__dirname, 'index.html'), 'utf8')

If I specify brfs in my package.json config, the above code will work fine when the end-user installs and uses my module with browserify (even if it's deep in their dependency tree). The asset gets statically inlined during the bundle step.

This is great since the module has a single entry point for both node and the browser, and I can test/develop with node, nodemon and tape.This approach can even handle large base64/binary files as a Buffer; while it may not be great for a web application, it can be great for quick prototyping in the browser.

However, this approach is incompatible with webpack and other bundlers. Examples of problematic modules which depend on a browserify transform:

solutions

transform-loader

Webpack has a thing called transform-loader. This is more of a hack, and it forces configuration on the end-user of your module. It's not automatically resolved, see #378 – whereas browserify transforms are resolved per module. This is especially annoying with dependencies-of-dependencies.

Further, it assumes that your end-user is either using browserify or webpack. Other bundlers like jspm and Rollup will not work.

Update Jan 2016

ify-loader can now be used in Webpack configs to resolve browserify transforms automatically across your node_modules, as browserify intends it to be. This fixes the automatic transform resolution, but does not solve the end-user configuration problem.

bundling

You can run your module through webpack or browserify, but this will lead to duplication in the end-user's final bundle, since some dependencies may be getting bundled twice and won't be de-duplicated by browserify/webpack.

It also makes your published code pretty ugly to look at, and you'll want to add source maps.

Update Dec 2015

With webpack, you can use external to avoid bundling a module and preserve de-duplication. See the discussion thread below.

babel plugin

I think it would be great if somebody developed a Babel plugin/transform that supports browserify transform streams listed in package.json "browserify" field. This would allow you to transpile the source rather than bundling it, fixing the dedupe issue.

I'm not sure how complex this is, and whether it carries other problems.

Update Jan 2016

babel-plugin-webpack-loaders can now be used which may help build "agnostic modules" by transpiling webpack loaders on pre-publish (like CSS or HTML inlining). However, it is a bit klunky since Babel is not designed to run asynchronous source transforms.

simple transpiler

Another solution is if somebody build a source transpiler that doesn't try to bundle code (like browserify/webpack) but instead just tries to support browserify transforms.

This way you can publish an npm dist that looks exactly like the source entry point, but with static files inlined so that it works in Node, browserify, webpack, etc.

Update Jan 2016

With transpilify (work in progress) you can transpile a source file with simple browserify transforms, but without adding the overhead of the bundler itself.


Sadly all of these solutions require more configuration and setup in one way or another, adding some overhead for module authors. :)

This is closely related with some of the things webpack tries to solve, like handling image and SVG assets. I think it would be novel if there was a standardized solution that supported Node and all modern bundlers... But I think a more realistic goal (for now) is just to get ASCII assets (HTML/GLSL/text/etc) working.

@AriaMinaei
Copy link

You can run your module through webpack or browserify, but this will lead to duplication in the end-user's final bundle, since some dependencies may be getting bundled twice and won't be de-duplicated by browserify/webpack.

I use webpack's externals option to bundle my npm package without its external dependencies. That way, I'm free to use any loader like transform?brfs inside the package, yet allow the consumer of the package to use it with any other bundler or none at all. And since external dependencies are left unbundled, the config wouldn't mess up de-duplication either. Example here.

@mattdesl
Copy link
Author

That's a pretty nice way to avoid the duplication. 😁 You still end up with some pretty ugly distribution code, and source maps on shapely do not seem to work:

screen shot 2015-12-29 at 9 40 37 am

@AriaMinaei
Copy link

You still end up with some pretty ugly distribution code

It is ugly indeed :D But would that matter much if the source map actually worked? I mean, babel's output, even unbundled, doesn't always look nice. And the more abstraction we add to the toolchain, the uglier the output code will get. Personally, I'd rather look at the original code via sourcemaps than look at the output code.

and source maps on shapely do not seem to work

Hey thanks for the heads up :) It used to work, so maybe I've broken it or something :D I'll look into it.

@mattdesl
Copy link
Author

Yeah, in theory source maps would solve everything. In practice, I find myself often needing to look at the original source for one reason or another. e.g. Debugging in legacy browsers, reading and tweaking a dependency in my node_modules/ to test a bug fix, looking through Promise stack traces which have destroyed source maps, or determining why Babel has introduced performance regressions in a hot code path.

To add to this, dependencies using a towering stack of language features tend to scare me off contributing. i.e. I probably won't submit PRs for a CoffeeScript-authored module since the source is mostly foreign to me. The same is becoming true with some of the experimental Babel features and plugins.

This is why I'd prefer to find a solution that doesn't involve "ugly code" for the end-user. 😄

@AriaMinaei
Copy link

Yeah, in theory source maps would solve everything. In practice, I find myself often needing to look at the original source for one reason or another. e.g. Debugging in legacy browsers, reading and tweaking a dependency in my node_modules/ to test a bug fix, looking through Promise stack traces which have destroyed source maps, or determining why Babel has introduced performance regressions in a hot code path.

Agreed. Been there in all cases. Output code must get better. But I think the long term solution is to just make source maps work better in practice. (Btw, promise stack traces can be patched to use sourcemaps).

To add to this, dependencies using a towering stack of language features tend to scare me off contributing. i.e. I probably won't submit PRs for a CoffeeScript-authored module since the source is mostly foreign to me. The same is becoming true with some of the experimental Babel features and plugins.

I see. I felt that way when trying to read react's source code. The bundled file was just too big to read, and the original source code seemed to have too much magic in it. Files require()ed packages that didn't actually exist (they were aliases for internal modules apparently). To me, that was just confusing.

Though generally I'm okay with different build systems and languages (they're all usually either JavaScript-ish or CoffeeScript-ish), as long as the build config is easy to understand, easy to set up, and there is no 'magic' in it.

All and all, I agree with you. Making bundled code look nicer will make everything easier.

@ahdinosaur
Copy link

@mattdesl: i reckon @mmckegg and i are keen for your simple transpiler idea. we were also thinking about a bundler for node using browserify, so require wouldn't have to synchronously look up the filesystem (sync require is slow!), but you're right that it's probably best to split those into separate modules.

@mattdesl
Copy link
Author

mattdesl commented Jan 3, 2016

Two new things:

@mattdesl
Copy link
Author

mattdesl commented Jan 4, 2016

@ahdinosaur
Copy link

@mattdesl 😻

@tbroyer
Copy link

tbroyer commented Feb 10, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment