We Need An Automated Module Build System

readme.md

We Need An Automated Module Build System

First, please read On Maintaining a Native Node Module

Problem: There is no standard way to build + distribute prebuilt binaries for node modules with native addons (e.g. C++ bindings).

There's a thing called NODE_MODULE_VERSION which is used to know if a native module was compiled for the correct version + flavor of node. If you are trying to load something compiled for e.g. iojs from February 2015 into node from March 2014 you will get an error something like Error: Module version mismatch. Expected 11, got 42. when you try and run your program. node uses numbers from 10 going up, and iojs uses numbers from 40 going up. The NODE_MODULE_VERSION gets bumped usually when V8 is upgraded, but sometimes for other reasons. It also gets even more complicated when you start talking about runtimes like Electron or Node-Webkit, but I won't get into that here. To my knowledge there is no list of all the NODE_MODULE_VERSIONs out there (EDIT here's one).

For maintainers, this results in a version matrix like this (duplicate this chart for every release they do of their native module):

	11	12	43	44
linux-x64
linux-x86
darwin-x64
win32-x64
win32-ia32
linux-armv7l
linux-armv6l

In my opinion all native modules should offer prebuilt binaries. An example of a module that switched to this recently is leveldown.

Previously, when you did npm install leveldown, it required you to have a C++ build chain installed so it could compile leveldb. This means doing e.g. apt-get install build-essential on Linux, downloading the multi-GB XCode from the Mac App Store on Mac, or downloading Visual Studio Free Edition on Windows. It's a big pain in the butt, especially for newbies (I have lots of empirical data on this from the nodeschool community).

Now, thanks to use of node-pre-gyp, npm looks at the leveldown releases for a prebuilt version matching the platform + architecture of the computer that is installing leveldown. If it finds a match, it just downloads the trusted prebuilt release. If it doesn't find a match, it falls back to the old method of compiling the sources.

To revisit our problem from above: Maintainers need to build every version in the above chart, for every version of their native addon!

Some people have hodge-podged together hacks that take advantage of Travis-CI (linux) or Appveyor (windows) to compile their modules and even upload them to GitHub Releases or S3. I'm proposing that someone in the community should take on the project of creating a dedicated, streamlined service that essentially does this, but focusing on making it easy specifically for native modules, because the tools that exist for this today aren't good enough, especially when you want to support building for platforms like Mac or ARM.

Feedback? Solutions? Use the comments below.

GNU R has this for the package repository called CRAN (http://cran.r-project.org/web/packages/). They have binary packages for Windows and OSX, but not for Linux. The system works very well, but it also needs a lot of pestering and manual software installations on the machines. They also have a lot of experience about what works with what. (E.g. mixing Fortran 90 and C++ code does not usually work.)

R does not have versioned dependencies, so they only build/keep binaries the latest versions of the packages.

I happen to know about (and will be possibly involved in) about a project that set out to improve the binary build system for R. Essentially we want a CI system that publishes the build binaries for several platforms: Windows (mingw), OS X (yosemite and snow leopard), various Linux versions and Solaris (!): http://cran.r-project.org/web/checks/check_flavors.html

While you need a node.js builder, and we need an R builder, there are also a lot of similarities. The general infrastructure can be essentially the same, and you face some of the same problems. We could have a common multi-platform CI-like infrastructure that has an API, and manages virtual machines and containers, locally and/or in the cloud. Similarly the handling of external libraries can be essentially the same: e.g. how a package specifies that it needs the libxml2 library, and what exactly one needs to install on the different platforms to fulfill this request.

I would love to collaborate on this.

Maybe a stupid idea but how about instead of building a centralized service it will work in a "greedy" like way ? First one to build the module on a platform will also publish it. Next one to install the module on the platform will download the already pre built binary.

This makes it hard for the first person on each platform, but makes it easy on the rest and saves the hassle of running and maintaining a centralized buikd service.

node-gyp-install is also a useful way to avoid plenty of headaches with iojs

https://github.com/mafintosh/node-gyp-install

Maybe a stupid idea but how about instead of building a centralized service it will work in a "greedy" like way ? First one to build the module on a platform will also publish it. Next one to install the module on the platform will download the already pre built binary.

That's a good idea to distribute the work, but I don't think it can be made safe. The builder needs to be a trusted source.

This sounds like it might be a great thing for the Node Foundation to support. They are already maintaining a similar environment just for builds of Node/io.js. They would also have the best chance of being able to fund such a project.

I'm sure a prototype of the service would go along way to getting more attention for this.

I shared some ideas in this io.js issue:

If npm packages were integrated with Travis CI or something similar, we'd have more accurate information about the build status of packages, which might be useful in general, especially for in this situation.

@RnbWd Excellent idea.

Many packages are already individually testing with Travis CI pulling from GitHub, but more general automated testing done straight from npm repository against available Node/io.js versions would be an awesome service. It could easily show compatibility statistics against any core version. I think running simple npm install (and npm test only if enabled by package author) would be enough.

I think someone is already working on this ;) If not, then someone should.

I suggest node-gyp, nan, etc. should be automated, not a responsibility of module maintainers. Even if this hypothetical (yet possible) build tool existed, it also needs to be incorporated in node-core and / or npm.

We should be able to determine whether or not a module is compatible on our system prior to install:

• metadata linked to npm packages (independent of maintainer)
• filter packages
• minimize build-time errors during install
• potentially use that information to build packages

During the install process, the correct versions of nan, node-gyp, or a prebuilt binary would be installed independent the original maintainer's package.json

• additional scripts would need to be included during the npm install process
• node-gyp, node-gyp-prebuilt, and nan should be pulled into the node foundation as indispensable assets
• the foundation should also help fund resources necessary to maintain host these resources

Sometimes, source code needs modification depending on the situation, and I don't know a good solution for that problem. Sometimes, the author just wrote a bad package.json file, and that would be trivial to modify. But IMO the weirdest scenario would be if an older / newer module works on your system, should npm inform us and let us choose the compatible alternative?

There may be other solutions, but I believe all of them will require the cooperation of companies like npm and the node-foundation, along with contributions from the open source community (even people like myself who've never programmed a line of C but still know how to use nan and node-gyp)

@mderazon it would be nice if it could be that simple, but there are SIGNIFICANT security problems with allowing a random person to upload a binary. If compilation was deterministic (compiling the same code produces a binary with the same hash) things would be much better, because if many people produced the same hash, you could be reasonablely confidant that build was untampered with.

could the compilation happen on the server in CI similar to what is done by homebrew?

@thealphanerd I agree this seems like something that should go hand-in-hand with automated testing

Hi,
What would be a good scheme for the native library dependency versions? For example hdf5-1.8.14 <= versions? Some users are back a few clicks.

@rvagg did a nice writeup related to this nodejs/build#151