Skip to content

Instantly share code, notes, and snippets.

@rvagg

rvagg/html

Created March 26, 2014 09:40
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save rvagg/9779764 to your computer and use it in GitHub Desktop.
Save rvagg/9779764 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
html
<h1>Testing code against many Node versions with Docker</h1>
<h2 id="the-problem-testing">The problem: testing</h2>
<p><strong><a href="https://github.com/rvagg/nan">NAN</a></strong> is a project designed to assist in building native (C++) Node.js add-ons while maintaining compatibility with Node and V8 from Node versions 0.8 onwards. V8 is undergoing <em>major</em> internal changes which is making add-on development very difficult; NAN's purpose is to abstract that pain. Instead of having to keep your code compatible across Node/V8 versions, NAN does it for you, and this is no simple task. But this means that we have to be sure to keep NAN tested and compatible with <em>all</em> of the versions it claims to support. This is no trivial exercise!</p>
<p><strong><a href="https://travis-ci.org/">Travis CI</a></strong> can help a little with this. It's possible to use <a href="https://github.com/creationix/nvm">nvm</a> to test across different versions of Node.js even beyond the officialy supported versions, we've tried this with NAN, without a whole lot of success. Ideally you'd have better choice of Node versions, but Travis have had some <a href="https://github.com/travis-ci/travis-ci/issues/1328">difficulty</a> keeping up. Also, historical npm bugs that ship with older versions of Node.js make it difficult, with a high failure rate from npm install problems, so we don't even publish the Travis badge on the NAN README because it just doesn't work.</p>
<p>The other problem with Travis is that it's a <em>CI</em> solution, not a proper <em>testing</em> solution. Even if it worked well, it's not really that helpful in the development process, you need rapid feedback that your code is working on your target platforms (this is one reason why I love back-end development more than front-end development!)</p>
<h2 id="the-solution-docker">The solution: Docker</h2>
<p>Enter Docker and <strong><a href="https://github.com/rvagg/dnt">DNT</a></strong></p>
<div style="margin: 0 auto;">
<img src="http://www.docker.io/static/img/homepage-docker-logo.png" width="138" height="114">
<img src="http://nodejs.org/images/logos/nodejs-dark.png" width="212" height="114">
<img src="http://img.pandawhale.com/29490-Picard-applause-clapping-gif-s5nz.gif" width="151" height="114">
</div>
<h3 id="dnt-docker-node-tester">DNT: Docker Node Tester</h3>
<p>Docker is a tool that simplifies the use of Linux containers to create lightweight, isolated compute "instances". Solaris and its variants have had this functionality for years in the form of "zones" but it's a relatively new concept for Linux and Docker makes the whole process a lot more friendly. Dockers relative simplicity has meant an amazing amout of activity in the Linux container space in recent months, it's become a huge ecosystem almost overnight.</p>
<p><strong>DNT</strong> is a very simple utility that contains two tools for working with Docker and Node.js. One tool helps to set-up containers for testing and the other runs your project's tests in those containers.</p>
<div style="margin: 0 auto;">
<img src="http://r.va.gg/images/2013/11/nan-dnt.png">
</div>
<p><strong>DNT</strong> includes a <code>setup-dnt</code> script that sets up the most basic Docker images required to run Node.js applications, nothing extra. It first creates an image called "dev_base" that uses the default Docker "ubuntu" image and adds the build tools required to compile and install Node.js</p>
<p>Next it creates a "node_dev" image that contains a complete copy of the Node.js <a href="http://github.com/joyent/node">source repository</a>. Finally, it creates a series of images that are required for the tests you want to run; for each Node version, it creates an image with Node installed and ready to use.</p>
<p>Setting up a project is a matter of creating a <em>.dntrc</em> file in the root directory of the project. This configuration file involves setting a <code>NODE_VERSIONS</code> variable with a list of all of the versions of Node you want to test against, and this can include "master" to test the latest code from the Node repository. You also set a <code>TEST_CMD</code> variable with a series of commands required to set up, compile and execute your tests. The <code>setup-dnt</code> command can be run against a <em>.dntrc</em> file to make sure that the appropriate Docker images are ready. The <code>dnt</code> command can then be used to execute the tests against all of the Node versions you specified.</p>
<p>Since Docker containers are completely isolated, <strong>DNT</strong> can run tests in parallel as long as the machine has the resources. The default is to use the number of cores on the computer as the concurrency level but this can be configured if not appropriate for the kinds of tests you want to run.</p>
<p>It's also possible to customise the base test image to include other, external tools and libraries required by your project, although this is a manual step in the set-up process.</p>
<p>Currently <strong>DNT</strong> is designed to parse TAP test output by reading the final line as either "ok" or "not ok" to report test status back on the command-line. It is configurable but you need to supply a command that will transform test output to either an "ok" or "not ok" (<code>sed</code> to the rescue?). The non-standard output of the Mocha TAP reporter is also supported out of the box.</p>
<h3 id="how-i-m-using-it">How I'm using it</h3>
<p>My primary use-case is for testing <strong>NAN</strong>. Being able to test against all the different V8 and Node APIs while coding is super helpful; particularly when tests run so quickly! My NAN <em>.dntrc</em> file tests against master, many of the 0.11 releases since 0.11.4 (0.11.0 to 0.11.3 are explicitly not supported by NAN and 0.11.11 and 0.11.12 are completely broken for native addons) and the last 5 releases of the 0.10 and 0.8 series. At the moment that's 18 versions of Node in all and on my computer the test suite takes approximately 20 seconds to complete across all of these releases.</p>
<p><strong>The NAN <a href="https://raw.github.com/rvagg/nan/master/.dntrc">.dntrc</a></strong></p>
<div class="highlight"><pre><span class="nv">NODE_VERSIONS</span><span class="o">=</span><span class="s2">"\</span>
<span class="s2"> master \</span>
<span class="s2"> v0.11.10 \</span>
<span class="s2"> v0.11.9 \</span>
<span class="s2"> v0.11.8 \</span>
<span class="s2"> v0.11.7 \</span>
<span class="s2"> v0.11.6 \</span>
<span class="s2"> v0.11.5 \</span>
<span class="s2"> v0.11.4 \</span>
<span class="s2"> v0.10.26 \</span>
<span class="s2"> v0.10.25 \</span>
<span class="s2"> v0.10.24 \</span>
<span class="s2"> v0.10.23 \</span>
<span class="s2"> v0.10.22 \</span>
<span class="s2"> v0.8.26 \</span>
<span class="s2"> v0.8.25 \</span>
<span class="s2"> v0.8.24 \</span>
<span class="s2"> v0.8.23 \</span>
<span class="s2"> v0.8.22 \</span>
<span class="s2">"</span>
<span class="nv">OUTPUT_PREFIX</span><span class="o">=</span><span class="s2">"nan-"</span>
<span class="nv">TEST_CMD</span><span class="o">=</span><span class="s2">"\</span>
<span class="s2"> cd /dnt/test/ &amp;&amp; \</span>
<span class="s2"> npm install &amp;&amp; \</span>
<span class="s2"> node_modules/.bin/node-gyp --nodedir /usr/src/node/ rebuild &amp;&amp; \</span>
<span class="s2"> node_modules/.bin/tap js/*-test.js; \</span>
<span class="s2">"</span>
</pre></div>
<p>Next I configured <strong><a href="https://github.com/rvagg/node-leveldown">LevelDOWN</a></strong> for <strong>DNT</strong>. LevelDOWN is a raw C++ binding that exposes LevelDB to Node.js, its main use is the backend for <a href="https://github.com/rvagg/node-levelup">LevelUP</a>. The needs are much simpler, the tests need to do a compile and run a lot of node-tap tests.</p>
<p><strong>The LevelDOWN <a href="https://raw.github.com/rvagg/node-leveldown/master/.dntrc">.dntrc</a></strong></p>
<div class="highlight"><pre><span class="nv">NODE_VERSIONS</span><span class="o">=</span><span class="s2">"\</span>
<span class="s2"> master \</span>
<span class="s2"> v0.11.10 \</span>
<span class="s2"> v0.11.9 \</span>
<span class="s2"> v0.10.26 \</span>
<span class="s2"> v0.10.25 \</span>
<span class="s2"> v0.8.26 \</span>
<span class="s2">"</span>
<span class="nv">OUTPUT_PREFIX</span><span class="o">=</span><span class="s2">"leveldown-"</span>
<span class="nv">TEST_CMD</span><span class="o">=</span><span class="s2">"\</span>
<span class="s2"> cd /dnt/ &amp;&amp; \</span>
<span class="s2"> npm install &amp;&amp; \</span>
<span class="s2"> node_modules/.bin/node-gyp --nodedir /usr/src/node/ rebuild &amp;&amp; \</span>
<span class="s2"> node_modules/.bin/tap test/*-test.js; \</span>
<span class="s2">"</span>
</pre></div>
<p>Another native Node add-on that I've set up with <strong>DNT</strong> is my <strong><a href="https://github.com/rvagg/node-libssh">libssh Node.js bindings</a></strong>. This one is a little more complicated because you need to have some non-standard libraries installed before compile. My <em>.dntrc</em> adds some extra <code>apt-get</code> sauce to fetch and install those packages. It means the tests take a little longer but it's not prohibitive. An alternative would be to configure the <em>node_dev</em> base-image to have these packages to all of my versioned images have them too.</p>
<p><strong>The node-libssh <a href="https://raw.github.com/rvagg/node-libssh/master/.dntrc">.dntrc</a></strong></p>
<div class="highlight"><pre><span class="nv">NODE_VERSIONS</span><span class="o">=</span><span class="s2">"master v0.11.10 v0.10.26"</span>
<span class="nv">OUTPUT_PREFIX</span><span class="o">=</span><span class="s2">"libssh-"</span>
<span class="nv">TEST_CMD</span><span class="o">=</span><span class="s2">"\</span>
<span class="s2"> apt-get install -y libkrb5-dev libssl-dev &amp;&amp; \</span>
<span class="s2"> cd /dnt/ &amp;&amp; \</span>
<span class="s2"> npm install &amp;&amp; \</span>
<span class="s2"> node_modules/.bin/node-gyp --nodedir /usr/src/node/ rebuild --debug &amp;&amp; \</span>
<span class="s2"> node_modules/.bin/tap test/*-test.js --stderr; \</span>
<span class="s2">"</span>
</pre></div>
<p><strong><a href="https://github.com/rvagg/node-levelup">LevelUP</a></strong> isn't a native add-on but it does use LevelDOWN which requires compiling. For the DNT config I'm removing <em>node_modules/leveldown/</em> prior to <code>npm install</code> so it gets rebuilt each time for each new version of Node.</p>
<p><strong>The <a href="https://raw.github.com/rvagg/node-levelup/master/.dntrc">LevelUP .dntrc</a></strong></p>
<div class="highlight"><pre><span class="nv">NODE_VERSIONS</span><span class="o">=</span><span class="s2">"\</span>
<span class="s2"> master \</span>
<span class="s2"> v0.11.10 \</span>
<span class="s2"> v0.11.9 \</span>
<span class="s2"> v0.10.26 \</span>
<span class="s2"> v0.10.25 \</span>
<span class="s2"> v0.8.26 \</span>
<span class="s2">"</span>
<span class="nv">OUTPUT_PREFIX</span><span class="o">=</span><span class="s2">"levelup-"</span>
<span class="nv">TEST_CMD</span><span class="o">=</span><span class="s2">"\</span>
<span class="s2"> cd /dnt/ &amp;&amp; \</span>
<span class="s2"> rm -rf node_modules/leveldown/ &amp;&amp; \</span>
<span class="s2"> npm install --nodedir=/usr/src/node &amp;&amp; \</span>
<span class="s2"> node_modules/.bin/tap test/*-test.js --stderr; \</span>
<span class="s2">#"</span>
</pre></div>
<h3 id="what-s-next-">What's next?</h3>
<p>It's not hard to imagine this forming the basis of a local CI system as well as a general testing tool. The speed even makes it tempting to run the tests on every git commit, or perhaps on every save. Already, the <a href="http://newrelic.com">New Relic</a> Node.js agent team are using an internal fork of DNT to test their agent against many versions of Node.js combined with tests for various common server frameworks; a very complicated job.</p>
<p>I'm always keen to have contributors, if you have particular needs and the skills to implement new functionality then I'd love to hear from you. I'm generally very open with my open source projects and happy to add contributors who add something valuable.</p>
<p>See the <strong><a href="https://github.com/rvagg/dnt">DNT</a></strong> GitHub repo for installation and detailed usage instructions.</p>
Raw
md
# Testing code against many Node versions with Docker
## The problem: testing
**[NAN](https://github.com/rvagg/nan)** is a project designed to assist in building native (C++) Node.js add-ons while maintaining compatibility with Node and V8 from Node versions 0.8 onwards. V8 is undergoing *major* internal changes which is making add-on development very difficult; NAN's purpose is to abstract that pain. Instead of having to keep your code compatible across Node/V8 versions, NAN does it for you, and this is no simple task. But this means that we have to be sure to keep NAN tested and compatible with *all* of the versions it claims to support. This is no trivial exercise!
**[Travis CI](https://travis-ci.org/)** can help a little with this. It's possible to use [nvm](https://github.com/creationix/nvm) to test across different versions of Node.js even beyond the officialy supported versions, we've tried this with NAN, without a whole lot of success. Ideally you'd have better choice of Node versions, but Travis have had some [difficulty](https://github.com/travis-ci/travis-ci/issues/1328) keeping up. Also, historical npm bugs that ship with older versions of Node.js make it difficult, with a high failure rate from npm install problems, so we don't even publish the Travis badge on the NAN README because it just doesn't work.
The other problem with Travis is that it's a *CI* solution, not a proper *testing* solution. Even if it worked well, it's not really that helpful in the development process, you need rapid feedback that your code is working on your target platforms (this is one reason why I love back-end development more than front-end development!)
## The solution: Docker
Enter Docker and **[DNT](https://github.com/rvagg/dnt)**
<div style="margin: 0 auto;">
<img src="http://www.docker.io/static/img/homepage-docker-logo.png" width="138" height="114">
<img src="http://nodejs.org/images/logos/nodejs-dark.png" width="212" height="114">
<img src="http://img.pandawhale.com/29490-Picard-applause-clapping-gif-s5nz.gif" width="151" height="114">
</div>
### DNT: Docker Node Tester
Docker is a tool that simplifies the use of Linux containers to create lightweight, isolated compute "instances". Solaris and its variants have had this functionality for years in the form of "zones" but it's a relatively new concept for Linux and Docker makes the whole process a lot more friendly. Dockers relative simplicity has meant an amazing amout of activity in the Linux container space in recent months, it's become a huge ecosystem almost overnight.
**DNT** is a very simple utility that contains two tools for working with Docker and Node.js. One tool helps to set-up containers for testing and the other runs your project's tests in those containers.
<div style="margin: 0 auto;">
<img src="http://r.va.gg/images/2013/11/nan-dnt.png">
</div>
**DNT** includes a `setup-dnt` script that sets up the most basic Docker images required to run Node.js applications, nothing extra. It first creates an image called "dev_base" that uses the default Docker "ubuntu" image and adds the build tools required to compile and install Node.js
Next it creates a "node_dev" image that contains a complete copy of the Node.js [source repository](http://github.com/joyent/node). Finally, it creates a series of images that are required for the tests you want to run; for each Node version, it creates an image with Node installed and ready to use.
Setting up a project is a matter of creating a *.dntrc* file in the root directory of the project. This configuration file involves setting a `NODE_VERSIONS` variable with a list of all of the versions of Node you want to test against, and this can include "master" to test the latest code from the Node repository. You also set a `TEST_CMD` variable with a series of commands required to set up, compile and execute your tests. The `setup-dnt` command can be run against a *.dntrc* file to make sure that the appropriate Docker images are ready. The `dnt` command can then be used to execute the tests against all of the Node versions you specified.
Since Docker containers are completely isolated, **DNT** can run tests in parallel as long as the machine has the resources. The default is to use the number of cores on the computer as the concurrency level but this can be configured if not appropriate for the kinds of tests you want to run.
It's also possible to customise the base test image to include other, external tools and libraries required by your project, although this is a manual step in the set-up process.
Currently **DNT** is designed to parse TAP test output by reading the final line as either "ok" or "not ok" to report test status back on the command-line. It is configurable but you need to supply a command that will transform test output to either an "ok" or "not ok" (`sed` to the rescue?). The non-standard output of the Mocha TAP reporter is also supported out of the box.
### How I'm using it
My primary use-case is for testing **NAN**. Being able to test against all the different V8 and Node APIs while coding is super helpful; particularly when tests run so quickly! My NAN *.dntrc* file tests against master, many of the 0.11 releases since 0.11.4 (0.11.0 to 0.11.3 are explicitly not supported by NAN and 0.11.11 and 0.11.12 are completely broken for native addons) and the last 5 releases of the 0.10 and 0.8 series. At the moment that's 18 versions of Node in all and on my computer the test suite takes approximately 20 seconds to complete across all of these releases.
**The NAN [.dntrc](https://raw.github.com/rvagg/nan/master/.dntrc)**
```sh
NODE_VERSIONS="\
master \
v0.11.10 \
v0.11.9 \
v0.11.8 \
v0.11.7 \
v0.11.6 \
v0.11.5 \
v0.11.4 \
v0.10.26 \
v0.10.25 \
v0.10.24 \
v0.10.23 \
v0.10.22 \
v0.8.26 \
v0.8.25 \
v0.8.24 \
v0.8.23 \
v0.8.22 \
"
OUTPUT_PREFIX="nan-"
TEST_CMD="\
cd /dnt/test/ && \
npm install && \
node_modules/.bin/node-gyp --nodedir /usr/src/node/ rebuild && \
node_modules/.bin/tap js/*-test.js; \
"
```
Next I configured **[LevelDOWN](https://github.com/rvagg/node-leveldown)** for **DNT**. LevelDOWN is a raw C++ binding that exposes LevelDB to Node.js, its main use is the backend for [LevelUP](https://github.com/rvagg/node-levelup). The needs are much simpler, the tests need to do a compile and run a lot of node-tap tests.
**The LevelDOWN [.dntrc](https://raw.github.com/rvagg/node-leveldown/master/.dntrc)**
```sh
NODE_VERSIONS="\
master \
v0.11.10 \
v0.11.9 \
v0.10.26 \
v0.10.25 \
v0.8.26 \
"
OUTPUT_PREFIX="leveldown-"
TEST_CMD="\
cd /dnt/ && \
npm install && \
node_modules/.bin/node-gyp --nodedir /usr/src/node/ rebuild && \
node_modules/.bin/tap test/*-test.js; \
"
```
Another native Node add-on that I've set up with **DNT** is my **[libssh Node.js bindings](https://github.com/rvagg/node-libssh)**. This one is a little more complicated because you need to have some non-standard libraries installed before compile. My *.dntrc* adds some extra `apt-get` sauce to fetch and install those packages. It means the tests take a little longer but it's not prohibitive. An alternative would be to configure the *node_dev* base-image to have these packages to all of my versioned images have them too.
**The node-libssh [.dntrc](https://raw.github.com/rvagg/node-libssh/master/.dntrc)**
```sh
NODE_VERSIONS="master v0.11.10 v0.10.26"
OUTPUT_PREFIX="libssh-"
TEST_CMD="\
apt-get install -y libkrb5-dev libssl-dev && \
cd /dnt/ && \
npm install && \
node_modules/.bin/node-gyp --nodedir /usr/src/node/ rebuild --debug && \
node_modules/.bin/tap test/*-test.js --stderr; \
"
```
**[LevelUP](https://github.com/rvagg/node-levelup)** isn't a native add-on but it does use LevelDOWN which requires compiling. For the DNT config I'm removing *node_modules/leveldown/* prior to `npm install` so it gets rebuilt each time for each new version of Node.
**The [LevelUP .dntrc](https://raw.github.com/rvagg/node-levelup/master/.dntrc)**
```sh
NODE_VERSIONS="\
master \
v0.11.10 \
v0.11.9 \
v0.10.26 \
v0.10.25 \
v0.8.26 \
"
OUTPUT_PREFIX="levelup-"
TEST_CMD="\
cd /dnt/ && \
rm -rf node_modules/leveldown/ && \
npm install --nodedir=/usr/src/node && \
node_modules/.bin/tap test/*-test.js --stderr; \
#"
```
### What's next?
It's not hard to imagine this forming the basis of a local CI system as well as a general testing tool. The speed even makes it tempting to run the tests on every git commit, or perhaps on every save. Already, the [New Relic](http://newrelic.com) Node.js agent team are using an internal fork of DNT to test their agent against many versions of Node.js combined with tests for various common server frameworks; a very complicated job.
I'm always keen to have contributors, if you have particular needs and the skills to implement new functionality then I'd love to hear from you. I'm generally very open with my open source projects and happy to add contributors who add something valuable.
See the **[DNT](https://github.com/rvagg/dnt)** GitHub repo for installation and detailed usage instructions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment