Building & Deploying Node Services

build-deploy-node.md

Building & Deploying Node Services

Chetan Desai @chetanddesai
DevOps Architect, Consumer Tax Group
Intuit, Inc.

When Node.js was first introduced to Intuit, our ops teams had a great challenge in front of them. What do you mean the whole thing dies if there is an exception?! Since Node.js is simply a command-line executable, and not a container, the build and deployment of node services is critical to the success of running Node.js services. I'd like to share some of our learnings with you all.

Build Tips

Never Install Global Packages

This one can be pretty controversial, especially when you get that silly warning npm WARN prefer global <pkgName>@<ver> should be installed with -g. There are a few issues with installing packages globally, but a very easy solution to not installing them globally.

1. You no longer have documentation of that module being used in your package.json.
2. Multiple developers could be using different versions depending on when they installed that package. By having it in the package.json, you can ensure they're pulling down either the same, or compatible versions.
3. It won't work in a shared build environment. This was the biggest issue for us. Build slave users rarely have sudo access for global installations, and even if they did, the version of a global module you are using might be different from a version someone else is using.

Fortunately, the solution is quite simple. When npm starts, it will bootstrap your path with any executables found in your node_modules folder. So there is no need to install command-line tools like Grunt, Gulp, Mocha, etc globally. Instead, add the command into a script in your package.json, such as:

"scripts": {
    "grunt": "grunt build",
    "test": "mocha unit-tests"
  }

Then update your documentation to tell people to execute npm run grunt instead of grunt build.

The Power of the package.json

The package.json of a node app should be the primary source of documentation. Through the scripts section, one can define how to build, test, and run your service. You can create any custom script and run it by typing npm run <script-name>. There are a few pre-defined scripts that you can use without the run keyword. npm test and npm start are a few examples. Here are the guidelines we give our node service developers:

1. Always define how to build, test, and start your service in the scripts section.
2. Provide a way to locally run your service through the start script via npm start. Provide examples of passing in a NODE_ENV and PORT.

  "scripts": {
    "start": "NODE_ENV=dev PORT=9000 node app.js"
  }

The barrier to settings up a local environment gets virtually eliminated through this sort of documentation. A second great benefit is that the easier to set up and use your service, the more likely people will clone it down and contribute back!!

Our Build Lifecycle

We follow the same pattern for building all of our node services.

1. npm install: install all the packages
2. npm shrinkwrap --dev: generate a shrinkwrap file with the dev dependencies so we can reproduce this build and scan for vulnerabilities.
3. npm run nsp: run the node security module script that calls nsp shrinkwrap to analyze our modules for known vulnerabilities. At the time of writing, the nsp cli and the requiresafe cli might be converging, so this call might change over to a requiresafe check execution in the future.
4. npm test: test the module
5. npm run <script-name>: run any custom scripts for building.
6. npm prune --production: remove the hundreds of megabytes of dev dependencies for building and testing that we don't need at runtime.

After these steps are complete, we zip up and version our service, so it's ready to be promoted through a number of environments.

Deployment Tips

Defining the Environment

The environment variable NODE_ENV needs to be used for loading environment specific configurations as well as changing behavior based on the environment. This environment variable is used as a best practice throughout many modules in npm. Specifically, for production, many projects use the value NODE_ENV=production.

Managing Stateless Services

Since your node service will throw an exception, and it will crash, we want to have a process manager ready to restart it. Node is designed to start up really fast. Because of this, and the single threaded nature of Node.js, we always want to have more than one process running. As a rule of thumb, we use the same number of CPUs on the box as node processes to run.

There are many different ways to manage your node services from modules like forever to built in functionality like cluster. A lot of new players are coming into this realm also. StrongLoop has their own Process Manager offering and put together a nice comparison chart. However, we don't use any of these. At Intuit we host our services on RHEL6, and we feel the best entity to manage processes is the OS itself! In RHEL6 we have a super easy way of managing processes via upstart. We are able to launch multiple instances of services and watch it by passing in a PORT environment variable. You will need to change your default port behavior to take an environment variable like the following.

server.listen(process.env.PORT || 8080, function () {
  logger.info('%s listening at %s', server.name, server.url);
});

That way we can run an upstart configuration that looks like the following:

/etc/init/nodejs-parent.conf

env name="nodejs parent"
env desc="Node.js"
env instances=4
env start_port=8080

# This script is a startup 'task' so we only define startup runlevels
# and Upstart won't exepct this script to be stoppable, the instance
# jobs are tracked and stoppable by Upstart
start on runlevel [2345]
task

script

  # For each instance ...
  for i in $(seq 1 1 $instances); do

    # Increment port number
    let port=$(expr $start_port + $i - 1)

    logger -s -t ${name} $"Starting ${desc} instance on port [${port}]"

    # Run the instance servers, passing any instance parameters as variables,
    # note that the parameters also must be defined in the `instance` stanza
    # in the instance configuration and are also required for manual control
    # of the job, e.g:
    # initctl stop nodejs-instance port=8080 foo=bar

    initctl start nodejs-instance port=${port}

  done

end script

The parent script will kick off and watch the number of specified instances, restarting any crashed instances on the specified port.

/etc/init/nodejs-instance.conf

instance ${port} # Incoming named argument from initctl

env name="nodejs-instance"
env desc="Node.js"
env user=nodejs
env directory=/opt/node/my-node-service
env ulimit="ulimit -HSn 10240"

# This script is started by a parent script so we only need to say when
# to stop it
stop on runlevel [S016]

respawn
respawn limit 20 10

# Variable expansion is only resolved in the script section
script

  logger -s -t ${name} $"Starting ${desc} instance on port [${port}]"

  $ulimit

  cd $directory

  # Start as non-privileged user, --session-command keeps the executable
  # within the child shell so it doesn't confuse Upstart and allows Upstart
  # to send appropriate signals directly to `node`

  exec bash <<EOT
    exec >> /app/node/my-node-service/logs/my-node-service-${port}.log 2>&1
    exec su ${user} --session-command "NODE_ENV=dev PORT=${port} node app.js"
  EOT

end script

I hope some of these tips were helpful, and I'd love to hear your thoughts and feedback. Leave a comment below!