Proof of concept to explore various Vagrant/Docker setups

.gitignore

node_modules/
.vagrant/

README.md

gist-vagrant-docker-explore

Proof of concept to explore various Vagrant/Docker setups

In the past, we have used Vagrant for our VM but ran into file system performance issues with Node.js based projects. The key grievance was syncing the local folder . caused significant delay in picking up file system changes on the remote machine. This slowed down developer workflows and caused frustration.

We want to see if other non-default setups will work as well as how to use Docker. For this exercise, we have bloated our node_modules folder (see package.json for details).

Host machine

As a baseline, we ran the watch script locally on the host machine.

Steps:

# Install our Node.js dependencies
npm install

# Start our script
npm start

Results:

• File system change events: nodemon restarted immediately
• File system read/write: index.js takes 300ms to load heavy dependencies

Vagrant with VirtualBox + default sync

https://www.vagrantup.com/docs/synced-folders/basic_usage.html

This setup uses VirtualBox and the default shared folders to sync a lot of node_modules between the local and remote machine.

For this experiment, run the following:

# Start our Vagrant machine
export ENV="default"
vagrant up

# SSH into the machine
vagrant ssh

# Start up a simple watch + reboot script
cd /vagrant
npm start

Results:

• File system change events: nodemon restarted immediately
• File system read/write: index.js took 2000ms to 3500ms to load heavy dependencies

Vagrant with VirtualBox + no sync

As a second baseline, we have a setup with no sync at all. For this, run the following:

# Start a default Vagrant machine
export ENV="default"
vagrant up

# SSH into the machine
vagrant ssh

# Copy the contents of `/vagrant` to a temporary folder
# DEV: This may take a while...
cd "$(mktemp -d)"
cp /vagrant/* . -r

# Start up a simple watch + reboot script
npm start

Results:

• File system change events: nodemon restarted immediately
• File system read/write: index.js takes 300ms to load heavy dependencies

Vagrant with VirtualBox + ignore

"Disabling" on https://www.vagrantup.com/docs/synced-folders/basic_usage.html

Add a disabled sync for node_modules to see if we can prevent Vagrant from syncing this folder

Steps are the same as "Vagrant with VirtualBox + default sync" but export ENV="ignore"

Unfortunately, we were unable to get this implementation working.

Additionally, we attempted to sync folders via Dir::glob + reject! but Vagrant isn't setup for syncing individual files.

Vagrant with VirtualBox + NFS

https://www.vagrantup.com/docs/synced-folders/nfs.html

Moved to NFS based sync for node_modules

Steps:

# Install NFS dependencies on host machine
# https://help.ubuntu.com/community/SettingUpNFSHowTo
sudo apt-get install -y nfs-kernel-server

# Start our Vagrant machine
export ENV="nfs"
vagrant up

# SSH into the machine
vagrant ssh

# Start up a simple watch + reboot script
cd /vagrant
npm start

Results:

• File system change events: nodemon starts within 300ms of save
• File system read/write: index.js takes 600ms to 1400ms to load a heavy dependency

Vagrant with VirtualBox + rsync

https://www.vagrantup.com/docs/synced-folders/rsync.html

We have skipped this due to lack of integration; we would need to run a separate vagrant rsync-auto process at all times.

Docker only

https://www.docker.com/

We explored using vanilla Docker (i.e. no Vagrant wrapper). To get our example running, perform the following:

# Install Docker
# See https://docs.docker.com/linux/step_one/

# Build our container
docker build -t docker-explore .

# Run our container with the local directory mounted to /vagrant
docker run -i -t -v $PWD:/vagrant docker-explore /bin/bash

# Run our script
cd /vagrant
npm start

Results:

• File system change events: nodemon restarted immediately (local performance)
• File system read/write: index.js takes 300ms to load a heavy dependency (local performance)

Notes:

• We should be able to use a bootstrap.sh strategy via ADD bootstrap.sh inside of Dockerfile
• We can connect to a running container via docker-exec
  • For example docker exec -it 3f0f94a39ae2 /bin/bash will connect to the container with id 3f0f94a39ae2
• To expose a port on an existing container, we must stop + commit (with change) + start
  • http://stackoverflow.com/a/26622041
  • It seems that docker-commit supports adding in EXPOSE instructions via change as well
• To persist data across container rebuilds, we can use a data-only container
  • https://docs.docker.com/engine/userguide/containers/dockervolumes/

Conclusions:

I'm on the fence about using Docker at the moment. It seems like it should work with some paradigm changes but there are definitely going to be unforeseen issues that we have previously solved with Vagrant.

We might be best off with sticking to Vagrant but keeping Node.js local for now.

Vagrant with Docker

We started getting this set up but decided to focus on Docker instead of learning both how it works + how it integrates into Vagrant.

I think the main benefit here is docker run {{container-id}} becomes vagrant docker-run (i.e. no id/name necessary).

Vagrant with LXC

https://github.com/fgrehm/vagrant-lxc

We are trying another container based solution for Vagrant. This one isn't as portable as Docker but it has the same syntax as Vagrant VirtualBox 👍

Steps:

# Install LXC dependencies
sudo apt-get install lxc redir

# Install vagrant-lxc plugin
vagrant plugin install lxc

# Start our Vagrant machine
export ENV="lxc"
vagrant up

# SSH into the machine
vagrant ssh

# Start up a simple watch + reboot script
cd /vagrant
npm start

Results:

• File system change events: nodemon restarted immediately (local performance)
• File system read/write: index.js takes 300ms to load a heavy dependency (local performance)

Notes:

• We are using a secondary user on our machine so the UIDs don't align. As a result, we need an additionl script during provisioning
  • https://github.com/fgrehm/vagrant-lxc/wiki/FAQ#help-my-shared-folders-have-the-wrong-owner
  • fgrehm/vagrant-lxc#151
• We need to use a special image for LXC support. However, it's well documented:
  • https://github.com/fgrehm/vagrant-lxc/blob/v1.2.1/BOXES.md
  • https://github.com/fgrehm/vagrant-lxc-base-boxes

Vagrant with VirtualBox + other sync options

• https://www.wildcardcorp.com/blog/making-vagrant-virtualbox-nfs-less-painful-with-sparse-disk-images
  • Seems to be OS X specific and other options listed are network based as well
• https://github.com/mrdavidlaing/vagrant-unison
  • VM + another sync program
  • TODO: Try it out but not confident in it (also need to do same separate command as rsync-auto)

bootstrap.sh

#!/usr/bin/env bash
# Exit on first error and output commands
set -e
set -x

# If we haven't updated apt-get, then update it now
if ! test -f .updated-apt-get; then
  sudo apt-get update
  touch .updated-apt-get
fi

# If we haven't installed Node.js, then install it
# https://github.com/nodesource/distributions/tree/21a7b37baeccfb8e09915fbcd57913afe5c34764#installation-instructions
if ! which node &> /dev/null; then
  curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -
  sudo apt-get install -y "nodejs=4.4.3-1nodesource1~trusty1"
fi

Dockerfile

# Base ourselves on Ubuntu 14.04 Trusty
# DEV: Script based on http://stackoverflow.com/a/27240910
FROM ubuntu:14.04

# Update all dependencies
RUN apt-get update

# Install Node.js dependencies
RUN apt-get install -y curl git

# Install Node.js
RUN curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -
RUN apt-get install -y "nodejs=4.4.3-1nodesource1~trusty1"

# By default, run our script
WORKDIR /vagrant
CMD npm start

index.js

// Load in a dependency heavy dependency
var start = Date.now();
var browserify = require('browserify');
var gulp = require('gulp');
var end = Date.now();

// Log something simple
console.log('Hello World! (' + (end - start) + 'ms)');

package.json

{
  "name": "gist-vagrant-docker-explore",
  "version": "1.0.0",
  "description": "Proof of concept to explore various Vagrant/Docker setups",
  "main": "index.js",
  "scripts": {
    "start": "nodemon -L index.js",
    "postinstall": "npm run webdriver-update",
    "test": "echo \"Error: no test specified\" && exit 1",
    "webdriver-start": "xvfb-run webdriver-manager start",
    "webdriver-update": "webdriver-manager update"
  },
  "author": "Todd Wolfson <todd@twolfson.com> (http://twolfson.com/)",
  "license": "Unlicense",
  "dependencies": {
    "browserify": "~13.0.0",
    "gemini": "~4.0.3",
    "gemini-gui": "~4.0.0",
    "gulp": "~3.9.1",
    "gulp-imagemin": "~2.4.0",
    "nodemon": "~1.9.1",
    "webdriver-manager": "~10.0.1"
  }
}

Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Resolve our options
env = ENV["ENV"]
if env.nil?
  raise("Environment variable \`ENV\` wasn't defined. Please define it")
end
options = {:sync_type => :default, :provider => :virtualbox}
if (env == "default")
  # Do nothing
elsif (env == "docker")
  options[:provider] = :docker
elsif (env == "ignore")
  options[:sync_type] = :ignore
elsif (env == "lxc")
  options[:provider] = :lxc
elsif (env == "nfs")
  options[:sync_type] = :nfs
else
  raise("Environment \"#{env}\" wasn't recognized. Please verify spelling")
end

# Define our Vagrant configuration
Vagrant.configure(2) do |config|
  # If we want to ignore a file, then ignore it
  if options.fetch(:sync_type) == :ignore
    config.vm.synced_folder(".", "/vagrant", :disabled => false)
    config.vm.synced_folder("./node_modules", "/vagrant/node_modules", :disabled => true)
  # Otherwise if we want to set up NFS, then reserve a private IP for it and set up the sync
  elsif options.fetch(:sync_type) == :nfs
    config.vm.network("private_network", :ip => "192.168.50.4")
    config.vm.synced_folder(".", "/vagrant", :type => "nfs")
  end

  # Give additional memory and CPU to VirtualBox provider
  # https://docs.vagrantup.com/v2/virtualbox/configuration.html
  if options.fetch(:provider) == :virtualbox
    # Set our box as Ubuntu@14.04 LTS
    # https://atlas.hashicorp.com/ubuntu/boxes/trusty64
    config.vm.box = "ubuntu/trusty64"

    # Configure VirtualBox
    config.vm.provider "virtualbox" do |vb|
      vb.memory = 2048
      vb.cpus = 2
    end
  elsif options.fetch(:provider) == :docker
    # Attribution for sanity https://github.com/bubenkoff/vagrant-docker-example/tree/5dd16820994b651a07edce616a76ea86807c6b4f
    ENV["VAGRANT_DEFAULT_PROVIDER"] = "docker"
    config.vm.provider "docker" do |d|
      d.build_dir = "."
      d.has_ssh = true
    end
  elsif options.fetch(:provider) == :lxc
    ENV["VAGRANT_DEFAULT_PROVIDER"] = "lxc"

    # Repair Vagrant UID/GID to match our current user
    uid = `id -u`.strip()
    gid = `id -g`.strip()
    config.vm.provision "shell", inline: <<-EOF
      # Exit on first error
      set -e

      # Resolve our UID and GID
      src_uid="$(id -u vagrant)"
      target_uid="#{uid}"
      src_gid="$(id -g vagrant)"
      target_gid="#{gid}"

      # If the user and group ids are aligned, then exit early
      if test "$src_uid" = "$target_uid" && test "$src_gid" = "$target_gid"; then
        exit 0
      fi

      # Otherwise, update our user id and group id
      # DEV: We cannot use \`usermod\` as it complains about \`vagrant\` having a process
      # Example: UID=100; GID=101
      #  /etc/shadow: libuuid:x:100:101::/var/lib/libuuid:
      #  /etc/group: libuuid:x:101:
      sed -E "s/(vagrant:.:)$src_uid:$src_gid:/\\1$target_uid:$target_gid:/g" --in-place /etc/passwd
      sed -E "s/(vagrant:.:)$src_gid:/\\1$target_gid:/g" --in-place /etc/group

      # Update all files to the proper user and group
      find / -uid "$src_uid" 2> /dev/null | grep --invert-match -E "^(/sys|/proc)" | xargs chown "$target_uid"
      find / -gid "$src_gid" 2> /dev/null | grep --invert-match -E "^(/sys|/proc)" | xargs chgrp "$target_gid"
    EOF

    config.vm.provider "lxc" do |lxc|
      # Configure our LXC setup
      config.vm.box = "fgrehm/trusty64-lxc"
      lxc.customize("cgroup.memory.limit_in_bytes", "2048M")
    end
  end

  # Provision our box with a script
  config.vm.provision "shell", path: "bootstrap.sh"
end