hdp sandbox vagrant

vbox-to-vagrant.md

Create a local Vagrant base box from an existing VirtualBox VM

What

• Setting up a new development VM should be as easy as 2 commands. And it is:
  • vagrant init; vagrant up
• In this example we are converting the HDP Sandbox to be used in this way. But the howto will work with any existing VM.

Why

• Why Vagrant
• Easier, faster and all the other reasons to use Vagrant instead of VMs directly
• Exposes the directory you are in as /vagrant on the VM. Making working with project files much much easier.
• A different VM for various projects instead of always hacking on the same VM and wiping it out
• Hopefully encourages us all to make tasks more repeatable

Creating the Vagrant base "box"

A Vagrant "box" is simply the base VM image which Vagrant uses each time you make a new Vagrant environment.

In this case we will be converting the HDP Sandbox VM into a Vagrant box.

Requirements

• Download the HDP Sandbox for VirtualBox file (will be named something like Sandbox_HDP_2.2_VirtualBox.ova)

• Ensure Vagrant is up-to-date:

  • OSX users with Homebrew can install like so: brew cask install vagrant
  • Or manually: http://www.vagrantup.com/downloads

Build the Vagrant "box"

• Open VirtualBox. Create a new VM from the .ova file you downloaded earlier (or an existing VM)

  • There is no need to boot it, unless you want to customize before creating your base machine (such as enabling Ambari).

• Create a new empty directory and the Vagrantfile which will be used for generating the base box.

  • Note: Every VM you create will have a Vagrantfile to define what it does. More on this later.
  • Example:

  mkdir vagrant-sandbox-builder
  cd vagrant-sandbox-builder
  vagrant init
  

• Update the Vagrantfile with a few things which are needed when building our "box", such as the fact that the Sandbox uses 'root' login:

  config.ssh.username = 'root'
  config.ssh.password = 'hadoop'
  config.ssh.insert_key = 'true'
  config.vm.box_check_update = false
  config.vm.network "private_network", type: "dhcp", :adapter=>2
  config.vm.provider "virtualbox" do |vb|
    vb.memory = "8192"
  end
  

• Create the Vagrant box: vagrant package --base HDP-Sandbox-2.2 --vagrantfile Vagrantfile

  • It will take a long time as the HDP-2.2 Sandbox is >4GB!

• Add the box to be available to Vagrant to use: vagrant box add --name HDP-Sandbox-2.2 package.box

Using the Box

• Note: The above steps are one-time. And the directory above is no longer needed, though you may want to keep the Vagrantfile to use the next time a new Sandbox comes out.

As an example, let's say you want to setup a new VM for a demo.

• You have many local files already in ~/src/myproject. And you want to keep using your local editor/IDE. You also don't want to lose that code when the VM goes away.

Simply create and boot the Vagrant VM

• (optional if it's a new project): mkdir -p ~/myproject
• cd ~/myproject
• generate the Vagrant environment: vagrant init HDP-Sandbox-2.2
• start the VM: vagrant up
• wait until it boots (might take a minute or 2 since the Sandbox is big. But this is only on the 1st time.)

Then you can:

• SSH in: vagrant ssh <-- it will know the password and all so nothing else is needed.
• It will be on a local IP so you don't need to do port forwarding.
  • Vagrant will list the IP or you can check with ifconfig: echo Access this box at: $(ifconfig eth1 | awk '/inet addr/ { gsub(/addr:/, ""); print $2}')

Putting that all together it looks like this for any new VM:

mkdir yourProjectsDir; cd yourProjectsDir
vagrant init HDP-Sandbox-2.2
vagrant up
vagrant ssh
echo Access this box at: $(ifconfig eth1 | awk '/inet addr/ { gsub(/addr:/, ""); print $2}')

Extra configuration

With the above configuration the VirtualBox VM will have:

• 8GB of RAM
• A host-only adapter so you can access the instance without port forwarding.

You may want to change the RAM and/or add a network adapter on the public network (VirtualBox calls these "bridged). Those and many other settings can be added easily.

After creating your new directory and Vagrantfile, simply update the Vagrantfile like so:

config.vm.network "public_network", :adapter=>3

config.vm.provider "virtualbox" do |vb|
  vb.memory = "4096"
end