This is a bash shell script for bootstrapping the Hyperledger Fabric development environment on an AWS Ubuntu AMI instance. It has been tested with Ubuntu v14.04 (AMI ID ami-2d39803a). While I wrote this to enable AWS deployments, this script should be usable in any Ubuntu based environment. See the first comment below for the detailed background.

setup-aws.sh

#!/bin/bash

# Install "git"
sudo apt-get update
sudo apt-get -y install git

# Clone the Hyperledger Fabric base image repository.  It would be preferable
# to use the "/tmp" directory as the destination, since this repository is no
# longer needed once the base image has been built.  However, this is
# problematic because one of the scripts that builds the base image removes
# everything from the "/tmp" directory, as opposed to just the artifacts that
# it created.
git clone http://gerrit.hyperledger.org/r/fabric-baseimage

# Execute the first part of the setup that builds the base image.  If in the
# past you have only setup a Fabric development environment using an existing
# Vagrant box (e.g. "hyperledger\fabric-baseimage"), then you have not seen
# this part run before since its final state is what is captured in the box.
# Normally, this part is executed via Packer to build a Vagrant box.
cd fabric-baseimage
BASEIMAGE_RELEASE=`cat ./release` sudo bash -c ./scripts/common/setup.sh
sudo bash -c ./scripts/devenv/setup.sh

# Setup the directory structure that will contain the Fabric main repository.
# When using Vagrant, this repository resides on the host and is made
# available to the guest via a VirtualBox shared folder.
sudo bash -c 'umask 00002 && mkdir -p /opt/gopath/src/github.com/hyperledger'
sudo chown -R ubuntu:ubuntu /opt/gopath

# Clone the Fabric repository
cd /opt/gopath/src/github.com/hyperledger
git clone http://gerrit.hyperledger.org/r/fabric

# Create a symbolic link to the repository.  Again, when using Vagrant, the
# repository on the host is exposed to the guest via another VirtualBox shared
# folder.
sudo ln -s /opt/gopath/src/github.com/hyperledger/fabric /hyperledger

# Execute the second part of the setup.  This part produces the majority of
# the console output when you use the 'vagrant up' command to start the
# Vagrant based development environment for the first time.  In this case, you
# won't see the output from the preliminary steps that Vagrant takes to
# provision a VirtualBox instance.
#
# Before executing the script, it is necessary to apply a few changes to it.  
# Specifically, references to the "vagrant" user and group names must be
# replaced with the equivalent "ubuntu" user and group names that are baked
# into the standard AWS Ubuntu AMI.
cd /hyperledger
sed -i 's#^usermod -a -G docker vagrant#usermod -a -G docker ubuntu#' ./devenv/setup.sh
sed -i 's#^sudo chown -R vagrant:vagrant#sudo chown -R ubuntu:ubuntu#' ./devenv/setup.sh
sed -i 's#>> /home/vagrant/.bashrc#>> /home/ubuntu/.bashrc#' ./devenv/setup.sh
sudo -H bash -c ./devenv/setup.sh

What is this?

This is a bash shell script for bootstrapping the Hyperledger Fabric development environment on an AWS Ubuntu AMI instance. It has been tested with Ubuntu v14.04 (AMI ID ami-2d39803a). This is the same Ubuntu release from which official Vagrant boxes containing the Fabric development environment are presently built.

While I wrote this script to enable AWS deployments, this script should be usable in any Ubuntu based environment with two caveats. First, if your environment has the absolute minimum of Ubuntu packages installed, you may need to resolve some additional package dependencies. If you do so, please document them in the comments here. Second, this script assumes the user name with which you authenticate yo your environment is "ubuntu", as opposed to what was originally "vagrant". If you are using some other user name, you'll need to adjust lines 27 and 28 of this script accordingly.

How do I use it?

Here are the basic steps that you'll need to follow:

1. You'll need to login to the AWS console and instantiate an appropriately sized EC2 instance. An official Vagrant box is presently sized as having two CPU cores, 4GB of memory, and 40GB of storage. The least expensive equivalent EC2 instance type is a t2.medium instance. (I am using one of these with 40GB of storage attached. If the storage ends up not being sufficiently utilized, I plan to downsize it later.)
2. You'll need to login to your EC2 instance and copy the setup-aws.sh script to it. You can do this on the instance using the curl or wget commands to download it directly from Github. If you do this, click the Raw link shown on this page and then copy the URL shown in your web browser's address bar. Alternatively, you can save the bash script to your local machine and then use SSH to copy it to the instance (i.e. using the scp command).
3. From the command line of your EC2 instance, just type source ./setup-aws.sh to start the bootstrap process.

Why does it exist?

I've successfully had the Vagrant based Fabric development environment running on my laptop. While it worked acceptably, my primary problem is that none of my laptops have or are upgradeable to more than 8GB of memory. Since the development environment defaults to using 4GB of memory, it made my laptop sluggish. Given the resource requirements for the development environment and my constraints, moving the development environment to an AWS EC2 instance seemed logical. Of course, hosting my development environment with AWS is also convenient for running experiments where I want to establish connectivity with other nodes, whether hosted with AWS or not.

Outstanding Questions and Issues

• This script is relatively straightforward and delegates all of the heavy lifting to existing scripts in the Fabric repositories. Hence, the most significant issue is that this AWS oriented solution now lives outside of these repositories. It would be nice if the deployment related artifacts in the repository were refactored to support multiple deployment scenarios, including one for AWS.
• One of the last steps in the final setup script creates the file /etc/profile.d/vagrant-devenv.sh. There are two problems with this. First, the file isn't named appropriately for the AWS deployment scenario, but the file name appears to be inconsequential. Second, there is a statement in the file that sets an environment variable as VAGRANT=1. This environment variables does not appear to be referenced in the repository though. Hence, I have left both as-is for now.

Feedback Wanted

I don't frequently write shell scripts, so I'd be happy to receive your recommendations regarding approach and style.

As soon as I completed the first reboot after getting Fabric running using AWS, I could no longer start the Docker daemon. In investigating, I realized that there is a critical difference between what this script does and how Fabric gets deployed via Vagrant. The difference is that there is an implicit reboot that happens between the execution of the first setup script and the second setup script. This is a result of the first setup script being executed via the Packer build and the second setup script being executed via the vagrant up command, with an image being taken and reconstitued between the two. This reboot is critical because the first setup script updates the Linux kernel, and the second setup script checks the Linux kernel version in-use to install the proper Docker storage driver version. If this reboot hasn't happened before the second setup script is executed, the wrong Docker storage driver for use with the final Linux kernel gets installed.

My preferred solution for working around this problem, before executing this script, is to manually update the Linux kernel and reboot. Here are the commands that will do this:

sudo apt-get update
sudo apt-get -y dist-upgrade
sudo reboot

UPDATE 2016, SEP 23: Revision 6 includes the necessary changes to again make this script work following the base image creation artifacts being split off from the fabric repository into to the new fabric-baseimage repository. The following is the commit record from the fabric repository showing when this change was merged into the master branch:

commit 3cfaeb582e962e1c9a7c147ac0f87bde8c6e0933
Merge: 3e74925 ac8ab84
Author: Jonathan Levi <jonathan@levi.name>
Date:   Thu Sep 22 20:46:44 2016 +0000

    Merge "Switch to new baseimage-pipeline"