Deploy AWS Storage Gateway on VirtualBox

README.md

Update Ubuntu

sudo apt-get update && sudo apt-get dist-upgrade && sudo apt-get autoremove

Install Required Linux Headers

sudo apt-get -y install gcc make linux-headers-$(uname -r) dkms ufw apt-transport-https ca-certificates curl software-properties-common

Add VirtualBox Repository and key

wget -q https://www.virtualbox.org/download/oracle_vbox_2016.asc -O- | sudo apt-key add -
wget -q https://www.virtualbox.org/download/oracle_vbox.asc -O- | sudo apt-key add -

Add VirtualBox repository

sudo sh -c 'echo "deb http://download.virtualbox.org/virtualbox/debian $(lsb_release -sc) contrib" >> /etc/apt/sources.list'

Install VirtualBox

sudo apt-get update && apt-get -y install virtualbox-5.2

Install VirtualBox Extension Pack

curl -O http://download.virtualbox.org/virtualbox/5.2.4/Oracle_VM_VirtualBox_Extension_Pack-5.2.4-119785.vbox-extpack
sudo VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-5.2.4-119785.vbox-extpack

Install Docker and Docker Compose

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" sudo apt-get update && sudo apt-get -y install docker-ce sudo usermod -aG docker ${USER} sudo curl -L https://github.com/docker/compose/releases/download/1.17.0/docker-compose-`uname -s-uname -m` -o /usr/local/bin/docker-compose sudo chmod +x /usr/local/bin/docker-compose

Configure vm autostart

Place Autostart configuration files

sudo mv virtualbox /etc/default/virtualbox
sudo mv vbox.cfg /etc/vbox/vbox.cfg

Lock down configurations and allow current user access

Log out and back in again to refresh the permissions after this step. This will also allow you to use docker

sudo chgrp vboxusers /etc/vbox
sudo chmod 1775 /etc/vbox
sudo usermod -a -G vboxusers ${USER}

VBoxManage setproperty autostartdbpath /etc/vbox
sudo service vboxautostart-service restart

Download Storage Gateway ESXi Image

mkdir ~/images
wget https://dslbbkfzjw91h.cloudfront.net/AWS-Appliance.zip && unzip AWS-Appliance.zip
mv AWS-Appliance-2019-07-24-1563990364.ova ~/images/AWS-Appliance.ova

## .ova file might have a different name, use the correct name
vboxmanage import ~/images/AWS-Appliance.ova --vsys 0 --vmname storage_gateway

Condigure vm to use NAT Networking

vboxmanage modifyvm storage_gateway --nic1 nat
vboxmanage modifyvm storage_gateway --natpf1 "nfs,tcp,,2049,,2049"
vboxmanage modifyvm storage_gateway --natpf2 "nfsudp,udp,,2049,,2049"
vboxmanage modifyvm storage_gateway --natpf3 "ntp,tcp,,8123,,123"

Ports below may not be required
vboxmanage modifyvm storage_gateway --natpf4 "dnstcp,tcp,,8053,,53"
vboxmanage modifyvm storage_gateway --natpf5 "dnsudp,udp,,8053,,53"
vboxmanage modifyvm storage_gateway --natpf6 "http,tcp,,8080,,80"
vboxmanage modifyvm storage_gateway --natpf7 "https,tcp,,8443,,443"
vboxmanage modifyvm storage_gateway --natpf1 "ssh_support,tcp,,2222,,22"
vboxmanage modifyvm storage_gateway --natpf1 "nfs2,tcp,,8111,,111"
vboxmanage modifyvm storage_gateway --natpf1 "nfs2udp,udp,,8111,,111"

Forward required ports for AWS Storage Gateway.

Decide on a static IP to assign the appliance. 192.168.15.2 will do. Also, under a non root user, VirtualBox only forwards ports > 1024 (no privileged ports). Make sure all your ports are acceptable. Because we will need to connect to http port 80 on the host to activate the appliance, we will use a docker ha-proxy container to forward 80 to 8080.

VBoxManage natnetwork modify --netname default --port-forward-4 "http:tcp:[]:8080:[192.168.15.2]:80"
VBoxManage natnetwork modify --netname default --port-forward-4 "https:tcp:[]:8443:[192.168.15.2]:443"
VBoxManage natnetwork modify --netname default --port-forward-4 "dnstcp:tcp:[]:8053:[192.168.15.2]:53"
VBoxManage natnetwork modify --netname default --port-forward-4 "dnsudp:udp:[]:8053:[192.168.15.2]:53"
VBoxManage natnetwork modify --netname default --port-forward-4 "ssh_support:tcp:[]:2222:[192.168.15.2]:22"
VBoxManage natnetwork modify --netname default --port-forward-4 "ntp:udp:[]:8123:[192.168.15.2]:123"
VBoxManage natnetwork modify --netname default --port-forward-4 "nfs:tcp:[]:2049:[192.168.15.2]:2049"
VBoxManage natnetwork modify --netname default --port-forward-4 "nfsudp:udp:[]:2049:[192.168.15.2]:2049"
VBoxManage natnetwork modify --netname default --port-forward-4 "nfs2:tcp:[]:8111:[192.168.15.2]:111"
VBoxManage natnetwork modify --netname default --port-forward-4 "nfs2udp:udp:[]:8111:[192.168.15.2]:111"

Configure and enable UFW. Make sure to allow port 22 for ssh

sudo ufw allow 22/tcp && sudo ufw allow 2222/tcp
sudo ufw allow 80/tcp && sudo ufw allow 8080/tcp
sudo ufw allow 443/tcp && sudo ufw allow 8443/tcp
sudo ufw allow 53 && sudo ufw allow 8053
sudo ufw allow 123/ucp && sudo ufw allow 8123/udp
sudo ufw allow 2049 && sudo ufw allow 2049
sudo ufw allow 111
sudo ufw default deny
sudo ufw enable

Configure Docker to forward ports

docker-compose up -d

Modify the vm to use the new nat network and start at boot, and also create and attach a 250 GB disk for caching

vboxmanage modifyvm storage_gateway --nic1 natnetwork --nat-network1 default --autostart-enabled on
vboxmanage createhd --filename ~/disks/cache.vdi --size 262144
mkdir -p ~/disks
vboxmanage storageattach storage_gateway --storagectl "SATA Controller" --port 1 --device 0 --type hdd --medium ~/disks/cache.vdi

Optionally, check your VM info before starting it

vboxmanage showvminfo storage_gateway

Start the Appliance VM.

vboxmanage startvm storage_gateway --type headless

RDP server will be available on port 5961, which you can connect to using any RCP client. Likely, you've setup the VM on a linux server, so it has no GUI, and we've not allowed port 5961 on UFW. You will need to forward the port to your workstation via ssh tunnel, like so:

ssh -i ssh_key_file -L 5961:127.0.0.1:5961 -fN remoteuser@remotehost

You can now connect to the AWS appliance via an RDP client like Microsoft's Remote Desktop 10 The client will ask for credentials, but the RDP connection is not secured, so type anything and connect When you see the Appliance's login screen, login with user: admin / password: password combination

• Follow the AWS guide to setup your gateway: https://docs.amazonaws.cn/en_us/storagegateway/latest/userguide/manage-on-premises.html
• After that is done, follow the getting started guide here: https://docs.aws.amazon.com/storagegateway/latest/userguide/GettingStarted.html

VM Controls

vboxmanage controlvm storage_gateway pause|resume|reset|poweroff|savestate

Mount NFS share

After creating a share on the AWS Console, you can mount it with:

# Install NFS helper utils
sudo apt-get -y install nfs-common

# Mount Share
sudo mount -t nfs -o nolock,hard gateway_ip:/bucket/prefix MOUNTPATH

docker-compose.yaml

version: "2.1"
services:
  tunnel:
    image: "dweomer/stunnel"
    restart: always
    container_name: stunnel
    network_mode: host
    volumes:
      - ./certs/stunnel.pem:/etc/stunnel/stunnel.pem:ro
      - ./certs/stunnel.key:/etc/stunnel/stunnel.key:ro
      - ./stunnel.conf:/srv/stunnel/stunnel.conf.template:ro
    environment:
      STUNNEL_SERVICE: nfs
      STUNNEL_ACCEPT: 2050
      STUNNEL_CONNECT: 127.0.0.1:2049
      STUNNEL2_ACCEPT: 111
      STUNNEL2_CONNECT: 127.0.0.1:8111
    logging:
      driver: "json-file"
      options:
        max-size: "10M"
        max-file: "3"

vbox.cfg

# Allow users to autostart VMs
default_policy = allow

virtualbox

# virtualbox defaults file
VBOXAUTOSTART_DB=/etc/vbox
VBOXAUTOSTART_CONFIG=/etc/vbox/vbox.cfg