Create nested dev environment on Azure

nested-dev-env-azure.md

Create a nested development envrionment on Azure VM. I use this for kernel specific sandboxes/dev. Admittedly, This is following my own quirky dev env. So apprach this with care.

What will you have:

1. Separate virtual network for your sandbox
2. QEMU (kvm enabled) vms running using this network
3. Additional data disk aufs mounted for home directory (to move that disk to new vms if needed)

Steps

0. Create VM that supports nested virtualization (_v3 typically does) Create VM (docs)[https://docs.microsoft.com/en-us/azure/virtual-machines/linux/quick-create-cli]

1. Setup network

Create this file

<network>
  <name>kernel-net</name>
  <uuid>5618453a-d6af-42f5-b77d-eb1727371b73</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='kerneldevbr0' stp='on' delay='0'/>
  <mac address='42:ef:c9:0b:a4:f1'/>
  <ip address='192.168.124.1' netmask='255.255.255.0'>
    <dhcp>
        <range start='192.168.124.2' end='192.168.124.254'/>
    </dhcp>
  </ip>
</network>

Run

virsh net-define <<PATH TO XML>>
virsh net-start kernel-net
# optional set auto start
# virsh net-autostart kernel-net 

I create VMs with large data disks, everything is expected to run off that datadisk

2. Prep the VM (I use debian based servers (ubuntu), modify for your favorite destro)

sudo apt install -y cloud-image-utils qemu-kvm libvirt-bin virtinst bridge-utils cpu-checker

3. Sanity check: Does your VM size supports nested virtualization

kvm-ok

for detailed walkthrough check vivik's docs (here)[https://www.cyberciti.biz/faq/installing-kvm-on-ubuntu-16-04-lts-server/]

4. Get the images (By now you created a directory and moved into it)

#don't make your life harder by going for UEFI images. it is not needed here
wget https://cloud-images.ubuntu.com/releases/16.04/release/ubuntu-16.04-server-cloudimg-amd64-disk1.img

5. Create the home directory disk

qemu-img create -f qcow2 ./data-disk.img 5G # Home directory disk -- you can resize later

6. Prepare could-init file to bootstrap the VM

File content (change things as needed)

#cloud-config
hostname: kerneldev
manage_resolv_conf: true
resolv_conf:
  nameservers: ['8.8.8.8']
users:
  - name: khenidak
    ssh-authorized-keys:
      - <<PUBLIC KEY HERE>>>
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    groups: sudo
    shell: /bin/bash
package_upgrade: true
packages:
  - apt-transport-https
  - ca-certificates
  - software-properties-common
  - aufs-tools
  - cgroupfs-mount
  - libltdl7
  - pigz
  - build-essential
  - curl
write_files:
  - content: |
        #!/bin/bash
        sudo blkid /dev/vdb
        if [[ "0" -ne "$?" ]];
        then
          sudo mkfs.ext4 /dev/vdb;
        else
          echo "**** /dev/vdb is formatted*****";
        fi
        sudo mkdir -p /mnt/data
        sudo chown -R khenidak:khenidak /mnt/data

        #ugly hack for aufs -- TODO: find something else
        mkdir -p /tmp/homes/
        sudo chown -R khenidak:khenidak /tmp/homes/
        sudo cp -rp /home/khenidak /tmp/homes/
    path: /opt/data-disk-prep.sh
    permissions: 0755
  - content: |
        [Unit]
        Description=format data disks
        After=network.target

        [Service]
        Type=oneshot
        ExecStart=/opt/data-disk-prep.sh

        [Install]
        WantedBy=multi-user.target
    path: /etc/systemd/system/format-disks.service
    permissions: 0755
  - content: |
        [Unit]
        Description=Automount datadisk
        After=format-disks.service

        [Mount]
        What=/dev/vdb
        Where=/mnt/data

        [Install]
        WantedBy=multi-user.target
    path: /etc/systemd/system/mnt-data.mount
    permissions: 0755
  - content: |
        [Unit]
        Description=Mount home
        After=mnt-data.mount

        [Mount]
        What=none
        Where=/home/khenidak
        Type=aufs
        Options=dirs=/mnt/data/=rw:/tmp/homes/khenidak/=ro

        [Install]
        WantedBy=multi-user.target
    path: /etc/systemd/system/home-khenidak.mount
    permissions: 0755
  - content: |
        [Unit]
        Description=change owner of home (fix for aufs in ability to set mount point owner)
        After=home-khenidak.mount

        [Service]
        Type=oneshot
        ExecStart=/bin/bash -c "sudo chown khenidak:khenidak /home/khenidak"

        [Install]
        WantedBy=multi-user.target
    path: /etc/systemd/system/fix-home-owner.service
    permissions: 0755

runcmd:
  - 'sudo echo 127.0.0.1 kerneldev >> /etc/hosts'
  - 'sudo systemctl daemon-reload'
  - 'sudo systemctl enable format-disks.service && sudo systemctl start format-disks.service'
  - 'sudo systemctl enable mnt-data.mount && sudo systemctl start mnt-data.mount'
  - 'sudo systemctl enable home-khenidak.mount && sudo systemctl start home-khenidak.mount'
  - 'sudo systemctl enable fix-home-owner.service && sudo systemctl start fix-home-owner.service'

Create cloud init img

cloud-localds cloud-init.img cloud-init.seed

7. Start the image

sudo qemu-system-x86_64 \
                  -smp 4 \
                   -m 4096 \
                   -enable-kvm \
                   -nographic \
                   --cdrom ./cloud-init.img \
                   --drive file=ubuntu-16.04-server-cloudimg-amd64-disk1.img,if=virtio,cache=writethrough,index=0 \
                   --drive file=data-disk.img,format=qcow2,if=virtio,cache=writethrough,index=1 \
                   --netdev bridge,br=kerneldevbr0,id=hn0 --device virtio-net-pci,netdev=hn0,id=nic

Quick FAQ

Why no libvirt? I like to see my VM console just in case i completely broke the VM's kernel.

But you are using libvirt network,no? Yes, because they have a better networking stack (than qemu) specifically dhcp, also enables easy assignment of static ips/multi vm in same net if needed.

What if i want to rebootstrap my VM? Get a new ubuntu cloud image .img file and re start it, it will go into bootstrap mood. If you are feeling brave try the merge disks feature (qemu-img create -b ..). Then just recreate the merge disk.

But this VM is ssh only,no? Yes, I use vim as dev env (my setup is not included here). You can add desktop things to cloud-init packages and x/rdp into your VM.

What is up with aufs? Yes it is a bit too hacky, but it works reliably. The idea is to keep home directory on a seprate disks to enable you to reset/delete/destroy/run another VM with the same home directory.

What is the ip of my VM? it will be in 192.168.124.1/24 address space. Easiest method is to run arp and get the ip (find the ip attached to kerneldevbr0). You can assign static MAC and edit the network to get a static ip.

PRs are welcomed