Using Local Virtual Machine Migration to perform fast VM launch using QEMU

qemu_migrate.md

Overview

QEMU supports migration of a virtual machine across machine over the network. It also has the capability to save the state of a virtual machine and resume from that state.

It is described in detail in https://www.linux-kvm.org/page/Migration

This capability can be leveraged to create an instant resume virtual machine image if the virtual machine is crafted with care.

Creating a Virtual Machine that can be fast resumed

Requirements

• The virtual machine should run purely from memory
• The virtual machine should save any persistent state to ram based storage
• The virtual machine should not rely on any host network devices (like tap, blk, 9p)
  • The state (including networking state) should be completely contained in the QEMU process itself (i.e. SLIRP)

How to create the Virtual machine

• Use kernel + initramfs + qemu + slirp networking to create the base image

Building the Kernel

• Use Clear Containers OS Builder https://github.com/clearcontainers/osbuilder#build-guest-kernel to build the kernel
• The default clear containers kernel configuration has to be updated to support initrd CONFIG_BLK_DEV_INITRD=y
• This will provide you with the bzImage needed

Building the initramfs

• (copied from https://gist.github.com/pacalet/660fa6472e0bcd53f71fc31f167c628f)
• Use a busybox based rootfs for ease of use

git clone https://github.com/mirror/busybox
cd busybox
make defconfig
make menuconfig (enable Busybox Settings ---> Build Options ---> Build BusyBox as a static binary (no shared libs) ---> yes)
make
make install

• Create the initramfs

mkdir $HOME/rootfs
cd $HOME/rootfs
mkdir -p bin sbin etc proc sys usr/bin usr/sbin
cp -a $BUSYBOX_BUILD/_install/* .
rm linuxrc

• Create an init binary with appropriate contents with executeable perms

#!/bin/sh
mount -t proc none /proc
mount -t sysfs none /sys
echo -e "Hello World\n"
exec /bin/sh

• Create the initramfs

cd $HOME/rootfs
find . -print0 | cpio --null -ov --format=newc | gzip -9 > $HOME/initramfs.cpio.gz

Booting the base image

Launch the VM with monitor enabled

 sudo ~/qemu/x86_64-softmmu/qemu-system-x86_64 \
         -machine pc,accel=kvm,kernel_irqchip \
         -cpu host \
         -kernel ./bzImage \
         -initrd ./initramfs.cpio.gz \
         -nographic \
         -nodefaults \
         -no-user-config \
         -m 256,maxmem=1G,slots=2 \
         -smp 2 -rtc base=utc,driftfix=slew \
         -global kvm-pit.lost_tick_policy=discard \
         -append 'console=hvc0 single iommu=false root=/dev/ram0 dhcp' \
         -netdev user,id=mynet0 \
         -device virtio-net-pci,netdev=mynet0 \
         -device virtio-serial-pci,id=virtio-serial0 \
         -chardev stdio,id=charconsole0 \
         -device virtconsole,chardev=charconsole0,id=console0     \
         -monitor telnet:127.0.0.1:1234,server,nowait 

Capturing its state

telnet localhost 127.0.0.1 1234
stop
migrate_set_speed 4095m                                            
migrate "exec:gzip -c > STATEFILE.gz"   
cont

Launching a new VM based on this VM's saved state

 sudo ~/qemu/x86_64-softmmu/qemu-system-x86_64 \
         -machine pc,accel=kvm,kernel_irqchip \
         -cpu host \
         -kernel ./bzImage \
         -initrd ./initramfs.cpio.gz \
         -nographic \
         -nodefaults \
         -no-user-config \
         -m 256,maxmem=1G,slots=2 \
         -smp 2 -rtc base=utc,driftfix=slew \
         -global kvm-pit.lost_tick_policy=discard \
         -append 'console=hvc0 single iommu=false root=/dev/ram0 dhcp' \
         -netdev user,id=mynet0 \
         -device virtio-net-pci,netdev=mynet0 \
         -device virtio-serial-pci,id=virtio-serial0 \
         -chardev stdio,id=charconsole0 \
         -device virtconsole,chardev=charconsole0,id=console0     \
         -incoming "exec: gzip -c -d STATEFILE.gz" \
         -monitor telnet:127.0.0.1:1235,server,nowait \

Resume it

As the VM was in a stopped state you need to resume it. This VM will resume instantly in exactly the same state as the previous one. As this is a pure RAM based VM there is no additional setup needed on host or guest side. The external network connectivity is also instantly restored.

Also you can launch as many VM's as you need from this single image

telnet localhost 127.0.0.1 1235
cont

Note

Using cpu=host means that image needs to be created on the same type of machine as the target