firecracker-boot-time.md

Measuring Firecracker boot time

Firecracker comes with an internal way of logging a timestamp that measures time elapsed between the very start of the guest VM and the moment a specific IO port has been written to.

That allows for marking specific moment along the boot process by having code writing to this port.

Artifacts build

Here we're going to measure the time it takes for a Firecracker guest VM to reach userspace. To do so we're going to build 3 components:

1. Firecracker
2. A guest kernel
3. A rootfs image.

We assume you created a dedicated build directory and export it to FC_BUILD. For example

export FC_BUILD=~/fc-build
mkdir -p $FC_BUILD

Firecracker

pushd $FC_BUILD
git clone https://github.com/firecracker-microvm/firecracker.git
cd firecracker
./tools/devtool/build
popd $FC_BUILD

The firecracker binary is going to be at $FC_BUILD/firecracker/build/debug/firecracker

Guest Kernel

pushd $FC_BUILD
git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git linux-stable
cd linux-stable
git reset --hard v4.14.72
curl https://raw.githubusercontent.com/firecracker-microvm/firecracker/master/resources/microvm-kernel-config -o .config
make oldconfig
make vmlinux -j `nproc`
popd $FC_BUILD

The Firecracker guest kernel is going to be at $FC_BUILD/linux-stable/vmlinux

Rootfs

See Firecracker rootfs preparation instructions

1. First we create an empty ext4 rootfs image:

export ROOTFS_DIR=$FC_BUILD/rootfs
mkdir -p $ROOTFS_DIR
dd if=/dev/zero of=$ROOTFS_DIR/fc-rootfs.ext4 bs=1M count=50
mkfs.ext4 $ROOTFS_DIR/fc-rootfs.ext4
mkdir /tmp/fc-rootfs
sudo mount $ROOTFS_DIR/fc-rootfs.ext4 /tmp/fc-rootfs

2. We can now populate it with an Alpine Linux image:

sudo docker run -it --rm -v /tmp/fc-rootfs:/fc-rootfs alpine

# You are now inside the container.
# First we will add an init system and some utilities
apk add openrc
apk add util-linux

# Set up a login terminal on the serial console (ttyS0):
ln -s agetty /etc/init.d/agetty.ttyS0
echo ttyS0 > /etc/securetty
rc-update add agetty.ttyS0 default

# Make sure special file systems are mounted on boot:
rc-update add devfs boot
rc-update add procfs boot

# Then, copy the newly configured system to the rootfs image:
for d in bin etc lib root sbin usr; do tar c "/$d" | tar x -C /fc-rootfs; done
for dir in dev proc run sys var; do mkdir /fc-rootfs/${dir}; done

# All done, exit docker shell
exit

3. Now we want to add a small utility that poke the Firecracker debug io port as soon as we reach userspace.

We first need to build a small piece of code to do so:

pushd $FC_BUILD
git clone https://github.com/sameo/firecracker-magic-port.git
cd firecracker-magic-port/
make
popd

Then we copy this binary into our rootfs:

sudo cp fc-magic-port /tmp/fc-rootfs/sbin/

4. The final step is to create a custom init command to have it first calling into fc-magic-port and then starting the regular init:

sudo bash
cat <<EOM >/tmp/fc-rootfs/init
#!/bin/sh
/sbin/fc-magic-port
exec /sbin/init
EOM
chmod a+x /tmp/fc-rootfs/init
exit

5. We can now unmount the rootfs

sudo umount /tmp/fc-rootfs

The Firecracker guest rootfs is now at $FC_BUILD/rootfs/fc-rootfs.ext4

Artifacts summary

Artifact	Location
Firecracker VMM	$FC_BUILD/firecracker/build/debug/firecracker
Guest kernel	$FC_BUILD/linux-stable/vmlinux
Guest rootfs	$FC_BUILD/rootfs/fc-rootfs.ext4

Running and measuring

1. Start a Firecracker instance:

rm -rf /tmp/firecracker.socket && $FC_BUILD/firecracker/build/debug/firecracker --seccomp-level 0

2. On another terminal, start a Firecracker guest:

$FC_BUILD/firecracker-magic-port/start_instance-debug.sh /tmp/firecracker.socket

Then the guest boot time timestamp can be found in the Firecracker log pipe:

grep Guest-boot-time /tmp/logs.fifo
2019-05-30T12:15:18.698648314 [anonymous-instance] Guest-boot-time = 279446 us 279 ms, 272485 CPU us 272 CPU ms

Thanks a lot for this write-up. I thought I update this information to keep it useful for others.

Since Firecracker v0.23 the boot timer device has slightly changed (see devices: btimer: Add boot timer device) and this change breaks your fc-magic-port utility.

This can however be easily fixed by changing the utility to something like:

// Based on https://github.com/firecracker-microvm/firecracker/blob/main/resources/tests/init.c

#include <sys/types.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>

// Base address values are defined in arch/src/lib.rs as arch::MMIO_MEM_START.
// Values are computed in arch/src/<arch>/mod.rs from the architecture layouts.
// Position on the bus is defined by MMIO_LEN increments, where MMIO_LEN is
// defined as 0x1000 in vmm/src/device_manager/mmio.rs.
#ifdef __x86_64__
#define MAGIC_MMIO_SIGNAL_GUEST_BOOT_COMPLETE 0xd0000000
#endif
#ifdef __aarch64__
#define MAGIC_MMIO_SIGNAL_GUEST_BOOT_COMPLETE 0x40000000
#endif

#define MAGIC_VALUE_SIGNAL_GUEST_BOOT_COMPLETE 123

int main () {
   int fd = open("/dev/mem", (O_RDWR | O_SYNC | O_CLOEXEC));
   int mapped_size = getpagesize();

   char *map_base = mmap(NULL,
        mapped_size,
        PROT_WRITE,
        MAP_SHARED,
        fd,
        MAGIC_MMIO_SIGNAL_GUEST_BOOT_COMPLETE);

   *map_base = MAGIC_VALUE_SIGNAL_GUEST_BOOT_COMPLETE;
   msync(map_base, mapped_size, MS_ASYNC);
}

Notice that you'll also have to start Firecracker with the --boot-timer command line parameter in order to activate the boot timer device.