Skip to content

Instantly share code, notes, and snippets.

What would you like to do?

Measuring Firecracker boot time

Firecracker comes with an internal way of logging a timestamp that measures time elapsed between the very start of the guest VM and the moment a specific IO port has been written to.

That allows for marking specific moment along the boot process by having code writing to this port.

Artifacts build

Here we're going to measure the time it takes for a Firecracker guest VM to reach userspace. To do so we're going to build 3 components:

  1. Firecracker
  2. A guest kernel
  3. A rootfs image.

We assume you created a dedicated build directory and export it to FC_BUILD. For example

export FC_BUILD=~/fc-build
mkdir -p $FC_BUILD


pushd $FC_BUILD
git clone
cd firecracker
popd $FC_BUILD

The firecracker binary is going to be at $FC_BUILD/firecracker/build/debug/firecracker

Guest Kernel

pushd $FC_BUILD
git clone git:// linux-stable
cd linux-stable
git reset --hard v4.14.72
curl -o .config
make oldconfig
make vmlinux -j `nproc`
popd $FC_BUILD

The Firecracker guest kernel is going to be at $FC_BUILD/linux-stable/vmlinux


See Firecracker rootfs preparation instructions

  1. First we create an empty ext4 rootfs image:
export ROOTFS_DIR=$FC_BUILD/rootfs
mkdir -p $ROOTFS_DIR
dd if=/dev/zero of=$ROOTFS_DIR/fc-rootfs.ext4 bs=1M count=50
mkfs.ext4 $ROOTFS_DIR/fc-rootfs.ext4
mkdir /tmp/fc-rootfs
sudo mount $ROOTFS_DIR/fc-rootfs.ext4 /tmp/fc-rootfs
  1. We can now populate it with an Alpine Linux image:
sudo docker run -it --rm -v /tmp/fc-rootfs:/fc-rootfs alpine

# You are now inside the container.
# First we will add an init system and some utilities
apk add openrc
apk add util-linux

# Set up a login terminal on the serial console (ttyS0):
ln -s agetty /etc/init.d/agetty.ttyS0
echo ttyS0 > /etc/securetty
rc-update add agetty.ttyS0 default

# Make sure special file systems are mounted on boot:
rc-update add devfs boot
rc-update add procfs boot

# Then, copy the newly configured system to the rootfs image:
for d in bin etc lib root sbin usr; do tar c "/$d" | tar x -C /fc-rootfs; done
for dir in dev proc run sys var; do mkdir /fc-rootfs/${dir}; done

# All done, exit docker shell
  1. Now we want to add a small utility that poke the Firecracker debug io port as soon as we reach userspace.

We first need to build a small piece of code to do so:

pushd $FC_BUILD
git clone
cd firecracker-magic-port/

Then we copy this binary into our rootfs:

sudo cp fc-magic-port /tmp/fc-rootfs/sbin/
  1. The final step is to create a custom init command to have it first calling into fc-magic-port and then starting the regular init:
sudo bash
cat <<EOM >/tmp/fc-rootfs/init
exec /sbin/init
chmod a+x /tmp/fc-rootfs/init
  1. We can now unmount the rootfs
sudo umount /tmp/fc-rootfs

The Firecracker guest rootfs is now at $FC_BUILD/rootfs/fc-rootfs.ext4

Artifacts summary

Artifact Location
Firecracker VMM $FC_BUILD/firecracker/build/debug/firecracker
Guest kernel $FC_BUILD/linux-stable/vmlinux
Guest rootfs $FC_BUILD/rootfs/fc-rootfs.ext4

Running and measuring

  1. Start a Firecracker instance:
rm -rf /tmp/firecracker.socket && $FC_BUILD/firecracker/build/debug/firecracker --seccomp-level 0
  1. On another terminal, start a Firecracker guest:
$FC_BUILD/firecracker-magic-port/ /tmp/firecracker.socket

Then the guest boot time timestamp can be found in the Firecracker log pipe:

grep Guest-boot-time /tmp/logs.fifo
2019-05-30T12:15:18.698648314 [anonymous-instance] Guest-boot-time = 279446 us 279 ms, 272485 CPU us 272 CPU ms
Copy link

simonis commented Jul 12, 2022

Thanks a lot for this write-up. I thought I update this information to keep it useful for others.

Since Firecracker v0.23 the boot timer device has slightly changed (see devices: btimer: Add boot timer device) and this change breaks your fc-magic-port utility.

This can however be easily fixed by changing the utility to something like:

// Based on

#include <sys/types.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>

// Base address values are defined in arch/src/ as arch::MMIO_MEM_START.
// Values are computed in arch/src/<arch>/ from the architecture layouts.
// Position on the bus is defined by MMIO_LEN increments, where MMIO_LEN is
// defined as 0x1000 in vmm/src/device_manager/
#ifdef __x86_64__
#ifdef __aarch64__


int main () {
   int fd = open("/dev/mem", (O_RDWR | O_SYNC | O_CLOEXEC));
   int mapped_size = getpagesize();

   char *map_base = mmap(NULL,

   msync(map_base, mapped_size, MS_ASYNC);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment