Talos on Firecracker

README.md

Talos on Firecracker

1. Build Firecracker from source (it's fast and easy, builds in docker): https://github.com/firecracker-microvm/firecracker/blob/master/docs/getting-started.md#building-from-source

2. Adjust /dev/kvm permissions: https://github.com/firecracker-microvm/firecracker/blob/master/docs/getting-started.md#prerequisites

3. Build Talos, you will need build/vmlinux & build/initramfs.xz.

4. Prepare configs for Talos: osctl config generate fire https://172.16.0.2:6443. Make following changes to the config: add resolvers to the machine.network config: nameservers: ["8.8.8.8", "1.1.1.1"] (IP config will come from kernel args), update install location: install: disk: /dev/vda (default is sda).

5. Prepare network for firecracker VM as described in https://github.com/firecracker-microvm/firecracker/blob/master/docs/network-setup.md#on-the-host (tap0 + iptables rules).

6. Run HTTP server on 172.16.0.1 (host end of tap device) to serve Talos configs from step 4 on port 8000.

7. Create a file which will be root block device for Talos node: dd if=/dev/zero of=sda bs=1M count=10000 (10Gb is enough, 1Gb is too small, not sure if we can go smaller than 10Gb).

This finalizes the preparation steps, now time to launch the thing.

Each time VM shuts down, you have to start firecracker process. This process provides unix socket API to configure/launch VM, and process output is VM console output, so easy to follow boot process:

$ rm -f /tmp/firecracker.socket
$ firecracker --api-sock /tmp/firecracker.socket

Now in another session, run the following to configure & launch VM (better to save as a script, this is just version with comments).

First, initial config for the VM with kernel & initramfs (see step3):

kernel_path="/home/smira/talos/build/vmlinux"
initrd_path="/home/smira/talos/build/initramfs.xz"

curl --unix-socket /tmp/firecracker.socket -i \
  -X PUT 'http://localhost/boot-source'   \
  -H 'Accept: application/json'           \
  -H 'Content-Type: application/json'     \
  -d "{
      \"kernel_image_path\": \"${kernel_path}\",
      \"initrd_path\": \"${initrd_path}\",
      \"boot_args\": \"console=ttyS0 reboot=k panic=1 pci=off talos.platform=metal talos.config=http://172.16.0.1:8000/init.yaml page_poison=1 slab_nomerge pti=on slub_debug=P ip=172.16.0.2::172.16.0.1:255.255.255.0::eth0:off random.trust_cpu=on printk.devkmsg=on\" 
  }"

Kernel boot args are really important, please note talos.config shoudl match whatever webserver was set up on step 6, ip config shoudl match network.

Next, VM resources (default is 128MB memory, not enough for Talos):

curl --unix-socket /tmp/firecracker.socket -i  \
   -X PUT 'http://localhost/machine-config' \
   -H 'Accept: application/json'            \
   -H 'Content-Type: application/json'      \
   -d '{
      "vcpu_count": 2,
      "mem_size_mib": 1024,
      "ht_enabled": false
    }'

Create data disk (host_path is path to the file from step 7):

curl --unix-socket /tmp/firecracker.socket -i \
   -X PUT 'http://localhost/drives/sda' \
   -H 'Accept: application/json'           \
   -H 'Content-Type: application/json'     \
   -d "{
         \"drive_id\": \"sda\",
         \"is_root_device\": false,
         \"is_read_only\": false,
         \"path_on_host\": \"./sda\"
     }"

Configure networking (from step 5):

curl -X PUT \
   --unix-socket /tmp/firecracker.socket \
        http://localhost/network-interfaces/eth0 \
    -H  'Accept: application/json'          \
    -H  'Content-Type: application/json'    \
    -d '{
      "iface_id": "eth0",
      "guest_mac": "AA:FC:00:00:00:01",
      "host_dev_name": "tap0"
     }'

Finally, start the VM:

curl --unix-socket /tmp/firecracker.socket -i \
   -X PUT 'http://localhost/actions'       \
   -H  'Accept: application/json'          \
   -H  'Content-Type: application/json'    \
   -d '{
      "action_type": "InstanceStart"
   }'

VM should boot Talos, run installer and attempt reboot. Firecracker treats reboot as shutdown, so firecracker is going to exit.

Restart firecracker process and run the same script as above, Talos node will boot once again and finish bootstrap process.

You should have working Talos node now, VM is available on IP 172.16.0.2 from the host.

up.sh

kernel_path="/home/smira/talos/build/vmlinux"
initrd_path="/home/smira/talos/build/initramfs.xz"

curl --unix-socket /tmp/firecracker.socket -i \
  -X PUT 'http://localhost/boot-source'   \
  -H 'Accept: application/json'           \
  -H 'Content-Type: application/json'     \
  -d "{
        \"kernel_image_path\": \"${kernel_path}\",
        \"initrd_path\": \"${initrd_path}\",
        \"boot_args\": \"console=ttyS0 reboot=k panic=1 pci=off talos.platform=metal talos.config=http://172.16.0.1:8000/init.yaml page_poison=1 slab_nomerge pti=on slub_debug=P ip=172.16.0.2::172.16.0.1:255.255.255.0::eth0:off random.trust_cpu=on printk.devkmsg=on\" 
   }"
  
curl --unix-socket /tmp/firecracker.socket -i  \
    -X PUT 'http://localhost/machine-config' \
    -H 'Accept: application/json'            \
    -H 'Content-Type: application/json'      \
    -d '{
        "vcpu_count": 2,
        "mem_size_mib": 1024,
        "ht_enabled": false
    }'

 curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT 'http://localhost/drives/sda' \
    -H 'Accept: application/json'           \
    -H 'Content-Type: application/json'     \
    -d "{
          \"drive_id\": \"sda\",
          \"is_root_device\": false,
          \"is_read_only\": false,
	  \"path_on_host\": \"./sda\"
     }"


curl -X PUT \
  --unix-socket /tmp/firecracker.socket \
  http://localhost/network-interfaces/eth0 \
    -H  'Accept: application/json'          \
    -H  'Content-Type: application/json'    \
  -d '{
      "iface_id": "eth0",
      "guest_mac": "AA:FC:00:00:00:01",
      "host_dev_name": "tap0"
    }'


curl --unix-socket /tmp/firecracker.socket -i \
    -X PUT 'http://localhost/actions'       \
    -H  'Accept: application/json'          \
    -H  'Content-Type: application/json'    \
    -d '{
        "action_type": "InstanceStart"
     }'