Skip to content

Instantly share code, notes, and snippets.

@akavel
Created March 21, 2017 16:47
Show Gist options
  • Save akavel/14d132b6e82b6cf5d8a7c632730cf3c7 to your computer and use it in GitHub Desktop.
Save akavel/14d132b6e82b6cf5d8a7c632730cf3c7 to your computer and use it in GitHub Desktop.
Instructions for Genode / NixOS dual-and-a-half-boot

Genode/NixOS dual-and-a-half-boot

If you've never used Genode or NixOS, or installed Gentoo, stop and consider what you are about to do.

The goal is too create a NixOS installation and a Genode installation, with the option of booting into NixOS, booting into Genode, and booting into Genode then booting the same NixOS install again in a virtual machine.

The trick will be to create a two boot partitions, one real, one virtual. NixOS will live on a third LUKS partition, Genode and NixOS will be able to boot from the real boot partition, and NixOS will also be able to boot from the virtual partition while again using the LUKS partition. The majority of the NixOS system configuration will be stored on the LUKS partition, and the hardware specific configuration will be stored on the respective boot partitions.

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Genode and GRUB partition ┃
┃ ┌───────────────────────┐ ┃
┃ │ NixOS boot disk image │ ┃
┃ └───────────────────────┘ ┃
┣━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┇ NixOS root LUKS partition ┇
┇                           ┇
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

NixOS install media

Fetch a NixOS minimal install image and dd to a USB stick. http://nixos.org/nixos/download.html Boot the image (use some real hardware, we're not virtualizing yet).

Partition block device

Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xffffffff

Device     Boot     Start       End   Sectors   Size Id Type
/dev/sda1            2048 268437503 268435456   128G 83 Linux
/dev/sda2       268568576 976773167 708204592 337.7G 83 Linux


Disk /dev/mapper/root: 337.7 GiB, 362598653952 bytes, 708200496 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Format partitions

# Genode / boot partition
mkfs.ext2 /dev/sda1

# NixOS encrypted root
cryptsetup luksFormat /dev/sda2
cryptsetup luksOpen /dev/sda2 root
mkfs.ext4 /dev/mapper/root

NixOS native install

Mount file systems

mount /dev/mapper/root /mnt
mkdir /mnt/boot
mount /dev/sda1 /mnt/boot

Edit NixOS global config

Move the system configuration to the boot partitition.

nixos-generate-config --root /mnt
mkdir /mnt/boot/nixos
# Important, move the native hardware config to the native boot partition.
mv /mnt/etc/nixos/hardware-configuration.nix /mnt/boot/nixos/hardware-configuration.nix

Create /mnt/boot/nixos/boot.nix with the following content:

{ config, pkgs, ... }:

{
  imports =
    [ # Include the results of the native hardware scan.
      ./hardware-configuration.nix
    ];

  boot.loader.grub =
    { enable = true;
      device = "/dev/sda";
      version = 2;
      extraEntries =
        ''menuentry "Genode" { legacy_configfile /boot/grub/menu.lst }'';

      extraEntriesBeforeNixOS = true;
    };


  # Unlock the LUKS partition during early boot
  boot.initrd.luks.devices =
    [ { device = "/dev/sda2"; name = "root"; } ];

  # Install VirtualBox and its kernel modules,
  # this is to manage the boot disk image natively.
  virtualisation.virtualbox.host.enable = true;

  # Wireless is only available in the native install,
  # so enable it here if you need it.
  #networking.wireless.enable = true;  
}

Now, modify /mnt/etc/nixos/configuration.nix to import the boot configuration.

   imports =
-    [ # Include the results of the hardware scan.
-      ./hardware-configuration.nix
+    [ # Include the boot and hardware configuration,
+      # from either the native or virtual boot device.
+      ../../boot/nixos/boot.nix
     ];

-  # Use the GRUB 2 boot loader.
-  boot.loader.grub.enable = true;
-  boot.loader.grub.version = 2;
-  # Define on which hard drive you want to install Grub.
-  # boot.loader.grub.device = "/dev/sda";
-

Desktop environments and a non-root user can be configured in /mnt/etc/nixos/configuration.nix, see the NixOS manual, available on virtual console 8 (press Alt+F8 to access), or at http://nixos.org/nixos/manual/index.html

The file /mnt/boot/nixos/hardware-configuration.nix contains the mount configuration for our partitions. The detection is quite good, but it doesn't hurt to give it a look over.

Creating a git repository in /etc/nixos is recommended.

Install NixOS

nixos-install

Cross your fingers and reboot. The first option in your boot menu will not work because we haven't gotten there yet.

Build and install Genode from NixOS

git clone https://github.com/genodelabs/genode.git
cd genode

Apply a patch to use the non-standard system paths in NixOS

git remote add ehmry https://github.com/ehmry/genode.git
git fetch ehmry
git cherry-pick ehmry/nixos

VirtualBox

Create the VirtualBox block devices.

If virtualbox.host was enabled in the configuration at /boot, the VirtualBox utilities will already be installed.

# the /boot directory is really our Genode install partition
mkdir -p /boot/vbox/nixos

# the virtualized NixOS will boot from a 1GB disk image...
VBoxManage createhd --filename /boot/vbox/nixos/boot.vdi --size 1024

# but the root file system will be a passthru to the LUKS partition.
VBoxManage internalcommands createrawvmdk -filename /boot/vbox/nixos/luks.vmdk -rawdisk /dev/sda2

# copy over or download the NixOS livecd
curl -L https://nixos.org/releases/nixos/latest-iso-minimal-x86_64-linux -o /boot/vbox/nixos/nixos-minimal-x86_64.iso

Configure Virtualbox

Install the following file to /boot/vbox/nixos/nixos.vbox and replace the device UUIDs with this script:

# if you don't want to install sed or awk you could replace these strings by hand
sed \
	-e "s/@BOOT_UUID@/$(VBoxManage showhdinfo /boot/vbox/nixos/boot.vdi | awk '{if($1 == "UUID:") print $2;}')/" \
	-e "s/@LUKS_UUID@/$(VBoxManage showhdinfo /boot/vbox/nixos/luks.vmdk | awk '{if($1 == "UUID:") print $2;}')/" \
	-i /boot/vbox/nixos/nixos.vbox
/boot/vbox/nixos/nixos.vbox
<?xml version="1.0"?>
<VirtualBox xmlns="http://www.innotek.de/VirtualBox-settings" version="1.15-linux">
  <Machine uuid="{0d91e36f-937d-4096-9631-dc52866aa4cc}" name="Nixos" OSType="Linux_64">
    <MediaRegistry>
      <HardDisks>
        <HardDisk uuid="{@BOOT_UUID@}" location="/boot.vdi" format="VDI" type="Normal"/>
        <HardDisk uuid="{@LUKS_UUID@}" location="/luks.vmdk" format="VMDK" type="Normal"/>
      </HardDisks>
      <DVDImages>
        <Image uuid="{d078721d-ee1f-4172-95c2-7c5a719d630b}" location="/nixos-minimal-x86_64.iso"/>
      </DVDImages>
      <FloppyImages/>
    </MediaRegistry>
    <ExtraData/>
    <Hardware version="2">
      <CPU count="4" hotplug="false">
        <HardwareVirtEx enabled="true"/>
        <HardwareVirtExNestedPaging enabled="true"/>
        <HardwareVirtExVPID enabled="true"/>
        <HardwareVirtExUX enabled="true"/>
        <PAE enabled="true"/>
        <LongMode enabled="true"/>
        <HardwareVirtExLargePages enabled="false"/>
        <HardwareVirtForce enabled="false"/>
      </CPU>
      <Memory RAMSize="1024" PageFusion="false"/>
      <HID Pointing="PS2Mouse" Keyboard="PS2Keyboard"/>
      <HPET enabled="false"/>
      <Chipset type="PIIX3"/>
      <Paravirt provider="Default"/>
      <Boot>
        <Order position="1" device="DVD"/>
        <Order position="2" device="HardDisk"/>
        <Order position="3" device="None"/>
      </Boot>
      <Display VRAMSize="12" monitorCount="1" accelerate3D="false" accelerate2DVideo="false"/>
      <VideoCapture enabled="false" screens="18446744073709551615" horzRes="1024" vertRes="768" rate="512" fps="25" maxTime="0" maxSize="0"/>
      <RemoteDisplay enabled="false" authType="Null"/>
      <BIOS>
        <ACPI enabled="true"/>
        <IOAPIC enabled="true"/>
        <Logo fadeIn="true" fadeOut="true" displayTime="0"/>
        <BootMenu mode="MessageAndMenu"/>
        <TimeOffset value="0"/>
        <PXEDebug enabled="false"/>
      </BIOS>
      <USB>
        <Controllers>
          <Controller name="OHCI" type="OHCI"/>
        </Controllers>
        <DeviceFilters/>
      </USB>
      <Network>
        <Adapter slot="0" enabled="true" MACAddress="0800271D7901" cable="true" speed="0" type="82540EM">
          <DisabledModes>
            <NAT>
              <DNS pass-domain="true" use-proxy="false" use-host-resolver="false"/>
              <Alias logging="false" proxy-only="false" use-same-ports="false"/>
            </NAT>
          </DisabledModes>
          <BridgedInterface name=""/>
        </Adapter>
      </Network>
      <UART>
        <Port slot="0" enabled="false" IOBase="0x3f8" IRQ="4" hostMode="Disconnected"/>
        <Port slot="1" enabled="false" IOBase="0x2f8" IRQ="3" hostMode="Disconnected"/>
      </UART>
      <LPT>
        <Port slot="0" enabled="false" IOBase="0x378" IRQ="7"/>
        <Port slot="1" enabled="false" IOBase="0x278" IRQ="5"/>
      </LPT>
      <AudioAdapter controller="AC97" driver="ALSA" enabled="true"/>
      <RTC localOrUTC="UTC"/>
      <SharedFolders>
        <SharedFolder name="shared" hostPath="/shared" writable="true" autoMount="true"/>
      </SharedFolders>
      <Clipboard mode="Disabled"/>
      <DragAndDrop mode="Disabled"/>
      <IO>
        <IoCache enabled="true" size="5"/>
        <BandwidthGroups/>
      </IO>
      <HostPci>
        <Devices/>
      </HostPci>
      <EmulatedUSB>
        <CardReader enabled="false"/>
      </EmulatedUSB>
      <Guest memoryBalloonSize="0"/>
      <GuestProperties/>
    </Hardware>
    <StorageControllers>
      <StorageController name="IDE" type="PIIX4" PortCount="2" useHostIOCache="true" Bootable="true">
        <AttachedDevice type="HardDisk" port="0" device="0">
          <Image uuid="{@BOOT_UUID@}"/>
        </AttachedDevice>
        <AttachedDevice type="HardDisk" port="0" device="1">
          <Image uuid="{@LUKS_UUID@}"/>
        </AttachedDevice>
        <AttachedDevice passthrough="false" type="DVD" port="1" device="0">
          <Image uuid="{d078721d-ee1f-4172-95c2-7c5a719d630b}"/>
        </AttachedDevice>
      </StorageController>
    </StorageControllers>
  </Machine>
</VirtualBox>

Configure NixOS subsystem of Genode

Save the following file to /boot/subsystems/nixos.subsystem.

/boot/subsystems/nixos.subsystem
<subsystem name="NixOS" help="NixOS in VirtualBox">
  <resource name="RAM" quantum="1536M" />
  <binary name="init" />
  <config prio_levels="2">
    <parent-provides>
      <service name="CAP"/>
      <service name="CPU"/>
      <service name="LOG"/>
      <service name="PD"/>
      <service name="RAM"/>
      <service name="RM"/>
      <service name="ROM"/>
      <service name="SIGNAL"/>
      <service name="Audio_out" />
      <service name="Block" />
      <service name="File_system" />
      <service name="Nic" />
      <service name="Nitpicker" />
      <service name="Report" />
      <service name="Rtc" />
      <service name="Usb" />
    </parent-provides>
    <default-route>
      <any-service> <parent/> <any-child/> </any-service>
    </default-route>

	<start name="timer">
		<resource name="RAM" quantum="1M"/>
		<provides><service name="Timer"/></provides>
	</start>

	<start name="nit_fb" priority="0">
		<resource name="RAM" quantum="8M" />
		<provides>
			<service name="Framebuffer" />
			<service name="Input" />
		</provides>
		<config width="800" height="600" />
	</start>

	<start name="vbox" priority="-1">
		<exit propagate="yes"/>
		<binary name="virtualbox" />
		<resource name="RAM" quantum="2176M"/>
			<config vbox_file="nixos.vbox" vm_name="NixOS">
			<libc stdout="/dev/log" stderr="/dev/log" rtc="/dev/rtc">
				<vfs>
					<dir name="dev">
						<log/>
						<rtc/>
						<block name="sda2" label="luks" block_buffer_count="128"/>
					</dir>
					<dir name="shared">
						<fs label="shared"/>
					</dir>
					<fs label="rump_fs"/>
				</vfs>
			</libc>
		</config>
		<route>
			<any-service>
				<any-child/> <parent/>
			</any-service>
		</route>
	</start>

  </config>
</subsystem>

NixOS virtualized install

Try and ignore the helpful systemd spew, it clears up after a while.

Create boot file system

TODO: an fdisk screenshot

mkfs.ext2 /dev/sda1 -L boot

Mount file systems and generate initial config

cryptsetup luksOpen /dev/sdb root
mount /dev/mapper/root /mnt
mount /dev/sda1 /mnt/boot

# mount point for the shared folder
mkdir /mnt/shared

Create boot configuration

nixos-generate-config --root /mnt
mkdir /mnt/boot/nixos
mv /mnt/etc/nixos/hardware-configuration.nix /mnt/boot/nixos

Create /mnt/boot/nixos/boot.nix with the following:

{ config, pkgs, ... }:

{
  imports =
    [ # Include the results of the virtual hardware scan.
      ./hardware-configuration.nix
    ];

  boot.loader.grub =
    { enable = true;
      device = "/dev/sda";
      version = 2;
    };

  # Unlock the LUKS partition during early boot
  boot.initrd.luks.devices =
    [ { device = "/dev/sdb"; name = "root"; } ];
}

Add the shared folder definition to /mnt/boot/nixos/hardware-configuration. nixos-generate-config does not get this one quite right, so we do it by hand.

  fileSystems."/shared" =
    { device = "/shared";
      fsType = "vboxsf";
    };

There is no need to edit /mnt/etc/nixos/configuration.nix again, this configuration is shared between the native and virtual install.

Install NixOS (again)

nixos-install
poweroff

Finalize nixos.vbox

Open a Noux terminal and edit vbox/nixos/nixos.vbox, commenting out reference to the livecd.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment