Skip to content

Instantly share code, notes, and snippets.

@jperkin

jperkin/netbsd-triton.md

Last active October 18, 2021 16:44
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jperkin/b6bece03be0cb64969324dd6e6e00a79 to your computer and use it in GitHub Desktop.
Save jperkin/b6bece03be0cb64969324dd6e6e00a79 to your computer and use it in GitHub Desktop.
Download ZIP
Generate NetBSD Triton Images
Raw
netbsd-triton.md

Creating NetBSD Bhyve/KVM images suitable for running in Triton.

Create Install VM

Create the VM json:

{
  "brand": "bhyve",
  "alias": "netbsd-install",
  "autoboot": "false",
  "bootrom": "bios",
  "ram": 8192,
  "disks": [
    {
      "boot": false,
      "model": "virtio",
      "size": 10240
    },
    {
      "boot": true,
      "model": "ahci",
      "media": "cdrom",
      "path": "/boot-com.iso"
    }
  ],
  "resolvers": [
    "10.0.84.48",
    "10.0.84.84"
  ],
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "10.0.83.220",
      "netmask": "255.255.255.0",
      "gateway": "10.0.83.1",
      "model": "virtio",
      "primary": true
    }
  ],
  "customer_metadata": {
    "root_authorized_keys": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBZ2pDopp8rpmfy8/kGP1Ex6rsl4Wj7ZjjfELeJgqs/q jperkin-ed25519-2020"
  }
}

Create VM but do not start it up yet.

$ vmadm create -f netbsd-install.json

Fetch NetBSD boot-com.iso and place it into root directory of the created zone.

$ curl -O http://ftp.netbsd.org/pub/NetBSD/NetBSD-9.1/amd64/installation/cdrom/boot-com.iso
$ cp boot-com.iso /zones/<uuid>/root

Boot Install VM

Now we can start it up and attach to the console.

$ vmadm start <uuid>
$ vmadm console <uuid>

During the boot, you must drop to a boot promptby pressing "3" before the timer runs out.

The xhci driver needs to be disabled, otherwise the host will panic during boot.

>> NetBSD/x86 BIOS Boot, Revision 5.11 (Sun Oct 18 19:24:30 UTC 2020) (from NetBSD 9.1)
>> Memory: 640/15360 k

     1. Boot normally
     2. Boot single user
     3. Drop to boot prompt

Choose an option; RETURN for default; SPACE to stop countdown.
Option 1 will be chosen in 0 seconds.     
type "?" or "help" for help.
> userconf disable xhci
> boot

Sysinst

Follow the NetBSD sysinst installer. I made the following changes from the defaults:

  • Set swap size to 2GB, leaving 8GB for /
  • Increase serial baud rate to 115200
  • Installation without X11
  • HTTP install
  • Configure network manually (no DHCP support)
  • Set a distinct hostname to make it easy to grep for things that need to be removed prior to image creation
  • Enable installation of binary packages (just pkgin)
  • Enable sshd, ntpd, ntpdate
  • Disable cgd, raidframe

After install, drop to a shell and edit /boot.cfg to fix the XHCI issue, then halt the machine.

$ mount /dev/dk0 /mnt2
$ echo "userconf=disable xhci" >>/mnt2/boot.cfg
$ umount /mnt2
$ poweroff

Post-Install Configuration

Remove the CD device

$ echo '{"remove_disks": ["/boot-com.iso"]}' | vmadm update <uuid>

Then boot the system back up again. You may also need to reconnect the console.

$ vmadm boot <uuid>
$ vmadm console <uuid>

Once booted, log in as root with the empty password.

Fetch support binaries and scripts:

#
# /usr/triton/bin contains our builds of mdata-client and jq.  These should only
# depend on libc and libm to be portable across different releases.
#
mkdir -p /usr/triton/bin
cd /usr/triton/bin
for file in jq mdata-delete mdata-get mdata-list mdata-put; do
    ftp https://us-east.manta.joyent.com/pkgsrc/public/support/images/netbsd/${file}
    chmod 0755 ${file}
done

#
# Install the /etc/rc.d/triton init script.  This sets up networking and the
# root SSH keys from customer_metadata.
#
ftp -o /etc/rc.d/triton https://us-east.manta.joyent.com/pkgsrc/public/support/images/netbsd/triton
chmod 0555 /etc/rc.d/triton

Update /etc/rc.conf:

  • Add triton=YES,
  • Remove hostname= and defaultroute= if set,
  • Remove wscons=YES.
  • Sort the remaining entries,
$ vi /etc/rc.conf

Turn off ttyE* entries in /etc/ttys, no need for virtual consoles:

$ vi /etc/ttys
ttyE1	"/usr/libexec/getty Pc"		wsvt25	off secure
ttyE2	"/usr/libexec/getty Pc"		wsvt25	off secure
ttyE3	"/usr/libexec/getty Pc"		wsvt25	off secure

Remove static entry added by sysinst to /etc/hosts:

$ vi /etc/hosts

Stop syslogd prior to removing files, avoiding them being written to on halt:

$ /etc/rc.d/syslogd stop

Remove or empty all files that will be created on first boot:

#
# Remove networking files created by /etc/rc.d/triton
#
for f in ifconfig.vioif0 mygate myname resolv.conf; do
    rm -f /etc/${f}
done
rm -f /var/spool/postfix/etc/resolv.conf

#
# Not sure where these come from
#
rm -f /etc/rc.conf.bak.*

#
# Remove SSH keys
#
rm -f /etc/ssh/ssh_host*
rm -f /root/.ssh/authorized_keys

#
# Remove and empty log files
#
rm -f /var/log/*.gz
for f in /var/log/*; do
    [ -f $f ] && >$f
done
rm -f /var/run/dmesg.boot
rm -f /var/run/rc.log
>/var/run/utmp
>/var/run/utmpx

#
# Remove pkgin database (it will almost certainly be out of date).
#
rm -f /var/db/pkgin/pkgin.db

#
# Remove the system entropy file, we do not want it to be deterministic.
#
rm -f /var/db/entropy-file

Note that /var/log/*.gz will be recreated every hour by the newsyslog cron, so double check when creating your image that you don't happen to be unlucky with timing in between removing the existing log files and shutting down the host.

Rebuild pkgdb files just in case:

$ pkg_admin rebuild
$ pkg_admin rebuild-tree

Finally, switch off. Make sure to use poweroff, as running shutdown -p now or similar will recreate the entropy file.

$ poweroff

Create Image

I don't really understand why this is required, but it is, otherwise you get "out of space" errors:

$ vmadm update <uuid> flexible_disk_size=20480

Now we can snapshot the image and generate the metadata.

$ zfs snapshot -r zones/<uuid>@final
$ zfs send zones/<uuid>/disk0@final | gzip -9 >netbsd-9.1.zfs.gz

The JSON manifest is below, the following variables need replacing:

  • @UUID@ with a fresh uuid (e.g. just run uuid).
  • @SIZE@ with the ls -l output for netbsd-9.1.zfs.gz.
  • @SHA1@ with the digest -a sha1 output for netbsd-9.1.zfs.gz.

Note to keep this order to ease diffs against imgadm show output.

{
  "v": 2,
  "uuid": "@UUID@",
  "owner": "00000000-0000-0000-0000-000000000000",
  "name": "netbsd-9.1",
  "version": "20210223",
  "state": "active",
  "disabled": false,
  "public": true,
  "published_at": "2020-02-22T22:22:22Z",
  "type": "zvol",
  "os": "bsd",
  "files": [
    {
      "sha1": "@SHA1@",
      "size": "@SIZE@",
      "compression": "gzip"
    }
  ],
  "description": "NetBSD 9.1.  Installation without X11.",
  "homepage": "https://www.netbsd.org/",
  "requirements": {},
  "nic_driver": "virtio",
  "disk_driver": "virtio",
  "cpu_type": "host",
  "image_size": 10240,
  "tags": {
    "role": "os"
  }
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment