Skip to content

Instantly share code, notes, and snippets.

@mgerdts
Created October 18, 2019 11:56
Show Gist options
  • Save mgerdts/d1dd0ae618262710be1895cb2e398260 to your computer and use it in GitHub Desktop.
Save mgerdts/d1dd0ae618262710be1895cb2e398260 to your computer and use it in GitHub Desktop.
Triton on Linux KVM

Triton on Linux KVM

I have a beefy Linux box that I use for development. One of the things this box does is run KVM instances that run a Triton headnode and several compute nodes.

Configuration

Networking

I have a single gigabit link to the outside world. All networks are able to reach the outside world via NAT.

[root@beast mgerdts]# virsh net-list
 Name            State    Autostart   Persistent
--------------------------------------------------
 coaladmin1      active   yes         yes
 coalexternal1   active   yes         yes
 default         active   yes         yes

I obtained the configuration for each of the coal networks with virst net-dumpxml <name>.

<network connections='4'>
  <name>coaladmin1</name>
  <uuid>85816e6e-3b51-4375-a317-b19c28fb6f50</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr1' stp='on' delay='0'/>
  <mac address='52:54:00:b9:d7:86'/>
  <domain name='coaladmin1'/>
  <ip address='10.99.99.1' netmask='255.255.255.0'>
  </ip>
</network>
<network connections='4'>
  <name>coalexternal1</name>
  <uuid>db5f922d-f782-44ee-a380-5cf5e22c1d74</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr2' stp='on' delay='0'/>
  <mac address='52:54:00:0d:9a:e8'/>
  <domain name='coalexternal1'/>
  <ip address='10.88.88.1' netmask='255.255.255.0'>
  </ip>
</network>

Storage

I'm using ZFS for storage, putting an entire Triton data center under one dataset. The idea is that if I'm experimenting, it becomes easy to snapshot an entire datacenter with one zfs snapshot -r. What I show below is not the result of careful planning, just what I did on the fly.

[root@beast mgerdts]# virsh vol-list data
 Name                   Path
-------------------------------------------------------------
 vms/coal-1/boot-usb    /dev/zvol/data/vms/coal-1/boot-usb
 vms/coal-1/cn1-disk0   /dev/zvol/data/vms/coal-1/cn1-disk0
 vms/coal-1/cn2-disk0   /dev/zvol/data/vms/coal-1/cn2-disk0
 vms/coal-1/cn3-disk0   /dev/zvol/data/vms/coal-1/cn3-disk0
 vms/coal-1/disk0       /dev/zvol/data/vms/coal-1/disk0
 vms/fedora30           /dev/zvol/data/vms/fedora30

[root@beast mgerdts]# virsh vol-dumpxml --pool data vms/coal-1/cn1-disk0
<volume type='block'>
  <name>vms/coal-1/cn1-disk0</name>
  <key>data/vms/coal-1/cn1-disk0</key>
  <source>
  </source>
  <capacity unit='bytes'>21474836480</capacity>
  <allocation unit='bytes'>0</allocation>
  <target>
    <path>/dev/zvol/data/vms/coal-1/cn1-disk0</path>
  </target>
</volume>

To ensure that I don't accidentally blow away my headnode disks, I created a snapshot of each of then and then used zfs hold to add a hold.

HN boot disk configuration

In order for the Triton setup program to recognize the USB image, I had to configure it (using the virtmanager gui, surely there's another way) to be a USB drive with the removable box ticked. Even though the USB image is on a pool made up of decent SSDs (mirrored), the performance of this virtual USB mimics the perfomance of a cheap thumb drive (single digit MB/s).

After the HN is set up, you can switch the drive over to a non-removable SATA drive. In virtmanager, I had to halt the VM, remove the USB disk, then add it back as a SATA disk to get past an error saying that "removable" (which was unchecked) is only applicable to USB. The zvol was not destroyed, but that may have been a result of the aforementioned zfs hold. You have been warned.

After making this change, you will probably need to update the boot order so that the newly added SATA disk is first in the boot order.

HD boot disk resize

Note: When I did this procedure, I had not yet switched the boot disk from USB to SATA.

The 4 GiB USB image has room for about 3 platform images. After bumping against this limit a couple times, I decided to grow this disk. The procedure went roughly like:

root@beast# zfs snapshot data/vms/coal-1/boot-usb@before-grow
root@beast# zfs set volsize=16g data/vms/coal-1/boot-usb

Halt the HN VM, then boot it. A reboot within the guest is not sufficient to get it to see the new size.

Save a copy of the usbkey contents. Be sure to use . rather than * so that dot files are not missed.

[root@headnode (coal-1) ~]# sdc-usbkey mount
/mnt/usbkey
[root@headnode (coal-1) ~]# cd /mnt/usbkey
[root@headnode (coal-1) /mnt/usbkey]# tar cvzf /var/tmp/mg-usbkey.tar.gz .
Compressing '/var/tmp/mg-usbkey.tar.gz' with '/usr/bin/gzip'...
...
[root@headnode (coal-1) /mnt/usbkey]# cd
[root@headnode (coal-1) ~]# sdc-usbkey unmount

Now we are ready to grow the partition and put a new file system on the partition.

[root@headnode (coal-1) ~]# format -e
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c1t0d0 <QEMU-QEMU HARDDISK-2.5+-16.00GB>  reserved
          /pci@0,0/pci1af4,1100@6,7/storage@4/disk@0,0
       1. c2t1d0 <QEMU-HARDDISK-2.5+-60.00GB>
          /pci@0,0/pci1af4,1100@9/disk@1,0
Specify disk (enter its number): 0
selecting c1t0d0: reserved
[disk formatted]
Note: capacity in disk label is smaller than the real disk capacity.
Select <partition> <expand> to adjust the label capacity.
...
format> pa


PARTITION MENU:
        0      - change `0' partition
        1      - change `1' partition
        2      - change `2' partition
        3      - change `3' partition
        4      - change `4' partition
        5      - change `5' partition
        6      - change `6' partition
        7      - change `7' partition
        8      - change '8' partition
        9      - change `9' partition
        expand - expand label to use whole disk
        select - select a predefined table
        modify - modify a predefined partition table
        name   - name the current table
        print  - display the current table
        label  - write partition map and label to the disk
        !<cmd> - execute <cmd>, then return
        quit
partition> expand
The expanded capacity is added to the unallocated space.
partition> pr
Volume:  reserved
Current partition table (original):
Total disk sectors available: 33537981 + 16384 (reserved sectors)

Part      Tag    Flag     First Sector        Size        Last Sector
  0     system    wm              2048     256.00MB         526335
  1       boot    wm            526336       1.00MB         528383
  2       root    wm            528384       3.46GB         7794687
  3 unassigned    wm                 0          0              0
  4 unassigned    wm                 0          0              0
  5 unassigned    wm                 0          0              0
  6 unassigned    wm                 0          0              0
  7 unassigned    wm                 0          0              0
  8   reserved    wm          33538015       8.00MB         33554398

partition> 2
Part      Tag    Flag     First Sector        Size        Last Sector
  2       root    wm            528384       3.46GB         7794687

Enter partition id tag[root]:
Enter partition permission flags[wm]:
Enter new starting Sector[528384]:
Enter partition size[7266304b, 7794687e, 3548mb, 3gb, 0tb]: 33538014e
partition> pr
Volume:  reserved
Current partition table (unnamed):
Total disk sectors available: 33537981 + 16384 (reserved sectors)

Part      Tag    Flag     First Sector        Size        Last Sector
  0     system    wm              2048     256.00MB         526335
  1       boot    wm            526336       1.00MB         528383
  2       root    wm            528384      15.74GB         33538014
  3 unassigned    wm                 0          0              0
  4 unassigned    wm                 0          0              0
  5 unassigned    wm                 0          0              0
  6 unassigned    wm                 0          0              0
  7 unassigned    wm                 0          0              0
  8   reserved    wm          33538015       8.00MB         33554398

partition> la
[0] SMI Label
[1] EFI Label
Specify Label type[1]:
Ready to label disk, continue? y

partition> ^D

The values I used for size were taken from the final pr output above.

[root@headnode (coal-1) ~]# mkfs -F pcfs -o b=SMARTSOBOOT,nofdisk,size=$((33538014-528384+1)),fat=32 /dev/rdsk/c1t0d0s2
Construct a new FAT file system on /dev/rdsk/c1t0d0s2: (y/n)? y

Now the content needs to be restored to the usb disk.

[root@headnode (coal-1) ~]# mount -F pcfs /dev/dsk/c1t0d0s2 /mnt/usbkey
[root@headnode (coal-1) ~]# cd /mnt/usbkey/
[root@headnode (coal-1) /mnt/usbkey]# tar xvzf /var/tmp/mg-usbkey.tar.gz
Decompressing '/var/tmp/mg-usbkey.tar.gz' with '/usr/bin/gzcat'...
...
[root@headnode (coal-1) /mnt/usbkey]# cd
[root@headnode (coal-1) ~]# df -h /mnt/usbkey
Filesystem             Size   Used  Available Capacity  Mounted on
/dev/dsk/c1t0d0s2     15.7G  3.36G      12.3G    22%    /mnt/usbkey
[root@headnode (coal-1) ~]# umount /mnt/usbkey

I expect at this point that the boot loader configuration on the usbdisk is busted, so I updated it by assigning a newer PI to the HN.

[root@headnode (coal-1) ~]# sdcadm platform list
VERSION           CURRENT_PLATFORM  BOOT_PLATFORM  LATEST  DEFAULT
20191017T032813Z  0                 1              true    false
20191017T000357Z  0                 1              false   true
20191014T204909Z  3                 1              false   false
20191009T235451Z  1                 1              false   false
20151126T062538Z  0                 0              false   false
[root@headnode (coal-1) ~]# sdcadm platform assign 20191017T032813Z headnode
updating headnode 55657228-6967-0142-a965-6b7be4e6e9b6 to 20191017T032813Z
Setting boot params for 55657228-6967-0142-a965-6b7be4e6e9b6
Updating booter cache for servers
Done updating booter caches
Verifying boot_platform updates
Cleaned up deprecated 'latest' symlink
[root@headnode (coal-1) ~]# reboot
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment