Skip to content

Instantly share code, notes, and snippets.

What would you like to do?
Using a zvol for a swap device for host and VMs
Swapping to Optane
Since i’m running virtual machines, there is another thing which should go to low-latency storage: swap. I try to conserve as much memory as possible, which means VMs sometimes use their swap space, which gets horribly slow in case it resides on spinning disks. For that reason i created another partition, created a separate ZFS pool and created disk images that will hold the VMs swap.
Creating a new pool is very simple and as i don’t need redundancy on swap it will just be one “device”, actually a partition. Using unique hardware identifiers instead of device paths (e.g. “/dev/nvme0n1p3”) is quite helpful as PCIe enumeration and partition order may change.
$ zpool create \
-O normalization=formD \
-O sync=always \
Now create a zvol as a swap device but you'll need to configure it appropriately.
- Set the volume block size to match your systems page size. This tuning prevents ZFS from having to perform read-modify-write options on a larger block while the system is already low on memory.
- Set the logbias=throughput and sync=always properties. Data written to the volume will be flushed immediately to disk freeing up memory as quickly as possible.
- Set primarycache=metadata to avoid keeping swap data in RAM via the ARC.
- Disable automatic snapshots of the swap device.
( 4GB works for me but size accordingly. I'm not sure if this duplication of settings is needed, i.e. sync=always, as the zvool should inheret these from the pool. In which case maybe all the options should go to the pool?)
$ zfs create -V 4G -b $(getconf PAGESIZE) \
-o logbias=throughput -o sync=always \
-o primarycache=metadata \
-o com.sun:auto-snapshot=false \
Now new virtual disks are created on this ZFS pool and get attached to their virtual machine.
$ zfs list
swaps 33.1M 96.8G 24K /swaps
swaps/vm-100-disk-1 30K 96.8G 30K -
swaps/vm-101-disk-1 1.02M 96.8G 1.02M -
Replacing old swap and re-claiming that space for the root partition is easy if the VMs are using LVM. /dev/sdb is the new virtual device available to the VM, stored at the ZFS “swaps” pool on Optane.
Add the new swap space to LVM:
$ pvcreate /dev/sdb
$ vgcreate swp /dev/sdb
$ lvcreate -l 100%FREE -n swap swp
Create the swap file system and use the UUID as device identifier in /etc/fstab:
$ mkswap /dev/vm-optane/swap
$ vim /etc/fstab
/dev/mapper/swp-swap none swap sw 0 0
Disable and remove the old swap partition:
$ swapoff /dev/vm-system/swap
$ lvremove /dev/vm-system/swap
Extend the root partition and file system to use the free’d up space:
$ lvextend -l +100%FREE /dev/vm-system/root
$ resize2fs /dev/vm-system/root
…and reboot the VM, just to be sure the file system is undamaged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment