The aim was to be able to:
- Run multiple Linux VMs on an Apple M1/ARM device
- Use Apple's HVF for native performance speeds
- Configure VMs to allow network access to each other
- Configure VMs to allow access to the internet
- Not rely on custom modifications of software
I had originally assumed I would be up and running with the above in a few hours, but I was wrong. After looking for examples online, of which there are many. The problem was that they were seemingly all based on either Intel based Apple devices, or for MacOS < 12. I'm using a 2021 MacBookPro with an M1 Pro chip running MacOS 12.6 (Monterey). I have therefore spent a lot of time finding out what doesn't work anymore.
I was also new to some of the tools I have used including QEMU and libvirt.
After much experimentation and failure, I have a working example to meet the goals set out. A lot of the issues faced seemed to be that changes to MacOS over the past few years and the time it takes to reflect this in software, has meant a period between what used to work, things not working, and now working again with different implementations. The end result achieved using the process defined below looks similar to the following:
Many examples I found online to help meet goal 3 relied on TAP devices. As I understand these were not natively supported by MacOS and required kernel extensions (kexts) to enable the functionality - commonly through a tool called 'tuntaposx'. As stated on their website the project has not been maintained since 2015 and references the requirement from Apple to use notarization on all updated or new kernel extensions which tuntaposx does not have. Tunnelblick details the situation here.
There are other helper tools available from QEMU/libvirt to deal with networking like qemu-bridge-helper / libvirt virtual networks - but they too don't yet seem to be working on my device for various reasons. Some of the current open issues from these projects detail more
I have tested this using:
- MacOS - 12.6.1 Monterey
- Chip - M1 Pro
- QEMU - 7.1.0
- libvirt - 8.9.0
- virt-manager - 4.1.0
As it's MacOS, I have used homebrew for package management. For convenience I am also using brew services as a wrapper for launchctl
brew install qemu libvirt virt-manager
# must use sudo as it affects qemu later
sudo brew services start libvirtFor native speeds you will need an OS ISO image for AARCH64 hosts (ARM). x86 ISO files will work but because Apple M1 devices use an ARM architecture they will need QEMU to emulate the necessary hardware which is much slower, and you may need to alter the command given later to use 'tcg' instead of 'hvf'.
I went with RHEL 9 as RHEL 7/8 wont work natively due to page sizes
Basic libvirt + QEMU knowledge is also required.
The working solution relies on the latest version of QEMU (7.1.0 as of Nov 2022). Some of the functionality has been added, with the man page updated, but with the QEMU wiki docs online not yet featuring the same detail on the invocation page.
With dependencies installed - you can now create a VM using a single command, and everything else will be handled automatically. You can run the same command again to create a second VM.
# create a libvirt 'domain' - specify some options using qemu command line pass through
sudo virt-install \
--name host1 \
--memory 2048 \
--vcpus 2 \
--disk size=30 \
--cdrom /path/to/ISO_files/rhel-baseos-9.0-aarch64-dvd.iso \
--os-variant rhel9.0 \
--virt-type hvf \
--qemu-commandline='-M highmem=off -netdev vmnet-shared,id=net0 -device virtio-net-device,netdev=net0,mac=54:54:00:55:54:51' \
--network user
# create a second libvirt 'domain' - ensure the friendly name and MAC address of the net device are unique and run the same command again
sudo virt-install \
--name host2 \
--memory 2048 \
--vcpus 2 \
--disk size=30 \
--cdrom /path/to/ISO_files/rhel-baseos-9.0-aarch64-dvd.iso \
--os-variant rhel9.0 \
--virt-type hvf \
--qemu-commandline='-M highmem=off -netdev vmnet-shared,id=net0 -device virtio-net-device,netdev=net0,mac=54:54:00:55:54:52' \
--network user
# create Nth libvirt domain - just keep name and MAC address unique
sudo virt-install \
--name hostN \
--memory 2048 \
--vcpus 2 \
--disk size=30 \
--cdrom /path/to/ISO_files/rhel-baseos-9.0-aarch64-dvd.iso \
--os-variant rhel9.0 \
--virt-type hvf \
--qemu-commandline='-M highmem=off -netdev vmnet-shared,id=net0 -device virtio-net-device,netdev=net0,mac=NN:NN:NN:NN:NN:NN' \
--network userNotable flags are:
--virt-type hvftell QEMU to use the Apple Hypervisor Framework instead of an alternative such as TCG. This enables the native speeds
--network userenable QEMU user mode networking, which gives us access to the host and the internet. Alternative is 'none' which removes access to both. Other named libvirt networks do not seem to work on the current version for my device i.e. '--network name=default'
From the
--qemu-commandlineflag
-netdev vmnet-sharedtells QEMU to use the vmnet APIs provided by Apple as part of HVF. This is the alternative to managing our own bridge/TAP devices commonly used in the past. Flag alternatives are vmnet-host and vmnet-bridged
-device mac=XX:YY...defines the MAC address used for the virtual device that will be attached to our VM. Must be unique per VM for DHCPI wasn't able to find virt-install equivalents for 'vmnet-*' netdev options yet hence the qemu-commandline workaround but this may come in a future release
In my case I connected to each VM via the console to complete the installation process before continuing.
This has now done a few things in the background:
- A bridge device has been created called 'bridge100' (sequential from 100 if you start multiple vmnet types at once)
- A vmenet device has been created for each VM requested, and then attached to the bridge
- An internal macOS DHCP service (bootpd) has been started to assign IP addresses to each VM (and the bridge)
- The QEMU user mode network (SLIRP) has been created which gives internet access over the default gateway 10.0.2.2
# some output omitted
root@apple-host$ ifconfig
...
vmenet0: flags=8b63<UP,BROADCAST,SMART,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
ether aa:70:70:52:86:d4
media: autoselect
status: active
# bridge100 has IP 192.168.64.1 assigned
# vmenet0 and vmenet1 are members
bridge100: flags=8a63<UP,BROADCAST,SMART,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
options=3<RXCSUM,TXCSUM>
ether be:d0:74:61:f6:64
inet 192.168.64.1 netmask 0xffffff00 broadcast 192.168.64.255
...
Configuration:
...
member: vmenet0 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 29 priority 0 path cost 0
member: vmenet1 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 31 priority 0 path cost 0
...
status: active
vmenet1: flags=8b63<UP,BROADCAST,SMART,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
ether b6:69:a5:54:74:45
media: autoselect
status: active<!-- Contents of /etc/bootpd.plist after creating VMs
dhcp_domain_name_server is same IP as bridge100-->
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Subnets</key>
<array>
<dict>
<key>_creator</key>
<string>com.apple.NetworkSharing</string>
<key>allocate</key>
<true/>
<key>dhcp_domain_name_server</key>
<array>
<string>192.168.64.1</string>
</array>
<key>dhcp_router</key>
<string>192.168.64.1</string>
<key>interface</key>
<string>bridge100</string>
<key>lease_max</key>
<integer>86400</integer>
<key>lease_min</key>
<integer>86400</integer>
<key>name</key>
<string>192.168.64/24</string>
<key>net_address</key>
<string>192.168.64.0</string>
<key>net_mask</key>
<string>255.255.255.0</string>
<key>net_range</key>
<array>
<string>192.168.64.2</string>
<string>192.168.64.254</string>
</array>
</dict>
</array>
...# IP config of first VM - some output omitted
# enp1s0 is on the QEMU SLIRP network and has access to host + internet
# eth1 is attached to bridge100 and has IP 192.160.64.2 assigned via DHCP - has access via bridge to other VMs we create
# eth1 MAC address matches that defined by 'virt-install'
[root@localhost ~] ip a s
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 52:54:00:d1:d1:8d brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute enp1s0
valid_lft 84794sec preferred_lft 84794sec
...
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 54:54:00:55:54:51 brd ff:ff:ff:ff:ff:ff
inet 192.168.64.2/24 brd 192.168.64.255 scope global dynamic noprefixroute eth1
valid_lft 84794sec preferred_lft 84794sec
...# IP config of second VM - some output omitted
# enp1s0 has same IP as VM1 as the SLIRP network for each VM is independent so it isn't clashing
# eth1 has IP 192.168.64.3 assigned
[root@localhost ~] ip a s
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 52:54:00:ae:7e:5c brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute enp1s0
valid_lft 85064sec preferred_lft 85064sec
...
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 54:54:00:55:54:52 brd ff:ff:ff:ff:ff:ff
inet 192.168.64.3/24 brd 192.168.64.255 scope global dynamic noprefixroute eth1
valid_lft 85064sec preferred_lft 85064sec
...# Routing tables used by both VMs
[root@localhost ~] ip route
default via 10.0.2.2 dev enp1s0 proto dhcp metric 100
default via 192.168.64.1 dev eth1 proto dhcp metric 101
10.0.2.0/24 dev enp1s0 proto kernel scope link src 10.0.2.15 metric 100
192.168.64.0/24 dev eth1 proto kernel scope link src <host_specific_IP> metric 101Each VM can now ping the other VMs that share the same bridge (all started with the same vmnet parameter). Each VM can also make outbound internet connections via the SLIRP network.
If you pass --network none to virt-install then you will not have the enp1s0 device attached within your VM and it will have no internet access, but it will still be able to access the other VMs using eth1
With everything configured as above the setup allows the following:
Access VM from Host
# Run from Apple host
user@apple-host$ ping 192.168.64.2
PING 192.168.64.2 (192.168.64.2): 56 data bytes
64 bytes from 192.168.64.2: icmp_seq=0 ttl=64 time=1.713 ms
--- 192.168.64.2 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.713/1.713/1.713/0.000 msAccess VM from VM
# Run from VM1 with IP 192.168.64.2
[root@localhost ~] ping 192.168.64.3
PING 192.168.64.3 (192.168.64.3) 56(84) bytes of data.
64 bytes from 192.168.64.3: icmp_seq=1 ttl=64 time=2.34 ms
--- 192.168.64.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.342/2.342/2.342/0.000 msAccess Host from VM
# Run from VM1
[root@localhost ~] ping 10.0.2.2
PING 10.0.2.2 (10.0.2.2) 56(84) bytes of data.
64 bytes from 10.0.2.2: icmp_seq=1 ttl=255 time=0.868 ms
--- 10.0.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.868/0.868/0.868/0.000 msSome errors I encountered along the way but not sure if they would be consistent for everyone
When running virt-install
Error:
error: failed to connect to the hypervisor
error: Operation not supported: Cannot use direct socket mode if no URI is set
Fix:
echo 'uri = qemu:///system' >> /opt/homebrew/etc/libvirt/libvirt.conf
brew services restart libvirtThis provides the default URI to connections to QEMU system instances
When starting libvirt - the expectation was that libvirt would manage virtlogd, which is used to manage logs from VMs as an alternative to writing them all to a file which need manual disk management.
Error:
error: failed to connect to the hypervisor
error: Failed to connect socket to '/opt/homebrew/var/run/libvirt/virtlogd-sock': No such file or directory
Fix:
sudo /opt/homebrew/sbin/virtlogd -dThis will start virlodg as a daemon process manually
I have seen some inconsistent behaviour when running virt-install which means the command hangs. I have had to send 'Ctrl + C' - wait for the process to exit (which can take up to a few minutes) then re-run the same command a second time and it works without issue.
Trying to run the same in a new terminal resulted in it hanging again, so running twice in the same session was what worked for me.
Now that this is working, my next steps to make it more usable are:
- Automate the OS installer, or see if disk images can be used
- Configure domains via XML template
- Wrap it with Vagrant to provision everything in one go
Great, I knew about
qemucommandline, but for some reason didn't think to use it before and kept looking for a way to use libvirt with virtual networks on Mac M1. That's a great job. Thank you.