max-i-mil/linux-vms-on-apple-m1-with-networking.md

## linux-vms-on-apple-m1-with-networking.md

      
    Raw
  

              linux-vms-on-apple-m1-with-networking.md
            
          
    Linux Virtual Machines with Private Network on an Apple M1 Device

Background

The aim was to be able to:

Run multiple Linux VMs on an Apple M1/ARM device
Use Apple's HVF for native performance speeds
Configure VMs to allow network access to each other
Configure VMs to allow access to the internet
Not rely on custom modifications of software

I had originally assumed I would be up and running with the above in a few hours, but I was wrong. After looking for examples online, of which there are many. The problem was that they were seemingly all based on either Intel based Apple devices, or for MacOS < 12. I'm using a 2021 MacBookPro with an M1 Pro chip running MacOS 12.6 (Monterey). I have therefore spent a lot of time finding out what doesn't work anymore.
I was also new to some of the tools I have used including QEMU and libvirt.
After much experimentation and failure, I have a working example to meet the goals set out. A lot of the issues faced seemed to be that changes to MacOS over the past few years and the time it takes to reflect this in software, has meant a period between what used to work, things not working, and now working again with different implementations. The end result achieved using the process defined below looks similar to the following:

Many examples I found online to help meet goal 3 relied on TAP devices. As I understand these were not natively supported by MacOS and required kernel extensions (kexts) to enable the functionality - commonly through a tool called 'tuntaposx'. As stated on their website the project has not been maintained since 2015 and references the requirement from Apple to use notarization on all updated or new kernel extensions which tuntaposx does not have. Tunnelblick details the situation here.
There are other helper tools available from QEMU/libvirt to deal with networking like qemu-bridge-helper / libvirt virtual networks - but they too don't yet seem to be working on my device for various reasons. Some of the current open issues from these projects detail more

Requirements

I have tested this using:

MacOS - 12.6.1 Monterey
Chip - M1 Pro
QEMU - 7.1.0
libvirt - 8.9.0
virt-manager - 4.1.0

As it's MacOS, I have used homebrew for package management. For convenience I am also using brew services as a wrapper for launchctl
brew install qemu libvirt virt-manager

# must use sudo as it affects qemu later
sudo brew services start libvirt
For native speeds you will need an OS ISO image for AARCH64 hosts (ARM). x86 ISO files will work but because Apple M1 devices use an ARM architecture they will need QEMU to emulate the necessary hardware which is much slower, and you may need to alter the command given later to use 'tcg' instead of 'hvf'.
I went with RHEL 9 as RHEL 7/8 wont work natively due to page sizes
Basic libvirt + QEMU knowledge is also required.
The working solution relies on the latest version of QEMU (7.1.0 as of Nov 2022). Some of the functionality has been added, with the man page updated, but with the QEMU wiki docs online not yet featuring the same detail on the invocation page.

Process

With dependencies installed - you can now create a VM using a single command, and everything else will be handled automatically. You can run the same command again to create a second VM.
# create a libvirt 'domain' - specify some options using qemu command line pass through
sudo virt-install \
--name host1 \
--memory 2048 \
--vcpus 2 \
--disk size=30 \
--cdrom /path/to/ISO_files/rhel-baseos-9.0-aarch64-dvd.iso \
--os-variant rhel9.0 \
--virt-type hvf \
--qemu-commandline='-M highmem=off -netdev vmnet-shared,id=net0 -device virtio-net-device,netdev=net0,mac=54:54:00:55:54:51' \
--network user

# create a second libvirt 'domain' - ensure the friendly name and MAC address of the net device are unique and run the same command again
sudo virt-install \
--name host2 \
--memory 2048 \
--vcpus 2 \
--disk size=30 \
--cdrom /path/to/ISO_files/rhel-baseos-9.0-aarch64-dvd.iso \
--os-variant rhel9.0 \
--virt-type hvf \
--qemu-commandline='-M highmem=off -netdev vmnet-shared,id=net0 -device virtio-net-device,netdev=net0,mac=54:54:00:55:54:52' \
--network user

# create Nth libvirt domain - just keep name and MAC address unique
sudo virt-install \
--name hostN \
--memory 2048 \
--vcpus 2 \
--disk size=30 \
--cdrom /path/to/ISO_files/rhel-baseos-9.0-aarch64-dvd.iso \
--os-variant rhel9.0 \
--virt-type hvf \
--qemu-commandline='-M highmem=off -netdev vmnet-shared,id=net0 -device virtio-net-device,netdev=net0,mac=NN:NN:NN:NN:NN:NN' \
--network user
Notable flags are:

--virt-type hvf tell QEMU to use the Apple Hypervisor Framework instead of an alternative such as TCG. This enables the native speeds

--network user enable QEMU user mode networking, which gives us access to the host and the internet. Alternative is 'none' which removes access to both. Other named libvirt networks do not seem to work on the current version for my device i.e. '--network name=default'


From the --qemu-commandline flag

-netdev vmnet-shared tells QEMU to use the vmnet APIs provided by Apple as part of HVF. This is the alternative to managing our own bridge/TAP devices commonly used in the past. Flag alternatives are vmnet-host and vmnet-bridged

-device mac=XX:YY... defines the MAC address used for the virtual device that will be attached to our VM. Must be unique per VM for DHCP

I wasn't able to find virt-install equivalents for 'vmnet-*' netdev options yet hence the qemu-commandline workaround but this may come in a future release

In my case I connected to each VM via the console to complete the installation process before continuing.
This has now done a few things in the background:
On the Apple host:


A bridge device has been created called 'bridge100' (sequential from 100 if you start multiple vmnet types at once)
A vmenet device has been created for each VM requested, and then attached to the bridge
An internal macOS DHCP service (bootpd) has been started to assign IP addresses to each VM (and the bridge)
The QEMU user mode network (SLIRP) has been created which gives internet access over the default gateway 10.0.2.2

# some output omitted
root@apple-host$ ifconfig
...
vmenet0: flags=8b63<UP,BROADCAST,SMART,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
        ether aa:70:70:52:86:d4 
        media: autoselect
        status: active
# bridge100 has IP 192.168.64.1 assigned
# vmenet0 and vmenet1 are members
bridge100: flags=8a63<UP,BROADCAST,SMART,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
        options=3<RXCSUM,TXCSUM>
        ether be:d0:74:61:f6:64 
        inet 192.168.64.1 netmask 0xffffff00 broadcast 192.168.64.255
        ...
        Configuration:
        ...
        member: vmenet0 flags=3<LEARNING,DISCOVER>
                ifmaxaddr 0 port 29 priority 0 path cost 0
        member: vmenet1 flags=3<LEARNING,DISCOVER>
                ifmaxaddr 0 port 31 priority 0 path cost 0
        ...
        status: active
vmenet1: flags=8b63<UP,BROADCAST,SMART,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
        ether b6:69:a5:54:74:45 
        media: autoselect
        status: active
<!-- Contents of /etc/bootpd.plist after creating VMs 
    dhcp_domain_name_server is same IP as bridge100-->
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>Subnets</key>
        <array>
                <dict>
                        <key>_creator</key>
                        <string>com.apple.NetworkSharing</string>
                        <key>allocate</key>
                        <true/>
                        <key>dhcp_domain_name_server</key>
                        <array>
                                <string>192.168.64.1</string>
                        </array>
                        <key>dhcp_router</key>
                        <string>192.168.64.1</string>
                        <key>interface</key>
                        <string>bridge100</string>
                        <key>lease_max</key>
                        <integer>86400</integer>
                        <key>lease_min</key>
                        <integer>86400</integer>
                        <key>name</key>
                        <string>192.168.64/24</string>
                        <key>net_address</key>
                        <string>192.168.64.0</string>
                        <key>net_mask</key>
                        <string>255.255.255.0</string>
                        <key>net_range</key>
                        <array>
                                <string>192.168.64.2</string>
                                <string>192.168.64.254</string>
                        </array>
                </dict>
        </array>
...
On the guest Linux VMs:

# IP config of first VM - some output omitted

# enp1s0 is on the QEMU SLIRP network and has access to host + internet
# eth1 is attached to bridge100 and has IP 192.160.64.2 assigned via DHCP - has access via bridge to other VMs we create
# eth1 MAC address matches that defined by 'virt-install'
[root@localhost ~] ip a s
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:d1:d1:8d brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute enp1s0
       valid_lft 84794sec preferred_lft 84794sec
    ...
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 54:54:00:55:54:51 brd ff:ff:ff:ff:ff:ff
    inet 192.168.64.2/24 brd 192.168.64.255 scope global dynamic noprefixroute eth1
       valid_lft 84794sec preferred_lft 84794sec
    ...
# IP config of second VM - some output omitted

# enp1s0 has same IP as VM1 as the SLIRP network for each VM is independent so it isn't clashing
# eth1 has IP 192.168.64.3 assigned
[root@localhost ~] ip a s
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:ae:7e:5c brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute enp1s0
       valid_lft 85064sec preferred_lft 85064sec
    ...
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 54:54:00:55:54:52 brd ff:ff:ff:ff:ff:ff
    inet 192.168.64.3/24 brd 192.168.64.255 scope global dynamic noprefixroute eth1
       valid_lft 85064sec preferred_lft 85064sec
    ...
# Routing tables used by both VMs
[root@localhost ~] ip route
default via 10.0.2.2 dev enp1s0 proto dhcp metric 100 
default via 192.168.64.1 dev eth1 proto dhcp metric 101 
10.0.2.0/24 dev enp1s0 proto kernel scope link src 10.0.2.15 metric 100 
192.168.64.0/24 dev eth1 proto kernel scope link src <host_specific_IP> metric 101
Each VM can now ping the other VMs that share the same bridge (all started with the same vmnet parameter). Each VM can also make outbound internet connections via the SLIRP network.
If you pass --network  none to virt-install then you will not have the enp1s0 device attached within your VM and it will have no internet access, but it will still be able to access the other VMs using eth1

Result

With everything configured as above the setup allows the following:
Access VM from Host
# Run from Apple host
user@apple-host$ ping 192.168.64.2
PING 192.168.64.2 (192.168.64.2): 56 data bytes
64 bytes from 192.168.64.2: icmp_seq=0 ttl=64 time=1.713 ms

--- 192.168.64.2 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.713/1.713/1.713/0.000 ms
Access VM from VM
# Run from VM1 with IP 192.168.64.2
[root@localhost ~] ping 192.168.64.3
PING 192.168.64.3 (192.168.64.3) 56(84) bytes of data.
64 bytes from 192.168.64.3: icmp_seq=1 ttl=64 time=2.34 ms

--- 192.168.64.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.342/2.342/2.342/0.000 ms
Access Host from VM
# Run from VM1
[root@localhost ~] ping 10.0.2.2
PING 10.0.2.2 (10.0.2.2) 56(84) bytes of data.
64 bytes from 10.0.2.2: icmp_seq=1 ttl=255 time=0.868 ms

--- 10.0.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.868/0.868/0.868/0.000 ms
Fixes

Some errors I encountered along the way but not sure if they would be consistent for everyone
1 - Add default URI for libvirt connections

When running virt-install
Error:

error: failed to connect to the hypervisor

error: Operation not supported: Cannot use direct socket mode if no URI is set

Fix:
echo 'uri = qemu:///system' >> /opt/homebrew/etc/libvirt/libvirt.conf

brew services restart libvirt
This provides the default URI to connections to QEMU system instances
2 - Manually start virtlogd

When starting libvirt - the expectation was that libvirt would manage virtlogd, which is used to manage logs from VMs as an alternative to writing them all to a file which need manual disk management.
Error:

error: failed to connect to the hypervisor

error: Failed to connect socket to '/opt/homebrew/var/run/libvirt/virtlogd-sock':  No such file or directory

Fix:
sudo /opt/homebrew/sbin/virtlogd -d
This will start virlodg as a daemon process manually
3 - Hanging when running virt-install

I have seen some inconsistent behaviour when running virt-install which means the command hangs. I have had to send 'Ctrl + C' - wait for the process to exit (which can take up to a few minutes) then re-run the same command a second time and it works without issue.
Trying to run the same in a new terminal resulted in it hanging again, so running twice in the same session was what worked for me.
Follow On

Now that this is working, my next steps to make it more usable are:

Automate the OS installer, or see if disk images can be used
Configure domains via XML template
Wrap it with Vagrant to provision everything in one go