How does Harvester mount volumes from Longhorn to virtual machine?

README.md

Q: How does Harvester mount volumes from Longhorn on virtual machine?

Q1: How does Harvester start a new virtual machine?

HTTP server route handler:

https://github.com/harvester/harvester/blob/77b7f00c07ef520e2d213438d3bf525227bbb5d3/pkg/api/vm/handler.go#L93-L96

Call Kubernetes REST client directly:

https://github.com/harvester/harvester/blob/7f458945f92f713daa01ac03b3fd7f52bd8fccdf/pkg/api/vm/handler.go#L174-L176

End of investigation on Harvester side.

Since I knew that Harvester is based on KubeVirt, I went to KubeVirt next.

Refer to https://kubevirt.io/user-guide/architecture/, I knew that virt-launcher is responsible to communicate with libvirt:

https://github.com/kubevirt/user-guide/blob/9c998557affa9ad478155637ed53d11932824109/docs/assets/architecture.png

So I searched something like CreateVMI in virt-launcher but I found nothing.

So I alternatively located VirtualMachineInstance struct, see if any method in virt-launcher used it, and I found about 143 usages. I filtered methods one by one, then I found a method called SyncVMI.

https://github.com/kubevirt/kubevirt/blob/62e3268e0dd58d06b292664c8aa149b9f6c552ed/pkg/virt-launcher/virtwrap/manager.go#L745-L874

It called CreateWithFlags to create virtual machine:

http://github.com/kubevirt/kubevirt/blob/62e3268e0dd58d06b292664c8aa149b9f6c552ed/pkg/virt-launcher/virtwrap/manager.go#L802-L802

...which was actually a C binding to libvirt:

https://github.com/kubevirt/kubevirt/blob/6c4c2b78eb2503d8ab87440ab324407d7b93cf73/vendor/libvirt.org/go/libvirt/domain.go#L1008-L1015

Surprisingly, libvirt called virtual machine domain, which was very unusual.

Q2: How does Harvester mount volumes?

Followed how I resolved the previous question, I focused on virt-launcher. SyncVMI called getAttachedDisks to get list of disk to be attached on virtual machine:

https://github.com/kubevirt/kubevirt/blob/62e3268e0dd58d06b292664c8aa149b9f6c552ed/pkg/virt-launcher/virtwrap/manager.go#L851-L870

...but after I check checkIfDiskReadyToUse method, I was getting confusing:

http://github.com/kubevirt/kubevirt/blob/62e3268e0dd58d06b292664c8aa149b9f6c552ed/pkg/virt-launcher/virtwrap/manager.go#L852-L852

It called os.Stat internally to check whether the disk is ready, so the disk must existed on node before SyncVMI got things done:

http://github.com/kubevirt/kubevirt/blob/62e3268e0dd58d06b292664c8aa149b9f6c552ed/pkg/virt-launcher/virtwrap/manager.go#L886-L915

Since Longhorn can provide volume as iSCSI target, I presumed one or more components must do the following things...

1. A component requests Longhorn mount a volume as iSCSI target
2. A component mounts the iSCSI target on specific nodes
3. A component on specific node receives virtual machine or virtual machine instance request from scheduler, starts a new virtual machine and mount the iSCSI target in traditional Linux way (that's why we need open-iscsi package installed on node), and manipulates qemu to mount the disk

Then I found nothing related in code-base, so I go back to documentation, search for keyword "iSCSI", then I found a Markdown file:

https://github.com/kubevirt/kubevirt/blob/b6946cdc0ea75cc862ceccc7e05a6f66b77ca9f1/docs/direct-pv-disks.md#virt-handler-behavior

So I put it simply:

1. User puts information about iSCSI target on Kubernetes cluster, which is built-in function: https://github.com/kubernetes/website/blob/b50819e5f9d64784f5de2ee620ef667d59280efe/content/en/docs/concepts/storage/volumes.md#iscsi
2. virt-handler converts VMI spec (YAML on Kubernetes) to libvirt configuration (XML on node)
3. libvirt mounts iSCSI target directly with XML configuration