Skip to content

Instantly share code, notes, and snippets.

@henry40408
Last active June 4, 2022 12:45
Show Gist options
  • Save henry40408/b079e3dc2dad9c0bc44f7d55b9e31b29 to your computer and use it in GitHub Desktop.
Save henry40408/b079e3dc2dad9c0bc44f7d55b9e31b29 to your computer and use it in GitHub Desktop.
How does Harvester mount volumes from Longhorn to virtual machine?

Q: How does Harvester mount volumes from Longhorn on virtual machine?

Q1: How does Harvester start a new virtual machine?

HTTP server route handler:

https://github.com/harvester/harvester/blob/77b7f00c07ef520e2d213438d3bf525227bbb5d3/pkg/api/vm/handler.go#L93-L96

Call Kubernetes REST client directly:

https://github.com/harvester/harvester/blob/7f458945f92f713daa01ac03b3fd7f52bd8fccdf/pkg/api/vm/handler.go#L174-L176

End of investigation on Harvester side.

Since I knew that Harvester is based on KubeVirt, I went to KubeVirt next.

Refer to https://kubevirt.io/user-guide/architecture/, I knew that virt-launcher is responsible to communicate with libvirt:

https://github.com/kubevirt/user-guide/blob/9c998557affa9ad478155637ed53d11932824109/docs/assets/architecture.png

So I searched something like CreateVMI in virt-launcher but I found nothing.

So I alternatively located VirtualMachineInstance struct, see if any method in virt-launcher used it, and I found about 143 usages. I filtered methods one by one, then I found a method called SyncVMI.

https://github.com/kubevirt/kubevirt/blob/62e3268e0dd58d06b292664c8aa149b9f6c552ed/pkg/virt-launcher/virtwrap/manager.go#L745-L874

It called CreateWithFlags to create virtual machine:

http://github.com/kubevirt/kubevirt/blob/62e3268e0dd58d06b292664c8aa149b9f6c552ed/pkg/virt-launcher/virtwrap/manager.go#L802-L802

...which was actually a C binding to libvirt:

https://github.com/kubevirt/kubevirt/blob/6c4c2b78eb2503d8ab87440ab324407d7b93cf73/vendor/libvirt.org/go/libvirt/domain.go#L1008-L1015

Surprisingly, libvirt called virtual machine domain, which was very unusual.

Q2: How does Harvester mount volumes?

Followed how I resolved the previous question, I focused on virt-launcher. SyncVMI called getAttachedDisks to get list of disk to be attached on virtual machine:

https://github.com/kubevirt/kubevirt/blob/62e3268e0dd58d06b292664c8aa149b9f6c552ed/pkg/virt-launcher/virtwrap/manager.go#L851-L870

...but after I check checkIfDiskReadyToUse method, I was getting confusing:

http://github.com/kubevirt/kubevirt/blob/62e3268e0dd58d06b292664c8aa149b9f6c552ed/pkg/virt-launcher/virtwrap/manager.go#L852-L852

It called os.Stat internally to check whether the disk is ready, so the disk must existed on node before SyncVMI got things done:

http://github.com/kubevirt/kubevirt/blob/62e3268e0dd58d06b292664c8aa149b9f6c552ed/pkg/virt-launcher/virtwrap/manager.go#L886-L915

Since Longhorn can provide volume as iSCSI target, I presumed one or more components must do the following things...

  1. A component requests Longhorn mount a volume as iSCSI target
  2. A component mounts the iSCSI target on specific nodes
  3. A component on specific node receives virtual machine or virtual machine instance request from scheduler, starts a new virtual machine and mount the iSCSI target in traditional Linux way (that's why we need open-iscsi package installed on node), and manipulates qemu to mount the disk

Then I found nothing related in code-base, so I go back to documentation, search for keyword "iSCSI", then I found a Markdown file:

https://github.com/kubevirt/kubevirt/blob/b6946cdc0ea75cc862ceccc7e05a6f66b77ca9f1/docs/direct-pv-disks.md#virt-handler-behavior

So I put it simply:

  1. User puts information about iSCSI target on Kubernetes cluster, which is built-in function: https://github.com/kubernetes/website/blob/b50819e5f9d64784f5de2ee620ef667d59280efe/content/en/docs/concepts/storage/volumes.md#iscsi
  2. virt-handler converts VMI spec (YAML on Kubernetes) to libvirt configuration (XML on node)
  3. libvirt mounts iSCSI target directly with XML configuration
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment