- Organization: QEMU
- Project: Fast Snapshot Load
- Contributor: Aadeshveer Singh aadeshveer07@gmail.com
- Mentor: Peter Xu peterx@redhat.com
The VM snapshot load feature in QEMU utilizes the precopy migration infrastructure. This mandates loading of entire snapshot image in memory before the guest can be allowed to run. Its performance and user experience can be significantly improved by using postcopy migration. This allows the guest to run nearly instantly with memory being loaded later.
Since QEMU primarily relies on qemu-devel mailing list for submitting changes,
most of my work has been submitted on the mailing list.
For a detailed dev log tracking my progress throughout the project, see my blog.
| Patch series version | Link to patch series | Summary |
|---|---|---|
| 1 | [RFC PATCH 0/5] | Basic structural implementation pushed as RFC to gather feedback of overall architectural design. Implemented a two thread model with on demand and background snapshot loading. A non polished working prototype |
| 2 | [RFC PATCH v2 0/7] | Made code modular and restructured code to prevent side effects based on input received on RFC v1. |
| 3 | [PATCH v3 00/11] | Added testing and documentation for new feature, dropping RFC tag. Addressed feedback received on RFC v2. |
| 4 | [PATCH v4 00/11] | Resolved bug related to handling different host and guest page sizes. |
| 5 | [PATCH v5 00/11] | Addressed side effects caused due to enabling new feature resulting in a stable 11 patch implementation of feature |
Post these changes, user will be able to enable postcopy-ram and mapped-ram simultaneously to enable fast snapshot load.
When enabled incoming migration from a file (snapshot load) will use postcopy migration to load the guest.
Initially only the guest device state is loaded, which is instantaneous allowing the guest to start running immediately.
As the guest tries to use some memory it will cause page faults which are caught and handled using userfaultfd.
This involved formulating a new page loading path that is robust to variations in page sizes and fits in current migration infrastructure.
A background thread also keeps loading pages sequentially to prevent relying on guest to access all pages for a complete migration.
The feature postcopy-blocktime is also enabled to gather data and observe latencies on handling page faults.
A smoke test and documentation have also been provided.
Usual snapshot load for a 16GiB guest can take a few seconds (on modern NVMe SSDs) or more during the copying of data from disk to memory phase. During this time the guest cannot run.
Using fast snapshot load (as implemented in this project), the load time becomes insignificant.
Guest CPUs are blocked waiting on page fault for a fraction of second with guest running readily.
Version 5 of the patch series is undergoing a final round of reviews on the list, with majority of patches already reviewed.
Improvements possible in future:
- Maintaining more data like bandwidth of load and progress.
- Implement features like bandwidth throttling.
- Add support for using
multifd - Handle cases if guest page size is greater than host page size.
During this project, I faced many challenges, starting from using mailing lists which were alien to me, to various bugs related to guest/host memory pages. During the first weeks, a major challenge was to actually understand the codebase. With the sheer size of the codebase, I would usually find myself getting lost and digressing into reading about stuff not really related to my project. This was compounded by my initial lack of knowledge about some virtualization concepts, which my mentor helped me through by always being there to answer my questions.
Once the coding started, I had to learn a lot of git gymnastics and the use of tools for interacting with the mailing list. The first-hand experience of getting my code reviewed rigorously was another fun thing I was introduced to.
In summary, this has been an amazing experience. I started as an undergrad with basic knowledge about virtualization and grew into someone who understands much more about it. I won't say I have mastered this concept, as it runs very deep and will take years, but it is a solid beginning for me.
Big thanks to my amazing mentor, Peter for guiding me at every step and helping me grow.