action plan for major Q1 project, RISC-V hypervisor

hypervisor.md

1. Initial definition of new SBI
   1. sbi_config_string_base is a problem because chain loaders may want to revise the configuration. Pass a pointer in a0 instead.
   2. Memory map in the config string instead of sbi_query_memory. Config string section for "bootstrap page tables" to eliminate walking requirement.
   3. ECALLs are defined in a new section of the config string rather than being simply assumed. Most (eventually all?) ECALLs become optional.
   4. ECALLs are individually versioned. Config string blocks become more explicit; sifive_uart_v0 and sifive_uart_v1 can't use the same drivers anyway.
   5. Yes you can chainload BBL-SBI to sorear-SBI.
   6. Define the missing ECALLs for enable/disable paging, mucounteren manipulation, etc. (Patch BBL to add them?)
   7. Maybe pin down a simpler load image format (not ELF, entry occurs with S-paging disabled). Needs research.
2. Write a classical hypervisor
   1. Run all guest code in U-mode
   2. Maintain one or more shadow page tables with LRU caching managed by the page fault handler
   3. SPTBR and SFENCE.VM are emulated to control the MMU
   4. Other CSRs are emulated to control the virtual privilege state and interrupt model
   5. Emulate MMIO hardware, PLIC etc etc.
   6. Implements the new and old SBI natively. Make sure it boots Linux and itself
   7. riscvemu + virtio forwarding?
3. SIE 1: S/U mode switches
   1. Assume the hardware has 4 modes, S U SV UV
   2. SV-mode has its own sscratch, etc separate from S-mode; also there is a PTBR for SV/UV separate from the one for S/U
   3. not accessed using CSRs; instead there is a sbi_sie ECALL which loads all SV-registers from a "siebk" data structure, runs in SV/UV until a trap is pending, then saves and switches back to S/U
   4. SV-mode still traps on SFENCE or SPTBR, this only speeds up "syscall activity"
   5. Hypervisor modified to use sbi_sie if available.
   6. Yes there is a patent covering SIE. Issued 1984, and I have documentation dated 1986.
   7. sbi_siestop for IPIs; details TBD
4. SIE 2: sbi_sie implementation
   1. From the hardware POV the only necessary difference between SV and S is that SV takes a trap on SPTBR + SFENCE
   2. Patch spike to add a mstatus bit for this. the mstatus bit is not on the standards track, sbi_sie is
   3. M-mode code can now implement sbi_sie by swapping register sets and leaving the CPU in S+trapping sfence mode
   4. The classical hypervisor can also easily implement sbi_sie (since it traps SPTBR anyway), so even without hw support this speeds up nested operation
5. SIE 3: passthrough
   1. Optional mode which the host can pass to SIE so that SIE operations are not intercepted but handled directly in M-mode; this makes deep nesting quadratic instead of exponential
6. SIE 4: use in OSes
   1. Linux could use SIE to run user apps instead of the default RISC-V trap protocol, which would have the curious advantage of reducing the number of exits in a classical virtualization context
   2. May want a config string boolean sie_is_faster_than_stvec
   3. If you take this far enough it's possible to have a design with only M+S modes
7. Nested paging
   1. Add a new mode to spike for nested paging operation, with m2pbtr translating addresses that come out of the spbtr translation
   2. Same page table format as SPTBR to make Ron Minnich happy. Although there's some interesting research that 1-level page tables work better here?
   3. Extend sbi_sie with an optionally-supported mode where the guest is allowed to set SPTBR and use SFENCE, but everything is translated using a second-level table (possibly taken from the host SPTBR, or a new slot in siebk)
   4. Invalidation TBD, nested operation also TBD
8. Device passthrough, self-virtualizing PLIC, IOMMUs
   1. Probably not feasible in Q1 timeframe
9. Get a FPGA dev kit and make a working hardware demo
   1. Probably not feasible in Q1, although it'd be a good excuse to become familiar with hw stuff
10. Port KVM
    1. May not be feasible in a 2017 timeframe
11. Explore the ECALL-only and bus-only ends of the design space
    1. For embedded devices and owned hardware, there's no meaningful administrative boundary between M-mode and S-mode.
    2. So from an alignment-of-incentives perspective it may make sense for such devices to either make all necessary functionality directly accessible from S-mode and boot in S-mode, or else to deliberately puncture the privilege boundary to reflect the lack of administrative division
    3. Conversely MMIO under a hypervisor requires some instruction set knowledge, and maximum robustness against 3rd-party ISA extensions would benefit from an ECALLs-only SBI design point