Notes about performance counters for Virtualbox

pmu_notes.txt

QUESTIONS ===
Which counters should be guest only and which should run in hypervisor as well?
    Vmware
        guest only
            instructions retired
            branches retired
        guest + hypervisor
            all other events

How much customization of the different kernel drivers are neccessary?
    Is it enough to use the wrappers (suplib)?

Which processors should be allowed?
    Nehalem and AMD Opteron gen 3 (greyhound)?
    Start with just Intel?
Which registers should be accessible?
    wmware:
        Intel CPUs
            IA32_PERFEVTSELx
            IA32_PMCx
            IA32_FIXED_CTRx
            IA32_PERF_GLOBAL_CTRL
            IA32_PERF_GLOBAL_STATUS
            IA32_PERF_GLOBAL_OVF_CTRL
            IA32_FIXED_CTR_CTRL
          AMD CPUs
            PERF_CTLx
            PERF_CTRx

Do we need hardware virtualization enabled?
    Looks like we can get hardware support for swapping the PMU registers with VT-x!
How make special case for inst retired and branches retired?
    What code to look in?
How check if performance counters are available?
    Need to run cpuid
        Need to check user has allowed performance counters as well
            Virtualbox conf directive
                Can wait 'til the end
How receive interrupts?
    Does Virtualbox emulate the APIC?
How store counter values on context switch?
How swap counter values on vm exit/entries?
Do we need to make adjustments between vbox notion of time and what gets
reported by the PMC? E.g. can ratios become misleading?
    Perhaps read up on performance measurements by others on VmWare and KVM
Do we ever need to emulate counters?
    vmware does it for 
        unhalted core cycles
        unhalted reference cycles 
            why?

TODO ===
[x] Build vbox
[x] Skim Intel docs for hw virtualization and performance counters
[ ] Send initial RFC mail
[ ] Experiment with builtin debugger
[ ] Figure out general call sequence when starting up a vm
[ ]

RESOURCES TO INVESTIGATE ===

Description of how to add performance counters to KVM.
Du, Jiaqing, Nipun Sehrawat, and Willy Zwaenepoel. "Performance Profiling in a Virtualized Environment." HotCloud. 2010.
https://www.usenix.org/legacy/event/hotcloud10/tech/full_papers/Du.pdf

Discussion about add performance counters to VmWare.
Serebrin, Benjamin, and Daniel Hecht. "Virtualizing performance counters." Euro-Par 2011: Parallel Processing Workshops. Springer Berlin Heidelberg, 2011.

User manual description of Vmware performance counters
[1] https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2030221

Discussions about tradeoffs when doing profiling with PMU's inside VmWare.
[2] http://performancebydesign.blogspot.se/2013/12/how-windows-performance-counters-are.html
[4] http://vpivot.com/2010/02/10/inaccuracy-of-in-guest-performance-counters/
[5] http://www.vmware.com/files/pdf/Timekeeping-In-VirtualMachines.pdf
[6] https://www.vmware.com/pdf/VI3.5_Performance.pdf

KVM patch series for ARM64 support on KVM. An example of how to deal with PMU registers (on another architecture though)
[7] http://thread.gmane.org/gmane.comp.emulators.kvm.devel/137754

x86 KVM PMU support
[8] http://lxr.free-electrons.com/source/arch/x86/kvm/pmu.c

KVM Delivering pmu events through local APIC on x86
[9] http://lxr.free-electrons.com/source/arch/x86/kvm/x86.c#L6421

Old patch for KVM in-guest performance monitoring
[10] http://thread.gmane.org/gmane.comp.emulators.kvm.devel/81637

Another old KVM patch-set
[11] http://thread.gmane.org/gmane.comp.emulators.kvm.devel/71892

VT-x and performance counters in KVM mode
[12] http://thread.gmane.org/gmane.comp.emulators.kvm.devel/1947/focus=1959

SOURCE CODE EXPEDITIONS ===

Where is the source code for the linux driver?
	There is a dir in out/linux.amd64/debug/bin/src/vboxdrv
		Appears to be generated
Looks like the Host driver is somehow associated with the VMM, but how?
    Do the VMM run in kernel context or as a user service?
        Expect kernel to improve performance?
Inside SUPDrv.c there's a massive list of global functions
    Are those the ones used to call from the host specific driver into the
    generic driver?
How do we do a context switch?

What is called during startup?
    Here reversed stacktrace. Functions called later apperas below. Functions
    called from inside another function are indented.

Console::i_powerUpThread(RTTHREAD Thread, void *pvUser)
    VMR3Create(...)
        vmR3CreateUVM(...)
            vmR3CreateU(...)
                PDMR3LdrLoadVMMR0U(pUVM);
                CFGMR3Init(...pUVM);
                vmR3InitRing3(pVM, pUVM);
                    HMR3Init(pVM);             // Init VT-x or AMD-v hardware virtualization
                    MMR3Init(pVM);             // Setup hypervisor memory
                    CPUMR3Init(pVM);           // Check cpu capabilities, setup context block
                    PGMR3Init(pVM);            // Init paging of VM
                    REMR3Init(pVM);            // Init recompiler
                    MMR3InitPaging(pVM);       // Init MM parts that depends on the Page manager
                    TMR3Init(pVM);             // Init Time manager
                    FTMR3Init(pVM);            // Init Fault Tolerance Manager
                    VMMR3Init(pVM);            // Init Virtual Machine Monitor
                    SELMR3Init(pVM);           // Init Selector Manager
                    TRPMR3Init(pVM);           // Init Trap Monitor
                    CSAMR3Init(pVM);           // Init Guest code Scan and Analysis Manager
                    PATMR3Init(pVM);           // Init Dynamic OS Guest Patch Manager
                    IOMR3Init(pVM);            // Input/Output monitor
                    EMR3Init(pVM);             // Init Execution Monitor
                    IEMR3Init(pVM);            // Init Interpreted Execution Monitor
                    DBGFR3Init(pVM);           // Init Debugger fascility
                    GIMR3Init(pVM);            // Init Guest Interface Manager
                    PDMR3Init(pVM);            // Init Pluggable Device Manager
                    PGMR3InitDynMap(pVM);
                    MMR3HyperInitFinalize(pVM);
                    PGMR3InitFinalize(pVM);
                    SELMR3InitFinalize(pVM);
                    TMR3InitFinalize(pVM);
                    REMR3InitFinalize(pVM);
                    PGMR3MemSetup(pVM, false /*fAtReset*/);
                    PDMR3MemSetup(pVM, false /*fAtReset*/);
                    vmR3InitDoCompleted(pVM, VMINITCOMPLETED_RING3);
                        VMMR3InitCompleted(pVM, enmWhat);
                        HMR3InitCompleted(pVM, enmWhat);
                        PGMR3InitCompleted(pVM, enmWhat);
                        SSMR3RegisterStub(pVM, "CSAM", 0);
                        rc = SSMR3RegisterStub(pVM, "PATM", 0);
                vmR3InitRing0(pVM);
                        VMMR3InitR0(pVM)
                        vmR3InitRC(pVM);
                VMMR3InitRC