Skip to content

Instantly share code, notes, and snippets.

@neggles
Last active June 18, 2024 13:09
Show Gist options
  • Save neggles/e35793da476095beac716c16ffba1d23 to your computer and use it in GitHub Desktop.
Save neggles/e35793da476095beac716c16ffba1d23 to your computer and use it in GitHub Desktop.
Hyper-V GPU Virtualization

Configuring GPU-PV on Hyper-V

This works on a Windows Pro 10 version 2004 or newer machine, confirmed to function with nVidia GPUs and recent AMD GPUs. Intel should work as well, but why do you want that?

I am not in any way responsible for working this out, that credit goes to Nimoa at cfx.re and reddit users on r/hyperv

  1. Make sure you have Hyper-V enabled (no way!) - there's a list of other features below that you might need, enable those if it doesn't work.
  2. Create a new VM using New-GPUPVirtualMachine.ps1 below and install Windows 10 on it.
  3. Gather driver files for your guest using one of the two methods below:

Using the driver gathering script (recommended)

Download the New-GPUPDriverPackage.ps1 script from this gist, it will gather the various files for you - it must be run as admin, make sure you either specify a destination path or run it from the folder you'd like the .zip created in.

  1. Run the script on your host, in an admin PowerShell session. It will create GPUPDriverPackage-[date].zip in the current directory, or a path specified with -Destination <path>.
  2. Copy the .zip to your guest VM and extract it.
  3. Copy the contents of the extracted GPUPDriverPackage folder into C:\Windows\ on the guest VM
  4. Reboot the guest, and enjoy your hardware acceleration!

This has been tested on nVidia, AMD, and Intel GPU drivers.
(Most Intel iGPUs have support for GPU-P, though I've not tested for Quick Sync Video support in guests yet)

Gathering driver files manually

This only covers nVidia drivers, but the process is very similar for Intel and AMD.

  1. On your host system:
    1. Browse to C:\Windows\system32\DriverStore\FileRepository
    2. Find the nvdispsi.inf_amd64_<guid> and/or nvltsi.inf_amd64_<guid> folders, and copy them to a temporary folder
    3. View driver details in device manager on your host, and copy all the files you see listed in System32 or SysWOW64 into matching folders inside your temporary folder.
  2. On your guest system:
    1. Browse to C:\Windows\system32\HostDriverStore\FileRepository
      (You will likely need to create the HostDriverStore and FileRepository directories)
    2. Copy the two driver folders you collected from your host to this path.
    3. Copy nvapi64.dll (and the other to C:\Windows\system32\ on the guest as well
  3. Shut down the guest VM,
  4. Make sure the VM's checkpoints are disabled, and automatic stop action is set to 'Turn Off'
    (The VM creation script covers this, but never hurts to be sure)
  5. Boot your VM, and enjoy your hardware acceleration!

Windows features that must be enabled:

  • Hyper-V
  • Windows Subsystem for Linux*
  • Virtual Machine Platform

* WSL/WSL2 is probably not necessary, but this functionality is present in Windows 10 to allow for CUDA support in WSL2, so it's probably a good idea to turn it on.

<#
.SYNOPSIS
Create a GPU-P Guest driver package.
.DESCRIPTION
Gathers the necessary files for a GPU-P enabled Windows guest to run.
.EXAMPLE
New-GPUPDriverPackage -DestinationPath '.'
.EXAMPLE
New-GPUPDriverPackage -Filter 'nvidia' -DestinationPath '.'
.INPUTS
None.
.OUTPUTS
A driver package .zip
.NOTES
This has some mildly dodgy use of CIM cmdlets...
.COMPONENT
PSHyperTools
.ROLE
GPUP
.FUNCTIONALITY
Creates a guest driver package.
#>
[CmdletBinding(
SupportsShouldProcess = $true,
PositionalBinding = $true,
DefaultParameterSetName = 'NoPathProvided',
HelpUri = 'http://www.microsoft.com/',
ConfirmImpact = 'Low')]
[Alias()]
[OutputType([String])]
Param (
# Path to output directory.
# If no file name is specified the filename will be GPUPDriverPackage-YYYYMMMDD.zip
[Parameter(
Mandatory = $false,
ParameterSetName = 'PathProvided',
HelpMessage = "Path to one or more locations.")]
[Alias("PSPath", "Path")]
[ValidateNotNullOrEmpty()]
[string]
$DestinationPath,
# Device friendly name filter.
# Only devices whose friendly names contain the supplied string will be processed
[Parameter(
Mandatory = $false,
HelpMessage = "Only add drivers for devices whose friendly names contain the supplied string.")]
[ValidateNotNullOrEmpty]
[String]
$Filter
)
process {
try {
# make me a temporary folder, and assemble the structure
$fTempFolder = Join-Path -Path $Env:TEMP -ChildPath "GPUPDriverPackage"
(New-Item -ItemType Directory -Path "$fTempFolder/System32/HostDriverStore/FileRepository" -Force -ErrorAction SilentlyContinue | Out-Null)
(New-Item -ItemType Directory -Path "$fTempFolder/SysWOW64" -Force -ErrorAction SilentlyContinue | Out-Null)
# Set default archive name
$ArchiveName = ('GPUPDriverPackage-{0}.zip' -f $(Get-Date -UFormat '+%Y%b%d'))
# Check if DestinationPath has been provided
if ($PSCmdlet.ParameterSetName -eq "PathProvided") {
switch ($DestinationPath) {
{ (Split-Path -Path $_ -Leaf) -match '(\.zip)' } {
# Path we've been provided is a full file path, so we'll just use it
$ArchiveFolder = Split-Path -Path $DestinationPath -Parent
$ArchiveName = Split-Path -Path $DestinationPath -Leaf
break
}
{ Test-Path -Path $_ -PathType Container } {
# Path exists and is a directory, so we place our file in it with the default name.
$ArchiveFolder = $DestinationPath
break
}
Default {
# Path doesn't end in .zip and doesn't exist, so we're going to assume it's a directory and place the file in it with the default name.
$ArchiveFolder = $DestinationPath
(New-Item -ItemType Directory -Path $ArchiveFolder -Force -ErrorAction SilentlyContinue | Out-Null)
break
}
}
} else {
# if DestinationPath not supplied, use current directory and default name
$ArchiveFolder = (Get-Location).Path
}
# just double check that one
if (-not $ArchiveFolder) { $ArchiveFolder = (Get-Location).Path }
Write-Output -InputObject ('Creating GPU-P driver package for host {0}' -f $Env:COMPUTERNAME)
Write-Output -InputObject ('Destination path: {0}' -f (Join-Path -Path $ArchiveFolder -ChildPath $ArchiveName))
<#
Determine which cmdlet we should use to gather the list of GPU-P capable GPUs.
On Windows builds before Server 2022/21H2, the cmdlet is 'Get-VMPartitionableGpu'
On later builds, it's 'Get-VMHostPartitionableGpu', and the old cmdlet just prints an error.
So, we check if Get-VMHostPartitionableGpu is a valid cmdlet to determine whether we should use it.
#>
Write-Output -InputObject "Getting all GPU-P capable GPUs in the current system..."
if (Get-Command -Name 'Get-VMHostPartitionableGpu' -ErrorAction SilentlyContinue) {
Write-Verbose -Message 'Using new Get-VMHostPartitionableGpu cmdlet'
$PVCapableGPUs = Get-VMHostPartitionableGpu
} else {
Write-Verbose -Message 'Using old Get-VMPartitionableGpu cmdlet'
$PVCapableGPUs = Get-VMPartitionableGpu
}
# if we found no GPU-P capable GPUs, throw an exception
if ($PVCapableGPUs.Count -lt 1) {
throw [System.Management.Automation.ItemNotFoundException]::new('Did not find any GPU-P capable GPUs in this system.')
} elseif ($PvGPUs.Count -gt 1) {
Write-Warning -Message (
("You have {0} GPU-P capable GPUs in this system. `n" -f $PvGPUs.Count) +
" At present, there is no way to control which one is assigned to a given VM.`n" +
" Unless one of the available GPUs is an intel IGP, it is highly recommended`n" +
" that you disable the GPU(s) you do not wish to use.`n")
$choices = '&Yes', '&No'
$question = 'Do you wish to proceed without disabling the extra GPU(s)?'
if ($Host.UI.PromptForChoice('', $question, $choices, 1) -eq 1) {
throw [System.Management.Automation.ActionPreferenceStopException]::new('User requested to cancel.')
}
}
# Map each PVCapableGPU to the corresponding PnPDevice. Regex (mostly) extracts the InstanceId from the VMPartitionableGpu 'name' property.
Write-Output -InputObject ('Mapping GPU-P capable GPUs to their corresponding PnPDevice objects...')
$InstanceExpr = [regex]::New('^\\\\\?\\(.+)#.*$')
$TargetGPUs = $PVCapableGPUs.Name | ForEach-Object -Process {
# I'm not proud of this dirty regex trick, but it works.
Get-PnpDevice -InstanceId $InstanceExpr.Replace($_, '$1').Replace('#', '\')
}
# OK, now that we have some actual device names, we can filter them if we've been asked to
if ($null -ne $Filter) {
Write-Output -InputObject ('Applying filter "{0}" to device list...' -f $Filter)
$TargetGPUs = $TargetGPUs | Where-Object { $_.FriendlyName -like ('*{0}*' -f $Filter) }
}
Write-Output -InputObject ('Will create driver package for {0} GPUs:' -f $TargetGPUs.Count)
$TargetGPUs.FriendlyName | ForEach-Object { Write-Output -InputObject (' - {0}' -f $_) }
} catch { throw $PSItem }
# Last chance to turn back, traveler. Are you sure?
if ($pscmdlet.ShouldProcess("Driver Package", "Create")) {
try {
Write-Output -InputObject ('The next few steps may take some time, depending on how many devices & driver packages are installed.')
Write-Output -InputObject ('If the script appears hung, please give it a few minutes to complete before terminating.')
# Get display class devices
Write-Output -InputObject ('Gathering display device CIM objects...')
$PnPEntities = Get-CimInstance -ClassName 'Win32_PnPEntity' | Where-Object { $_.Class -like 'Display' }
Write-Verbose -Message ('Found {0} display devices' -f $PnPEntities.Count)
($PnPEntities | Format-Table -AutoSize | Out-String).Trim().Split("`n") | ForEach-Object -Process { Write-Verbose -Message (' {0}' -f $_) }
# Get display class drivers
Write-Output -InputObject ('Gathering display device driver CIM objects...')
$PnPSignedDrivers = Get-CimInstance -ClassName 'Win32_PnPSignedDriver' -Filter "DeviceClass = 'DISPLAY'"
Write-Verbose -Message ('Found {0} display device drivers' -f $PnPSignedDrivers.Count)
$PnPSignedInfo = ($PnPSignedDrivers | Select-Object -Property DeviceName,DriverProviderName,InfName,DriverVersion,Description | Format-Table -AutoSize | Out-String).Trim().Split("`n")
$PnPSignedInfo | ForEach-Object -Process { Write-Verbose -Message (' {0}' -f $_) }
# next we have to get every PnPSignedDriverCIMDataFile, because Get-CimAssociatedInstance doesn't wanna play ball
Write-Output -InputObject ('Gathering all driver file objects... (this is the slow one. Blame Microsoft.)') # or me not understanding CIM i guess?
$SignedDriverFiles = Get-CimInstance -ClassName 'Win32_PNPSignedDriverCIMDataFile'
Write-Output -InputObject ('Found {0} files across all system drivers.' -f $SignedDriverFiles.Count)
foreach ($GPU in $TargetGPUs) {
Write-Output -InputObject ('Getting driver package for {0}' -f $GPU.FriendlyName)
$PnPEntity = $PnPEntities | Where-Object { $_.InstanceId -eq $GPU.InstanceId }[0]
Write-Verbose -Message ('Device PnP Entity:')
($PnPEntity | Format-Table -AutoSize | Out-String).Trim().Split("`n") | ForEach-Object -Process { Write-Verbose -Message (' {0}' -f $_) }
$PnPSignedDriver = $PnPSignedDrivers | Where-Object { $_.DeviceId -eq $GPU.InstanceId }
Write-Verbose -Message ('Device PnPSignedDriver:')
($PnPSignedDriver | Format-Table -AutoSize | Out-String).Trim().Split("`n") | ForEach-Object -Process { Write-Verbose -Message (' {0}' -f $_) }
$SystemDriver = Get-CimAssociatedInstance -InputObject $PnPEntity -Association Win32_SystemDriverPNPEntity
Write-Verbose -Message ('Device SystemDriver:')
($SystemDriver | Format-Table -AutoSize | Out-String).Trim().Split("`n") | ForEach-Object -Process { Write-Verbose -Message (' {0}' -f $_) }
Write-Verbose -Message ('SystemDriver main/anchor file: {0}' -f $SystemDriver.PathName)
$DriverStoreFolder = Get-Item -Path (Split-Path -Path $SystemDriver.PathName -Parent)
while ((Get-Item (Split-Path $DriverStoreFolder)).Name -notlike 'FileRepository') {
$DriverStoreFolder = Get-Item -Path (Split-Path -Path $DriverStoreFolder -Parent)
}
Write-Verbose -Message ('Device DriverStoreFolder: {0}' -f $DriverStoreFolder.FullName)
Write-Output -InputObject ('Found package {0}, copying DriverStore folder {1} to temporary directory' -f $PnPSignedDriver.InfName, (Split-Path $DriverStoreFolder -Leaf))
$TempDriverStore = ('{0}/System32/HostDriverStore/FileRepository/{1}' -f $fTempFolder, $DriverStoreFolder.Name)
$DriverStoreFolder | Copy-Item -Destination $TempDriverStore -Recurse -Force
Write-Output -InputObject ('Copied {0} of {1} files to temporary directory' -f (Get-ChildItem -Path $TempDriverStore -Recurse).Count, (Get-ChildItem -Path $DriverStoreFolder -Recurse).Count)
# Get driver files from system32 etc and copy
Write-Output -InputObject ('Gathering files from System32 and SysWOW64')
$DriverFiles = ($SignedDriverFiles | Where-Object { $_.Antecedent.DeviceID -like $GPU.DeviceID }).Dependent.Name | Sort-Object
$NonDriverStoreFiles = $DriverFiles.Where{$_ -notlike '*DriverStore*'}
Write-Output -InputObject ('Found {0} files, copying to temporary directory...' -f $NonDriverStoreFiles.Count)
$NonDriverStoreFiles | ForEach-Object -Process {
$TargetPath = Join-Path -Path $fTempFolder -ChildPath $_.ToLower().Replace(('{0}\' -f $Env:SYSTEMROOT.ToLower()),'')
# make sure the parent folder exists
(New-Item -ItemType directory -Path (Split-Path -Path $TargetPath -Parent) -Force -ErrorAction SilentlyContinue | Out-Null)
Write-Output -InputObject (' - {0} -> {1}' -f $_, $TargetPath)
Copy-Item -Path $_ -Destination $TargetPath -Force -Recurse
}
Write-Output -InputObject ('Finished gathering files for {0}' -f $GPU.FriendlyName)
}
Write-Output -InputObject ('All driver files have been collected, creating archive file')
$Location = (Get-Location).Path
Set-Location -Path (Split-Path -Path $fTempFolder -Parent)
Compress-Archive -Path $fTempFolder -DestinationPath (Join-Path -Path $ArchiveFolder -ChildPath $ArchiveName) -CompressionLevel Fastest -Confirm:$false
Set-Location -Path $Location
Write-Output -InputObject ('GPU driver package has been created at path {0}\{1}' -f $ArchiveFolder, $ArchiveName)
} catch {
throw $PSItem
} finally {
Write-Output -InputObject ('Cleaning up temporary directory {0}' -f $fTempFolder)
Remove-Item -Recurse -Force -Path $fTempFolder
}
}
Write-Output -InputObject ('Driver package generation complete.')
Write-Output -InputObject ('Please copy it to your guest and extract the archive into C:\Windows\')
}
$Config = @{
VMName = 'GPU-VM' # Edit this to match your existing VM. If you don't have a VM with this name, it will be created.
VMMemory = 8192MB # Set appropriately
VMCores = 4 # likewise
MinRsrc = 80000000 # We don't really know what these values do - my GPU reports 100,000,000 available units...
MaxRsrc = 100000000 # I suspect in the current implementation they do nothing, but this is known to work - play around if you like!
OptimalRsrc = 100000000
}
### actual execution
try {
# Get VM host capabilities
$VMHost = Get-VMHost
# Get highest VM config version available
$VMVersion = $VMHost.SupportedVmVersions[$VMHost.SupportedVmVersions.Count - 1]
# Get existing VM if it exists
$VMObject = (Get-VM -Name $Config.VMName -ErrorAction SilentlyContinue)
# Create VM if it doesn't already exist
if (-not $VMObject) {
$NewVM = @{
Name = $Config.VMName
MemoryStartupBytes = $Config.VMMemory
Generation = 2
Version = $VMVersion
}
New-VM @NewVM
}
# Enable VM features required for this to work
$SetParams = @{
VMName = $Config.VMName
GuestControlledCacheTypes = $true
LowMemoryMappedIoSpace = 1Gb
HighMemoryMappedIoSpace = 32GB
AutomaticStopAction = 'TurnOff'
CheckpointType = 'Disabled'
}
Set-VM @SetParams
# Disable secure boot
Set-VMFirmware -VMName $Config.VMName -EnableSecureBoot 'Off'
# Parameters for vAdapter
$GPUParams = @{
VMName = $Config.VMName
MinPartitionVRAM = $Config.MinRsrc
MaxPartitionVRAM = $Config.MaxRsrc
OptimalPartitionVRAM = $Config.OptimalRsrc
MinPartitionEncode = $Config.MinRsrc
MaxPartitionEncode = $Config.MaxRsrc
OptimalPartitionEncode = $Config.OptimalRsrc
MinPartitionDecode = $Config.MinRsrc
MaxPartitionDecode = $Config.MaxRsrc
OptimalPartitionDecode = $Config.OptimalRsrc
MinPartitionCompute = $Config.MinRsrc
MaxPartitionCompute = $Config.MaxRsrc
OptimalPartitionCompute = $Config.OptimalRsrc
}
# Get adapter if it exists
$VMAdapter = (Get-VMGpuPartitionAdapter -VMName $Config.VMName -ErrorAction SilentlyContinue)
# Add adapter if not present, update if present
if ($VMAdapter) {
Set-VMGpuPartitionAdapter @GPUParams
} else {
Add-VMGpuPartitionAdapter @GPUParams
}
} catch {
Write-Error "Something went wrong with creation. Error details below:"
Write-Error $PSItem.ErrorDetails
throw $PSItem
}
@mmikeww
Copy link

mmikeww commented Jul 6, 2022

I'd like to expand this to automatically transfer the driver bundle to the guest (using Copy-VMFile) and unpack it into the right place (using PowerShell Direct) but that's a whole other nightmare to deal with... One day...

That Easy-GPU-PV script does this too. Instead of using Copy-VMFile, it mounts the VHD so that it gets a normal drive letter, and then performs the copy. See this file:

https://github.com/jamesstringerparsec/Easy-GPU-PV/blob/main/Update-VMGpuPartitionDriver.ps1

The full repo creates brand new VMs, but I don't use it for that. I just use these two files I've linked to manually perform the driver copy into my existing VMs

@RomaKoks
Copy link

RomaKoks commented Jan 7, 2023

gpu appears in VM's device manager, but nvidia-smi says:
Failed to initialize NVML: Unknown Error

It seems they (Nvidia) have changed something again.
I installed Nvidia drivers (527.41) on host machine via nvidia cuda toolkit 12.0 if it helps somehow.

Configuration:
Host:
Windows 10 22H2
2x RTX 2070

VM:
Windows 10 22H2
RTX 2070

list of all files in prepared by script zip:
https://gist.github.com/RomaKoks/420bd9e06ed0af2e88c7c1b62cc66ee7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment