r/windows: https://www.reddit.com/r/windows/comments/10ff15p/solved_disable_switching_default_audio_device/ r/VoiceMeeter: https://www.reddit.com/r/VoiceMeeter/comments/10fm3cr/solved_disable_switching_default_audio_device/

SetDefaulDeviceOnConnection.xml

<?xml version="1.0" encoding="UTF-16"?>
<Task version="1.2" xmlns="http://schemas.microsoft.com/windows/2004/02/mit/task">
  <RegistrationInfo>
    <Date>2023-01-18T18:19:11.4118961</Date>
    <Author>NULL234\null234</Author>
    <URI>\SetDefaulDeviceOnConnection</URI>
    <Description>Stop annoying windows! i want to decide what stadart playback device i use.</Description>
  </RegistrationInfo>
  <Triggers>
    <EventTrigger>
      <Enabled>true</Enabled>
      <Subscription>&lt;QueryList&gt;&lt;Query Id="0" Path="Microsoft-Windows-DriverFrameworks-UserMode/Operational"&gt;&lt;Select Path="Microsoft-Windows-DriverFrameworks-UserMode/Operational"&gt;*[System[(Level=1  or Level=2 or Level=3 or Level=4 or Level=0 or Level=5)]]&lt;/Select&gt;&lt;/Query&gt;&lt;/QueryList&gt;</Subscription>
    </EventTrigger>
  </Triggers>
  <Principals>
    <Principal id="Author">
      <UserId>S-1-5-21-2611742254-942843215-3789948414-1001</UserId>
      <LogonType>InteractiveToken</LogonType>
      <RunLevel>LeastPrivilege</RunLevel>
    </Principal>
  </Principals>
  <Settings>
    <MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy>
    <DisallowStartIfOnBatteries>true</DisallowStartIfOnBatteries>
    <StopIfGoingOnBatteries>true</StopIfGoingOnBatteries>
    <AllowHardTerminate>true</AllowHardTerminate>
    <StartWhenAvailable>false</StartWhenAvailable>
    <RunOnlyIfNetworkAvailable>false</RunOnlyIfNetworkAvailable>
    <IdleSettings>
      <StopOnIdleEnd>true</StopOnIdleEnd>
      <RestartOnIdle>false</RestartOnIdle>
    </IdleSettings>
    <AllowStartOnDemand>true</AllowStartOnDemand>
    <Enabled>true</Enabled>
    <Hidden>false</Hidden>
    <RunOnlyIfIdle>false</RunOnlyIfIdle>
    <WakeToRun>false</WakeToRun>
    <ExecutionTimeLimit>PT72H</ExecutionTimeLimit>
    <Priority>7</Priority>
  </Settings>
  <Actions Context="Author">
    <Exec>
      <Command>C:\Temp\SoundVolumeView.exe</Command>
      <Arguments>/SetDefault "VB-Audio VoiceMeeter VAIO\Device\VoiceMeeter Input\Render" 0</Arguments>
    </Exec>
    <Exec>
      <Command>C:\Temp\SoundVolumeView.exe</Command>
      <Arguments>/SetDefault "VB-Audio VoiceMeeter VAIO\Device\VoiceMeeter Input\Render" 1</Arguments>
    </Exec>
    <Exec>
      <Command>C:\Temp\SoundVolumeView.exe</Command>
      <Arguments>/SetDefault "VB-Audio VoiceMeeter VAIO\Device\VoiceMeeter Input\Render" 2</Arguments>
    </Exec>
  </Actions>
</Task>

Quick note for anyone wondering: You can also also set the default input device with this task.
For example when you're using VoiceMeeter you can add these three commands:

    <Exec>
      <Command>C:\Temp\SoundVolumeView.exe</Command>
      <Arguments>/SetDefault "VB-Audio VoiceMeeter VAIO\Device\VoiceMeeter Output\Capture" 0</Arguments>
    </Exec>
    <Exec>
      <Command>C:\Temp\SoundVolumeView.exe</Command>
      <Arguments>/SetDefault "VB-Audio VoiceMeeter VAIO\Device\VoiceMeeter Output\Capture" 1</Arguments>
    </Exec>
    <Exec>
      <Command>C:\Temp\SoundVolumeView.exe</Command>
      <Arguments>/SetDefault "VB-Audio VoiceMeeter VAIO\Device\VoiceMeeter Output\Capture" 2</Arguments>
    </Exec>

For anybody in the future looking at this - Would suggest changing line 12

# Old
-      <Subscription>&lt;QueryList&gt;&lt;Query Id="0" Path="Microsoft-Windows-DriverFrameworks-UserMode/Operational"&gt;&lt;Select Path="Microsoft-Windows-DriverFrameworks-UserMode/Operational"&gt;*[System[(Level=1  or Level=2 or Level=3 or Level=4 or Level=0 or Level=5)]]&lt;/Select&gt;&lt;/Query&gt;&lt;/QueryList&gt;</Subscription>

# New
+      <Subscription>&lt;QueryList&gt;&lt;Query Id="0" Path="Microsoft-Windows-Audio-UserMode/Operational"&gt;&lt;Select Path="Microsoft-Windows-Audio-UserMode/Operational"&gt;*[System[(Level=1  or Level=2 or Level=3 or Level=4 or Level=0 or Level=5)]]&lt;/Select&gt;&lt;/Query&gt;&lt;/QueryList&gt;</Subscription>

Or more specifically

# Remove this
-DriverFrameworks

# Replace with this
+Audio

This makes the task trigger on audio device change, instead of driver change. I assume if you're doing this script you never want to switch off voicemeeter anyway, so it won't matter!