auxsetup.exe claims to have a button to delete settings from the registry but the version I had deletes settings under a different (HKEY_USERS\*\Software\Freeware\VirtualDub\
) which I think might be out of date.
The settings live under this key: Computer\HKEY_USERS\<UID>\SOFTWARE\VirtualDub.org
Deleting that key and restarting VirtualDub will reset everything to defaults.