VPN/Routing troubleshooting

README.md

Troubleshoot

The script

Run attached script.

Script output samples

OS X: Access to key vault not using P2S VPN

Detected platform: Unix
Hostname: redacted.vault.azure.net
Checking hosts file
Match on hosts file(/etc/hosts) for 'redacted.vault.azure.net'
10.2.2.2 redacted.vault.azure.net

Resolving IP addresses for 'redacted.vault.azure.net'
Resolved IP addresses: 
10.2.2.2
Using first IP address for route and connection testing: 10.2.2.2
Next hop information:
   route to: redacted.vault.azure.net
destination: default
       mask: default
    gateway: 172.17.44.1
  interface: en0
      flags: <UP,GATEWAY,DONE,STATIC,PRCLONING,GLOBAL>
 recvpipe  sendpipe  ssthresh  rtt,msec    rttvar  hopcount      mtu     expire
       0         0         0         0         0         0      1500         0 
Curl with HTTPS
*   Trying 10.2.2.2:443...
* Connected to redacted.vault.azure.net (10.2.2.2) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* Connection timed out after 10004 milliseconds
* Closing connection 0
curl: (28) Connection timed out after 10004 milliseconds
Curl with HTTP
*   Trying 10.2.2.2:80...
* Connected to redacted.vault.azure.net (10.2.2.2) port 80 (#0)
> GET / HTTP/1.1
> Host: redacted.vault.azure.net
> User-Agent: curl/8.1.2
> Accept: */*
> 
* Operation timed out after 10005 milliseconds with 0 bytes received
* Closing connection 0
curl: (28) Operation timed out after 10005 milliseconds with 0 bytes received

OS X: Access to key vault using P2S VPN

Detected platform: Unix
Hostname: redacted.vault.azure.net
Checking hosts file
Match on hosts file(/etc/hosts) for 'redacted.vault.azure.net'
10.2.2.2 redacted.vault.azure.net

Resolving IP addresses for 'redacted.vault.azure.net'
Resolved IP addresses: 
10.2.2.2
Using first IP address for route and connection testing: 10.2.2.2
Next hop information:
   route to: redacted.vault.azure.net
destination: 10.2.2.0
       mask: 255.255.192.0
  interface: utun4
      flags: <UP,DONE,CLONING,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    rttvar  hopcount      mtu     expire
       0         0         0         0         0         0      1500         0 
Curl with HTTPS
*   Trying 10.2.2.2:443...
* Connected to redacted.vault.azure.net (10.2.2.2) port 443 (#0)
Removed for clarity
> GET / HTTP/2
> Host: redacted.vault.azure.net
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/2 403 
< content-type: text/html
< x-content-type-options: nosniff
< strict-transport-security: max-age=31536000;includeSubDomains
< date: Sun, 13 Aug 2023 14:06:32 GMT
< content-length: 1233
< 
Removed for clarity
* Connection #0 to host redacted.vault.azure.net left intact
Curl with HTTP
*   Trying 10.2.2.2:80...
* Connected to redacted.vault.azure.net (10.2.2.2) port 80 (#0)
> GET / HTTP/1.1
> Host: redacted.vault.azure.net
> User-Agent: curl/8.1.2
> Accept: */*
> 
* Operation timed out after 10005 milliseconds with 0 bytes received
* Closing connection 0
curl: (28) Operation timed out after 10005 milliseconds with 0 bytes received

route get command's output changed from interface: en to interface: utun4 and destination: default to destination: 10.2.2.0 mask: 255.255.192.0 therefore non-default route is being picked and TCP connection flows trough P2S VPN.

Windows: No VPN connection

Next hop information:

IPAddress         : 172.17.44.20
InterfaceIndex    : 8
InterfaceAlias    : WiFi
AddressFamily     : IPv4
Type              : Unicast
PrefixLength      : 24
PrefixOrigin      : Dhcp
SuffixOrigin      : Dhcp
AddressState      : Preferred
ValidLifetime     : 22:39:45
PreferredLifetime : 22:39:45
SkipAsSource      : False
PolicyStore       : ActiveStore


Caption            :
Description        :
ElementName        :
InstanceID         : :8:8:8:9:55B55;A<8;@8><8;55;
AdminDistance      :
DestinationAddress :
IsStatic           :
RouteMetric        : 0
TypeOfRoute        : 3
AddressFamily      : IPv4
CompartmentId      : 1
DestinationPrefix  : 0.0.0.0/0
InterfaceAlias     : WiFi
InterfaceIndex     : 8
InterfaceMetric    : 35
NextHop            : 172.17.44.1
PreferredLifetime  : 1.00:00:00
Protocol           : NetMgmt
Publish            : No
State              : Alive
Store              : ActiveStore
ValidLifetime      : 1.00:00:00
PSComputerName     :
ifIndex            : 8

• 172.17.44.1: LAN gateway address
• 172.17.44.20: computer LAN IP
• InterfaceAlias : WiFi local network interface

Windows: VPN connection

Next hop information:


IPAddress         : 10.3.1.3
InterfaceIndex    : 70
InterfaceAlias    : <vpn connection related interface name>
AddressFamily     : IPv4
Type              : Unicast
PrefixLength      : 32
PrefixOrigin      : Other
SuffixOrigin      : Other
AddressState      : Preferred
ValidLifetime     : Infinite ([TimeSpan]::MaxValue)
PreferredLifetime : Infinite ([TimeSpan]::MaxValue)
SkipAsSource      : False
PolicyStore       : ActiveStore


Caption            :
Description        :
ElementName        :
InstanceID         : ;:8;C<8;<B8:9;B55A:55:8:8:8:55;
AdminDistance      :
DestinationAddress :
IsStatic           :
RouteMetric        : 256
TypeOfRoute        : 3
AddressFamily      : IPv4
CompartmentId      : 1
DestinationPrefix  : 10.2.2.0/18
InterfaceAlias     : <vpn connection related interface name>
InterfaceIndex     : 70
InterfaceMetric    : 55
NextHop            : 0.0.0.0
PreferredLifetime  : 10675199.02:48:05.4775807
Protocol           : NetMgmt
Publish            : No
State              : Alive
Store              : ActiveStore
ValidLifetime      : 10675199.02:48:05.4775807
PSComputerName     :
ifIndex            : 70

• 10.3.1.3: VPN tunnel IP
• 10.2.2.0/18: subnet on other end of P2S VPN connection
• InterfaceAlias: VPN connection

Reset DNS caches after hosts file edit

OS X

sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder

Windows

Changes on hosts file are used by resolver as soon as file is being saved.

p2s-troubleshooting.ps1

# TODO: parametrize
$hostname = "redacted.vault.azure.net"
$httpAddress = "http://${hostname}/"
$httpsAddress = "https://${hostname}/"

if ($PSVersionTable.PSEdition -eq "Desktop") {
    $platform = "Windows"
} elseif ($PSVersionTable.Platform -eq "Unix") {
    $platform = "Unix"
} else {
    $platform = "Windows"
}

Write-Host "Detected platform: $platform"
Write-Host "Hostname: $hostname"

# Check if hosts file has entry for $hostname
Write-Host "Checking hosts file"
if ($platform -eq "Windows") {
    $hostsfile = "c:/Windows/System32/Drivers/etc/hosts"
} else {
    $hostsfile = "/etc/hosts"
}

$result = Get-content $hostsfile | Select-String $hostname
if ($Null -eq $result) {
    Write-Host "No entry for '$hostname' on hosts file($hostsfile)"
} else {
    Write-Host "Match on hosts file($hostsfile) for '$hostname'"
    Write-Host $result
}


# Resolve IP from hostname
Write-Host
Write-Host "Resolving IP addresses for '$hostname'"
$resolvedIPAddresses = [System.Net.Dns]::GetHostAddresses($hostname) # This command also reads hosts file, host command on OS X will not
if ($Null -eq $resolvedIPAddresses) {
    Write-Host "Failed to resolve ${hostname}: `$resolvedIPAddresses is `$Null. Exiting...."
    $resolvFailed = $true
}
if ($resolvedIPAddresses.Length -eq 0) {
    Write-Host "Failed to resolve ${hostname}: `$resolvedIPAddresses is empty. Exiting...."
    $resolvFailed = $true
}

if (-not $resolvFailed) {
    Write-Host "Resolved IP addresses: "
    $resolvedIPAddresses.IPAddressToString


    $resolvedIPAddress = $resolvedIPAddresses[0].IPAddressToString
    Write-Host "Using first IP address for route and connection testing: $resolvedIPAddress"

    # Find the next hop
    Write-host "Next hop information:"
    if ($platform -eq "Windows") {
        Find-NetRoute -RemoteIPAddress $resolvedIPAddress
    } else {
        route get $resolvedIPAddress
    }

    if ($platform -eq "Windows") {
        $curl = "curl.exe"
    } else {
        $curl = "curl"
    }

    Write-Host "Curl with HTTPS"
    & $curl $httpsAddress -v -k --max-time 10

    Write-Host "Curl with HTTP"
    & $curl $httpAddress -v --max-time 10
}