Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Workaround for WSL2 network broken on VPN

Overview

Internet connection and DNS routing are broken from WSL2 instances, when some VPNs are active. The workaround breaks down into two problems:

  1. Network connection to internet
  2. DNS in WSL2

This problem is tracked in multiple microsoft/WSL issues including, but not limited to:

  • microsoft/WSL#5068
  • microsoft/WSL#4277
  • microsoft/WSL#4246

Network connection

When the VPN connection is active, network traffic out of WSL2 is not passed to the internet.

Changing the Interface Metric 1 -> 6000 for AnyConnect VPN Adapter resolves the connection issue, but this has to be done after each time the VPN connects.

By default, the Interface Metrics for AnyConnect are:

  • IPv6: 6000
  • IPv4: 1

ping times out from WSL Shell.

Changing the Interface Metrics for AnyConnect to:

  • IPv6: 6000
  • IPv4: 6000

ping to IP Addresses succeed, but still no DNS Resolution.

DNS Resolution

When the VPN is active, the autogenerated /etc/resolv.conf does not work. The list of nameservers must be manually built to include some sane default DNS Name Servers and the DNS from the VPN.

First, disable automatically generating /etc/resolv.conf. Add the following configuration, or create the file if it doesn't exist. The path to this file is from the shell prompt of your WSL2 instance.

/etc/wsl.conf

[network]
generateResolvConf = false

Next, manually add the corportate DNS Server as the first nameserver in /etc/resolv.conf.

/etc/resolv.conf

nameserver <corporateDNS1>
nameserver <corporateDNS2>
nameserver 1.1.1.1

To get <corporateDNS> addresses, use ipconfig /all from CMD or Powershell prompt, and check the details of the VPN adapter:

Description . . . . . . . . . . . : Cisco AnyConnect Secure Mobility Client Virtual Miniport Adapter for Windows x64
Physical Address. . . . . . . . . : XX-XX-XX-XX-XX-XX
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
IPv6 Address. . . . . . . . . . . : xxxx:xxxx:xxxx:xxxx(Preferred)
Link-local IPv6 Address . . . . . : xxxx:xxxx:xxxx:xxxx(Preferred)
IPv4 Address. . . . . . . . . . . : 10.20.30.40(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.255
Default Gateway . . . . . . . . . : ::
                                    0.0.0.0
DHCPv6 IAID . . . . . . . . . . . :
DHCPv6 Client DUID. . . . . . . . : 
DNS Servers . . . . . . . . . . . : 123.45.67.89    <- Corporate DNS 1
                                    123.45.67.90    <- Corporate DNS 2
Primary WINS Server . . . . . . . : xxx.xx.xxx.xx
NetBIOS over Tcpip. . . . . . . . : Enabled

Automatically update Interface Metric

To automate this, I put the PS command in a script and created a Scheduled Task to run every time there is a network change.

Save the script in a file

First, create the script. I have a 'scripts' directory in my Windows user home, so I put it at:

%HOMEPATH%\scripts\UpdateAnyConnectInterfaceMetric.ps1

Get-NetAdapter | Where-Object {$_.InterfaceDescription -Match "Cisco AnyConnect"} | Set-NetIPInterface -InterfaceMetric 6000

You can save it where you want, just make sure to use that path in step 13 below.

Create the scheduled task:

  1. Open 'Task Scheduler'
  2. Click "Create Task" on Right Sidebar
  3. Name: Update Anyconnect Adapter Interface Metric for WSL2
  4. Set Security Options
    • Check box: 'Run with highest priveleges'
  5. Select 'Triggers' Tab
  6. Click 'New' at bottom of Window
  7. Open 'Begin the task' drop-down
  8. Select 'On an Event'
  9. Configure Event:
    • option 1: Trigger on any Network Change
      • Log: 'Microsoft-Windows-NetworkProfile/Operational'
      • Source: 'NetworkProfile'
      • Event ID: '10000'
    • option 2: Trigger only when AnyConnect Client successfully connects to VPN
      • Log: 'Cisco AnyCOnnect Secure Mobility Client'
      • Source: 'acvpnagent'
      • Event ID: '2039'
  10. Click 'OK'
  11. Select 'Actions' Tab
  12. Click 'New'
  13. Configure Action:
    • Action: 'Start a Program'
    • Program/script: 'Powershell.exe'
    • Add arguments: '-ExecutionPolicy Bypass -File %HOMEPATH%\scripts\UpdateAnyConnectInterfaceMetric.ps1'
  14. Click 'OK'
  15. Select 'Conditions' Tab
  16. Uncheck box:
    • Power -> Start the task only if the computer is on AC Power
  17. Click 'OK'

When AnyConnect finishes connecting, a Powershell window pops up for a couple seconds and WSL can reach the network.

@sebukoleth

This comment has been minimized.

Copy link

@sebukoleth sebukoleth commented Feb 15, 2021

The above steps has saved my sanity on WSL2 - Ubuntu 20.04 - and Cisco AnyConnect, too.

@phelps-matthew

This comment has been minimized.

Copy link

@phelps-matthew phelps-matthew commented Feb 28, 2021

This works excellently. [WSL2 - Ubuntu 20.04 - and Cisco AnyConnect]. I did, however, notice that my /etc/resolv.conf disappeared after reboot. Any ideas on this behavior? Have confirmed /etc/wsl.conf is

[network]
generateResolvConf = false
@AddAndy

This comment has been minimized.

Copy link

@AddAndy AddAndy commented Mar 2, 2021

When i disable auto generation, i can't seem to create a /etc/resolve.conf (even with sudo i can't create the file)

@machuu

This comment has been minimized.

Copy link
Owner Author

@machuu machuu commented Mar 2, 2021

When i disable auto generation, i can't seem to create a /etc/resolve.conf (even with sudo i can't create the file)

Have you restarted WSL? It might help if you do wsl.exe --shutdown from CMD terminal, then reopen WSL.

@ecool

This comment has been minimized.

Copy link

@ecool ecool commented Mar 4, 2021

So, I'm confused at how this is a good idea to do. Increasing the metric value on your VPN's connection is basically telling your operating system to ignore your VPN connection. Which means all network traffic is no longer using it and that means it is the same as just disconnecting your VPN. Maybe I'm not understanding something or missing something but it just doesn't seem like anyone should do this if they really want to use their VPN. If I am, I would love to know what.

I have been researching why WSL 2 seems to have issues with VPNs and I haven't found an answer that works yet. This probably would work but as I said you're basically just disabling the VPN and I don't want that.

@ecool

This comment has been minimized.

Copy link

@ecool ecool commented Mar 4, 2021

Also @AddAndy, after disabling the auto generation you're basically left with a broken symlink at /etc/resolv.conf if you wish to edit it with a custom nameserver you'll need to sudo rm /etc/resolv.conf and then sudo touch /etc/resolv.conf to create the file. After that you can edit as you would normally.

@phelps-matthew

This comment has been minimized.

Copy link

@phelps-matthew phelps-matthew commented Mar 4, 2021

So, I'm confused at how this is a good idea to do. Increasing the metric value on your VPN's connection is basically telling your operating system to ignore your VPN connection. Which means all network traffic is no longer using it and that means it is the same as just disconnecting your VPN. Maybe I'm not understanding something or missing something but it just doesn't seem like anyone should do this if they really want to use their VPN. If I am, I would love to know what.

I have been researching why WSL 2 seems to have issues with VPNs and I haven't found an answer that works yet. This probably would work but as I said you're basically just disabling the VPN and I don't want that.

Just as a datapoint, I require VPN to access ssh onto work remote computers and I have not noticed any real issues or lag.

@ecool

This comment has been minimized.

Copy link

@ecool ecool commented Mar 4, 2021

Just as a datapoint, I require VPN to access ssh onto work remote computers and I have not noticed any real issues or lag.

I can see that working fine, since I believe you're probably using the IP address of the VPN network to ssh to. That lets Windows know that it needs to route through that network interface. I have Hamachi as a VPN LAN for my computers and that is how I am able to ssh into them. Using the hamachi IP address. Metric applies as a priority level. So if you had 2 VPN networks and one had a metric of 2000 and the other 3000. Even if they both go to the same VPN network it would use the one with the metric of 2000 before trying the 3000 if the prior one failed.

@machuu

This comment has been minimized.

Copy link
Owner Author

@machuu machuu commented Mar 4, 2021

@ecool

So, I'm confused at how this is a good idea to do. Increasing the metric value on your VPN's connection is basically telling your operating system to ignore your VPN connection. Which means all network traffic is no longer using it and that means it is the same as just disconnecting your VPN. Maybe I'm not understanding something or missing something but it just doesn't seem like anyone should do this if they really want to use their VPN. If I am, I would love to know what.

I have been researching why WSL 2 seems to have issues with VPNs and I haven't found an answer that works yet. This probably would work but as I said you're basically just disabling the VPN and I don't want that.

From my observations, changing the Anyconnect adapter metric does not allow anything to bypass the Anyconnect VPN.
traceroute from WSL2 and tracert from windows side both report going through the corportate network before hitting the internet.

My sense is that the VPN is more invasive than just becoming the lowest metric interface, and I agree with your idea that the real problem is in vEthernet(WSL) or more likely vEthernet(Default Switch). Anyconnect is probably stepping on something in vEthernet.

My working theory is that Anyconnect is intercepting WSL2 traffic, so it doesn't properly go through vEthernet, and responses don't make it back. When Anyconnect metric is increased, vEthernet traffic is routed in a way that Anyconnect can grab and re-route it through the VPN and get the responses back to WSL2.

I tested the Anyconnect interface metric, to find the exact value where WSL stops working, and the minimum working value is 270.

Below, Ethernet 2 is the Anyconnect adapter.

73      vEthernet (WSL)                 IPv4                  1500              15 Disabled Connected       ActiveStore
25      Bluetooth Network Connection    IPv4                  1500              65 Enabled  Disconnected    ActiveStore
17      Ethernet 3                      IPv4                 65536              25 Enabled  Connected       ActiveStore
42      vEthernet (Default Switch)      IPv4                  1500              15 Disabled Connected       ActiveStore
19      Ethernet 2                      IPv4                  1390             270 Disabled Connected       ActiveStore
21      Local Area Connection* 2        IPv4                  1500              25 Enabled  Disconnected    ActiveStore
10      Local Area Connection* 1        IPv4                  1500              25 Enabled  Disconnected    ActiveStore
13      Ethernet                        IPv4                  1500               5 Enabled  Disconnected    ActiveStore
4       Wi-Fi                           IPv4                  1500              50 Enabled  Connected       ActiveStore
1       Loopback Pseudo-Interface 1     IPv4            4294967295              75 Disabled Connected       ActiveStore
@ecool

This comment has been minimized.

Copy link

@ecool ecool commented Mar 5, 2021

@machuu,

Thanks for responding. VPN being more invasive does seem like something that may be true. I'll have to test it on my own networking setup since I don't use Anyconnect VPN and not sure if all VPNs do the same thing.

Hopefully, the base issue gets resolved without needing workarounds.

@maxtorete

This comment has been minimized.

Copy link

@maxtorete maxtorete commented Mar 15, 2021

The point 17 is not working for me, it doesn't trigger the script after selecting my corporate network name as condition.

I changed trigger's Event to:

  • Log: 'Cisco AnyConnect Mobility Security Client'
  • Source: 'acvpnagent'
  • Event ID: '2039'

So it will be launched only at a successful connection to any vpn network with the Cisco AnyConnect client without adding a network name as condition.

@machuu

This comment has been minimized.

Copy link
Owner Author

@machuu machuu commented Mar 15, 2021

The point 17 is not working for me, it doesn't trigger the script after selecting my corporate network name as condition.

I changed trigger's Event to:

* Log: 'Cisco AnyConnect Mobility Security Client'

* Source: 'acvpnagent'

* Event ID: '2039'

So it will be launched only at an successful connection to any vpn network with the Cisco AnyConnect client without adding a network name as condition.

Thanks, this is a cleaner trigger than just a network change. I'll update with those Event details.

@wesleymusgrove

This comment has been minimized.

Copy link

@wesleymusgrove wesleymusgrove commented Apr 1, 2021

@machuu,

Thanks for putting this together is such an organized format! I followed all these steps exactly and very carefully and have also been trying very similar steps provided by various other people in the issues you mentioned.

I'm able to ping external sites on the internet, but I can't curl or wget them for some reason.

ping works:

$ ping example.com
PING example.com (93.184.216.34) 56(84) bytes of data.
64 bytes from 93.184.216.34 (93.184.216.34): icmp_seq=1 ttl=54 time=54.5 ms
64 bytes from 93.184.216.34 (93.184.216.34): icmp_seq=2 ttl=54 time=55.4 ms
64 bytes from 93.184.216.34 (93.184.216.34): icmp_seq=3 ttl=54 time=55.7 ms
64 bytes from 93.184.216.34 (93.184.216.34): icmp_seq=4 ttl=54 time=53.5 ms
64 bytes from 93.184.216.34 (93.184.216.34): icmp_seq=5 ttl=54 time=54.9 ms
^C
--- example.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 53.563/54.846/55.704/0.765 ms

curl does not work:

$ curl -v https://www.example.com
* Rebuilt URL to: https://www.example.com/
*   Trying 93.184.216.34...
* TCP_NODELAY set
* connect to 93.184.216.34 port 443 failed: Connection refused
*   Trying 2606:2800:220:1:248:1893:25c8:1946...
* TCP_NODELAY set
* Immediate connect fail for 2606:2800:220:1:248:1893:25c8:1946: Network is unreachable
*   Trying 2606:2800:220:1:248:1893:25c8:1946...
* TCP_NODELAY set
* Immediate connect fail for 2606:2800:220:1:248:1893:25c8:1946: Network is unreachable
* Failed to connect to www.example.com port 443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to www.example.com port 443: Connection refused

In addition to using Cisco AnyConnect VPN on my Windows 10 host, all my traffic also goes through a Zscaler proxy running on Windows on http://localhost:9000. That means that while I'm connected to my VPN, in Windows I also have to specify HTTP_PROXY=http://localhost:9000 pretty much everywhere you can imagine so that things can access the internet. Inside of WSL, I'm able to refer to this Zscaler proxy as http://host.docker.internal:9000. However I'm getting this error: curl: (56) Recv failure: Connection reset by peer and I don't know at what layer it's failing...

curl -v -x http://host.docker.internal:9000 https://www.example.com
* Rebuilt URL to: https://www.example.com/
*   Trying 10.99.148.144...
* TCP_NODELAY set
* Connected to host.docker.internal (10.99.148.144) port 9000 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to www.example.com:443
> CONNECT www.example.com:443 HTTP/1.1
> Host: www.example.com:443
> User-Agent: curl/7.58.0
> Proxy-Connection: Keep-Alive
>
* Recv failure: Connection reset by peer
* Received HTTP code 0 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

In, Powershell, if I run wsl hostname -I, the IP of my Windows host is 172.25.167.201, but trying to use this proxy as 172.25.167.201:9000 instead of http://host.docker.internal:9000 inside WSL doesn't work. So I don't think that's the correct way to go about doing this.

$ curl -v -x 172.25.167.201:9000 https://www.example.com
* Rebuilt URL to: https://www.example.com/
*   Trying 172.25.167.201...
* TCP_NODELAY set
* connect to 172.25.167.201 port 9000 failed: Connection refused
* Failed to connect to 172.25.167.201 port 9000: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 172.25.167.201 port 9000: Connection refused

Any thoughts or suggestions?

@phelps-matthew

This comment has been minimized.

Copy link

@phelps-matthew phelps-matthew commented Apr 17, 2021

@machuu Do you have a link to the parent github issue thiat this gist spawned from?

@craibuc

This comment has been minimized.

Copy link

@craibuc craibuc commented May 18, 2021

The script and task allows me to connect to the external network, but I'm having SSL-certificate issues:

Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [109 kB]
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Ign:3 https://dl.bintray.com/sbt/debian  InRelease
Get:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Err:5 https://dl.bintray.com/sbt/debian  Release
  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 52.43.14.61 443]
Get:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [983 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [777 kB]
Reading package lists... Done
E: The repository 'https://dl.bintray.com/sbt/debian  Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Is there something else that I need to do?

@doublespaces

This comment has been minimized.

Copy link

@doublespaces doublespaces commented May 20, 2021

When i disable auto generation, i can't seem to create a /etc/resolve.conf (even with sudo i can't create the file)

This is probably happening because that /etc/resolv.conf is actually an existing symlink to another file. You need to delete the symlink and create the new file or you need to rename the file it points to and create a new version of it with the nameservers you desire.

I also recommend putting your own local wifi router's nameservers first on the list so that WSL2 will also function when you're not on the VPN without having to wait for the VPN DNS to timeout.

@neradp

This comment has been minimized.

Copy link

@neradp neradp commented May 21, 2021

@craibuc The script and task allows me to connect to the external network, but I'm having SSL-certificate issues: Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.

I had the same problem. My corporate network was protected by Zscaler proxy.
I just had to add Zscaler CA authorities to my distribution certification store.

@craibuc

This comment has been minimized.

Copy link

@craibuc craibuc commented May 21, 2021

@neradp How would I know if the organization is using that proxy? If it is, where do I find these certificates on Windows?

@moore-automation

This comment has been minimized.

Copy link

@moore-automation moore-automation commented Jun 15, 2021

I found by changing the run options to 'Run whether the user is logged in or not' was a good option as it hides the powershell prompt with the negative that you need to provide the password.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment