Skip to content

Instantly share code, notes, and snippets.

@kylemanna

kylemanna/0_gluetun_network_bug.md

Last active March 29, 2023 01:43
Star
Embed
What would you like to do?
Learn more about clone URLs
Download ZIP
Bug notes for gluetun multiple networks issue.
Raw
0_gluetun_network_bug.md

README

Issue

The gluetun container doesn't work correctly with multiple networks attached.

This is an example of a working and broken config with only minor differences.

Expected Output

Simple single network case

Update: This example is actually incorrect despite working in some instances. What's happening in this case is some of the packets are routed through the gateway and other's are bridged directly resultin in incorrect connection tracking state. On a router this test fails because the SYN-ACK packet is flagged as invalid by netfilter: forward rule ct state invalid counter packets 0 bytes 0 drop (verdict drop)

Working log entries:

gluetun-traefik-1  | Hostname: 45758167d34b
gluetun-traefik-1  | IP: 127.0.0.1
gluetun-traefik-1  | IP: 10.9.0.9
gluetun-traefik-1  | IP: 192.168.160.3
gluetun-traefik-1  | IP: 192.168.176.2
gluetun-traefik-1  | RemoteAddr: 192.168.160.2:48778
gluetun-traefik-1  | GET / HTTP/1.1
gluetun-traefik-1  | Host: gluetun
gluetun-traefik-1  | User-Agent: Wget
gluetun-traefik-1  | Connection: close
gluetun-traefik-1  |
gluetun-traefik-1  | Hostname: 45758167d34b
gluetun-traefik-1  | IP: 127.0.0.1
gluetun-traefik-1  | IP: 10.9.0.9
gluetun-traefik-1  | IP: 192.168.160.3
gluetun-traefik-1  | IP: 192.168.176.2
gluetun-traefik-1  | RemoteAddr: 192.168.160.2:48786
gluetun-traefik-1  | GET / HTTP/1.1
gluetun-traefik-1  | Host: gluetun
gluetun-traefik-1  | User-Agent: Wget
gluetun-traefik-1  | Connection: close
...

Broken multi-network case

Failing log entries:

broke-traefik-1  | wget: can't connect to remote host (192.168.160.3): Connection timed out
broke-traefik-1  | wget: can't connect to remote host (192.168.160.3): Connection timed out
broke-traefik-1  | wget: can't connect to remote host (192.168.160.3): Connection timed out
...

Possible Fix

The policy routing causes the response to be sent on the wrong interface.

~/c/broke ❯❯❯ docker compose exec gluetun sh -c 'apk add -u iproute2 curl tcpdump; ip -c addr ls; tcpdump -ni any port 80'
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz
(1/12) Upgrading musl (1.2.3-r0 -> 1.2.3-r2)
(2/12) Upgrading libcrypto1.1 (1.1.1q-r0 -> 1.1.1t-r1)
(3/12) Upgrading libssl1.1 (1.1.1q-r0 -> 1.1.1t-r1)
(4/12) Installing brotli-libs (1.0.9-r6)
(5/12) Installing nghttp2-libs (1.47.0-r0)
(6/12) Installing libcurl (7.83.1-r6)
(7/12) Installing curl (7.83.1-r6)
(8/12) Installing iproute2-tc (5.17.0-r0)
(9/12) Installing iproute2-ss (5.17.0-r0)
(10/12) Installing iproute2 (5.17.0-r0)
Executing iproute2-5.17.0-r0.post-install
(11/12) Installing libpcap (1.10.1-r0)
(12/12) Installing tcpdump (4.99.1-r3)
Executing busybox-1.35.0-r17.trigger
Executing ca-certificates-20220614-r0.trigger
OK: 26 MiB in 42 packages
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tun0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default
    link/none
    inet 10.9.0.9/24 brd 10.9.0.255 scope global tun0
       valid_lft forever preferred_lft forever
188: eth1@if189: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:c0:a8:a0:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.160.3/20 brd 192.168.175.255 scope global eth1
       valid_lft forever preferred_lft forever
190: eth0@if191: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:c0:a8:b0:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.176.2/20 brd 192.168.191.255 scope global eth0
       valid_lft forever preferred_lft forever
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
05:32:17.026835 eth1  In  IP 192.168.160.2.58646 > 192.168.160.3.80: Flags [S], seq 1257491925, win 64240, options [mss 1460,sackOK,TS val 3907762244 ecr 0,nop,wscale 7], length 0
05:32:17.026880 tun0  Out IP 192.168.160.3.80 > 192.168.160.2.58646: Flags [S.], seq 3376706450, ack 1257491926, win 64296, options [mss 1380,sackOK,TS val 2663978277 ecr 3907762244,nop,wscale 7], length 0
05:32:18.048113 tun0  Out IP 192.168.160.3.80 > 192.168.160.2.58646: Flags [S.], seq 3376706450, ack 1257491926, win 64296, options [mss 1380,sackOK,TS val 2663979299 ecr 3907762244,nop,wscale 7], length 0
05:32:18.048203 eth1  In  IP 192.168.160.2.58646 > 192.168.160.3.80: Flags [S], seq 1257491925, win 64240, options [mss 1460,sackOK,TS val 3907763266 ecr 0,nop,wscale 7], length 0
05:32:18.048232 tun0  Out IP 192.168.160.3.80 > 192.168.160.2.58646: Flags [S.], seq 3376706450, ack 1257491926, win 64296, options [mss 1380,sackOK,TS val 2663979299 ecr 3907762244,nop,wscale 7], length 0
05:32:20.074746 tun0  Out IP 192.168.160.3.80 > 192.168.160.2.58646: Flags [S.], seq 3376706450, ack 1257491926, win 64296, options [mss 1380,sackOK,TS val 2663981325 ecr 3907762244,nop,wscale 7], length 0
05:32:20.074839 eth1  In  IP 192.168.160.2.58646 > 192.168.160.3.80: Flags [S], seq 1257491925, win 64240, options [mss 1460,sackOK,TS val 3907765292 ecr 0,nop,wscale 7], length 0
05:32:20.074867 tun0  Out IP 192.168.160.3.80 > 192.168.160.2.58646: Flags [S.], seq 3376706450, ack 1257491926, win 64296, options [mss 1380,sackOK,TS val 2663981325 ecr 3907762244,nop,wscale 7], length 0
05:32:24.127890 tun0  Out IP 192.168.160.3.80 > 192.168.160.2.58646: Flags [S.], seq 3376706450, ack 1257491926, win 64296, options [mss 1380,sackOK,TS val 2663985378 ecr 3907762244,nop,wscale 7], length 0
05:32:24.127913 eth1  In  IP 192.168.160.2.58646 > 192.168.160.3.80: Flags [S], seq 1257491925, win 64240, options [mss 1460,sackOK,TS val 3907769345 ecr 0,nop,wscale 7], length 0
05:32:24.127924 tun0  Out IP 192.168.160.3.80 > 192.168.160.2.58646: Flags [S.], seq 3376706450, ack 1257491926, win 64296, options [mss 1380,sackOK,TS val 2663985378 ecr 3907762244,nop,wscale 7], length 0

Initial rules

$ docker compose exec gluetun ip rule ls
0:      from all lookup local
100:    from 192.168.176.2 lookup 200
101:    not from all fwmark 0xca6c lookup 51820
32766:  from all lookup main
32767:  from all lookup default

Fix

The main routing table is correct, it's just overridden by the VPN policies.

$ docker compose exec gluetun ip rule add to 192.168.160.0/24 lookup main

System recovers and expected logs are printed from wget of the whoami service.

Raw
docker-compose.broke.yaml
version: "3"
services:
gluetun:
#image: qmcgaw/gluetun
image: qmcgaw/gluetun:v3.32
cap_add:
- NET_ADMIN
environment:
- VPN_SERVICE_PROVIDER=custom
- VPN_TYPE=wireguard
- VPN_ENDPOINT_IP
- VPN_ENDPOINT_PORT
- WIREGUARD_PUBLIC_KEY
- WIREGUARD_PRIVATE_KEY
- WIREGUARD_PRESHARED_KEY
- WIREGUARD_ADDRESSES
#- FIREWALL_VPN_INPUT_PORTS=80
#- FIREWALL_OUTBOUND_SUBNETS=173.16.80.0/24
networks:
default:
traefik:
whoami:
image: traefik/whoami
network_mode: "service:gluetun"
traefik:
# Emulate a traefik load balancer by making requests from the "traefik
# reverse proxy" to the upsteram whoami service.
image: busybox
command: sh -c 'while true; do wget -qO- http://gluetun; sleep 1; done'
#command: sleep infinity
networks:
traefik:
networks:
default: # Used to access Internet / VPN
traefik: # Internal network to access reverse proxy
internal: true
Raw
docker-compose.single.yaml
version: "3"
services:
gluetun:
#image: qmcgaw/gluetun
image: qmcgaw/gluetun:v3.32
cap_add:
- NET_ADMIN
environment:
- VPN_SERVICE_PROVIDER=custom
- VPN_TYPE=wireguard
- VPN_ENDPOINT_IP
- VPN_ENDPOINT_PORT
- WIREGUARD_PUBLIC_KEY
- WIREGUARD_PRIVATE_KEY
- WIREGUARD_PRESHARED_KEY
- WIREGUARD_ADDRESSES
#- FIREWALL_VPN_INPUT_PORTS=80
#- FIREWALL_OUTBOUND_SUBNETS=173.16.80.0/24
networks:
default:
whoami:
image: traefik/whoami
network_mode: "service:gluetun"
traefik:
# Emulate a traefik load balancer by making requests from the "traefik
# reverse proxy" to the upsteram whoami service.
image: busybox
command: sh -c 'while true; do wget -qO- http://gluetun; sleep 1; done'
#command: sleep infinity
networks:
default:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment