Minimal test to verify if direct server return is actually working.

direct-server-return-test.md

Direct Server Return Test

Summary

According to the Google Cloud documentation and the technical support team, creating a LoadBalancer kubernetes service will create a load balancer with direct server return, and therefore the incoming connection source IP address will be that of the originating client and not the load balancer. However that does not appear to be the case. This is a test kubernetes service + deployment that logs the incoming connection IP in order to validate whether Direct Server Return is working correctly.

Setup

Step 1: Create the service + deployment

Run kubectl apply -f ./direct-server-return-test.yaml (See attached yaml file)

If this works correctly, you should see a pod running for the deployment:

maxweisel@Maxs-MacBook-Pro GCP-LoadBalancer-Test % kubectl get pods
NAME                                         READY   STATUS      RESTARTS   AGE
direct-server-return-test-97bb7b4dd-d7pcm   1/1     Running     1          10s

Step 2: Follow the logs for the deployment

Start following the logs for the pod using the pod name retrieved from the previous step. These logs are where we'll see the source IP of the incoming request.

maxweisel@Maxs-MacBook-Pro GCP-LoadBalancer-Test % kubectl logs direct-server-return-test-97bb7b4dd-d7pcm --follow
Ign:1 http://deb.debian.org/debian stretch InRelease
Get:2 http://deb.debian.org/debian stretch-updates InRelease [93.6 kB]
Get:3 http://deb.debian.org/debian stretch Release [118 kB]
Get:4 http://deb.debian.org/debian stretch Release.gpg [2410 B]
Get:5 http://security.debian.org/debian-security stretch/updates InRelease [53.0 kB]
Get:6 http://deb.debian.org/debian stretch-updates/main amd64 Packages [2596 B]
Get:7 http://deb.debian.org/debian stretch/main amd64 Packages [7080 kB]
Get:8 http://security.debian.org/debian-security stretch/updates/main amd64 Packages [558 kB]
Fetched 7907 kB in 1s (4086 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  netcat-traditional
The following NEW packages will be installed:
  netcat netcat-traditional
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 76.0 kB of archives.
After this operation, 173 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian stretch/main amd64 netcat-traditional amd64 1.10-41+b1 [67.0 kB]
Get:2 http://deb.debian.org/debian stretch/main amd64 netcat all 1.10-41 [8962 B]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 76.0 kB in 0s (5497 kB/s)
Selecting previously unselected package netcat-traditional.
(Reading database ... 6503 files and directories currently installed.)
Preparing to unpack .../netcat-traditional_1.10-41+b1_amd64.deb ...
Unpacking netcat-traditional (1.10-41+b1) ...
Selecting previously unselected package netcat.
Preparing to unpack .../netcat_1.10-41_all.deb ...
Unpacking netcat (1.10-41) ...
Setting up netcat-traditional (1.10-41+b1) ...
update-alternatives: using /bin/nc.traditional to provide /bin/nc (nc) in auto mode
Setting up netcat (1.10-41) ...
listening on [any] 4444 ...

Step 3: Get the service public IP

From a second terminal run kubectl get services

At first it may say <pending> under EXTERNAL-IP, give it about a minute and it will be assigned.

maxweisel@Maxs-MacBook-Pro GCP-LoadBalancer-Test % kubectl get services        
NAME                        TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
direct-server-return-test   LoadBalancer   10.113.8.74     35.199.30.241    4444:30281/TCP               62s

Once assigned, we're ready to test.

Step 4: Open a connection to the service

Use telnet to open a new connection to the service external IP address:

maxweisel@Maxs-MacBook-Pro GCP-LoadBalancer-Test % telnet 35.199.30.241 4444
Trying 35.199.30.241...
Connected to 241.30.199.35.bc.googleusercontent.com.
Escape character is '^]'.

If you see this, the connection was established successfully, and we should see a log for it in the first terminal window:

maxweisel@Maxs-MacBook-Pro GCP-LoadBalancer-Test % kubectl logs direct-server-return-test-97bb7b4dd-d7pcm --follow
[...]
listening on [any] 4444 ...
connect to [10.48.0.13] from gke-us-east4-b-default-pool-cef30296-z3wb.c.normal.internal [10.150.0.12] 58100

Normally, I would expect to see the IP address of the machine that I ran the telnet command from, however, netcat displays a local internal IP address of either the google load balancer, or another node in my node pool. The source IP address is never displayed.

Note: Once you've successfully connected once, the pod will need to be restarted in order to connect again.

direct-server-return-test.yaml

# Service
apiVersion: v1
kind: Service
metadata:
  name: direct-server-return-test
  labels:
    app: direct-server-return-test
spec:
  type: LoadBalancer
  selector:
    app: direct-server-return-test
  ports:
  - protocol: TCP
    port: 4444
    targetPort: 4444

---

# Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: direct-server-return-test
  labels:
    app: direct-server-return-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: direct-server-return-test
  template:
    metadata:
      labels:
        app: direct-server-return-test
    spec:
      containers:
      - name: direct-server-return-test
        image: debian:stretch
        command: ["/bin/sh","-c"]
        args: ["apt-get update && apt-get install -y netcat && nc -vv -l -p 4444"]