Manual test procedure for client cert authentication for backend TLS

README.md

Test procedure for client cert authentication for backend TLS

This document describes manual test procedure for projectcontour/contour#2910

Preparation

Generate the certificates for the test case by using https://github.com/tsaarni/certyaml

$ cat >certs.yaml <<EOF
subject: cn=internal-root-ca
---
subject: cn=httpbin
issuer: cn=internal-root-ca
sans:
- DNS:httpbin
---
subject: cn=envoy
issuer: cn=internal-root-ca
---
subject: cn=untrusted-client
ca: false
key_usage:
- KeyEncipherment
- DigitalSignature
EOF

$ mkdir certs
$ go get github.com/tsaarni/certyaml
$ go run github.com/tsaarni/certyaml --destination certs
Loading manifest file: certs.yaml
Reading certificate state file: certs/certs.state
Writing: certs/internal-root-ca.pem certs/internal-root-ca-key.pem
Writing: certs/httpbin.pem certs/httpbin-key.pem
Writing: certs/envoy.pem certs/envoy-key.pem
Writing: certs/untrusted-client.pem certs/untrusted-client-key.pem
Writing state: certs/certs.state

Following certificates were created:

• internal-root-ca: CA that issues certificates that are considered valid between services running within the Kubernetes cluster.
• httpbin: Server certificate for the backend service. The certificate is issued by internal-root-ca.
• envoy: Client certificate for Envoy. The certificate is issued by internal-root-ca.
• untrusted-client: Client certificate for Envoy. The certificate is self-signed and NOT issued by internal-root-ca, therefore backend will reject connections with this certificate.

Store the certificates and keys as secrets

$ kubectl create secret generic httpbin --dry-run=client -o yaml --from-file=certs/httpbin.pem --from-file=certs/httpbin-key.pem | kubectl apply -f -
$ kubectl create secret generic internal-root-ca --from-file=ca.crt=certs/internal-root-ca.pem --dry-run=client -o yaml | kubectl apply -f -
$ kubectl -n projectcontour create secret tls client --cert=certs/envoy.pem --key=certs/envoy-key.pem --dry-run=client -o yaml | kubectl apply -f -
$ kubectl -n projectcontour create secret tls untrusted-client --cert=certs/untrusted-client.pem --key=certs/untrusted-client-key.pem --dry-run=client -o yaml | kubectl apply -f -

Deploy backend service (see httpbin.yaml in this gist)

$ kubectl apply -f https://gist.githubusercontent.com/tsaarni/245584e8b4e40b9aa5a9924c7d13eadf/raw//httpbin.yaml

Note: The HTTPProxy in the manifest has been set up with FQDN assuming Envoy is listening at host1.127-0-0-101.nip.io. As well as curl requests in the following test cases are written to use host1.127-0-0-101.nip.io as the host name. Change these to fit your test environment!

Test cases

TC1: successful connection with valid client certificate

Run Contour with configuration file including envoy-client-certificate parameter:

$ cat >contour-config.yaml <<EOF
tls:
  envoy-client-certificate:
    name: client
    namespace: projectcontour
EOF

Wait for configuration to be propagated to Envoy.

Check that you get successful response from the backend service

$ curl http://host1.127-0-0-101.nip.io/headers
{
  "headers": {
    "Accept": "*/*",
    "Host": "host1.127-0-0-101.nip.io",
    "User-Agent": "curl/7.68.0",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000",
    "X-Envoy-Internal": "true"
  }
}

TC2: rotate client certificate

Remove envoy certificate and private key and regenerate them:

$ rm certs/envoy*
$ go run github.com/tsaarni/certyaml --destination certs
Loading manifest file: certs.yaml
Reading certificate state file: certs/certs.state
No changes: skipping internal-root-ca
No changes: skipping httpbin
Writing: certs/envoy.pem certs/envoy-key.pem
No changes: skipping untrusted-client
Writing state: certs/certs.state

Update the secret with the newly generated certificate and private key:

$ kubectl -n projectcontour create secret tls client --cert=certs/envoy.pem --key=certs/envoy-key.pem --dry-run=client -o yaml | kubectl apply -f -

Observe from Contour logs that the secret update was picked up.

Note: Following part of the procedure works only on Linux + kind. I don't know if it is possible to do this on macOS as well?

To prove that the new certificate was taken into use by Envoy, the network traffic needs to be captured between Envoy and httpbin to check that the client certificate sent by Envoy has really changed. One way to do this is by checking that the TLS handshake has the correct client certificate with the updated Not Before / Not After dates.

First check the new validity dates from the new certificate:

$ openssl x509 -in certs/envoy.pem -text | grep -A2 Validity
        Validity
            Not Before: Sep 16 05:38:44 2020 GMT
            Not After : Sep 16 05:38:44 2021 GMT

Run Wireshak in the network namespace of httpbin container, which in this case runs on hypercorn web server (similar to unicorn but http2 capable):

$ sudo nsenter --target $(pgrep hypercorn) --net wireshark -f "port 443" -k

# make a new request which will be captured by wireshark
$ curl http://host1.127-0-0-101.nip.io/headers
{
  "headers": {
    "Accept": "*/*",
    "Content-Length": "0",
    "Host": "host1.127-0-0-101.nip.io",
    "User-Agent": "curl/7.68.0",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000",
    "X-Envoy-Internal": "true"
  }
}

In wireshark, look for TLS 1.2 protocol message Certificate (the packet originating from Envoy, which is after Server Hello) and unfold the TLS message until you see the certificate validity times. Check that they match with the expected dates printed by openssl.

Note: if you do not see Certificate message, it is because authentication was already executed during previous request and TLS session resumption was used during this TLS handshake. This triggers shorter TLS handshake that does not include the certificates. You can rotate Envoy certificate again to trigger reset for the TLS state or alternatively restart httpbin pod (which then requires restarting wireshark as well).

TC3: unsuccessful connection with invalid client certificate

Run Contour with configuration file including envoy-client-certificate parameter:

$ cat >contour-config.yaml <<EOF
tls:
  envoy-client-certificate:
    name: untrusted-client
    namespace: projectcontour
EOF

Wait for configuration to be propagated to Envoy.

Check that request with invalid certificate is rejected by the backend service

$ curl http://host1.127-0-0-101.nip.io/headers
upstream connect error or disconnect/reset before headers. reset reason: connection failure

You can additionally run wireshark to observe that the Certificate message has a certificate with subject CN=untrusted-client.

TC4: unsuccessful connection without client certificate

Run Contour without envoy-client-certificate parameter.

Wait for configuration to be propagated to Envoy.

Check that request without certificate is rejected by the backend service

$ curl http://host1.127-0-0-101.nip.io/headers
upstream connect error or disconnect/reset before headers. reset reason: connection failure

You can additionally run wireshark to observe that the Certificate message has field Certificates Length with value 0. The Certificate message is still sent as a response to the backend server Certificate Request, but the lenght is zero, because Envoy was not configured with a client certificate.

httpbin.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      containers:
      - name: httpbin
        image: tsaarni/httpbin@sha256:d99e457f1b68400d5888d7b52c9e12e5de70ac76c46e454ef4dc8d6d1c0f0b6a
        ports:
          - containerPort: 443
        env:
          - name: X_SERVER_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
        volumeMounts:
        - mountPath: /run/secrets/certs/
          name: httpbin-cert
          readOnly: true
        - mountPath: /run/secrets/ca
          name: ca-cert
          readOnly: true
        command: ["/bin/sh"]
        # Configure HTTP server to require client certificate and validate it against trusted CA cert
        #     --cert-reqs 0 == ssl.CERT_NONE
        #     --cert-reqs 1 == ssl.CERT_OPTIONAL
        #     --cert-reqs 2 == ssl.CERT_REQUIRED
        args:
          - "-c"
          - "hypercorn -b 0.0.0.0:443 --access-logfile - --cert-reqs 2 --ca-cert /run/secrets/ca/ca.crt --certfile /run/secrets/certs/httpbin.pem --keyfile /run/secrets/certs/httpbin-key.pem httpbin:asyncio_app -k asyncio"
      volumes:
      - name: httpbin-cert
        secret:
          secretName: httpbin
      - name: ca-cert
        secret:
          secretName: internal-root-ca
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  annotations:
  labels:
    app: httpbin
spec:
  ports:
  - name: https
    port: 443
  selector:
    app: httpbin
---
apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: httpbin
spec:
  virtualhost:
    fqdn: host1.127-0-0-101.nip.io
  routes:
    - services:
        - name: httpbin
          port: 443
          protocol: tls
          validation:
            caSecret: internal-root-ca
            subjectName: httpbin