NGINX Plus Ingress Controller TLS Passthrough

README.md

TLS Passthrough in NGINX Plus Ingress Controller

This gist shows how to configure TLS passthrough in NGINX Plus Ingress Controller.

Currently, TLS passthrough is not supported with NGINX Plus Ingress Controller. Long-term, we will be adding support for TLS passthrough via our Custom resources. Short term, there is a workaround solution to enable TLS passthrough which you can find below.

In our example, we will deploy an Ingress resource with TLS termination and a TLS passthrough configuration and test that both work.

Prerequisites

1. Install the Ingress Controller v1.5.6. We're using the latest stable version as of October 7th, 2019.
2. Save the public IP address of the Ingress controller into a shell variable:

   $ IC_IP=XXX.YYY.ZZZ.III
   

3. Save the HTTPS port of the Ingress controller into a shell variable:

   $ IC_HTTPS_PORT=<port number>
   

Example

1. Deploy the cafe app:

   $ kubectl apply -f cafe-app.yaml
   

2. Create a secret and an Ingress resource for the cafe app:

   $ kubectl apply -f cafe-secret.yaml
   $ kubectl apply -f cafe-ingress.yaml
   

3. Deploy the secure app:

   $ kubectl apply -f secure-app.yaml
   

   As the app configures TLS termination in the pod, in the next step, we will use the TLS Passthrough feature to route connections to this app without decrypting the traffic.

4. Update the ConfigMap with custom templates and stream-snippets, required for TLS passthrough configuration.

   $ kubectl apply -f nginx-config.yaml
   

5. Verify that NGINX Plus routes the connection:

   For the cafe app:

   $ curl --resolve cafe.example.com:$IC_HTTPS_PORT:$IC_IP https://cafe.example.com:$IC_HTTPS_PORT/coffee -k
   Server address: 10.16.0.10:80
   Server name: coffee-more-676894c74b-wpjd2
   Date: 07/Oct/2019:19:11:30 +0000
   URI: /coffee
   Request ID: d306db1021210ac3d24986dfaf8603b2
   

   For the secure app:

   $ curl --resolve app.example.com:$IC_HTTPS_PORT:$IC_IP https://app.example.com:$IC_HTTPS_PORT -k
   here is your response
   

6. Verify the access logs include the correct client IP (see this doc about viewing the access logs):

   For the cafe app:

   127.0.0.1 - - [07/Oct/2019:19:11:30 +0000] "GET /coffee HTTP/1.1" 200 163 "-" "curl/7.54.0" "-"
   127.0.0.1 [07/Oct/2019:19:11:30 +0000] TCP 200 1911 244 0.241 cafe.example.com
   

   For the secure app:

   127.0.0.1 [07/Oct/2019:19:11:37 +0000] TCP 200 1726 237 0.189 app.example.com
   127.0.0.1 [07/Oct/2019:19:11:37 +0000] TCP 200 1726 237 0.246 app.example.com
   

Explanation of the ConfigMap

The ConfigMap includes the following changes to the templates. Those changes are represented as diffs:

• Ingress template:

  diff --git a/internal/configs/version1/nginx-plus.ingress.tmpl b/internal/configs/version1/nginx-plus.ingress.tmpl
  index 4f17c614..d877bcff 100644
  --- a/internal/configs/version1/nginx-plus.ingress.tmpl
  +++ b/internal/configs/version1/nginx-plus.ingress.tmpl
  @@ -27,7 +27,9 @@ server {
          {{end}}
          {{if $server.SSL}}
          {{- range $port := $server.SSLPorts}}
  -       listen {{$port}} ssl{{if $server.HTTP2}} http2{{end}}{{if $server.ProxyProtocol}} proxy_protocol{{end}};
  +       listen unix:/var/lib/nginx/ingress-https.sock ssl{{if $server.HTTP2}} http2{{end}} proxy_protocol;
  +       set_real_ip_from unix:;
  +       real_ip_header proxy_protocol;
          {{- end}}
          ssl_certificate {{$server.SSLCertificate}};
          ssl_certificate_key {{$server.SSLCertificateKey}};

  Here instead of binding to an SSL port(s), which is typically 443, we're biding to a unix socket. A separate server in the stream block in the main configuration file will pass connections for any Ingress hosts with TLS termination to this unix socket. We need to enable PROXY protocol so that the client IP address is preserved.

• Main template:

  diff --git a/internal/configs/version1/nginx-plus.tmpl b/internal/configs/version1/nginx-plus.tmpl
  index 8855ec2b..0835911c 100644
  --- a/internal/configs/version1/nginx-plus.tmpl
  +++ b/internal/configs/version1/nginx-plus.tmpl
  @@ -76,7 +76,9 @@ http {
  
      server {
          listen 80 default_server{{if .ProxyProtocol}} proxy_protocol{{end}};
  -        listen 443 ssl default_server{{if .HTTP2}} http2{{end}}{{if .ProxyProtocol}} proxy_protocol{{end}};
  +        listen unix:/var/lib/nginx/ingress-https.sock ssl{{if .HTTP2}} http2{{end}} default_server proxy_protocol;
  +        set_real_ip_from unix:;
  +        real_ip_header proxy_protocol;
  
          ssl_certificate /etc/nginx/secrets/default;
          ssl_certificate_key /etc/nginx/secrets/default;
  @@ -153,4 +155,26 @@ stream {
  
      {{range $value := .StreamSnippets}}
      {{$value}}{{end}}
  +
  +    server {
  +        listen 443;
  +
  +        # if PROXY protocol is required
  +        # listen 443  proxy_protocol;
  +        # set_real_ip_from 0.0.0.0/0;
  +
  +        ssl_preread on;
  +
  +        proxy_protocol on;
  +        proxy_pass $dest_internal_passthrough;
  +    }
  +
  +    server {
  +        listen unix:/var/lib/nginx/passthrough.sock proxy_protocol;
  +        set_real_ip_from unix:;
  +
  +        ssl_preread on;
  +
  +        proxy_pass $dest_upstream;
  +    }
  }

  Here we make a change to the default server, similar to the change in the Ingress template.

  We also have a stream server that listens on 443. This server will examine the connection SNI and pass the connection to $dest_upstream, which is set in the stream-snippets, see below. The PROXY protocol is enabled (proxy_protocol on;), so that the client IP is preserved.

We also configure:

• stream-snippets:

  resolver kube-dns.kube-system.svc.cluster.local valid=5s;
  
  map $ssl_preread_server_name $dest_internal_passthrough {
    app.example.com unix:/var/lib/nginx/passthrough.sock;
    default unix:/var/lib/nginx/ingress-https.sock; 
  }
  
  map $ssl_preread_server_name $dest_upstream {
     app.example.com secure-app;
  }
  
  upstream secure-app {
     zone secure-app 256k;
  
     server secure-app.default.svc.cluster.local service=_https._tcp resolve; 
  }

  Here we set the resolver to KubeDNS so that we can dynamically resolve the IP addresses behind the secure-app service. For the secure-app service we create a corresponding upstream. Note that the service must be created as Headless. Otherwise, the server secure-app.default.svc.cluster.local will be resolved into the service VIP.

  We also configure two maps that will route connections to appropriate servers.

  As a rule, for each new passthrough route, we need to do the following:

  1. Add an entry to the map $ssl_preread_server_name $dest_internal_passthrough map.
  2. Add an entry to the map $ssl_preread_server_name $dest_upstream map.
  3. Add an upstream.

  All those changes can be done by modifying the stream-snippets key. No other changes are necessary.

• And stream-log-format.

cafe-app.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: coffee
spec:
  replicas: 2
  selector:
    matchLabels:
      app: coffee
  template:
    metadata:
      labels:
        app: coffee
    spec:
      containers:
      - name: coffee
        image: nginxdemos/hello:plain-text
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: coffee-svc
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: coffee
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tea
spec:
  replicas: 3
  selector:
    matchLabels:
      app: tea 
  template:
    metadata:
      labels:
        app: tea 
    spec:
      containers:
      - name: tea 
        image: nginxdemos/hello:plain-text
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: tea-svc
  labels:
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: tea

cafe-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - cafe.example.com
    secretName: cafe-secret
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80
      - path: /coffee
        backend:
          serviceName: coffee-svc
          servicePort: 80

cafe-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: cafe-secret
type: Opaque
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURMakNDQWhZQ0NRREFPRjl0THNhWFdqQU5CZ2txaGtpRzl3MEJBUXNGQURCYU1Rc3dDUVlEVlFRR0V3SlYKVXpFTE1Ba0dBMVVFQ0F3Q1EwRXhJVEFmQmdOVkJBb01HRWx1ZEdWeWJtVjBJRmRwWkdkcGRITWdVSFI1SUV4MApaREViTUJrR0ExVUVBd3dTWTJGbVpTNWxlR0Z0Y0d4bExtTnZiU0FnTUI0WERURTRNRGt4TWpFMk1UVXpOVm9YCkRUSXpNRGt4TVRFMk1UVXpOVm93V0RFTE1Ba0dBMVVFQmhNQ1ZWTXhDekFKQmdOVkJBZ01Ba05CTVNFd0h3WUQKVlFRS0RCaEpiblJsY201bGRDQlhhV1JuYVhSeklGQjBlU0JNZEdReEdUQVhCZ05WQkFNTUVHTmhabVV1WlhoaApiWEJzWlM1amIyMHdnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFDcDZLbjdzeTgxCnAwanVKL2N5ayt2Q0FtbHNmanRGTTJtdVpOSzBLdGVjcUcyZmpXUWI1NXhRMVlGQTJYT1N3SEFZdlNkd0kyaloKcnVXOHFYWENMMnJiNENaQ0Z4d3BWRUNyY3hkam0zdGVWaVJYVnNZSW1tSkhQUFN5UWdwaW9iczl4N0RsTGM2SQpCQTBaalVPeWwwUHFHOVNKZXhNVjczV0lJYTVyRFZTRjJyNGtTa2JBajREY2o3TFhlRmxWWEgySTVYd1hDcHRDCm42N0pDZzQyZitrOHdnemNSVnA4WFprWldaVmp3cTlSVUtEWG1GQjJZeU4xWEVXZFowZXdSdUtZVUpsc202OTIKc2tPcktRajB2a29QbjQxRUUvK1RhVkVwcUxUUm9VWTNyemc3RGtkemZkQml6Rk8yZHNQTkZ4MkNXMGpYa05MdgpLbzI1Q1pyT2hYQUhBZ01CQUFFd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFLSEZDY3lPalp2b0hzd1VCTWRMClJkSEliMzgzcFdGeW5acS9MdVVvdnNWQTU4QjBDZzdCRWZ5NXZXVlZycTVSSWt2NGxaODFOMjl4MjFkMUpINnIKalNuUXgrRFhDTy9USkVWNWxTQ1VwSUd6RVVZYVVQZ1J5anNNL05VZENKOHVIVmhaSitTNkZBK0NuT0Q5cm4yaQpaQmVQQ0k1ckh3RVh3bm5sOHl3aWozdnZRNXpISXV5QmdsV3IvUXl1aTlmalBwd1dVdlVtNG52NVNNRzl6Q1Y3ClBwdXd2dWF0cWpPMTIwOEJqZkUvY1pISWc4SHc5bXZXOXg5QytJUU1JTURFN2IvZzZPY0s3TEdUTHdsRnh2QTgKN1dqRWVxdW5heUlwaE1oS1JYVmYxTjM0OWVOOThFejM4Zk9USFRQYmRKakZBL1BjQytHeW1lK2lHdDVPUWRGaAp5UkU9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBcWVpcCs3TXZOYWRJN2lmM01wUHJ3Z0pwYkg0N1JUTnBybVRTdENyWG5LaHRuNDFrCkcrZWNVTldCUU5semtzQndHTDBuY0NObzJhN2x2S2wxd2k5cTIrQW1RaGNjS1ZSQXEzTVhZNXQ3WGxZa1YxYkcKQ0pwaVJ6ejBza0lLWXFHN1BjZXc1UzNPaUFRTkdZMURzcGRENmh2VWlYc1RGZTkxaUNHdWF3MVVoZHErSkVwRwp3SStBM0kreTEzaFpWVng5aU9WOEZ3cWJRcCt1eVFvT05uL3BQTUlNM0VWYWZGMlpHVm1WWThLdlVWQ2cxNWhRCmRtTWpkVnhGbldkSHNFYmltRkNaYkp1dmRySkRxeWtJOUw1S0Q1K05SQlAvazJsUkthaTAwYUZHTjY4NE93NUgKYzMzUVlzeFR0bmJEelJjZGdsdEkxNURTN3lxTnVRbWF6b1Z3QndJREFRQUJBb0lCQVFDUFNkU1luUXRTUHlxbApGZlZGcFRPc29PWVJoZjhzSStpYkZ4SU91UmF1V2VoaEp4ZG01Uk9ScEF6bUNMeUw1VmhqdEptZTIyM2dMcncyCk45OUVqVUtiL1ZPbVp1RHNCYzZvQ0Y2UU5SNThkejhjbk9SVGV3Y290c0pSMXBuMWhobG5SNUhxSkpCSmFzazEKWkVuVVFmY1hackw5NGxvOUpIM0UrVXFqbzFGRnM4eHhFOHdvUEJxalpzVjdwUlVaZ0MzTGh4bndMU0V4eUZvNApjeGI5U09HNU9tQUpvelN0Rm9RMkdKT2VzOHJKNXFmZHZ5dGdnOXhiTGFRTC94MGtwUTYyQm9GTUJEZHFPZVBXCktmUDV6WjYvMDcvdnBqNDh5QTFRMzJQem9idWJzQkxkM0tjbjMyamZtMUU3cHJ0V2wrSmVPRmlPem5CUUZKYk4KNHFQVlJ6NWhBb0dCQU50V3l4aE5DU0x1NFArWGdLeWNrbGpKNkY1NjY4Zk5qNUN6Z0ZScUowOXpuMFRsc05ybwpGVExaY3hEcW5SM0hQWU00MkpFUmgySi9xREZaeW5SUW8zY2czb2VpdlVkQlZHWTgrRkkxVzBxZHViL0w5K3l1CmVkT1pUUTVYbUdHcDZyNmpleHltY0ppbS9Pc0IzWm5ZT3BPcmxEN1NQbUJ2ek5MazRNRjZneGJYQW9HQkFNWk8KMHA2SGJCbWNQMHRqRlhmY0tFNzdJbUxtMHNBRzR1SG9VeDBlUGovMnFyblRuT0JCTkU0TXZnRHVUSnp5K2NhVQprOFJxbWRIQ2JIelRlNmZ6WXEvOWl0OHNaNzdLVk4xcWtiSWN1YytSVHhBOW5OaDFUanNSbmU3NFowajFGQ0xrCmhIY3FIMHJpN1BZU0tIVEU4RnZGQ3haWWRidUI4NENtWmlodnhicFJBb0dBSWJqcWFNWVBUWXVrbENkYTVTNzkKWVNGSjFKelplMUtqYS8vdER3MXpGY2dWQ0thMzFqQXdjaXowZi9sU1JxM0hTMUdHR21lemhQVlRpcUxmZVpxYwpSMGlLYmhnYk9jVlZrSkozSzB5QXlLd1BUdW14S0haNnpJbVpTMGMwYW0rUlk5WUdxNVQ3WXJ6cHpjZnZwaU9VCmZmZTNSeUZUN2NmQ21mb09oREN0enVrQ2dZQjMwb0xDMVJMRk9ycW40M3ZDUzUxemM1em9ZNDR1QnpzcHd3WU4KVHd2UC9FeFdNZjNWSnJEakJDSCtULzZzeXNlUGJKRUltbHpNK0l3eXRGcEFOZmlJWEV0LzQ4WGY2ME54OGdXTQp1SHl4Wlp4L05LdER3MFY4dlgxUE9ucTJBNWVpS2ErOGpSQVJZS0pMWU5kZkR1d29seHZHNmJaaGtQaS80RXRUCjNZMThzUUtCZ0h0S2JrKzdsTkpWZXN3WEU1Y1VHNkVEVXNEZS8yVWE3ZlhwN0ZjanFCRW9hcDFMU3crNlRYcDAKWmdybUtFOEFSek00NytFSkhVdmlpcS9udXBFMTVnMGtKVzNzeWhwVTl6WkxPN2x0QjBLSWtPOVpSY21Vam84UQpjcExsSE1BcWJMSjhXWUdKQ2toaVd4eWFsNmhZVHlXWTRjVmtDMHh0VGwvaFVFOUllTktvCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==

nginx-config.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-config
  namespace: nginx-ingress
data:
  ingress-template: |
    {{range $upstream := .Upstreams}}
    upstream {{$upstream.Name}} {
      zone {{$upstream.Name}} 256k;
      {{if $upstream.LBMethod }}{{$upstream.LBMethod}};{{end}}
      {{range $server := $upstream.UpstreamServers}}
      server {{$server.Address}}:{{$server.Port}} max_fails={{$server.MaxFails}} fail_timeout={{$server.FailTimeout}}
          {{- if $server.SlowStart}} slow_start={{$server.SlowStart}}{{end}}{{if $server.Resolve}} resolve{{end}};{{end}}
      {{if $upstream.StickyCookie}}
      sticky cookie {{$upstream.StickyCookie}};
      {{end}}
      {{if $.Keepalive}}keepalive {{$.Keepalive}};{{end}}
      {{- if $upstream.UpstreamServers -}}
      {{- if $upstream.Queue}}
      queue {{$upstream.Queue}} timeout={{$upstream.QueueTimeout}}s;
      {{- end -}}
      {{- end}}
     }
    {{- end}}

    {{range $server := .Servers}}
    server {
      {{if not $server.GRPCOnly}}
      {{range $port := $server.Ports}}
      listen {{$port}}{{if $server.ProxyProtocol}} proxy_protocol{{end}};
      {{- end}}
      {{end}}
      {{if $server.SSL}}
      {{- range $port := $server.SSLPorts}}
      listen unix:/var/lib/nginx/ingress-https.sock ssl{{if $server.HTTP2}} http2{{end}} proxy_protocol;
      set_real_ip_from unix:;
      real_ip_header proxy_protocol;
      {{- end}}
      ssl_certificate {{$server.SSLCertificate}};
      ssl_certificate_key {{$server.SSLCertificateKey}};
      {{if $server.SSLCiphers}}
      ssl_ciphers {{$server.SSLCiphers}};
      {{end}}
      {{end}}
      {{range $setRealIPFrom := $server.SetRealIPFrom}}
      set_real_ip_from {{$setRealIPFrom}};{{end}}
      {{if $server.RealIPHeader}}real_ip_header {{$server.RealIPHeader}};{{end}}
      {{if $server.RealIPRecursive}}real_ip_recursive on;{{end}}

      server_tokens "{{$server.ServerTokens}}";

      server_name {{$server.Name}};

      status_zone {{$server.StatusZone}};

      {{if not $server.GRPCOnly}}
      {{range $proxyHideHeader := $server.ProxyHideHeaders}}
      proxy_hide_header {{$proxyHideHeader}};{{end}}
      {{range $proxyPassHeader := $server.ProxyPassHeaders}}
      proxy_pass_header {{$proxyPassHeader}};{{end}}
      {{end}}

      {{if $server.SSL}}
      {{if not $server.GRPCOnly}}
      {{- if $server.HSTS}}
      set $hsts_header_val "";
      proxy_hide_header Strict-Transport-Security;
      {{- if $server.HSTSBehindProxy}}
      if ($http_x_forwarded_proto = 'https') {
      {{else}}
      if ($https = on) {
      {{- end}}
        set $hsts_header_val "max-age={{$server.HSTSMaxAge}}; {{if $server.HSTSIncludeSubdomains}}includeSubDomains; {{end}}preload";
      }

      add_header Strict-Transport-Security "$hsts_header_val" always;
      {{end}}

      {{- if $server.SSLRedirect}}
      if ($scheme = http) {
        return 301 https://$host:{{index $server.SSLPorts 0}}$request_uri;
      }
      {{- end}}
      {{end}}
      {{- end}}

      {{- if $server.RedirectToHTTPS}}
      if ($http_x_forwarded_proto = 'http') {
        return 301 https://$host$request_uri;
      }
      {{- end}}

      {{with $jwt := $server.JWTAuth}}
      auth_jwt_key_file {{$jwt.Key}};
      auth_jwt "{{.Realm}}"{{if $jwt.Token}} token={{$jwt.Token}}{{end}};

      {{- if $jwt.RedirectLocationName}}
      error_page 401 {{$jwt.RedirectLocationName}};
      {{end}}
      {{end}}

      {{- if $server.ServerSnippets}}
      {{range $value := $server.ServerSnippets}}
      {{$value}}{{end}}
      {{- end}}

      {{- range $healthCheck := $server.HealthChecks}}
      location @hc-{{$healthCheck.UpstreamName}} {
        {{- range $name, $header := $healthCheck.Headers}}
        proxy_set_header {{$name}} "{{$header}}";
        {{- end }}
        proxy_connect_timeout {{$healthCheck.TimeoutSeconds}}s;
        proxy_read_timeout {{$healthCheck.TimeoutSeconds}}s;
        proxy_send_timeout {{$healthCheck.TimeoutSeconds}}s;
        proxy_pass {{$healthCheck.Scheme}}://{{$healthCheck.UpstreamName}};
        health_check {{if $healthCheck.Mandatory}}mandatory {{end}}uri={{$healthCheck.URI}} interval=
          {{- $healthCheck.Interval}}s fails={{$healthCheck.Fails}} passes={{$healthCheck.Passes}};
      }
      {{end -}}

      {{- range $location := $server.JWTRedirectLocations}}
      location {{$location.Name}} {
        internal;
        return 302 {{$location.LoginURL}};
      }
      {{end -}}

      {{range $location := $server.Locations}}
      location {{$location.Path}} {
        {{with $location.MinionIngress}}
        # location for minion {{$location.MinionIngress.Namespace}}/{{$location.MinionIngress.Name}}
        {{end}}
        {{if $location.GRPC}}
        {{if not $server.GRPCOnly}}
        error_page 400 @grpcerror400;
        error_page 401 @grpcerror401;
        error_page 403 @grpcerror403;
        error_page 404 @grpcerror404;
        error_page 405 @grpcerror405;
        error_page 408 @grpcerror408;
        error_page 414 @grpcerror414;
        error_page 426 @grpcerror426;
        error_page 500 @grpcerror500;
        error_page 501 @grpcerror501;
        error_page 502 @grpcerror502;
        error_page 503 @grpcerror503;
        error_page 504 @grpcerror504;
        {{end}}

        {{- if $location.LocationSnippets}}
        {{range $value := $location.LocationSnippets}}
        {{$value}}{{end}}
        {{- end}}

        {{with $jwt := $location.JWTAuth}}
        auth_jwt_key_file {{$jwt.Key}};
        auth_jwt "{{.Realm}}"{{if $jwt.Token}} token={{$jwt.Token}}{{end}};
        {{end}}

        grpc_connect_timeout {{$location.ProxyConnectTimeout}};
        grpc_read_timeout {{$location.ProxyReadTimeout}};
        grpc_set_header Host $host;
        grpc_set_header X-Real-IP $remote_addr;
        grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        grpc_set_header X-Forwarded-Host $host;
        grpc_set_header X-Forwarded-Port $server_port;
        grpc_set_header X-Forwarded-Proto $scheme;

        {{- if $location.ProxyBufferSize}}
        grpc_buffer_size {{$location.ProxyBufferSize}};
        {{- end}}

        {{if $location.SSL}}
        grpc_pass grpcs://{{$location.Upstream.Name}}
        {{else}}
        grpc_pass grpc://{{$location.Upstream.Name}};
        {{end}}
        {{else}}
        proxy_http_version 1.1;
        {{if $location.Websocket}}
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        {{- else}}
        {{- if $.Keepalive}}proxy_set_header Connection "";{{end}}
        {{- end}}

        {{- if $location.LocationSnippets}}
        {{range $value := $location.LocationSnippets}}
        {{$value}}{{end}}
        {{- end}}

        {{ with $jwt := $location.JWTAuth }}
        auth_jwt_key_file {{$jwt.Key}};
        auth_jwt "{{.Realm}}"{{if $jwt.Token}} token={{$jwt.Token}}{{end}};
        {{if $jwt.RedirectLocationName}}
        error_page 401 {{$jwt.RedirectLocationName}};
        {{end}}
        {{end}}

        proxy_connect_timeout {{$location.ProxyConnectTimeout}};
        proxy_read_timeout {{$location.ProxyReadTimeout}};
        client_max_body_size {{$location.ClientMaxBodySize}};
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto {{if $server.RedirectToHTTPS}}https{{else}}$scheme{{end}};
        proxy_buffering {{if $location.ProxyBuffering}}on{{else}}off{{end}};
        {{- if $location.ProxyBuffers}}
        proxy_buffers {{$location.ProxyBuffers}};
        {{- end}}
        {{- if $location.ProxyBufferSize}}
        proxy_buffer_size {{$location.ProxyBufferSize}};
        {{- end}}
        {{- if $location.ProxyMaxTempFileSize}}
        proxy_max_temp_file_size {{$location.ProxyMaxTempFileSize}};
        {{- end}}
        {{if $location.SSL}}
        proxy_pass https://{{$location.Upstream.Name}}{{$location.Rewrite}};
        {{else}}
        proxy_pass http://{{$location.Upstream.Name}}{{$location.Rewrite}};
        {{end}}
        {{end}}
      }{{end}}
      {{if $server.GRPCOnly}}
      error_page 400 @grpcerror400;
      error_page 401 @grpcerror401;
      error_page 403 @grpcerror403;
      error_page 404 @grpcerror404;
      error_page 405 @grpcerror405;
      error_page 408 @grpcerror408;
      error_page 414 @grpcerror414;
      error_page 426 @grpcerror426;
      error_page 500 @grpcerror500;
      error_page 501 @grpcerror501;
      error_page 502 @grpcerror502;
      error_page 503 @grpcerror503;
      error_page 504 @grpcerror504;
      {{end}}
      {{if $server.HTTP2}}
      location @grpcerror400 { default_type application/grpc; return 400 "\n"; }
      location @grpcerror401 { default_type application/grpc; return 401 "\n"; }
      location @grpcerror403 { default_type application/grpc; return 403 "\n"; }
      location @grpcerror404 { default_type application/grpc; return 404 "\n"; }
      location @grpcerror405 { default_type application/grpc; return 405 "\n"; }
      location @grpcerror408 { default_type application/grpc; return 408 "\n"; }
      location @grpcerror414 { default_type application/grpc; return 414 "\n"; }
      location @grpcerror426 { default_type application/grpc; return 426 "\n"; }
      location @grpcerror500 { default_type application/grpc; return 500 "\n"; }
      location @grpcerror501 { default_type application/grpc; return 501 "\n"; }
      location @grpcerror502 { default_type application/grpc; return 502 "\n"; }
      location @grpcerror503 { default_type application/grpc; return 503 "\n"; }
      location @grpcerror504 { default_type application/grpc; return 504 "\n"; }
      {{end}}
    }{{end}}
  main-template: |
    user  nginx;
    worker_processes  {{.WorkerProcesses}};
    {{- if .WorkerRlimitNofile}}
    worker_rlimit_nofile {{.WorkerRlimitNofile}};{{end}}
    {{- if .WorkerCPUAffinity}}
    worker_cpu_affinity {{.WorkerCPUAffinity}};{{end}}
    {{- if .WorkerShutdownTimeout}}
    worker_shutdown_timeout {{.WorkerShutdownTimeout}};{{end}}

    daemon off;

    error_log  /var/log/nginx/error.log {{.ErrorLogLevel}};
    pid        /var/run/nginx.pid;

    {{- if .MainSnippets}}
    {{range $value := .MainSnippets}}
    {{$value}}{{end}}
    {{- end}}

    events {
        worker_connections  {{.WorkerConnections}};
    }

    http {
        include       /etc/nginx/mime.types;
        default_type  application/octet-stream;

        {{- if .HTTPSnippets}}
        {{range $value := .HTTPSnippets}}
        {{$value}}{{end}}
        {{- end}}

        {{if .LogFormat -}}
        log_format  main  '{{.LogFormat}}';
        {{- else -}}
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';
        {{- end}}

        {{if .AccessLogOff}}
        access_log off;
        {{else}}
        access_log  /var/log/nginx/access.log  main;
        {{end}}

        sendfile        on;
        #tcp_nopush     on;

        keepalive_timeout {{.KeepaliveTimeout}};
        keepalive_requests {{.KeepaliveRequests}};

        #gzip  on;

        server_names_hash_max_size {{.ServerNamesHashMaxSize}};
        {{if .ServerNamesHashBucketSize}}server_names_hash_bucket_size {{.ServerNamesHashBucketSize}};{{end}}

        variables_hash_bucket_size {{.VariablesHashBucketSize}};
        variables_hash_max_size {{.VariablesHashMaxSize}};

        map $http_upgrade $connection_upgrade {
            default upgrade;
            ''      close;
        }

        {{if .SSLProtocols}}ssl_protocols {{.SSLProtocols}};{{end}}
        {{if .SSLCiphers}}ssl_ciphers "{{.SSLCiphers}}";{{end}}
        {{if .SSLPreferServerCiphers}}ssl_prefer_server_ciphers on;{{end}}
        {{if .SSLDHParam}}ssl_dhparam {{.SSLDHParam}};{{end}}

        {{if .ResolverAddresses}}
        resolver {{range $resolver := .ResolverAddresses}}{{$resolver}}{{end}}{{if .ResolverValid}} valid={{.ResolverValid}}{{end}}{{if not .ResolverIPV6}} ipv6=off{{end}};
        {{if .ResolverTimeout}}resolver_timeout {{.ResolverTimeout}};{{end}}
        {{end}}

        server {
            listen 80 default_server{{if .ProxyProtocol}} proxy_protocol{{end}};
          listen unix:/var/lib/nginx/ingress-https.sock ssl{{if .HTTP2}} http2{{end}} default_server proxy_protocol;
            set_real_ip_from unix:;
            real_ip_header proxy_protocol;

            ssl_certificate /etc/nginx/secrets/default;
            ssl_certificate_key /etc/nginx/secrets/default;

            server_name _;
            server_tokens "{{.ServerTokens}}";
            access_log off;

            {{if .HealthStatus}}
            location /nginx-health {
                default_type text/plain;
                return 200 "healthy\n";
            }
            {{end}}

            location / {
              return 404;
            }
        }

        {{- if .NginxStatus}}
        # NGINX Plus APIs
        server {
            listen {{.NginxStatusPort}};

            root /usr/share/nginx/html;

            access_log off;

            location  = /dashboard.html {
            }
            {{range $value := .NginxStatusAllowCIDRs}}
            allow {{$value}};{{end}}

            deny all;
            location /api {
                api write=off;
            }
        }
        {{- end}}

        # NGINX Plus API over unix socket
        server {
            listen unix:/var/run/nginx-plus-api.sock;
            access_log off;

            # $config_version_mismatch is defined in /etc/nginx/config-version.conf
            location /configVersionCheck {
                if ($config_version_mismatch) {
                    return 503;
                }
                return 200;
            }

            location /api {
                api write=on;
            }
        }

        include /etc/nginx/config-version.conf;
        include /etc/nginx/conf.d/*.conf;
    }

    stream {
        {{if .StreamLogFormat -}}
        log_format  stream-main  '{{.StreamLogFormat}}';
        {{- else -}}
        log_format  stream-main  '$remote_addr [$time_local] '
                          '$protocol $status $bytes_sent $bytes_received '
                          '$session_time';
        {{- end}}

        access_log  /var/log/nginx/stream-access.log  stream-main;

        {{range $value := .StreamSnippets}}
        {{$value}}{{end}}

        server {
            listen 443;

            # if PROXY protocol is required
            # listen 443  proxy_protocol;
            # set_real_ip_from 0.0.0.0/0;

            ssl_preread on;

            proxy_protocol on;
            proxy_pass $dest_internal_passthrough;
        }

        server {
            listen unix:/var/lib/nginx/passthrough.sock proxy_protocol;
            set_real_ip_from unix:;

            ssl_preread on;

            proxy_pass $dest_upstream; 
        }
    }
  stream-log-format: "$remote_addr [$time_local] $protocol $status $bytes_sent $bytes_received $session_time $ssl_preread_server_name"
  stream-snippets: |
    resolver kube-dns.kube-system.svc.cluster.local valid=5s;

    map $ssl_preread_server_name $dest_internal_passthrough {
      app.example.com unix:/var/lib/nginx/passthrough.sock;
      default unix:/var/lib/nginx/ingress-https.sock; 
    }

    map $ssl_preread_server_name $dest_upstream {
       app.example.com secure-app;
    }

    upstream secure-app {
       zone secure-app 256k;

       server secure-app.default.svc.cluster.local service=_https._tcp resolve; 
    }
    

secure-app.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-app 
spec:
  replicas: 1
  selector:
    matchLabels:
      app: secure-app
  template:
    metadata:
      labels:
        app: secure-app
    spec:
      containers:
      - name: secure-app
        image: nginx 
        ports:
        - containerPort: 443 
        volumeMounts:
          - name: secret
            mountPath: "/etc/nginx/ssl"
            readOnly: true
          - name: config-volume
            mountPath: /etc/nginx/conf.d
      volumes:
      - name: secret
        secret:
          secretName: app-tls-secret
      - name: config-volume
        configMap:
          name: secure-config 
---
apiVersion: v1
kind: Service
metadata:
  name: secure-app
spec:
  clusterIP: None
  ports:
  - port: 443 
    targetPort: 443 
    protocol: TCP
    name: https
  selector:
    app: secure-app
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: secure-config
data:
  app.conf: |-
    server {
      listen 443 ssl;

      server_name app.example.com;

      ssl_certificate /etc/nginx/ssl/tls.crt;
      ssl_certificate_key /etc/nginx/ssl/tls.key;

      default_type text/plain;

      location / {
        return 200 "here is your response\n";
      }
    }
---
apiVersion: v1
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVCakNDQXU2Z0F3SUJBZ0lKQUpicWVnTHB0U2JhTUEwR0NTcUdTSWIzRFFFQkJRVUFNRjh4Q3pBSkJnTlYKQkFZVEFrZENNUk13RVFZRFZRUUlFd3BUYjIxbExWTjBZWFJsTVNFd0h3WURWUVFLRXhoSmJuUmxjbTVsZENCWAphV1JuYVhSeklGQjBlU0JNZEdReEdEQVdCZ05WQkFNVEQyRndjQzVsZUdGdGNHeGxMbU52YlRBZUZ3MHhPREF5Ck1USXdNREF6TkRWYUZ3MHhPVEF5TVRJd01EQXpORFZhTUY4eEN6QUpCZ05WQkFZVEFrZENNUk13RVFZRFZRUUkKRXdwVGIyMWxMVk4wWVhSbE1TRXdId1lEVlFRS0V4aEpiblJsY201bGRDQlhhV1JuYVhSeklGQjBlU0JNZEdReApHREFXQmdOVkJBTVREMkZ3Y0M1bGVHRnRjR3hsTG1OdmJUQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQCkFEQ0NBUW9DZ2dFQkFLenNmMll0R2hVU0lyYWpTS1ZLSVBrTmFzODcrTzJDaHlsZTduL212V012WFJZZWI2R3oKQktKV3FkSS9UajlQQlJxTWVzajByMjF5UlAwaVc0VVBTYjZNT3psNisyYjBJeS9nTEhJRGxJN0NDTVU5cThHSAorL3Y4ZjAyMXJWYmUrNGdsWmZWVTZJbXg2Vlc0ODkzVTcwQXR6Y1hGNnFDUGRUWDNjWW02MTVmNE02M1YzdTdqClJGN1JINzBDL1NScVVvN29FVmZxR0thN1prdWVodnlLSWZURE5hQUt0WFhDLzlCeDlYSDIyREFxcTVKRUhHVHAKSVluRFE4eFdFRXlUQmx1V2JwU0JwUEVMRDcyUHhwQW9DU0trdVdXSzJYbmlKOG9BTFZJWlhaaHFvamw4Sk5SMgpiNWE3RFJEcTNTYzNNSzhwMEwzZXFsLzRPcnhjUGdJUVdtTUNBd0VBQWFPQnhEQ0J3VEFkQmdOVkhRNEVGZ1FVCmtvK2owNGJWaDZyTjdCbk8wbjRLMUo4S2tIRXdnWkVHQTFVZEl3U0JpVENCaG9BVWtvK2owNGJWaDZyTjdCbk8KMG40SzFKOEtrSEdoWTZSaE1GOHhDekFKQmdOVkJBWVRBa2RDTVJNd0VRWURWUVFJRXdwVGIyMWxMVk4wWVhSbApNU0V3SHdZRFZRUUtFeGhKYm5SbGNtNWxkQ0JYYVdSbmFYUnpJRkIwZVNCTWRHUXhHREFXQmdOVkJBTVREMkZ3CmNDNWxlR0Z0Y0d4bExtTnZiWUlKQUpicWVnTHB0U2JhTUF3R0ExVWRFd1FGTUFNQkFmOHdEUVlKS29aSWh2Y04KQVFFRkJRQURnZ0VCQUc3RUxMUGVrQXJkYy9COUxsZXZsMCtLNWtYN2JsZDBqa1JmZjRzalA5MTdkSFliem0zMQoxNi9QT0ZKc3ZmOTFhNXdOTnNzL3JOVG13ZEZuSC8xNTJJVEgyamJiUEd5bGIyMkNiemgvU09XWVUzcnJEeHk3ClVtMFNqMmdJUHRWdjc3WTY4Y1ZtOTNVK3oxNjM1akVNUUtXcUpYRlBCSU9iWVd1SWNManJ1WTg5dGhpdUtVNTcKNGlraFlqT0t2ZnU4NVNyUDQybGV5Qk1PMHROVVNCZWl6SmZpWDA1N3RtR0xwaXhRYnBsaTlXUjc5bXpLcFJwZApEaEdFMHpxZ1ZSMDlOeGF2cmpNcjdtNHpvRGg1d09McFVQSEVCU2FhU2QzNzA4WGwrTFVDSTNQajhHcEtvUWRlCm11b2t3MndVTFQzR0ZTZjd4OTZSdUJqTmRWb3NSRkJpZjM0PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBck94L1ppMGFGUklpdHFOSXBVb2crUTFxenp2NDdZS0hLVjd1ZithOVl5OWRGaDV2Cm9iTUVvbGFwMGo5T1AwOEZHb3g2eVBTdmJYSkUvU0piaFE5SnZvdzdPWHI3WnZRakwrQXNjZ09VanNJSXhUMnIKd1lmNysveC9UYld0VnQ3N2lDVmw5VlRvaWJIcFZianozZFR2UUMzTnhjWHFvSTkxTmZkeGliclhsL2d6cmRYZQo3dU5FWHRFZnZRTDlKR3BTanVnUlYrb1lwcnRtUzU2Ry9Jb2g5TU0xb0FxMWRjTC8wSEgxY2ZiWU1DcXJra1FjClpPa2hpY05EekZZUVRKTUdXNVp1bElHazhRc1B2WS9Ha0NnSklxUzVaWXJaZWVJbnlnQXRVaGxkbUdxaU9Yd2sKMUhadmxyc05FT3JkSnpjd3J5blF2ZDZxWC9nNnZGdytBaEJhWXdJREFRQUJBb0lCQVFDSGd2WDdmbEM0UG5RUgpxRGZmd0EzQzNtN2JZK1laU25iZFJ0V2tTWkFVMENNa21FbG04RUVyYnJxNlZuM2RRdkYrOHFPdUk0SHVST1FuCmN1dEJoTStIa2FFLzFFNTdTY3JoVTgzQXMybVJ6aUROWVJ6ZUZ0Q3praFc1TWl4YXJYZDBJOHFZelNkRjhMUW8KUno1a2t0L0M3YUlaNEpXVHFaaHk2Q3lEZ0hZL1VpcEFGZG5mTE1NWG00Q1R3OTVGV1VpNGRaUHY4ZzVNNFZVUQo1K25sMUdPUFdsdGpNaWRlY0VxYVlYdzh3amVYQ1JNMDZLeWJSaDU1cU5reHladDZ4YWU4d0JLaE1PV2VRcmVPCjZhQ0tBQjNaOC9vM21JeTd2WjUzWE51WEFQaHhLR3E2OFVkQTgrQ1lKb3dPNFdscGhKRkE2QUcrYjZpRnJwNkgKZmFybVkxRkJBb0dCQU5wT0xZUGVhNEY1eXhxU3NpYkFpTXJiYkZROTlDejRZampkTUlEM25zWnpFblVrdjF0QwpGbTUwOHhodFk0TFRiUEQ0c2RPcmNETVlqYTM0UENKQTFOV0p5UXdZWkdMaEhxTXg1NWJjQ2VaL1Y0S3FlN29ZCm5aK2tPb29RbFBsQUFTZVViYU0vWCtHdDRUdjdwVGxjQWJETWVTd1Z0R2I2ZXFhZUpNUFJmZlhUQW9HQkFNckkKVGVBSjhkejQ2TC9raWZOMjMrVVFkSGdZdzZ5cExSZ0JuRllweGpIM1VWOWRFZVdXdUZzdzhraTVha25TVmIwVgpFaWJqb1BCdjZxd1RvWmhKNHE5L0lOdzJmZE4xTGV3N3ZOaG5vL1A5MHJpUDZ4b2llczJsN2c3bmlEUDJ5ZnRaCnpJSkU0OUs2SVJzT3c4ZHkrS3hrQzJaZDdWcy9BM2x1Q2hYZWVWOHhBb0dBSG9yTGdXU1A0K2gzU3Z0MUkwalMKbXBjQ1cvTGpBNXVvbWs0UDZDczhzb1VNOHdpMklQMXBDQUVpdGFzd1BmQjRrR29xN3ZOUVdrVzRKTHZUSmZPdQpFMFlZczdHQjhmZVBBc1FMbzZhYlYvMCs4QkFNQ1doQ1BVQ0wxQjhueUl0MDNlVzlSUmFyd25aQ1NkTVdOYVV5CnMxcVlKVnZRQm94S3RwN3ZnOW4rWm5NQ2dZRUFsUmJmNnJCbEdzb0dsYzg1ZmI4UXJpR0RBQ2wwOUNVTituQjAKdVFUTnF6N2luUEtZamV4YWJ2RjFzUEpocXhUeDVLcnhSWlptWldCamNWQ2RwcEhzRUl1dlpUakxHZ1UxVmxJMQpiZ1lGRFFhNVB1alJPYzNQN0JMckRCbytrYllJbXJ4VEdCUCtUSmg4YnFCVVlQZXV6VkJnOFVwdGtJQ3IxVU9LCk5ybnpFb0VDZ1lFQXh5a3JTblQreGdjZHFSaXBQenFnL1NJT0VJTXp2VHp2alQ2cG9nb1FhOGhYUkJxTTQ3NUoKVnJlMWlIUXF5b2tDcEM0d0wvTWx2SkhhNW1FMExkdGdyNG9UOUVsMkwrNy9qNHlUL01CUHB2a2M2UWtKaEFLcgpYQ2pIN29seHhWVmhjaVUwZG9JUlYwL0VjRjAwS1NnQnBXR1dOU2UyVm44cTdFelhISHpQVC80PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
kind: Secret
metadata:
  name: app-tls-secret
type: Opaque

This is a mandatory feature in today's cloud and container era.
Why is this complex to implement TLS/SSL pass through in Nginx-Plus. It is quiet simple in the community edition.
I think i kind of lost in the lengthy documentation.

@pleshakov Thanks for providing a gist for accomplishing ssl-passthroughs using NGINX-Plus. Will the above also work with NGINX community edition Ingress Controller?

Hi @rahulsharma2020

The workaround relies on NGINX Plus (specifically, DNS service discovery), so it cannot be used with NGINX OSS.

However, we just released a new version of the Ingress Controller with support for TLS passthrough. Please take a look at the doc
https://docs.nginx.com/nginx-ingress-controller/configuration/transportserver-resource/ and the TLS Passthrough example https://github.com/nginxinc/kubernetes-ingress/tree/v1.7.0/examples-of-custom-resources/tls-passthrough It works both NGINX OSS and NGINX Plus.

Thanks @pleshakov. Appreciate your quick response. Can you tell me the version of IngressController that supports the TLS passthrough?

I am running NGINX in 2 different configurations:

• One on Kubernetes has version 1.17.9

nginx-ingress-7c7cf574cf-qssf6:/$ nginx -v
nginx version: nginx/1.17.9

The image is nginx/nginx-ingress:1.6.3

• Another one running directly on Ubuntu is at version 1.10.3

nginx-vm-1:/var/log/nginx$ nginx -v
nginx version: nginx/1.10.3 (Ubuntu)

Looking at the docs, I see v3 of Controller is released (https://docs.nginx.com/nginx-controller/admin-guide/installing-nginx-controller/#). How does the Controller versioning line-up with the versioning that nginx -v command reports?

The needed release is 1.7.0. So the image should be nginx/nginx-ingress:1.7.0.

Looking at the docs, I see v3 of Controller is released (https://docs.nginx.com/nginx-controller/admin-guide/installing-nginx-controller/#). How does the Controller versioning line-up with the versioning that nginx -v command reports?

this is actually a different product. For the Ingress Controller, please take a look at https://docs.nginx.com/nginx-ingress-controller/

Thank you @pleshakov. I successfully executed both of my use-cases:

• TLS Passthrough
• TCP/UDP forwarding

using the CRDs for TransportServers and GlobalConfigurations in the new improved version 1.7.0.

Appreciate your guidance and recommendations!

Great! Thank you