NGINX Plus Ingress Controller with custom annotations for sticky learn session persistence with sessions sharing among multiple IC replicas.

README.md

Description

This demo shows how to enable sticky learn session persistence for an Ingress resource with the sessions shared among multiple NGINX Plus Ingress Controller pods.

Setup

1. Deploy the Ingress Controller as a deployment -- https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/installation.md
2. Deploy a headless service for the Ingress Controller that will be used for zone synchronization:

   $ kubectl apply -f nginx-ingress-zonesync.yaml
   

3. Apply the ConfigMap with the stream-snippets for configuring zone synchronization and a custom template for custom sticky learn annotations. Note that we also added add_header nginx-ingress-hostname $hostname; to the location block in the template to allow us to understand which NGINX Plus IC pod sends a response:

   $ kubectl apply -f nginx-config.yaml
   

4. Scale the number of the Ingress Controller pods to 3:

   $ kubectl scale deploy nginx-ingress --replicas=3 -n nginx-ingress
   

5. Deploy the backend app and the Ingress resource:

   $ kubectl apply -f backend.yaml
   $ kubectl apply -f ingress.yaml
   

Demo

Notes:

• We assume that NGINX Plus Ingress Controller pods are exposed through a load balancer with the IP saved into $IC_IP variable.
• The backend app sets the cookie session. If a request includes the cookie session, the backend will not set the session cookie.

1. Send two requests to example.com and see that the responses come from different backend pods through different NGINX Plus IC pods:

   $ curl $IC_IP -H "host: example.com" -v
   * Rebuilt URL to: <REDACTED>/
   *   Trying <REDACTED>...
   * TCP_NODELAY set
   * Connected to <REDACTED> (<REDACTED>) port 80 (#0)
   > GET / HTTP/1.1
   > host: example.com
   > User-Agent: curl/7.54.0
   > Accept: */*
   >
   < HTTP/1.1 200 OK
   < Server: nginx/1.15.10
   < Date: Thu, 13 Jun 2019 11:26:05 GMT
   < Content-Type: text/plain
   < Content-Length: 59
   < Connection: keep-alive
   < Set-Cookie: session=b66b69306c303c72d7876b0297dd334a
   < nginx-ingress-hostname: nginx-ingress-6c9c85d755-mn24r
   <
   Response from 10.60.1.33:80 (backend-app-85f495985c-whftm)
   

   The response came from 10.60.1.33:80 (backend-app-85f495985c-whftm) backend through nginx-ingress-6c9c85d755-mn24r.

   curl $IC_IP -H "host: example.com" -v
   * Rebuilt URL to: <REDACTED>/
   *   Trying <REDACTED>...
   * TCP_NODELAY set
   * Connected to <REDACTED> (<REDACTED>) port 80 (#0)
   > GET / HTTP/1.1
   > host: example.com
   > User-Agent: curl/7.54.0
   > Accept: */*
   >
   < HTTP/1.1 200 OK
   < Server: nginx/1.15.10
   < Date: Thu, 13 Jun 2019 11:26:06 GMT
   < Content-Type: text/plain
   < Content-Length: 59
   < Connection: keep-alive
   < Set-Cookie: session=8e985030b254c50a817273a6a8572a37
   < nginx-ingress-hostname: nginx-ingress-6c9c85d755-99cvf
   <
   Response from 10.60.2.50:80 (backend-app-85f495985c-vkhg4)
   

   The response came from 10.60.2.50:80 (backend-app-85f495985c-vkhg4) backend through nginx-ingress-6c9c85d755-99cvf.

2. Send one request and then send a few requests with the session cookie from the response to the first request. All the responses should come from the same backend pod but through different NGINX Plus IC pods:

   curl $IC_IP -H "host: example.com" -v
   * Rebuilt URL to: <REDACTED>/
   *   Trying <REDACTED>...
   * TCP_NODELAY set
   * Connected to <REDACTED> (<REDACTED>) port 80 (#0)
   > GET / HTTP/1.1
   > host: example.com
   > User-Agent: curl/7.54.0
   > Accept: */*
   >
   < HTTP/1.1 200 OK
   < Server: nginx/1.15.10
   < Date: Thu, 13 Jun 2019 11:33:23 GMT
   < Content-Type: text/plain
   < Content-Length: 59
   < Connection: keep-alive
   < Set-Cookie: session=6300a29c34f19718f85955b4b23ca609
   < nginx-ingress-hostname: nginx-ingress-6c9c85d755-4w767
   <
   Response from 10.60.1.33:80 (backend-app-85f495985c-whftm)
   * Connection #0 to host <REDACTED> left intact
   

   $ curl $IC_IP -H "host: example.com" -v --cookie session=6300a29c34f19718f85955b4b23ca609
   * Rebuilt URL to: <REDACTED>/
   *   Trying <REDACTED>...
   * TCP_NODELAY set
   * Connected to <REDACTED> (<REDACTED>) port 80 (#0)
   > GET / HTTP/1.1
   > host: example.com
   > User-Agent: curl/7.54.0
   > Accept: */*
   > Cookie: session=6300a29c34f19718f85955b4b23ca609
   >
   < HTTP/1.1 200 OK
   < Server: nginx/1.15.10
   < Date: Thu, 13 Jun 2019 11:33:53 GMT
   < Content-Type: text/plain
   < Content-Length: 59
   < Connection: keep-alive
   < nginx-ingress-hostname: nginx-ingress-6c9c85d755-99cvf
   <
   Response from 10.60.1.33:80 (backend-app-85f495985c-whftm)
   

   $ curl $IC_IP -H "host: example.com" -v --cookie session=6300a29c34f19718f85955b4b23ca609
   * Rebuilt URL to: <REDACTED>/
   *   Trying <REDACTED>...
   * TCP_NODELAY set
   * Connected to <REDACTED> (<REDACTED>) port 80 (#0)
   > GET / HTTP/1.1
   > host: example.com
   > User-Agent: curl/7.54.0
   > Accept: */*
   > Cookie: session=6300a29c34f19718f85955b4b23ca609
   >
   < HTTP/1.1 200 OK
   < Server: nginx/1.15.10
   < Date: Thu, 13 Jun 2019 11:33:56 GMT
   < Content-Type: text/plain
   < Content-Length: 59
   < Connection: keep-alive
   < nginx-ingress-hostname: nginx-ingress-6c9c85d755-4w767
   <
   Response from 10.60.1.33:80 (backend-app-85f495985c-whftm)
   

Notes

Currently, the custom annotation custom.nginx.org/sticky-learn enables session persistence for every service in an Ingress resource. If you would like to enable session persistence per service, use Mergeable Ingresses to split one Ingress resource into multiple Mergeable minions and change the custom annotation to support Mergeable Ingesses -- https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/custom-annotations.md#custom-annotations-with-mergeable-ingress-resources Once this is done, you can enable session persistence only for services of particular minions.

backend.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: backend-app 
spec:
  replicas: 3
  selector:
    matchLabels:
      app: backend-app
  template:
    metadata:
      labels:
        app: backend-app
    spec:
      containers:
      - name: backend-app
        image: nginx 
        ports:
        - containerPort: 80
        volumeMounts:
          - name: config-volume
            mountPath: /etc/nginx/conf.d
      volumes:
      - name: config-volume
        configMap:
          name: backend-config 
---
apiVersion: v1
kind: Service
metadata:
  name: backend-app
spec:
  ports:
  - port: 80 
    targetPort: 80 
    protocol: TCP
    name: http
  selector:
    app: backend-app
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: backend-config
data:
  app.conf: |-
    map $cookie_session $session {
      '' 'session=$request_id';
      default '';
    }
    server {
      listen 80 default_server;

      server_name example.com;

      default_type text/plain;

      location / {
        return 200 'Response from $server_addr:$server_port ($hostname)\n'; 
        add_header Set-Cookie $session;
      }
    }

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
     kubernetes.io/ingress.class: "nginx"
     custom.nginx.org/sticky-learn: "on"
     custom.nginx.org/sticky-learn-create: "$upstream_cookie_session"
     custom.nginx.org/sticky-learn-lookup: "$cookie_session"
     custom.nginx.org/sticky-learn-timeout: "1h"
spec:
  rules:
  - host: example.com 
    http:
      paths:
      - path: /
        backend:
          serviceName: backend-app 
          servicePort: 80

nginx-config.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-config
  namespace: nginx-ingress
data:
  stream-snippets: |
    keyval_zone zone=zone_test_sync:32k timeout=5s sync; # a test zone to make sure we can see the number of online nodes via /api/4/stream/zone_sync

    server {
      listen 7777;

      resolver kube-dns.kube-system.svc.cluster.local valid=5s;

      zone_sync;
      zone_sync_server nginx-ingress-zonesync.nginx-ingress.svc.cluster.local:7777 resolve;
    }
  ingress-template: |
    {{range $upstream := .Upstreams}}
    upstream {{$upstream.Name}} {
      zone {{$upstream.Name}} 256k;
      {{if $upstream.LBMethod }}{{$upstream.LBMethod}};{{end}}
      {{range $server := $upstream.UpstreamServers}}
      server {{$server.Address}}:{{$server.Port}} max_fails={{$server.MaxFails}} fail_timeout={{$server.FailTimeout}}
          {{- if $server.SlowStart}} slow_start={{$server.SlowStart}}{{end}}{{if $server.Resolve}} resolve{{end}};{{end}}
      {{if $upstream.StickyCookie}}
      sticky cookie {{$upstream.StickyCookie}};
      {{end}}
      {{if $.Keepalive}}keepalive {{$.Keepalive}};{{end}}
      {{- if $upstream.UpstreamServers -}}
      {{- if $upstream.Queue}}
      queue {{$upstream.Queue}} timeout={{$upstream.QueueTimeout}}s;
      {{- end -}}
      {{- end}}
      {{if eq (index $.Ingress.Annotations "custom.nginx.org/sticky-learn") "on"}}
        {{$create := index $.Ingress.Annotations "custom.nginx.org/sticky-learn-create"}}
        {{$lookup := index $.Ingress.Annotations "custom.nginx.org/sticky-learn-lookup"}}
        {{$timeout := index $.Ingress.Annotations "custom.nginx.org/sticky-learn-timeout"}}
        {{$zone := print $.Ingress.Namespace "-" $.Ingress.Name}}
      sticky learn
        create={{$create}}
        lookup={{$lookup}}
        zone={{$zone}}:1m
        timeout={{$timeout}}
        sync;
      {{end}}
    }
    {{- end}}

    {{range $server := .Servers}}
    server {
      {{if not $server.GRPCOnly}}
      {{range $port := $server.Ports}}
      listen {{$port}}{{if $server.ProxyProtocol}} proxy_protocol{{end}};
      {{- end}}
      {{end}}
      {{if $server.SSL}}
      {{- range $port := $server.SSLPorts}}
      listen {{$port}} ssl{{if $server.HTTP2}} http2{{end}}{{if $server.ProxyProtocol}} proxy_protocol{{end}};
      {{- end}}
      ssl_certificate {{$server.SSLCertificate}};
      ssl_certificate_key {{$server.SSLCertificateKey}};
      {{if $server.SSLCiphers}}
      ssl_ciphers {{$server.SSLCiphers}};
      {{end}}
      {{end}}
      {{range $setRealIPFrom := $server.SetRealIPFrom}}
      set_real_ip_from {{$setRealIPFrom}};{{end}}
      {{if $server.RealIPHeader}}real_ip_header {{$server.RealIPHeader}};{{end}}
      {{if $server.RealIPRecursive}}real_ip_recursive on;{{end}}

      server_tokens "{{$server.ServerTokens}}";

      server_name {{$server.Name}};

      status_zone {{$server.StatusZone}};

      {{if not $server.GRPCOnly}}
      {{range $proxyHideHeader := $server.ProxyHideHeaders}}
      proxy_hide_header {{$proxyHideHeader}};{{end}}
      {{range $proxyPassHeader := $server.ProxyPassHeaders}}
      proxy_pass_header {{$proxyPassHeader}};{{end}}
      {{end}}

      {{if $server.SSL}}
      {{if not $server.GRPCOnly}}
      {{- if $server.HSTS}}
      set $hsts_header_val "";
      proxy_hide_header Strict-Transport-Security;
      {{- if $server.HSTSBehindProxy}}
      if ($http_x_forwarded_proto = 'https') {
      {{else}}
      if ($https = on) {
      {{- end}}
        set $hsts_header_val "max-age={{$server.HSTSMaxAge}}; {{if $server.HSTSIncludeSubdomains}}includeSubDomains; {{end}}preload";
      }

      add_header Strict-Transport-Security "$hsts_header_val" always;
      {{end}}

      {{- if $server.SSLRedirect}}
      if ($scheme = http) {
        return 301 https://$host:{{index $server.SSLPorts 0}}$request_uri;
      }
      {{- end}}
      {{end}}
      {{- end}}

      {{- if $server.RedirectToHTTPS}}
      if ($http_x_forwarded_proto = 'http') {
        return 301 https://$host$request_uri;
      }
      {{- end}}

      {{with $jwt := $server.JWTAuth}}
      auth_jwt_key_file {{$jwt.Key}};
      auth_jwt "{{.Realm}}"{{if $jwt.Token}} token={{$jwt.Token}}{{end}};

      {{- if $jwt.RedirectLocationName}}
      error_page 401 {{$jwt.RedirectLocationName}};
      {{end}}
      {{end}}

      {{- if $server.ServerSnippets}}
      {{range $value := $server.ServerSnippets}}
      {{$value}}{{end}}
      {{- end}}

      {{- range $healthCheck := $server.HealthChecks}}
      location @hc-{{$healthCheck.UpstreamName}} {
        {{- range $name, $header := $healthCheck.Headers}}
        proxy_set_header {{$name}} "{{$header}}";
        {{- end }}
        proxy_connect_timeout {{$healthCheck.TimeoutSeconds}}s;
        proxy_read_timeout {{$healthCheck.TimeoutSeconds}}s;
        proxy_send_timeout {{$healthCheck.TimeoutSeconds}}s;
        proxy_pass {{$healthCheck.Scheme}}://{{$healthCheck.UpstreamName}};
        health_check {{if $healthCheck.Mandatory}}mandatory {{end}}uri={{$healthCheck.URI}} interval=
          {{- $healthCheck.Interval}}s fails={{$healthCheck.Fails}} passes={{$healthCheck.Passes}};
      }
      {{end -}}

      {{- range $location := $server.JWTRedirectLocations}}
      location {{$location.Name}} {
        internal;
        return 302 {{$location.LoginURL}};
      }
      {{end -}}

      {{range $location := $server.Locations}}
      location {{$location.Path}} {
        {{with $location.MinionIngress}}
        # location for minion {{$location.MinionIngress.Namespace}}/{{$location.MinionIngress.Name}}
        {{end}}
        {{if $location.GRPC}}
        {{if not $server.GRPCOnly}}
        error_page 400 @grpcerror400;
        error_page 401 @grpcerror401;
        error_page 403 @grpcerror403;
        error_page 404 @grpcerror404;
        error_page 405 @grpcerror405;
        error_page 408 @grpcerror408;
        error_page 414 @grpcerror414;
        error_page 426 @grpcerror426;
        error_page 500 @grpcerror500;
        error_page 501 @grpcerror501;
        error_page 502 @grpcerror502;
        error_page 503 @grpcerror503;
        error_page 504 @grpcerror504;
        {{end}}

        {{- if $location.LocationSnippets}}
        {{range $value := $location.LocationSnippets}}
        {{$value}}{{end}}
        {{- end}}

        {{with $jwt := $location.JWTAuth}}
        auth_jwt_key_file {{$jwt.Key}};
        auth_jwt "{{.Realm}}"{{if $jwt.Token}} token={{$jwt.Token}}{{end}};
        {{end}}

        grpc_connect_timeout {{$location.ProxyConnectTimeout}};
        grpc_read_timeout {{$location.ProxyReadTimeout}};
        grpc_set_header Host $host;
        grpc_set_header X-Real-IP $remote_addr;
        grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        grpc_set_header X-Forwarded-Host $host;
        grpc_set_header X-Forwarded-Port $server_port;
        grpc_set_header X-Forwarded-Proto $scheme;

        {{- if $location.ProxyBufferSize}}
        grpc_buffer_size {{$location.ProxyBufferSize}};
        {{- end}}

        {{if $location.SSL}}
        grpc_pass grpcs://{{$location.Upstream.Name}}
        {{else}}
        grpc_pass grpc://{{$location.Upstream.Name}};
        {{end}}
        {{else}}
        proxy_http_version 1.1;
        {{if $location.Websocket}}
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        {{- else}}
        {{- if $.Keepalive}}proxy_set_header Connection "";{{end}}
        {{- end}}

        {{- if $location.LocationSnippets}}
        {{range $value := $location.LocationSnippets}}
        {{$value}}{{end}}
        {{- end}}

        {{ with $jwt := $location.JWTAuth }}
        auth_jwt_key_file {{$jwt.Key}};
        auth_jwt "{{.Realm}}"{{if $jwt.Token}} token={{$jwt.Token}}{{end}};
        {{if $jwt.RedirectLocationName}}
        error_page 401 {{$jwt.RedirectLocationName}};
        {{end}}
        {{end}}

        proxy_connect_timeout {{$location.ProxyConnectTimeout}};
        proxy_read_timeout {{$location.ProxyReadTimeout}};
        client_max_body_size {{$location.ClientMaxBodySize}};
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto {{if $server.RedirectToHTTPS}}https{{else}}$scheme{{end}};
        proxy_buffering {{if $location.ProxyBuffering}}on{{else}}off{{end}};
        {{- if $location.ProxyBuffers}}
        proxy_buffers {{$location.ProxyBuffers}};
        {{- end}}
        {{- if $location.ProxyBufferSize}}
        proxy_buffer_size {{$location.ProxyBufferSize}};
        {{- end}}
        {{- if $location.ProxyMaxTempFileSize}}
        proxy_max_temp_file_size {{$location.ProxyMaxTempFileSize}};
        {{- end}}
        {{if $location.SSL}}
        proxy_pass https://{{$location.Upstream.Name}}{{$location.Rewrite}};
        {{else}}
        proxy_pass http://{{$location.Upstream.Name}}{{$location.Rewrite}};
        {{end}}
        add_header nginx-ingress-hostname $hostname;
        {{end}}
      }{{end}}
      {{if $server.GRPCOnly}}
      error_page 400 @grpcerror400;
      error_page 401 @grpcerror401;
      error_page 403 @grpcerror403;
      error_page 404 @grpcerror404;
      error_page 405 @grpcerror405;
      error_page 408 @grpcerror408;
      error_page 414 @grpcerror414;
      error_page 426 @grpcerror426;
      error_page 500 @grpcerror500;
      error_page 501 @grpcerror501;
      error_page 502 @grpcerror502;
      error_page 503 @grpcerror503;
      error_page 504 @grpcerror504;
      {{end}}
      {{if $server.HTTP2}}
      location @grpcerror400 { default_type application/grpc; return 400 "\n"; }
      location @grpcerror401 { default_type application/grpc; return 401 "\n"; }
      location @grpcerror403 { default_type application/grpc; return 403 "\n"; }
      location @grpcerror404 { default_type application/grpc; return 404 "\n"; }
      location @grpcerror405 { default_type application/grpc; return 405 "\n"; }
      location @grpcerror408 { default_type application/grpc; return 408 "\n"; }
      location @grpcerror414 { default_type application/grpc; return 414 "\n"; }
      location @grpcerror426 { default_type application/grpc; return 426 "\n"; }
      location @grpcerror500 { default_type application/grpc; return 500 "\n"; }
      location @grpcerror501 { default_type application/grpc; return 501 "\n"; }
      location @grpcerror502 { default_type application/grpc; return 502 "\n"; }
      location @grpcerror503 { default_type application/grpc; return 503 "\n"; }
      location @grpcerror504 { default_type application/grpc; return 504 "\n"; }
      {{end}}
    }{{end}}

nginx-ingress-zonesync.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress-zonesync
  namespace: nginx-ingress
spec:
  clusterIP: None
  ports:
  - port: 7777
    targetPort: 7777 
    protocol: TCP
    name: zonesync 
  selector:
    app: nginx-ingress

This feature isn't available in NGINX OSS