kubernetes-ingress websockets with nodejs

frontend-ws-connection.ts

export const ws = webSocket<WebsocketMessage>(`wss://${location.hostname}:${location.protocol === 'https:' ? 443 : 80}/ws/`);

export const wsObserver = ws
  .pipe(
    retryWhen(errors =>
      errors.pipe(
        delay(1000)
      )
    )
  );

wsObserver.subscribe(console.log);

ingress-service.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-service
  namespace: <YOUR_NAMESPACE>
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
    nginx.ingress.kubernetes.io/proxy-send-timeout: 3600
spec:
  rules:
    - http:
        paths:
          - path: /
            backend:
              serviceName: client-cluster-ip-service
              servicePort: 3000
          # Below is the important part!
          - path: /ws/
            backend:
              serviceName: server-cluster-ip-service
              servicePort: 40510

server-cluster-ip-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: server-cluster-ip-service
  namespace: <YOUR_NAMESPACE>
spec:
  type: ClusterIP
  selector:
    component: server
  ports:
    - port: 40510
      targetPort: 40510
      # The below line isn't required.
      protocol: TCP

server-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: server-deployment
  namespace: <YOUR_NAMESPACE>
spec:
  replicas: 1
  selector:
    matchLabels:
      component: server
  template:
    metadata:
      labels:
        component: server
    spec:
      containers:
        - name: server
          image: <YOUR_DOCKER_IMAGE>
          ports:
            - containerPort: 40510

server.js

import { Server as WebSocketServer } from 'ws';

// IMPORTANT: not a secure connection
const wss = new WebSocketServer({
    path: '/ws/',
    port: 40510,
});

wss.on('connection', function (ws) {
    console.log('connection!');
});

wss.on('close', function close() {
    console.log('ws disconnected');
});

@nxz91 You can use kustomize which has now been build in to Kubernetes to apply difference configs in different environments so that you can use a different ingress.

where does serviceName "client-cluster-ip-service" come from?

That is just a name I chose. You will need to chose one to. See the documentation here: https://kubernetes.io/docs/concepts/services-networking/ingress/#the-ingress-resource

And does this mean that only requests on paths / and /ws/ get forwarded to the container on port 40510? Why are we using ports 3000 and 40510? Why can't we just use one port?

You can use one port if you are only using WebSockets. In a lot of applications, however, you have multiple different services, some of which use different protocols such as https and WebSockets. In the example above, my static files (HTML, CSS, JS) are being served at port 3000 and my WS server is available at port 40510.

Based on your questions it seems to me like you need to you need to learn the basics of Kubernetes. I can personally recommend https://www.udemy.com/docker-and-kubernetes-the-complete-guide/ If you can't afford it, you can always do the interactive tutorials and then read the documentation.

I hope this helps

Any idea how to use SSL termination in the ingress to work with websockets?

@huegelc no idea sorry

@jsdevtom have you ever had any issues with health checks? I can't get the service exposed through the ingress because my GKE console keeps telling me that the service is in UNHEALTHY state (tries to send a liveness and readiness probes). I haven't configured any, just like you have it in your YAML file.

Thanks a lot for sharing!

@svachmic Yes I have. I can't quite remember what the issue was though. Looking over my browser history, I saw I liked this comment: kubernetes/kubernetes#20555 (comment) So I assume this fixed it for me. Hope this helps!

It's a silly thing - but GKE only accepts the kubernetes.io/ingress.class: nginx when the nginx part is in quotes: kubernetes.io/ingress.class: "nginx". Otherwise it ignores it and supplies the L7 GLB instead of the NGINX controller I deployed in the cluster.

So in fact the health problem went away when the correct annotation was assigned, as the nginx load balancer does not require the websocket server to have a health check.

Hi, I have issue on setting in bare metal kubeadm behind SLB. Do you have solution for this ?

I always back to this page in any stuck page, and this solution didn't works for me

Chapeau bas!

in my case ingress was not available directly, but via Classic Load Balancer which does not support sockets :(
https://aws.amazon.com/elasticloadbalancing/features/

Does anyone know any Guide for configuring haproxy-ingress to support WebSockets?

Fantastic post by the way has really helped me.

So following the OP instructions, I have the ingress using SSL and using wss: from the client to connect to the ingress, but then the back-end is just HTTP.

I am getting this error: failed: Error in connection establishment: net::ERR_SSL_PROTOCOL_ERROR

So my follow up question:

Do I need to configure the back-end service with an SSL cert or is there some ingress magic configuration that would allow it to up stream the request as ws: ?

thx

If you are using kubernetes you should probably instlal https://github.com/jetstack/cert-manager and update your ingress to handle SSL certificate.

If you are using kubernetes you should probably instlal https://github.com/jetstack/cert-manager and update your ingress to handle SSL certificate.

I have SSL terminating at the ingress controller, with a certificate configured. Do I also need to configure an additional certificate on the backend service?

Thanks @jsdevtom!!!
I just had the issue with the websocket connection close after 30 seconds and therefore I find your gist.

I'm using GKE with the L7 GLB. For those, who are using the Google cloud infrastructure it is pretty simple to get websocket connections working with google managed ssl certificates. It looks a little bit different:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: service-backend-config
spec:
  timeoutSec: 3600

---

apiVersion: v1
kind: Service
metadata:
  labels:
    app: service
  name: service
  annotations:
    cloud.google.com/backend-config: '{"ports": {"8080":"service-backend-config"}}'
spec:
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    app: service
  type: NodePort

---

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: public-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "your-ip"
    networking.gke.io/managed-certificates: "your-certificates"
spec:
  rules: 
  - host: "your-domain.com"
    http:
      paths:
      - backend:
          serviceName: service
          servicePort: 8080
        path: /*

in k8s 1.18.8 the annotations need a string not a number:
nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
nginx.ingress.kubernetes.io/proxy-send-timeout: 3600
==>
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"

still not working though

I'm sure that missing trailing forward slash on /ws/ would have cost me 10 hours too. So, thanks!

I cant get this to work with socket io and typescript. Any suggestion will be highly appreciated

After some hours trying to fix it, i found the proper config for my ingress ( k8s azure )

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  namespace: development
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
  rules:
    - host: svc-host-domain.com
      http:
        paths:
        - backend:
            service:
              name: my-service
              port:
                number: 8084
          path: /publish
          pathType: Prefix
        - backend:
            service:
              name: my-service
              port:
                number: 8080
          path: /ws/
          pathType: Prefix

Thank you for the great snippet

( i'm still struggling to have the keep-alive working well, it disconnects my websocket clients suddenly )

I'm sure that missing trailing forward slash on /ws/ would have cost me 10 hours too. So, thanks!

What should happen if the trailing slash is missing? I'm trying to debug an error related to my k8s websocket setup, and I'm using /ws right now, but I'm not exactly sure it's failing...

A working e2e complete setup of this scenario is available here: https://github.com/martencassel/nodejs-websocket-nginx-ingress-setup

A working e2e complete setup of this scenario is available here: https://github.com/martencassel/nodejs-websocket-nginx-ingress-setup

🔥

Thank you you save me a lot of time. I have been struggling with this issue, at the time I didn't know that I have to configure my ingress for websocket. Thank you again.