kubernetes-ingress websockets with nodejs

frontend-ws-connection.ts

export const ws = webSocket<WebsocketMessage>(`wss://${location.hostname}:${location.protocol === 'https:' ? 443 : 80}/ws/`);

export const wsObserver = ws
  .pipe(
    retryWhen(errors =>
      errors.pipe(
        delay(1000)
      )
    )
  );

wsObserver.subscribe(console.log);

ingress-service.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-service
  namespace: <YOUR_NAMESPACE>
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
    nginx.ingress.kubernetes.io/proxy-send-timeout: 3600
spec:
  rules:
    - http:
        paths:
          - path: /
            backend:
              serviceName: client-cluster-ip-service
              servicePort: 3000
          # Below is the important part!
          - path: /ws/
            backend:
              serviceName: server-cluster-ip-service
              servicePort: 40510

server-cluster-ip-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: server-cluster-ip-service
  namespace: <YOUR_NAMESPACE>
spec:
  type: ClusterIP
  selector:
    component: server
  ports:
    - port: 40510
      targetPort: 40510
      # The below line isn't required.
      protocol: TCP

server-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: server-deployment
  namespace: <YOUR_NAMESPACE>
spec:
  replicas: 1
  selector:
    matchLabels:
      component: server
  template:
    metadata:
      labels:
        component: server
    spec:
      containers:
        - name: server
          image: <YOUR_DOCKER_IMAGE>
          ports:
            - containerPort: 40510

server.js

import { Server as WebSocketServer } from 'ws';

// IMPORTANT: not a secure connection
const wss = new WebSocketServer({
    path: '/ws/',
    port: 40510,
});

wss.on('connection', function (ws) {
    console.log('connection!');
});

wss.on('close', function close() {
    console.log('ws disconnected');
});

It took me roughly a total of 28 hours to get my frontend to be able to access the node.js websocket server with Kubernetes in MiniKube.

Mainly due to my lack of knowledge/ plain stupidity in the realms of networking. I tried many debugging techniques. For people who run in to similar errors, here is a list of pitfalls mixed with a checklist of tips to debug/ solve this problem:

• There is a confusing difference between kubernetes-ingress and ingress-nginx. You probably want ingress-nginx. When looking at GitHub issues/ docs, make sure you're reading from the correct project.
• Ensure the path of the websocket is correct and consistent across files. For example,
  • in the ingress-service.yaml you have - path: /ws/ NOT - path: /ws
  • in your server you have path: '/ws/'
  • in your websocket client you have wss://${location.hostname}:${location.protocol === 'https:' ? 443 : 80}/ws/
• Support for websockets is provided by NGINX out of the box. No special configuration required. The only requirement to avoid the close of connections is the increase of the values of proxy-read-timeout and proxy-send-timeout.¹ (as shown above)
• Kubernetes tab completion will save you hours on finding pod names make sure you have it
• Ensure your server's websocket is 'insecure' (using ws and NOT wss). This is because the ingress will most likely be managing the certificate for you. Trying to use wss inside your server will result in mysterious 502 Bad Gateway errors that don't show up in the in ingress' pod's logs (which can be found using kubectl exec -n ingress-nginx <name of ingress controller> cat nginx.conf)
• Kubernetes-ingress' troubleshooting page is very helpful but the debug logging didn't do anything for me

Criticism and suggestions are extremely welcome!

Thanks for posting this.
The links in the first checklist item are all the same. I'm guessing the last two should be https://github.com/kubernetes/ingress-nginx/?

@wilsonianb Thanks! Nice catch. I have updated the comment

First of all, thank you for sharing this! There is no ingress controller pod showing up for me and my ingress does not have any events. Do I need to add an ingress controller? Is a YAML missing or am I missing something?

@nxz91 are you minikube with ingress-nginx? If so, have you followed the installation instructions found here?

Hi @jsdevtom, after you said this I checked further and installed ingress-nginx on my DigitalOcean kubernetes cluster. That's where I'm trying to get it to work. How could I move something to the cloud that works locally on minikube at some point?

I'm wondering: where does serviceName "client-cluster-ip-service" come from? And does this mean that only requests on paths / and /ws/ get forwarded to the container on port 40510? Why are we using ports 3000 and 40510? Why can't we just use one port?

Here it says that we should increase the timeout and make sure that TCP is used. Why this path handling and port stuff?

@nxz91 You can use kustomize which has now been build in to Kubernetes to apply difference configs in different environments so that you can use a different ingress.

where does serviceName "client-cluster-ip-service" come from?

That is just a name I chose. You will need to chose one to. See the documentation here: https://kubernetes.io/docs/concepts/services-networking/ingress/#the-ingress-resource

And does this mean that only requests on paths / and /ws/ get forwarded to the container on port 40510? Why are we using ports 3000 and 40510? Why can't we just use one port?

You can use one port if you are only using WebSockets. In a lot of applications, however, you have multiple different services, some of which use different protocols such as https and WebSockets. In the example above, my static files (HTML, CSS, JS) are being served at port 3000 and my WS server is available at port 40510.

Based on your questions it seems to me like you need to you need to learn the basics of Kubernetes. I can personally recommend https://www.udemy.com/docker-and-kubernetes-the-complete-guide/ If you can't afford it, you can always do the interactive tutorials and then read the documentation.

I hope this helps

Any idea how to use SSL termination in the ingress to work with websockets?

@huegelc no idea sorry

@jsdevtom have you ever had any issues with health checks? I can't get the service exposed through the ingress because my GKE console keeps telling me that the service is in UNHEALTHY state (tries to send a liveness and readiness probes). I haven't configured any, just like you have it in your YAML file.

Thanks a lot for sharing!

@svachmic Yes I have. I can't quite remember what the issue was though. Looking over my browser history, I saw I liked this comment: kubernetes/kubernetes#20555 (comment) So I assume this fixed it for me. Hope this helps!

It's a silly thing - but GKE only accepts the kubernetes.io/ingress.class: nginx when the nginx part is in quotes: kubernetes.io/ingress.class: "nginx". Otherwise it ignores it and supplies the L7 GLB instead of the NGINX controller I deployed in the cluster.

So in fact the health problem went away when the correct annotation was assigned, as the nginx load balancer does not require the websocket server to have a health check.

Hi, I have issue on setting in bare metal kubeadm behind SLB. Do you have solution for this ?

I always back to this page in any stuck page, and this solution didn't works for me

Chapeau bas!

in my case ingress was not available directly, but via Classic Load Balancer which does not support sockets :(
https://aws.amazon.com/elasticloadbalancing/features/

Does anyone know any Guide for configuring haproxy-ingress to support WebSockets?

Fantastic post by the way has really helped me.

So following the OP instructions, I have the ingress using SSL and using wss: from the client to connect to the ingress, but then the back-end is just HTTP.

I am getting this error: failed: Error in connection establishment: net::ERR_SSL_PROTOCOL_ERROR

So my follow up question:

Do I need to configure the back-end service with an SSL cert or is there some ingress magic configuration that would allow it to up stream the request as ws: ?

thx

If you are using kubernetes you should probably instlal https://github.com/jetstack/cert-manager and update your ingress to handle SSL certificate.

If you are using kubernetes you should probably instlal https://github.com/jetstack/cert-manager and update your ingress to handle SSL certificate.

I have SSL terminating at the ingress controller, with a certificate configured. Do I also need to configure an additional certificate on the backend service?

Thanks @jsdevtom!!!
I just had the issue with the websocket connection close after 30 seconds and therefore I find your gist.

I'm using GKE with the L7 GLB. For those, who are using the Google cloud infrastructure it is pretty simple to get websocket connections working with google managed ssl certificates. It looks a little bit different:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: service-backend-config
spec:
  timeoutSec: 3600

---

apiVersion: v1
kind: Service
metadata:
  labels:
    app: service
  name: service
  annotations:
    cloud.google.com/backend-config: '{"ports": {"8080":"service-backend-config"}}'
spec:
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    app: service
  type: NodePort

---

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: public-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "your-ip"
    networking.gke.io/managed-certificates: "your-certificates"
spec:
  rules: 
  - host: "your-domain.com"
    http:
      paths:
      - backend:
          serviceName: service
          servicePort: 8080
        path: /*

in k8s 1.18.8 the annotations need a string not a number:
nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
nginx.ingress.kubernetes.io/proxy-send-timeout: 3600
==>
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"

still not working though

I'm sure that missing trailing forward slash on /ws/ would have cost me 10 hours too. So, thanks!

I cant get this to work with socket io and typescript. Any suggestion will be highly appreciated

After some hours trying to fix it, i found the proper config for my ingress ( k8s azure )

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  namespace: development
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
  rules:
    - host: svc-host-domain.com
      http:
        paths:
        - backend:
            service:
              name: my-service
              port:
                number: 8084
          path: /publish
          pathType: Prefix
        - backend:
            service:
              name: my-service
              port:
                number: 8080
          path: /ws/
          pathType: Prefix

Thank you for the great snippet

( i'm still struggling to have the keep-alive working well, it disconnects my websocket clients suddenly )

I'm sure that missing trailing forward slash on /ws/ would have cost me 10 hours too. So, thanks!

What should happen if the trailing slash is missing? I'm trying to debug an error related to my k8s websocket setup, and I'm using /ws right now, but I'm not exactly sure it's failing...

A working e2e complete setup of this scenario is available here: https://github.com/martencassel/nodejs-websocket-nginx-ingress-setup

A working e2e complete setup of this scenario is available here: https://github.com/martencassel/nodejs-websocket-nginx-ingress-setup

🔥

Thank you you save me a lot of time. I have been struggling with this issue, at the time I didn't know that I have to configure my ingress for websocket. Thank you again.