-
-
Save jsdevtom/7045c03c021ce46b08cb3f41db0d76da to your computer and use it in GitHub Desktop.
export const ws = webSocket<WebsocketMessage>(`wss://${location.hostname}:${location.protocol === 'https:' ? 443 : 80}/ws/`); | |
export const wsObserver = ws | |
.pipe( | |
retryWhen(errors => | |
errors.pipe( | |
delay(1000) | |
) | |
) | |
); | |
wsObserver.subscribe(console.log); |
apiVersion: extensions/v1beta1 | |
kind: Ingress | |
metadata: | |
name: ingress-service | |
namespace: <YOUR_NAMESPACE> | |
annotations: | |
kubernetes.io/ingress.class: nginx | |
nginx.ingress.kubernetes.io/proxy-read-timeout: 3600 | |
nginx.ingress.kubernetes.io/proxy-send-timeout: 3600 | |
spec: | |
rules: | |
- http: | |
paths: | |
- path: / | |
backend: | |
serviceName: client-cluster-ip-service | |
servicePort: 3000 | |
# Below is the important part! | |
- path: /ws/ | |
backend: | |
serviceName: server-cluster-ip-service | |
servicePort: 40510 |
apiVersion: v1 | |
kind: Service | |
metadata: | |
name: server-cluster-ip-service | |
namespace: <YOUR_NAMESPACE> | |
spec: | |
type: ClusterIP | |
selector: | |
component: server | |
ports: | |
- port: 40510 | |
targetPort: 40510 | |
# The below line isn't required. | |
protocol: TCP |
apiVersion: apps/v1 | |
kind: Deployment | |
metadata: | |
name: server-deployment | |
namespace: <YOUR_NAMESPACE> | |
spec: | |
replicas: 1 | |
selector: | |
matchLabels: | |
component: server | |
template: | |
metadata: | |
labels: | |
component: server | |
spec: | |
containers: | |
- name: server | |
image: <YOUR_DOCKER_IMAGE> | |
ports: | |
- containerPort: 40510 |
import { Server as WebSocketServer } from 'ws'; | |
// IMPORTANT: not a secure connection | |
const wss = new WebSocketServer({ | |
path: '/ws/', | |
port: 40510, | |
}); | |
wss.on('connection', function (ws) { | |
console.log('connection!'); | |
}); | |
wss.on('close', function close() { | |
console.log('ws disconnected'); | |
}); |
Thanks for posting this.
The links in the first checklist item are all the same. I'm guessing the last two should be https://github.com/kubernetes/ingress-nginx/?
@wilsonianb Thanks! Nice catch. I have updated the comment
First of all, thank you for sharing this! There is no ingress controller pod showing up for me and my ingress does not have any events. Do I need to add an ingress controller? Is a YAML missing or am I missing something?
Hi @jsdevtom, after you said this I checked further and installed ingress-nginx on my DigitalOcean kubernetes cluster. That's where I'm trying to get it to work. How could I move something to the cloud that works locally on minikube at some point?
I'm wondering: where does serviceName "client-cluster-ip-service" come from? And does this mean that only requests on paths / and /ws/ get forwarded to the container on port 40510? Why are we using ports 3000 and 40510? Why can't we just use one port?
Here it says that we should increase the timeout and make sure that TCP is used. Why this path handling and port stuff?
@nxz91 You can use kustomize which has now been build in to Kubernetes to apply difference configs in different environments so that you can use a different ingress.
where does serviceName "client-cluster-ip-service" come from?
That is just a name I chose. You will need to chose one to. See the documentation here: https://kubernetes.io/docs/concepts/services-networking/ingress/#the-ingress-resource
And does this mean that only requests on paths / and /ws/ get forwarded to the container on port 40510? Why are we using ports 3000 and 40510? Why can't we just use one port?
You can use one port if you are only using WebSockets. In a lot of applications, however, you have multiple different services, some of which use different protocols such as https and WebSockets. In the example above, my static files (HTML, CSS, JS) are being served at port 3000 and my WS server is available at port 40510.
Based on your questions it seems to me like you need to you need to learn the basics of Kubernetes. I can personally recommend https://www.udemy.com/docker-and-kubernetes-the-complete-guide/ If you can't afford it, you can always do the interactive tutorials and then read the documentation.
I hope this helps
Any idea how to use SSL termination in the ingress to work with websockets?
@huegelc no idea sorry
@jsdevtom have you ever had any issues with health checks? I can't get the service exposed through the ingress because my GKE console keeps telling me that the service is in UNHEALTHY state (tries to send a liveness and readiness probes). I haven't configured any, just like you have it in your YAML file.
Thanks a lot for sharing!
@svachmic Yes I have. I can't quite remember what the issue was though. Looking over my browser history, I saw I liked this comment: kubernetes/kubernetes#20555 (comment) So I assume this fixed it for me. Hope this helps!
It's a silly thing - but GKE only accepts the kubernetes.io/ingress.class: nginx
when the nginx part is in quotes: kubernetes.io/ingress.class: "nginx"
. Otherwise it ignores it and supplies the L7 GLB instead of the NGINX controller I deployed in the cluster.
So in fact the health problem went away when the correct annotation was assigned, as the nginx load balancer does not require the websocket server to have a health check.
Hi, I have issue on setting in bare metal kubeadm behind SLB. Do you have solution for this ?
I always back to this page in any stuck page, and this solution didn't works for me
Chapeau bas!
in my case ingress was not available directly, but via Classic Load Balancer which does not support sockets :(
https://aws.amazon.com/elasticloadbalancing/features/
Does anyone know any Guide for configuring haproxy-ingress to support WebSockets?
Fantastic post by the way has really helped me.
So following the OP instructions, I have the ingress using SSL and using wss: from the client to connect to the ingress, but then the back-end is just HTTP.
I am getting this error: failed: Error in connection establishment: net::ERR_SSL_PROTOCOL_ERROR
So my follow up question:
Do I need to configure the back-end service with an SSL cert or is there some ingress magic configuration that would allow it to up stream the request as ws: ?
thx
If you are using kubernetes you should probably instlal https://github.com/jetstack/cert-manager and update your ingress to handle SSL certificate.
If you are using kubernetes you should probably instlal https://github.com/jetstack/cert-manager and update your ingress to handle SSL certificate.
I have SSL terminating at the ingress controller, with a certificate configured. Do I also need to configure an additional certificate on the backend service?
Thanks @jsdevtom!!!
I just had the issue with the websocket connection close after 30 seconds and therefore I find your gist.
I'm using GKE with the L7 GLB. For those, who are using the Google cloud infrastructure it is pretty simple to get websocket connections working with google managed ssl certificates. It looks a little bit different:
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: service-backend-config
spec:
timeoutSec: 3600
---
apiVersion: v1
kind: Service
metadata:
labels:
app: service
name: service
annotations:
cloud.google.com/backend-config: '{"ports": {"8080":"service-backend-config"}}'
spec:
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 8080
selector:
app: service
type: NodePort
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: public-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: "your-ip"
networking.gke.io/managed-certificates: "your-certificates"
spec:
rules:
- host: "your-domain.com"
http:
paths:
- backend:
serviceName: service
servicePort: 8080
path: /*
in k8s 1.18.8 the annotations need a string not a number:
nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
nginx.ingress.kubernetes.io/proxy-send-timeout: 3600
==>
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
still not working though
I'm sure that missing trailing forward slash on /ws/
would have cost me 10 hours too. So, thanks!
I cant get this to work with socket io and typescript. Any suggestion will be highly appreciated
After some hours trying to fix it, i found the proper config for my ingress ( k8s azure )
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-service
namespace: development
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
rules:
- host: svc-host-domain.com
http:
paths:
- backend:
service:
name: my-service
port:
number: 8084
path: /publish
pathType: Prefix
- backend:
service:
name: my-service
port:
number: 8080
path: /ws/
pathType: Prefix
Thank you for the great snippet
( i'm still struggling to have the keep-alive working well, it disconnects my websocket clients suddenly )
I'm sure that missing trailing forward slash on
/ws/
would have cost me 10 hours too. So, thanks!
What should happen if the trailing slash is missing? I'm trying to debug an error related to my k8s websocket setup, and I'm using /ws
right now, but I'm not exactly sure it's failing...
A working e2e complete setup of this scenario is available here: https://github.com/martencassel/nodejs-websocket-nginx-ingress-setup
A working e2e complete setup of this scenario is available here: https://github.com/martencassel/nodejs-websocket-nginx-ingress-setup
🔥
Thank you you save me a lot of time. I have been struggling with this issue, at the time I didn't know that I have to configure my ingress for websocket. Thank you again.
Thank you for this, was struggling with something very similar when trying to use an ingress for my websocket client when deployed in minikube. If it helps anyone else:
Websocket client:
const websocketUrl = "ws://minikube.local/ws/"; // Replace with your WebSocket server URL
const socket = new WebSocket(websocketUrl);
(for local deployment, edit /etc/hosts so that minikube.local resolves to your minikube ip)
Websocket-server ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: websocket-server-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: minikube.local
http:
paths:
- path: /ws/
pathType: Prefix
backend:
service:
name: websocket-server
port:
number: 8080
Websocket-server service:
apiVersion: v1
kind: Service
metadata:
name: websocket-server
spec:
selector:
app: websocket-server
ports:
- protocol: TCP
port: 8080
It took me roughly a total of 28 hours to get my frontend to be able to access the node.js websocket server with Kubernetes in MiniKube.
Mainly due to my lack of knowledge/ plain stupidity in the realms of networking. I tried many debugging techniques. For people who run in to similar errors, here is a list of pitfalls mixed with a checklist of tips to debug/ solve this problem:
- path: /ws/
NOT- path: /ws
path: '/ws/'
wss://${location.hostname}:${location.protocol === 'https:' ? 443 : 80}/ws/
ws
and NOTwss
). This is because the ingress will most likely be managing the certificate for you. Trying to use wss inside your server will result in mysterious 502 Bad Gateway errors that don't show up in the in ingress' pod's logs (which can be found usingkubectl exec -n ingress-nginx <name of ingress controller> cat nginx.conf
)Criticism and suggestions are extremely welcome!