for Docker compose deployment, see https://gist.github.com/wey-gu/950e4f4c673badae375e59007d80d372
a. create services per metad and storaged pod with LoadBalancer type to expose outside the cluster
b. Use TCP Proxy(b.1) or/and DNS(b.2) to resolve endpoint as their : inside the cluster
In this example, in b. I give a demo where both TCP Proxy and DNS are used, actually only DNS is enough if the exposed port remains same of src and target.
a. service for metad and storaged
kubectl apply -f svc.yaml
❯ kubectl get svc -n nebula
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
metad-0 LoadBalancer 10.96.221.218 192.168.49.2 19559:31915/TCP,9559:31808/TCP 38m
metad-1 LoadBalancer 10.108.35.180 192.168.49.3 19560:31443/TCP,9559:32029/TCP 38m
metad-2 LoadBalancer 10.98.148.165 192.168.49.4 19559:31254/TCP,9559:31904/TCP 38m
nebula-graphd-svc ClusterIP 10.98.76.0 <none> 9669/TCP,19669/TCP,19670/TCP 43h
nebula-metad-headless ClusterIP None <none> 9559/TCP,19559/TCP,19560/TCP 43h
nebula-storaged-headless ClusterIP None <none> 9779/TCP,19779/TCP,19780/TCP,9778/TCP 43h
storaged-0 LoadBalancer 10.100.96.93 192.168.49.5 19779:32716/TCP,9779:31209/TCP 3s
storaged-1 LoadBalancer 10.98.60.202 192.168.49.6 19779:30429/TCP,9779:32210/TCP 3s
storaged-2 LoadBalancer 10.105.215.225 192.168.49.7 19779:30055/TCP,9779:31030/TCP 3s
svc.yaml:
apiVersion: v1
kind: Service
metadata:
name: metad-0
namespace: nebula
spec:
selector:
statefulset.kubernetes.io/pod-name: nebula-metad-0
ports:
- protocol: TCP
port: 19559
targetPort: 19559
name: http
- protocol: TCP
port: 9559
targetPort: 9559
name: thrift
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
name: metad-1
namespace: nebula
spec:
selector:
statefulset.kubernetes.io/pod-name: nebula-metad-1
ports:
- protocol: TCP
port: 19560
targetPort: 19559
name: http
- protocol: TCP
port: 9559
targetPort: 9559
name: thrift
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
name: metad-2
namespace: nebula
spec:
selector:
statefulset.kubernetes.io/pod-name: nebula-metad-2
ports:
- protocol: TCP
port: 19559
targetPort: 19559
name: http
- protocol: TCP
port: 9559
targetPort: 9559
name: thrift
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
name: storaged-0
namespace: nebula
spec:
selector:
statefulset.kubernetes.io/pod-name: nebula-storaged-0
ports:
- protocol: TCP
port: 19779
targetPort: 19779
name: http
- protocol: TCP
port: 9779
targetPort: 9779
name: thrift
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
name: storaged-1
namespace: nebula
spec:
selector:
statefulset.kubernetes.io/pod-name: nebula-storaged-1
ports:
- protocol: TCP
port: 19779
targetPort: 19779
name: http
- protocol: TCP
port: 9779
targetPort: 9779
name: thrift
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
name: storaged-2
namespace: nebula
spec:
selector:
statefulset.kubernetes.io/pod-name: nebula-storaged-2
ports:
- protocol: TCP
port: 19779
targetPort: 19779
name: http
- protocol: TCP
port: 9779
targetPort: 9779
name: thrift
type: LoadBalancer
b.1 Nginx as TCP proxy(haproxy or equivalent service by infra provider) , configuration in /etc/nginx/nginx.conf
Note: This example only covers the storaged instances.
stream {
map_hash_bucket_size 128;
map_hash_max_size 2048;
map $server_addr $name {
10.1.1.168 storage0;
10.1.1.69 storage1;
10.1.1.70 storage2;
default https_default_backend;
}
upstream storage0 {
server 192.168.49.5:9779;
}
upstream storage1 {
server 192.168.49.6:9779;
}
upstream storage2 {
server 192.168.49.7:9779;
}
upstream https_default_backend {
server 127.0.0.1:443;
}
server {
listen 10.1.1.168:9779;
listen 10.1.1.69:9779;
listen 10.1.1.70:9779;
proxy_pass $name;
}
}And the arch looks like:
┌──────────────────────────────────────────────────────┐
│ K8s Cluster │
│ │
│ │
│ ┌──────────────────────────────────────┐ │
│ │ NebulaGraph Cluster │ │
│ .─────. │ ┌──────────────┐ │ │
┌────┐ │╱ ╲ │ │ Storaged-0 │ │ │
│ ├─▶( Service )──┼─────────▶│ │ │ │
│ │ │`. ,' │ │ │ │ │
│ │ │ `───' │ └──────────────┘ │ │
│ │ │ │ │ │
┌─────┐ │ │ │ .─────. │ ┌──────────────┐ │ │
│ │ │ │ │╱ ╲ │ │ Storaged-1 │ │ │
━┫ DNS ┣━━▶│ngx ├─▶( Service )──┼─────────▶│ │ │ │
│ │ │ │ │`. ,' │ │ │ │ │
└─────┘ │ │ │ `───' │ └──────────────┘ │ │
│ │ │ │ │ │
│ │ │ .─────. │ ┌──────────────┐ │ │
│ │ │╱ ╲ │ │ Storaged-2 │ │ │
│ ├─▶( Service )──┼─────────▶│ │ │ │
│ │ │`. ,' │ │ │ │ │
└────┘ │ `───' │ └──────────────┘ │ │
│ │ │ │
│ └──────────────────────────────────────┘ │
│ │
│ │
└──────────────────────────────────────────────────────┘
b.2 DNS A record of TCP Proxy frontend
10.1.1.168 nebula-storaged-0.nebula-metad-headless.nebula.svc.cluster.local
10.1.1.69 nebula-storaged-1.nebula-metad-headless.nebula.svc.cluster.local
10.1.1.70 nebula-storaged-2.nebula-metad-headless.nebula.svc.cluster.local