expose NebulaGraph for Spark

expose-nebulagraph-for-spark-docker.md

For k8s deployment, see https://gist.github.com/wey-gu/699b9a2ef5dff5f0fb5f288d692ddfd5

Create extral interfaces for host

If not leveraging multiple interfaces, we have to use TLS instead to leverage SNI routing

ip address add 10.1.1.157/24 dev eth0
ip address add 10.1.1.156/24 dev eth0
ip address add 10.1.1.155/24 dev eth0
ip address add 10.1.1.154/24 dev eth0
ip address add 10.1.1.152/24 dev eth0

Now we have 6 hosts:

Option1 needs 3 hosts while Option2 needs 6 hosts

If we cannot create multiple interfaces, we need then use different ip_port instead.

$ ip address show eth0

3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 0e:b2:a1:8e:61:d7 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.152/24 brd 10.1.1.255 scope global dynamic noprefixroute eth0
       valid_lft 33513sec preferred_lft 33513sec
    inet 10.1.1.151/24 brd 10.1.1.255 scope global secondary noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 10.1.1.154/24 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet 10.1.1.157/24 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet 10.1.1.156/24 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet 10.1.1.155/24 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::c359:9b82:d450:4c25/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

Option1: expose ports

modifying nebula-docker-compose/docker-compose.yaml

--- a/docker-compose.yaml
+++ b/docker-compose.yaml
@@ -22,7 +22,7 @@ services:
       retries: 3
       start_period: 20s
     ports:
-      - 9559
+      - 10.1.1.151:9559:9559
       - 19559
       - 19560
     volumes:
@@ -56,7 +56,7 @@ services:
       retries: 3
       start_period: 20s
     ports:
-      - 9559
+      - 10.1.1.152:9559:9559
       - 19559
       - 19560
     volumes:
@@ -90,7 +90,7 @@ services:
       retries: 3
       start_period: 20s
     ports:
-      - 9559
+      - 10.1.1.154:9559:9559
       - 19559
       - 19560
     volumes:
@@ -128,7 +128,7 @@ services:
       retries: 3
       start_period: 20s
     ports:
-      - 9779
+      - 10.1.1.151:9779:9779
       - 19779
       - 19780
     volumes:
@@ -166,7 +166,7 @@ services:
       retries: 3
       start_period: 20s
     ports:
-      - 9779
+      - 10.1.1.152:9779:9779
       - 19779
       - 19780
     volumes:
@@ -204,7 +204,7 @@ services:
       retries: 3
       start_period: 20s
     ports:
-      - 9779
+      - 10.1.1.154:9779:9779
       - 19779
       - 19780
     volumes:

Configure dns/hosts in client host

/etc/hosts

10.1.1.151 graphd metad0 storaged0
10.1.1.152 metad1 storaged1
10.1.1.154 metad2 storaged2

Option2: Create a nginx for handling the traffic:

In this case, we don't need to modify nebula-docker-compose.yaml.

Get exposed ports

docker ps --format "table {{.Names}}\t{{.Ports}}" | grep "9559\|9779"

Create a reverse proxy

Create a docker-compose.yaml under a folder named reverseproxy.

docker-compose.yaml:

version: '3.7'
services:

  nginx:
    image: nginx:1.19.2-alpine
    container_name: nginx
    hostname: nginx
    volumes:
      - ${PWD}/nginx.conf:/etc/nginx/nginx.conf:ro
    network_mode: "host"

Create a nginx.conf under same folder, where we need to put the port got in above step:

nginx.conf:

events {}
stream {

    map_hash_bucket_size 128;
    map_hash_max_size 2048;

    map $server_addr $name {
        10.1.1.155 storaged0;
        10.1.1.156 storaged1;
        10.1.1.157 storaged2;
        10.1.1.151 metad0;
        10.1.1.152 metad1;
        10.1.1.154 metad2;
        default https_default_backend;
    }

    upstream storaged0 {
        server localhost:49255;
    }
    upstream storaged1 {
        server localhost:49256;
    }
    upstream storaged2 {
        server localhost:49257;
    }
    upstream metad0 {
        server localhost:49252;
    }
    upstream metad1 {
        server localhost:49253;
    }
    upstream metad2 {
        server localhost:49254;
    }

    upstream https_default_backend {
        server 127.0.0.1:443;
    }

    server {
        listen 10.1.1.157:9779;
        listen 10.1.1.156:9779;
        listen 10.1.1.155:9779;
        listen 10.1.1.151:9559;
        listen 10.1.1.152:9559;
        listen 10.1.1.154:9559;
        proxy_pass $name;
    }
}

configure dns/hosts on client host

/etc/hosts

10.1.1.151 graphd metad0
10.1.1.155 storaged0
10.1.1.156 storaged1
10.1.1.157 storaged2
10.1.1.152 metad1
10.1.1.154 metad2

Another obvious option that requires no extra interfaces is:

• expose all 9559/9779 ports like the first graphd, but with different ports
• --meta_server_addrs modified accordingly
• in client side create hosts/dns to resolve metad0-2, storaged0-2 to the IP of docker host