Dgraph + Nginx Load Balancing

README.md

Download this gist's ZIP file and extract it to a directory called dgraph-nginx.

mkdir dgraph-nginx
cd dgraph-nginx
wget -O dgraph-nginx.zip https://gist.github.com/danielmai/0cf7647b27c7626ad8944c4245a9981e/archive/5a2f1a49ca2f77bc39981749e4783e3443eb3ad9.zip
unzip -j dgraph-nginx.zip

This creates two files: docker-compose.yml and nginx.conf.

Start the 6-node Dgraph cluster (3 Dgraph Zero, 3 Dgraph Alpha, replication setting 3) by starting the Docker Compose config:

docker-compose up

In a different shell, run the dgraph increment (docs) tool against the Nginx gRPC load balancer (nginx:9080):

docker-compose exec alpha1 dgraph increment --alpha nginx:9080 --num=10

If you have dgraph installed on your host machine, then you can also run this from the host:

dgraph increment --alpha localhost:9080 --num=10

The increment tool uses the Dgraph Go client to establish a gRPC connection against the --alpha flag and transactionally increments a counter predicate --num times.

In the Nginx access logs (in the docker-compose up shell window), you'll see access logs like the following:

nginx_1   | [15/Jan/2020:03:12:02 +0000] 172.20.0.9 - - -  nginx to: 172.20.0.7:9080: POST /api.Dgraph/Query HTTP/2.0 200 upstream_response_time 0.008 msec 1579057922.135 request_time 0.009
nginx_1   | [15/Jan/2020:03:12:02 +0000] 172.20.0.9 - - -  nginx to: 172.20.0.2:9080: POST /api.Dgraph/Query HTTP/2.0 200 upstream_response_time 0.012 msec 1579057922.149 request_time 0.013
nginx_1   | [15/Jan/2020:03:12:02 +0000] 172.20.0.9 - - -  nginx to: 172.20.0.5:9080: POST /api.Dgraph/Query HTTP/2.0 200 upstream_response_time 0.008 msec 1579057922.162 request_time 0.012
nginx_1   | [15/Jan/2020:03:12:02 +0000] 172.20.0.9 - - -  nginx to: 172.20.0.7:9080: POST /api.Dgraph/Query HTTP/2.0 200 upstream_response_time 0.012 msec 1579057922.176 request_time 0.013
nginx_1   | [15/Jan/2020:03:12:02 +0000] 172.20.0.9 - - -  nginx to: 172.20.0.2:9080: POST /api.Dgraph/Query HTTP/2.0 200 upstream_response_time 0.012 msec 1579057922.188 request_time 0.011
nginx_1   | [15/Jan/2020:03:12:02 +0000] 172.20.0.9 - - -  nginx to: 172.20.0.5:9080: POST /api.Dgraph/Query HTTP/2.0 200 upstream_response_time 0.016 msec 1579057922.202 request_time 0.013

The logs say that it load balanced traffic to the following upstream addresses defined in alpha_grpc in nginx.conf:

• nginx to: 172.20.0.7
• nginx to: 172.20.0.2
• nginx to: 172.20.0.5

By default, Nginx load balancing is done round-robin.

docker-compose.yml

# Auto-generated with: [compose -a 3 -z 3 --port_offset=0 --local=false --expose_ports=false]
#
version: "3.5"
services:
  nginx:
    image: nginx:1.17.7
    depends_on:
      # Hostnames referenced in nginx.conf need to be available
      # before Nginx starts
      - zero1
      - zero2
      - zero3
      - alpha1
      - alpha2
      - alpha3
    ports:
      - 80:80
      - 8080:8080
      - 9080:9080
    volumes:
      - type: bind
        source: ./nginx.conf
        target: /etc/nginx/conf.d/dgraph.conf
        read_only: true
  alpha1:
    image: dgraph/dgraph:latest
    hostname: alpha1
    ports:
    - 8080
    - 9080
    command: dgraph alpha --my=alpha1:7080 --lru_mb=1024 --zero=zero1:5080 --logtostderr
      -v=2 --idx=1
  alpha2:
    image: dgraph/dgraph:latest
    hostname: alpha2
    depends_on:
    - alpha1
    labels:
      cluster: test
    ports:
    - 8080
    - 9080
    command: dgraph alpha --my=alpha2:7080 --lru_mb=1024 --zero=zero1:5080 --logtostderr
      -v=2 --idx=2
  alpha3:
    image: dgraph/dgraph:latest
    hostname: alpha3
    ports:
    - 8080
    - 9080
    command: dgraph alpha --my=alpha3:7080 --lru_mb=1024 --zero=zero1:5080 --logtostderr
      -v=2 --idx=3
  zero1:
    image: dgraph/dgraph:latest
    hostname: zero1
    ports:
    - 5080
    - 6080
    command: dgraph zero --idx=1 --my=zero1:5080 --replicas=3 --logtostderr -v=2
      --bindall
  zero2:
    image: dgraph/dgraph:latest
    hostname: zero2
    ports:
    - 5080
    - 6080
    command: dgraph zero --idx=2 --my=zero2:5080 --replicas=3 --logtostderr -v=2
      --peer=zero1:5080
  zero3:
    image: dgraph/dgraph:latest
    hostname: zero3
    ports:
    - 5080
    - 6080
    command: dgraph zero --idx=3 --my=zero3:5080 --replicas=3 --logtostderr -v=2
      --peer=zero1:5080
volumes: {}

nginx.conf

upstream alpha_grpc {
  server alpha1:9080;
  server alpha2:9080;
  server alpha3:9080;
}

upstream alpha_http {
  server alpha1:8080;
  server alpha2:8080;
  server alpha3:8080;
}

# $upstream_addr is the ip:port of the Dgraph Alpha defined in the upstream
# Example: 172.25.0.2, 172.25.0.7, 172.25.0.5 are the IP addresses of alpha1, alpha2, and alpha3
# /var/log/nginx/access.log will contain these logs showing "localhost to <upstream address>"
# for the different backends. By default, Nginx load balancing is round robin.

# [15/Jan/2020:00:28:11 +0000] 172.25.0.1 - - -  localhost to: 172.25.0.2:9080: POST /api.Dgraph/Query HTTP/2.0 200 upstream_response_time 0.028 msec 1579048091.865 request_time 0.027
# [15/Jan/2020:00:28:11 +0000] 172.25.0.1 - - -  localhost to: 172.25.0.7:9080: POST /api.Dgraph/Query HTTP/2.0 200 upstream_response_time 0.032 msec 1579048091.897 request_time 0.031
# [15/Jan/2020:00:28:11 +0000] 172.25.0.1 - - -  localhost to: 172.25.0.5:9080: POST /api.Dgraph/Query HTTP/2.0 200 upstream_response_time 0.028 msec 1579048091.926 request_time 0.028

log_format upstreamlog '[$time_local] $remote_addr - $remote_user - $server_name $host to: $upstream_addr: $request $status upstream_response_time $upstream_response_time msec $msec request_time $request_time';

server {
  listen 9080 http2;
  access_log /var/log/nginx/access.log upstreamlog;
  location / {
    grpc_pass grpc://alpha_grpc;
  }
}

server {
  listen 8080;
  access_log /var/log/nginx/access.log upstreamlog;
  location / {
    proxy_pass http://alpha_http;
  }
}