Skip to content

Instantly share code, notes, and snippets.

@MichelDiz MichelDiz/README.md forked from danielmai/README.md
Last active Feb 17, 2020

Embed
What would you like to do?
Dgraph + Nginx Load Balancing

First thing

One thing that I have to mention is that this Gist is a personal test env. Some things can be configured in a way that you don't want it to be.

Add the host information in the host file to your system (or do some DNS config)

Go to http://grafana_http:3000/ and import the grafana_dgraph.json

You have HTTP and GRPC endpoints.

You can remove zero_grpc, zero1 and alpha1.

You also have jaeger_http and ratel_http.

You can use docker-compose logs -f alpha1 in a new terminal to follow a specific container. Not all containers you can get logs. Cuz they're deactivated.

Tips

Use docker container prune to delete all containers in your Docker. Use docker volume prune to delete all volumes.

Basics

Download this gist's ZIP file and extract it to a directory called dgraph-nginx.

mkdir dgraph-nginx
cd dgraph-nginx
wget -O dgraph-nginx.zip https://gist.github.com/danielmai/0cf7647b27c7626ad8944c4245a9981e/archive/5a2f1a49ca2f77bc39981749e4783e3443eb3ad9.zip
unzip -j dgraph-nginx.zip

This creates two files: docker-compose.yml and nginx.conf.

Start the 6-node Dgraph cluster (3 Dgraph Zero, 3 Dgraph Alpha, replication setting

  1. by starting the Docker Compose config:
docker-compose up

In a different shell, run the dgraph increment (docs) tool against the Nginx gRPC load balancer (nginx:9080):

docker-compose exec alpha1 dgraph increment --alpha nginx:9080 --num=10

If you have dgraph installed on your host machine, then you can also run this from the host:

dgraph increment --alpha localhost:9080 --num=10

The increment tool uses the Dgraph Go client to establish a gRPC connection against the --alpha flag and transactionally increments a counter predicate --num times.

In the Nginx access logs (in the docker-compose up shell window), you'll see access logs like the following:

nginx_1   | [15/Jan/2020:03:12:02 +0000] 172.20.0.9 - - -  nginx to: 172.20.0.7:9080: POST /api.Dgraph/Query HTTP/2.0 200 upstream_response_time 0.008 msec 1579057922.135 request_time 0.009
nginx_1   | [15/Jan/2020:03:12:02 +0000] 172.20.0.9 - - -  nginx to: 172.20.0.2:9080: POST /api.Dgraph/Query HTTP/2.0 200 upstream_response_time 0.012 msec 1579057922.149 request_time 0.013
nginx_1   | [15/Jan/2020:03:12:02 +0000] 172.20.0.9 - - -  nginx to: 172.20.0.5:9080: POST /api.Dgraph/Query HTTP/2.0 200 upstream_response_time 0.008 msec 1579057922.162 request_time 0.012
nginx_1   | [15/Jan/2020:03:12:02 +0000] 172.20.0.9 - - -  nginx to: 172.20.0.7:9080: POST /api.Dgraph/Query HTTP/2.0 200 upstream_response_time 0.012 msec 1579057922.176 request_time 0.013
nginx_1   | [15/Jan/2020:03:12:02 +0000] 172.20.0.9 - - -  nginx to: 172.20.0.2:9080: POST /api.Dgraph/Query HTTP/2.0 200 upstream_response_time 0.012 msec 1579057922.188 request_time 0.011
nginx_1   | [15/Jan/2020:03:12:02 +0000] 172.20.0.9 - - -  nginx to: 172.20.0.5:9080: POST /api.Dgraph/Query HTTP/2.0 200 upstream_response_time 0.016 msec 1579057922.202 request_time 0.013

The logs say that it load balanced traffic to the following upstream addresses defined in alpha_grpc in nginx.conf:

  • nginx to: 172.20.0.7
  • nginx to: 172.20.0.2
  • nginx to: 172.20.0.5

By default, Nginx load balancing is done round-robin.

Using Grafana

1 - add datasources.

  • Find the Prometheus.
  • In "HTTP" http://prometheus:9090
  • go to "import" and poste the JSON file "grafana_dgraph.json"

Done

global:
scrape_interval: 2s
scrape_configs:
- job_name: "dgraph"
metrics_path: "/debug/prometheus_metrics"
scrape_interval: "2s"
static_configs:
- targets:
- zero1:6080
- zero2:6080
- zero3:6080
- zero4:6080
- zero5:6080
- zero6:6080
- alpha1:8080
- alpha2:8080
- alpha3:8080
- alpha4:8080
- alpha5:8080
- alpha6:8080
- alpha7:8080
- alpha8:8080
- alpha9:8080
- alpha10:8080
- alpha11:8080
- alpha12:8080
- alpha13:8080
- alpha14:8080
- job_name: "node-exporter"
scrape_interval: 2s
metrics_path: "/metrics"
static_configs:
- targets:
- "node-exporter:9100"
version: "3.5"
services:
nginx:
image: nginx:1.17.7
depends_on:
# Hostnames referenced in nginx.conf need to be available
# before Nginx starts
- zero1
- zero2
- zero3
- zero4
- zero5
- zero6
- alpha1
- alpha2
- alpha3
- alpha4
- alpha5
- alpha6
- alpha7
- alpha8
- alpha9
- alpha10
- alpha11
- alpha12
- jaeger
- ratel
ports:
- 80:80
- 8080:8080
- 9080:9080
- 8000:8000
- 9090:9090
- 14268:14268
- 16686:16686
- 3000:3000
volumes:
- type: bind
source: ./nginx.conf
target: /etc/nginx/conf.d/dgraph.conf
read_only: true
zero1:
image: dgraph/dgraph:v2.0.0-rc1
hostname: zero1
ports:
- 5080
- 6080
restart: on-failure
command: dgraph zero --idx=1 --my=zero1:5080 --logtostderr -v=2
--bindall --jaeger.collector=http://jaeger:14268
zero2:
image: dgraph/dgraph:v2.0.0-rc1
logging:
driver: none
hostname: zero2
ports:
- 5080
- 6080
restart: on-failure
command: dgraph zero --idx=2 --my=zero2:5080 --replicas=3 --logtostderr -v=2
--peer=zero1:5080 --jaeger.collector=http://jaeger:14268
zero3:
image: dgraph/dgraph:v2.0.0-rc1
logging:
driver: none
hostname: zero3
ports:
- 5080
- 6080
restart: on-failure
command: dgraph zero --idx=3 --my=zero3:5080 --replicas=3 --logtostderr -v=2
--peer=zero1:5080 --jaeger.collector=http://jaeger:14268
zero4:
image: dgraph/dgraph:v2.0.0-rc1
logging:
driver: none
hostname: zero1
ports:
- 5080
- 6080
restart: on-failure
command: dgraph zero --idx=4 --my=zero4:5080 --logtostderr -v=2
--bindall --jaeger.collector=http://jaeger:14268
zero5:
image: dgraph/dgraph:v2.0.0-rc1
logging:
driver: none
hostname: zero2
ports:
- 5080
- 6080
restart: on-failure
command: dgraph zero --idx=5 --my=zero5:5080 --replicas=3 --logtostderr -v=2
--peer=zero1:5080 --jaeger.collector=http://jaeger:14268
zero6:
image: dgraph/dgraph:v2.0.0-rc1
logging:
driver: none
hostname: zero3
ports:
- 5080
- 6080
restart: on-failure
command: dgraph zero --idx=6 --my=zero6:5080 --replicas=3 --logtostderr -v=2
--peer=zero1:5080 --jaeger.collector=http://jaeger:14268
alpha1:
image: dgraph/dgraph:v2.0.0-rc1
hostname: alpha1
ports:
- 8080
- 9080
restart: on-failure
command:
dgraph alpha --my=alpha1:7080 --lru_mb=4096 --zero=zero1:5080 --logtostderr
-v=2 --idx=1 --jaeger.collector=http://jaeger:14268
alpha2:
image: dgraph/dgraph:v2.0.0-rc1
hostname: alpha2
depends_on:
- alpha1
ports:
- 8080
- 9080
restart: on-failure
command:
dgraph alpha --my=alpha2:7080 --lru_mb=4096 --zero=zero1:5080 --logtostderr
-v=2 --idx=2 --jaeger.collector=http://jaeger:14268
alpha3:
image: dgraph/dgraph:v2.0.0-rc1
hostname: alpha3
depends_on:
- alpha2
ports:
- 8080
- 9080
restart: on-failure
command:
dgraph alpha --my=alpha3:7080 --lru_mb=4096 --zero=zero1:5080 --logtostderr
-v=2 --idx=3 --jaeger.collector=http://jaeger:14268
alpha4:
image: dgraph/dgraph:v2.0.0-rc1
logging:
driver: none
hostname: alpha4
depends_on:
- alpha3
ports:
- 8080
- 9080
restart: on-failure
command:
dgraph alpha --my=alpha4:7080 --lru_mb=4096 --zero=zero1:5080 --logtostderr
-v=2 --idx=4 --jaeger.collector=http://jaeger:14268
alpha5:
image: dgraph/dgraph:v2.0.0-rc1
logging:
driver: none
hostname: alpha5
depends_on:
- alpha4
ports:
- 8080
- 9080
restart: on-failure
command:
dgraph alpha --my=alpha5:7080 --lru_mb=4096 --zero=zero1:5080 --logtostderr
-v=2 --idx=5 --jaeger.collector=http://jaeger:14268
alpha6:
image: dgraph/dgraph:v2.0.0-rc1
logging:
driver: none
hostname: alpha6
depends_on:
- alpha5
ports:
- 8080
- 9080
restart: on-failure
command:
dgraph alpha --my=alpha6:7080 --lru_mb=4096 --zero=zero1:5080 --logtostderr
-v=2 --idx=6 --jaeger.collector=http://jaeger:14268
alpha7:
image: dgraph/dgraph:v2.0.0-rc1
logging:
driver: none
hostname: alpha7
depends_on:
- alpha6
ports:
- 8080
- 9080
restart: on-failure
command:
dgraph alpha --my=alpha7:7080 --lru_mb=4096 --zero=zero1:5080 --logtostderr
-v=2 --idx=7 --jaeger.collector=http://jaeger:14268
alpha8:
image: dgraph/dgraph:v2.0.0-rc1
logging:
driver: none
hostname: alpha8
depends_on:
- alpha7
ports:
- 8080
- 9080
restart: on-failure
command:
dgraph alpha --my=alpha8:7080 --lru_mb=4096 --zero=zero1:5080 --logtostderr
-v=2 --idx=8 --jaeger.collector=http://jaeger:14268
alpha9:
image: dgraph/dgraph:v2.0.0-rc1
logging:
driver: none
hostname: alpha9
depends_on:
- alpha8
ports:
- 8080
- 9080
restart: on-failure
command:
dgraph alpha --my=alpha9:7080 --lru_mb=4096 --zero=zero1:5080 --logtostderr
-v=2 --idx=9 --jaeger.collector=http://jaeger:14268
alpha10:
image: dgraph/dgraph:v2.0.0-rc1
logging:
driver: none
hostname: alpha10
depends_on:
- alpha9
ports:
- 8080
- 9080
restart: on-failure
command:
dgraph alpha --my=alpha10:7080 --lru_mb=4096 --zero=zero1:5080 --logtostderr
-v=2 --idx=10 --jaeger.collector=http://jaeger:14268
alpha11:
image: dgraph/dgraph:v2.0.0-rc1
logging:
driver: none
hostname: alpha11
depends_on:
- alpha10
ports:
- 8080
- 9080
restart: on-failure
command:
dgraph alpha --my=alpha11:7080 --lru_mb=4096 --zero=zero1:5080 --logtostderr
-v=2 --idx=11 --jaeger.collector=http://jaeger:14268
alpha12:
image: dgraph/dgraph:v2.0.0-rc1
logging:
driver: none
hostname: alpha12
depends_on:
- alpha11
ports:
- 8080
- 9080
restart: on-failure
command:
dgraph alpha --my=alpha12:7080 --lru_mb=4096 --zero=zero1:5080 --logtostderr
-v=2 --idx=12 --jaeger.collector=http://jaeger:14268
alpha13:
image: dgraph/dgraph:v2.0.0-rc1
logging:
driver: none
hostname: alpha13
depends_on:
- alpha12
ports:
- 8080
- 9080
restart: on-failure
command:
dgraph alpha --my=alpha13:7080 --lru_mb=4096 --zero=zero1:5080 --logtostderr
-v=2 --idx=13 --jaeger.collector=http://jaeger:14268
alpha14:
image: dgraph/dgraph:v2.0.0-rc1
logging:
driver: none
hostname: alpha14
depends_on:
- alpha13
ports:
- 8080
- 9080
restart: on-failure
command:
dgraph alpha --my=alpha14:7080 --lru_mb=4096 --zero=zero1:5080 --logtostderr
-v=2 --idx=14 --jaeger.collector=http://jaeger:14268
ratel:
image: dgraph/dgraph:v2.0.0-rc1
logging:
driver: none
hostname: ratel
ports:
- 8000
command: dgraph-ratel
jaeger:
image: jaegertracing/all-in-one:latest
container_name: jaeger
working_dir: /working/jaeger
logging:
driver: none
ports:
- 14268
- 16686
command: --memory.max-traces=1000000
node-exporter:
image: quay.io/prometheus/node-exporter
container_name: node-exporter
pid: host
working_dir: /working/jaeger
logging:
driver: none
volumes:
- type: bind
source: ./
target: /host
read_only: true
prometheus:
image: prom/prometheus:latest
logging:
driver: none
hostname: prometheus
volumes:
- type: volume
source: prometheus-volume
target: /prometheus
read_only: false
- type: bind
source: ./dgraph_prometheus.yml
target: /etc/prometheus/prometheus.yml
read_only: true
command:
- "--config.file=/etc/prometheus/prometheus.yml"
ports:
- 9090
grafana:
image: grafana/grafana:latest
hostname: "grafana"
logging:
driver: none
environment:
# - GF_SECURITY_ADMIN_PASSWORD=secret
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
ports:
- 3000
volumes:
- type: volume
source: grafana-volume
target: /var/lib/grafana
read_only: false
volumes:
grafana-volume: {}
prometheus-volume: {}
{
"__inputs": [
{
"name": "DS_DGRAPH",
"label": "localhost",
"description": "Datasource with Dgraph metrics",
"type": "datasource",
"pluginId": "prometheus",
"pluginname": "prometheus"
}
],
"version": 4,
"id": 4,
"title": "Dgraph",
"originalTitle": "Dgraph",
"tags": [],
"style": "dark",
"timezone": "browser",
"editable": true,
"hideControls": false,
"sharedCrosshair": false,
"gnetId": null,
"graphTooltip": 0,
"iteration": 1581648260240,
"refresh": "5s",
"schemaVersion": 22,
"uid": "I3MEcmwZz",
"panels": [
{
"datasource": "Prometheus",
"gridPos": {
"h": 6,
"w": 12,
"x": 0,
"y": 0
},
"id": 34,
"links": [],
"options": {
"fieldOptions": {
"calcs": [
"last"
],
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"title": "Memory in use",
"unit": "decbytes"
},
"limit": 3,
"overrides": [],
"values": false
},
"orientation": "auto",
"showThresholdLabels": false,
"showThresholdMarkers": true
},
"pluginVersion": "6.6.1",
"targets": [
{
"expr": "(dgraph_memory_idle_bytes{instance=~'$Instance'}+dgraph_memory_inuse_bytes{instance=~'$Instance'})",
"intervalFactor": 1,
"legendFormat": "",
"metric": "dgraph_memory_idle_bytes",
"refId": "A",
"step": 2
}
],
"timeFrom": null,
"timeShift": null,
"title": "",
"transparent": true,
"type": "gauge"
},
{
"cacheTimeout": null,
"datasource": "Prometheus",
"gridPos": {
"h": 2,
"w": 4,
"x": 13,
"y": 0
},
"hideTimeOverride": false,
"id": 30,
"links": [],
"options": {
"colorMode": "background",
"fieldOptions": {
"calcs": [
"max"
],
"defaults": {
"mappings": [
{
"id": 0,
"op": "=",
"text": "N/A",
"type": 1,
"value": "null"
}
],
"nullValueMode": "connected",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 1
}
]
},
"title": "Health Status -",
"unit": "short"
},
"overrides": [],
"values": false
},
"graphMode": "none",
"justifyMode": "center",
"orientation": "vertical"
},
"pluginVersion": "6.6.1",
"targets": [
{
"expr": "dgraph_alpha_health_status{instance=~'$Instance'}-1",
"format": "heatmap",
"hide": false,
"instant": false,
"intervalFactor": 1,
"legendFormat": "",
"metric": "dgraph_active_mutations_total",
"refId": "A",
"step": 2
}
],
"timeFrom": null,
"timeShift": null,
"title": "Zero and Alpha",
"transparent": true,
"type": "stat"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorPostfix": false,
"colorPrefix": false,
"colorValue": true,
"colors": [
"#299c46",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": null,
"decimals": null,
"description": "",
"format": "short",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 3,
"w": 7,
"x": 17,
"y": 0
},
"id": 28,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"options": {},
"pluginVersion": "6.6.1",
"postfix": " /RDFs",
"postfixFontSize": "70%",
"prefix": "",
"prefixFontSize": "20%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false,
"ymax": null,
"ymin": null
},
"tableColumn": "",
"targets": [
{
"expr": "sum(dgraph_num_edges_total{instance=~'$Instance'})",
"intervalFactor": 1,
"legendFormat": "",
"refId": "A"
}
],
"thresholds": "",
"timeFrom": null,
"timeShift": null,
"title": "",
"transparent": true,
"type": "singlestat",
"valueFontSize": "200%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "max"
},
{
"cacheTimeout": null,
"datasource": null,
"gridPos": {
"h": 3,
"w": 6,
"x": 12,
"y": 3
},
"id": 22,
"links": [],
"options": {
"fieldOptions": {
"calcs": [
"last"
],
"defaults": {
"mappings": [],
"nullValueMode": "connected",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"title": "Goroutines",
"unit": "none"
},
"overrides": [],
"values": false
},
"orientation": "horizontal",
"showThresholdLabels": false,
"showThresholdMarkers": false
},
"pluginVersion": "6.6.1",
"targets": [
{
"expr": "go_goroutines{instance=~'$Instance'}",
"format": "time_series",
"intervalFactor": 1,
"refId": "A"
}
],
"timeFrom": null,
"timeShift": null,
"title": "",
"transparent": true,
"type": "gauge"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorPostfix": false,
"colorPrefix": false,
"colorValue": true,
"colors": [
"#299c46",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": null,
"decimals": null,
"description": "",
"format": "short",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 2,
"w": 6,
"x": 18,
"y": 3
},
"id": 29,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"options": {},
"pluginVersion": "6.6.1",
"postfix": " /queries",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "20%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false,
"ymax": null,
"ymin": null
},
"tableColumn": "",
"targets": [
{
"expr": "sum(dgraph_num_queries_total{instance=~'$Instance'})",
"intervalFactor": 1,
"legendFormat": "",
"refId": "A"
}
],
"thresholds": "",
"timeFrom": null,
"timeShift": null,
"title": "",
"transparent": true,
"type": "singlestat",
"valueFontSize": "150%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "total"
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"editable": true,
"error": false,
"fill": 1,
"fillGradient": 0,
"grid": {},
"gridPos": {
"h": 9,
"w": 24,
"x": 0,
"y": 6
},
"hiddenSeries": false,
"id": 1,
"isNew": true,
"legend": {
"alignAsTable": false,
"avg": false,
"current": false,
"max": true,
"min": false,
"rightSide": false,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "connected",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "dgraph_memory_inuse_bytes+dgraph_memory_idle_bytes{instance=~'$Instance'}",
"intervalFactor": 2,
"legendFormat": "",
"metric": "dgraph_memory_idle_bytes",
"refId": "A",
"step": 2
},
{
"expr": "dgraph_memory_proc_bytes{instance=~'$Instance'}",
"intervalFactor": 2,
"legendFormat": "",
"metric": "dgraph_memory_proc_bytes",
"refId": "B",
"step": 2
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Total memory",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "bytes",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": "",
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"editable": true,
"error": false,
"fill": 1,
"fillGradient": 0,
"grid": {},
"gridPos": {
"h": 7,
"w": 12,
"x": 0,
"y": 15
},
"hiddenSeries": false,
"id": 17,
"isNew": true,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"max": true,
"min": false,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "connected",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "dgraph_active_mutations_total{instance=~'$Instance'}",
"intervalFactor": 2,
"legendFormat": "",
"metric": "dgraph_active_mutations_total",
"refId": "A",
"step": 2
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Active mutations",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"editable": true,
"error": false,
"fill": 0,
"fillGradient": 0,
"grid": {},
"gridPos": {
"h": 7,
"w": 12,
"x": 12,
"y": 15
},
"hiddenSeries": false,
"id": 5,
"isNew": true,
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"hideEmpty": false,
"hideZero": false,
"max": false,
"min": false,
"rightSide": false,
"show": true,
"total": true,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "connected",
"options": {
"dataLinks": []
},
"percentage": false,
"pluginVersion": "6.6.1",
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "dgraph_pending_proposals_total{instance=~'$Instance'}",
"intervalFactor": 2,
"legendFormat": "",
"metric": "dgraph_pending_proposals_total",
"refId": "A",
"step": 2
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Pending Proposals",
"tooltip": {
"shared": true,
"sort": 1,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"label": "",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": "",
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"editable": true,
"error": false,
"fill": 1,
"fillGradient": 0,
"grid": {},
"gridPos": {
"h": 7,
"w": 12,
"x": 0,
"y": 22
},
"hiddenSeries": false,
"id": 14,
"isNew": true,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "connected",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "dgraph_memory_idle_bytes{instance=~'$Instance'}",
"interval": "",
"intervalFactor": 2,
"legendFormat": "",
"metric": "dgraph_memory_idle_bytes",
"refId": "A",
"step": 2
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Memory Heap Idle",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "bytes",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"cacheTimeout": null,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 7,
"w": 12,
"x": 12,
"y": 22
},
"hiddenSeries": false,
"hideTimeOverride": false,
"id": 23,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pluginVersion": "6.6.1",
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "dgraph_num_queries_total",
"format": "heatmap",
"hide": false,
"instant": false,
"intervalFactor": 1,
"legendFormat": "",
"metric": "dgraph_active_mutations_total",
"refId": "A",
"step": 2
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Processed Queries",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"editable": true,
"error": false,
"fill": 1,
"fillGradient": 0,
"grid": {},
"gridPos": {
"h": 7,
"w": 12,
"x": 0,
"y": 29
},
"hiddenSeries": false,
"id": 6,
"isNew": true,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "connected",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "rate(go_gc_duration_seconds_sum{instance=~'$Instance'}[30s])",
"intervalFactor": 2,
"legendFormat": "",
"metric": "go_gc_duration_seconds_sum",
"refId": "A",
"step": 2
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "GC second sum rate(30s)",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"editable": true,
"error": false,
"fill": 1,
"fillGradient": 0,
"grid": {},
"gridPos": {
"h": 7,
"w": 12,
"x": 12,
"y": 29
},
"hiddenSeries": false,
"id": 16,
"isNew": true,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "connected",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "dgraph_pending_queries_total{instance=~'$Instance'}",
"intervalFactor": 2,
"legendFormat": "",
"metric": "dgraph_pending_queries_total",
"refId": "A",
"step": 2
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Pending Queries",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"cacheTimeout": null,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 7,
"w": 12,
"x": 0,
"y": 36
},
"hiddenSeries": false,
"hideTimeOverride": false,
"id": 31,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pluginVersion": "6.6.1",
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "dgraph_raft_applied_index{instance=~'$Instance'}",
"format": "heatmap",
"hide": false,
"instant": false,
"intervalFactor": 1,
"legendFormat": "",
"metric": "dgraph_active_mutations_total",
"refId": "A",
"step": 2
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Raft Applied Index",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"editable": true,
"error": false,
"fill": 1,
"fillGradient": 0,
"grid": {},
"gridPos": {
"h": 7,
"w": 12,
"x": 12,
"y": 36
},
"hiddenSeries": false,
"id": 18,
"isNew": true,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "connected",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "dgraph_alpha_health_status{instance=~'$Instance'}",
"intervalFactor": 2,
"legendFormat": "",
"metric": "dgraph_alpha_health_status",
"refId": "A",
"step": 2
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Server Health",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "cumulative"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"cacheTimeout": null,
"dashLength": 10,
"dashes": false,
"datasource": "Prometheus",
"description": "Last threads used by go.",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 7,
"w": 12,
"x": 12,
"y": 43
},
"hiddenSeries": false,
"hideTimeOverride": false,
"id": 32,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pluginVersion": "6.6.1",
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "go_threads",
"format": "heatmap",
"hide": false,
"instant": false,
"intervalFactor": 1,
"legendFormat": "",
"metric": "dgraph_active_mutations_total",
"refId": "A",
"step": 2
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Threads",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"timepicker": {
"now": true,
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"templating": {
"list": [
{
"allFormat": "regex wildcard",
"allValue": null,
"current": {
"selected": false,
"text": "All",
"value": "$__all"
},
"datasource": "Prometheus",
"definition": "",
"hide": 0,
"includeAll": true,
"label": null,
"multi": false,
"multiFormat": "glob",
"name": "Instance",
"options": [],
"query": "label_values(instance)",
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},
"time": {
"from": "now-12h",
"to": "now"
},
"links": []
}
192.168.99.108 alpha_http
192.168.99.108 alpha_grpc
192.168.99.108 zero_grpc
192.168.99.108 zero1
192.168.99.108 alpha1
192.168.99.108 jaeger_http
192.168.99.108 ratel_http
192.168.99.108 grafana_http
upstream alpha_grpc {
server alpha1:9080;
server alpha2:9080;
server alpha3:9080;
server alpha4:9080;
server alpha5:9080;
server alpha6:9080;
server alpha7:9080;
server alpha8:9080;
server alpha9:9080;
server alpha10:9080;
server alpha11:9080;
server alpha12:9080;
server alpha13:9080;
server alpha14:9080;
}
upstream alpha_http {
server alpha1:8080;
server alpha2:8080;
server alpha3:8080;
server alpha4:8080;
server alpha5:8080;
server alpha6:8080;
server alpha7:8080;
server alpha8:8080;
server alpha9:8080;
server alpha10:8080;
server alpha11:8080;
server alpha12:8080;
server alpha13:8080;
server alpha14:8080;
}
upstream ratel_http {
server ratel:8000;
}
upstream jaeger_http {
server jaeger:16686;
}
upstream grafana_http {
server grafana:3000;
}
upstream prometheus_http {
server prometheus:9090;
}
# $upstream_addr is the ip:port of the Dgraph Alpha defined in the upstream
# Example: 172.25.0.2, 172.25.0.7, 172.25.0.5 are the IP addresses of alpha1, alpha2, and alpha3
# /var/log/nginx/access.log will contain these logs showing "localhost to <upstream address>"
# for the different backends. By default, Nginx load balancing is round robin.
# [15/Jan/2020:00:28:11 +0000] 172.25.0.1 - - - localhost to: 172.25.0.2:9080: POST /api.Dgraph/Query HTTP/2.0 200 upstream_response_time 0.028 msec 1579048091.865 request_time 0.027
# [15/Jan/2020:00:28:11 +0000] 172.25.0.1 - - - localhost to: 172.25.0.7:9080: POST /api.Dgraph/Query HTTP/2.0 200 upstream_response_time 0.032 msec 1579048091.897 request_time 0.031
# [15/Jan/2020:00:28:11 +0000] 172.25.0.1 - - - localhost to: 172.25.0.5:9080: POST /api.Dgraph/Query HTTP/2.0 200 upstream_response_time 0.028 msec 1579048091.926 request_time 0.028
log_format upstreamlog '[$time_local] $remote_addr - $remote_user - $server_name $host to: $upstream_addr: $request $status upstream_response_time $upstream_response_time msec $msec request_time $request_time';
server {
listen 9080 http2;
server_name alpha_grpc;
access_log /var/log/nginx/access.log upstreamlog;
location / {
grpc_pass grpc://alpha_grpc;
}
}
server {
listen 8080;
server_name alpha_http;
access_log /var/log/nginx/access.log upstreamlog;
location / {
proxy_pass http://alpha_http;
}
}
server {
listen 8000;
server_name ratel_http;
access_log /var/log/nginx/access.log upstreamlog;
location / {
proxy_pass http://ratel_http;
}
}
server {
listen 16686;
server_name jaeger;
access_log /var/log/nginx/access.log upstreamlog;
location / {
proxy_pass http://jaeger_http;
}
}
server {
listen 3000;
server_name grafana;
access_log /var/log/nginx/access.log upstreamlog;
location / {
proxy_pass http://grafana_http;
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.