Running a Django app with gunicorn in a Docker container gets a bit tricky, when you try to expose Prometheus metrics (with e.g. https://github.com/korfuri/django-prometheus).
Every worker thread in gunicorn will be an entirely separate process, all RRD behind a single port. So, if you are running more than one worker thread (which you most likely do and should) you will need to let the workers each listen on a dedicated port, so your metrics will not get confused. You could try to add additional labels, but that still would come with a lot of problems. Running on a range of ports is supported quite well with korfuri/django-prometheus, as described here: https://github.com/korfuri/django-prometheus/blob/master/documentation/exports.md#exporting-metrics-in-a-wsgi-application-with-multiple-processes
This will be a bit uncomfortable though, especially if your app is hidden behind an ingress reverse proxy and you do not want to punch lots of holes into your firewall config, or create lots and lots of vhosts.
As long as the ingress proxy can access all those dedicated ports, you're golden. In this case we're running nginx, which allows you to use HTTP GET Parameters directly in your config.
server {
listen 80;
server_name _;
location / {
return 404;
}
location /metrics {
# ONLY allow Ports in the range of 9100-9199, i.e. matching the following regex:
if ($arg_port !~ "91\d{2}") {
return 403;
}
proxy_pass http://127.0.0.1:$arg_port;
}
}
As you can see, the /metrics location uses the GET Parameter port to modify the proxy_pass directive. For example, accessing /metrics?port=9100 will use proxy_pass http://127.0.0.1:9100;.
With this trick, we've essentially mimicked the same way that you would acces the prometheus_blackbox_exporter, and thus can use a very similar scrape_config:
- job_name: 'django'
metrics_path: /metrics # path of the fancy nginx location block
static_configs:
- targets:
- 9100
- 9101
- 9102
...
relabel_configs:
- source_labels: [__address__] # use target name as GET parameter 'port'
target_label: __param_port
- source_labels: [__param_target] # distinguish between workers via ports
target_label: port
- target_label: __address__
replacement: 192.168.178.12:80 # ingress proxy
This is fucking dangerous. You are essentially allowing anyone that can access your ingess proxy (which is probably everyone, because, ingress. duh.) to open a HTTP connection to any port on that target IP. Holy shitballs.
At the very least, enable Basic/Digest Auth for this location block, and limit the port range with if-directives!
you've been warned.