Skip to content

Instantly share code, notes, and snippets.

@xmrdsc xmrdsc/rpc_bandwidth.md
Last active Feb 12, 2019

Embed
What would you like to do?
Monero RPC node bandwidth monitoring

In this document, $YOUR_IP refers to your public ip address.

Keeping track of bandwidth for Monero RPC nodes

As Monero RPC traffic is HTTP, we can use nginx to reverse proxy and record bandwidth passing through. This implies that monerod's RPC port stays on localhost

Where normally we would use:

./monerod --max-concurrency 2 --rpc-bind-ip YOUR.IP.GOES.HERE --rpc-bind-port 18089 --restricted-rpc

We would now use:

./monerod --max-concurrency 2 --rpc-bind-port 18089 --restricted-rpc

And let it bind to localhost:

netstat -tulpn | grep 18089
tcp        0      0 127.0.0.1:18089         0.0.0.0:*               LISTEN      11242/./monerod 

So that we have some room on $YOURIP:18089 for nginx.

Setting up Nginx

First install sudo apt install nginx nginx-extras (ubuntu).

Edit /etc/nginx/nginx.conf and add this code inside the http { block:

log_format monerod_custom '<ip> [$time_iso8601] "$request" $status $bytes_sent $request_length';

Create a new server config /etc/nginx/sites-enabled/monerod.conf

server {
    listen $YOURIP:18089;
    more_clear_headers Server; 

    access_log /var/log/nginx/monerod.log monerod_custom;
    error_log /var/log/nginx/monerod-err.log;

    root /var/www/html;
    index index.html;

    location / {
        allow all;
        add_header Server Epee-based always;
        proxy_hide_header Server;

        proxy_pass http://127.0.0.1:18089;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP  $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Host $host;
    }

    location ~ /\.ht {
            deny all;
    }
}

Now restart nginx: sudo systemctl restart nginx

And observe the custom log format in /var/log/nginx/monerod.log.

rpc_bandwidth.rpc

To gather bandwidth statistics from the nginx logs, we can use the following hacky python3 script:

#/usr/bin/python3
import sys
import re
import gzip
from glob import glob
from datetime import datetime

print("Monero RPC bandwidth")
print("=================")
print()

NGINX_LOG_PATH = '/var/log/nginx/monerod.log'
DATE_FILTER = sys.argv[1]
DATE_FILTER = datetime.strptime(DATE_FILTER, '%Y-%m-%d')


def size_human(size, precision=2):
    suffixes = ['B', 'KB', 'MB', 'GB', 'TB']
    _index = 0
    while size > 1024 and _index < 4:
        _index += 1
        size = size / 1024.0
    return "%.*f%s" % (precision, size, suffixes[_index])


def read_nginx_log(fn, date_filter):
    lines = []
    _date_filter = date_filter.strftime('%Y-%m-%d')
    if fn.endswith('.gz'):
        with gzip.open(fn, 'rb') as f:
            for line in f:
                if not line:
                    continue

                line = line.decode('utf8')
                if _date_filter not in line:
                    continue

                line = line.strip()
                lines.append(line)
        return lines

    with open(fn, 'r') as f:
        for line in f.readlines():
            if not line or _date_filter not in line:
                continue
            lines.append(line.strip())
    return lines


def parse_nginx_logs(date_filter):
    global NGINX_LOG_PATH
    data = []

    for logfile in glob("%s*" % NGINX_LOG_PATH):
        print('reading %s' % logfile)
        lines = read_nginx_log(logfile, date_filter)
        for line in lines:
            date = re.findall(r'\[(.*)\]', line)
            if not date:
                continue
            date = date[0]

            try:
                date = date[:date.find('+')]
                # date = datetime.strptime(date.strip(), '%d/%b/%Y:%H:%M:%S')
                date = datetime.strptime(date.strip(), '%Y-%m-%dT%H:%M:%S')
            except Exception as ex:
                continue

            if ']' not in line:
                continue

            line = line[line.find(']')+2:]
            bytes_sent, bytes_received = map(int, line.split(' ')[-2:])
            data.append({
                'date': date,
                'bytes_sent': bytes_sent,
                'bytes_received': bytes_received
            })
    return data


data = parse_nginx_logs(DATE_FILTER)

# sort on date
data = sorted(data, key=lambda k: k['date'], reverse=False)
min_date = data[0]['date']
max_date = data[-1]['date']

bytes_received = sum([z['bytes_received'] for z in data])
bytes_sent = sum([z['bytes_sent'] for z in data])

print()
print(min_date.strftime('%Y-%m-%d %H:%M:%S') + " <-> " + max_date.strftime('%Y-%m-%d %H:%M:%S'))
print("=================")
print("Bytes received: %s" % size_human(bytes_received))
print("Bytes sent: %s" % size_human(bytes_sent))

Usage:

$ python3 rpc_bandwidth.py 2018-12-15

Where the first input is the date you want bandwidth of.

Example output:

Monero RPC bandwidth
=================

reading /var/log/nginx/monerod.log
reading /var/log/nginx/monerod.log.2.gz
reading /var/log/nginx/monerod.log.1

2018-12-15 00:00:17 <-> 2018-12-15 19:21:59
=================
Bytes received: 14.54MB
Bytes sent: 8.74GB

author: dsc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.