grafana-dashboard-install.md

grafana-dashboard-install.md

NODE

sudo ceph mgr module enable prometheus
sudo ceph config set mgr mgr/prometheus/server_addr 0.0.0.0
sudo ceph config set mgr mgr/prometheus/server_port 9283

wget https://github.com/prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.linux-amd64.tar.gz
tar xvf node_exporter-1.2.2.linux-amd64.tar.gz
sudo cp node_exporter-1.2.2.linux-amd64/node_exporter /usr/local/bin/

sudo apt-get install gosu
sudo useradd -M prometheus
sudo usermod -L prometheus

sudo chown prometheus:prometheus /usr/local/bin/node_exporter
sudo vi /etc/init.d/node_exporter
sudo chmod +x /etc/init.d/node_exporter
sudo update-rc.d node_exporter defaults
sudo service node_exporter status
sudo vi /etc/default/node_exporter
sudo service node_exporter start
ps auxww | grep node_exporter
cat /var/log/prometheus/node_exporter.log

sudo ceph dashboard set-grafana-api-ssl-verify False
sudo ceph dashboard set-grafana-api-url http://node1:3000?orgId=1

GRAFANA

sudo apt-get install -y adduser libfontconfig1
wget https://dl.grafana.com/enterprise/release/grafana-enterprise_8.1.2_arm64.deb
sudo dpkg -i grafana-enterprise_8.1.2_arm64.deb
sudo vi /etc/grafana/grafana.ini
sudo /bin/systemctl enable grafana-server
sudo /bin/systemctl start grafana-server

sudo grafana-cli --homepath "/usr/share/grafana" admin reset-admin-password admin

PROMETHEUS

sudo apt-get install gosu
sudo useradd -M prometheus
sudo usermod -L prometheus
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.29.2/prometheus-2.29.2.linux-arm64.tar.gz
tar xvf prometheus-2.29.2.linux-arm64.tar.gz

sudo cp prometheus-2.29.2.linux-arm64/prometheus /usr/local/bin/
sudo cp prometheus-2.29.2.linux-arm64/promtool /usr/local/bin/
sudo cp -r prometheus-2.29.2.linux-arm64/consoles /etc/prometheus
sudo cp -r prometheus-2.29.2.linux-arm64/console_libraries /etc/prometheus
sudo vi /etc/prometheus/prometheus.yml
sudo chown -R prometheus:prometheus /etc/prometheus/

sudo vi /etc/init.d/prometheus
sudo chmod +x /etc/init.d/prometheus
sudo update-rc.d prometheus defaults
sudo service prometheus status
sudo vi /etc/default/prometheus
sudo service prometheus start
ps auxww | grep prometheus
cat /var/log/prometheus/prometheus.log

sudo grafana-cli plugins install vonage-status-panel
sudo grafana-cli plugins install grafana-piechart-panel

sudo mkdir -p /etc/grafana/dashboards/ceph-dashboard/
cd /etc/grafana/dashboards/ceph-dashboard/

sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/ceph-cluster.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/cephfs-overview.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/host-details.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/hosts-overview.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/osd-device-details.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/osds-overview.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/pool-detail.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/pool-overview.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/radosgw-overview.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/rbd-overview.json

sudo mkdir -p /etc/grafana/provisioning/dashboards
sudo vi ceph-dashboard.yml

sudo /bin/systemctl restart grafana-server
sudo tail -f /var/log/grafana/grafana.log

Ok, I got it working with the provisioning way you showed, the .yaml files are very picky...., but the dedicated dashboards were not picked up by the ceph dashboard....strange...

.

Ok, solved...you can remove all my posts:
I had to set at the end:

sudo ceph dashboard set-grafana-api-url https://grafana.theater.net:3000

without ?orgId=1
Now it picked up only the frame itself!

Thanks for your input!

Regards,

Christian

Hi @peters77

If you don't mind I would like to keep them. All resources I can host for people to find solutions to their problems the better.

Your correspondence have a lot of good gotchas that could help someone else that are struggling.

Best regards
Daniel

Daniel,

no problem. Maybe it's really useful?

I have again one question: I installed grafana and prometheus on a dedicated VM.
In
/etc/prometheus/prometheus.yml

I set

global:
scrape_interval: 5s

scrape_configs:

• job_name: 'ceph'
  static_configs:
  • targets: ['ceph1.theater.net:9283', 'ceph2.theater.net:9283', 'ceph3.theater.net:9283']
• job_name: 'node_exporter'
  static_configs:
  • targets: ['ceph1.theater.net:9100', 'ceph2.theater.net:9100', 'ceph3.theater.net:9100']

Is this ok!? Should I add the prometheus target:

• job_name: 'prometheus'
  static_configs:
  • targets: ['ceph1.theater.net:9090', 'ceph2.theater.net:9090', 'ceph3.theater.net:9090']

Did I understand it right that the MGRs provide the info in the same way as the node_exporter (installed on all ceph nodes) and the prometheus instance on the extrenal VM scraped the info from all three Ceph nodes!?
Do I have to install prometheus on all three nodes, too!? I don't think so, it's not clear in your video as you provide prometheus und grafana download links in your document as arm64 binarys so I think you did an install of this packages on an extrenal machine, too?

I just wonder if I select the "Host Details" garafan ceph-dashboard page and select e.g. my node1, I get no raw capacity value but if I switch to node2 (where my MGR runs at the moment), I get a value for Raw Capacity!? So maybe something seems to lack on my configuration? Or is this ok?
On OSD Overview I have no Latencies (Read/Writes), too. So something seems not perfekt (datasource?)

...ok....at http://ip:9283 all metrics are provided by the ceph cluster (where ip is the ip of the active MGR). So the active manager gathers all inormation of the cluster nodes...? Maybe I should be pleased with the result, 99% is working. :-)

Regards,

Christian

Hi @peters77

Well, the way to understand it is that each node has information to give, some come from managers others are collected by the node exporters. Prometheus is calling each node to fetch the information and put it into a data stream. You could have multiple Prometheus servers if required for the data flow but often only one server is required. Grafana is used to create pretty graphs.

So I install node exporter on all hosts to get OS information and then I connect gather information from other services that provide it.

Best regards
Daniel

The links for the boards have changed ...

wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/ceph-cluster.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/cephfs-overview.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/host-details.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/hosts-overview.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/osd-device-details.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/osds-overview.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/pool-detail.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/pool-overview.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/radosgw-detail.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/radosgw-overview.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/radosgw-overview.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/radosgw-sync-overview.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/rbd-details.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/rbd-overview.json