Skip to content

Instantly share code, notes, and snippets.

@kalaspuffar
Last active February 28, 2022 14:25
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save kalaspuffar/b0faa4089b7f3920dbafbcedcccbaf16 to your computer and use it in GitHub Desktop.
Save kalaspuffar/b0faa4089b7f3920dbafbcedcccbaf16 to your computer and use it in GitHub Desktop.
grafana-dashboard-install.md

NODE

sudo ceph mgr module enable prometheus
sudo ceph config set mgr mgr/prometheus/server_addr 0.0.0.0
sudo ceph config set mgr mgr/prometheus/server_port 9283
wget https://github.com/prometheus/node_exporter/releases/download/v1.2.2/node_exporter-1.2.2.linux-amd64.tar.gz
tar xvf node_exporter-1.2.2.linux-amd64.tar.gz
sudo cp node_exporter-1.2.2.linux-amd64/node_exporter /usr/local/bin/
sudo apt-get install gosu
sudo useradd -M prometheus
sudo usermod -L prometheus
sudo chown prometheus:prometheus /usr/local/bin/node_exporter
sudo vi /etc/init.d/node_exporter
sudo chmod +x /etc/init.d/node_exporter
sudo update-rc.d node_exporter defaults
sudo service node_exporter status
sudo vi /etc/default/node_exporter
sudo service node_exporter start
ps auxww | grep node_exporter
cat /var/log/prometheus/node_exporter.log
sudo ceph dashboard set-grafana-api-ssl-verify False
sudo ceph dashboard set-grafana-api-url http://node1:3000?orgId=1

GRAFANA

sudo apt-get install -y adduser libfontconfig1
wget https://dl.grafana.com/enterprise/release/grafana-enterprise_8.1.2_arm64.deb
sudo dpkg -i grafana-enterprise_8.1.2_arm64.deb
sudo vi /etc/grafana/grafana.ini
sudo /bin/systemctl enable grafana-server
sudo /bin/systemctl start grafana-server
sudo grafana-cli --homepath "/usr/share/grafana" admin reset-admin-password admin

PROMETHEUS

sudo apt-get install gosu
sudo useradd -M prometheus
sudo usermod -L prometheus
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.29.2/prometheus-2.29.2.linux-arm64.tar.gz
tar xvf prometheus-2.29.2.linux-arm64.tar.gz
sudo cp prometheus-2.29.2.linux-arm64/prometheus /usr/local/bin/
sudo cp prometheus-2.29.2.linux-arm64/promtool /usr/local/bin/
sudo cp -r prometheus-2.29.2.linux-arm64/consoles /etc/prometheus
sudo cp -r prometheus-2.29.2.linux-arm64/console_libraries /etc/prometheus
sudo vi /etc/prometheus/prometheus.yml
sudo chown -R prometheus:prometheus /etc/prometheus/
sudo vi /etc/init.d/prometheus
sudo chmod +x /etc/init.d/prometheus
sudo update-rc.d prometheus defaults
sudo service prometheus status
sudo vi /etc/default/prometheus
sudo service prometheus start
ps auxww | grep prometheus
cat /var/log/prometheus/prometheus.log
sudo grafana-cli plugins install vonage-status-panel
sudo grafana-cli plugins install grafana-piechart-panel
sudo mkdir -p /etc/grafana/dashboards/ceph-dashboard/
cd /etc/grafana/dashboards/ceph-dashboard/
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/ceph-cluster.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/cephfs-overview.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/host-details.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/hosts-overview.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/osd-device-details.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/osds-overview.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/pool-detail.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/pool-overview.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/radosgw-overview.json
sudo wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/rbd-overview.json
sudo mkdir -p /etc/grafana/provisioning/dashboards
sudo vi ceph-dashboard.yml
sudo /bin/systemctl restart grafana-server
sudo tail -f /var/log/grafana/grafana.log
@peters77
Copy link

Hi Daniel,

sorry to bother you, hope it's ok. I tried to email you but it doesn't work, so I try this way.

I watch lots of your Ceph videos (great!), did some VM Ceph installs (manual deploying as with cephadmin I run in trouble quite quick).
My cluster is running now and I tried to setup the Grafana/Prometheus monitoring of the Ceph cluster.
With automatic deploying of the Dashboard1 I wasn't successfull, but I added my Prometheus database in Grafana by hand (grafana server and Prometheus running on an extra VM). The grafana dashboards are now running well, see screenshot. Thanks for you notes you placed as a link at your youtube video. I would like if you could add the init.d scripts for node_exporter and prometheus, too. Was a little bit tricky to find the right ones.
Now I stuck to view the grafana page as an iframe in the Ceph dasboard.
I access my grafana main page with http://grafana.theater.net:3000.
That works.
As example I choose the ceph-cluster dashboard which displays fine with URL http://grafana.theater.net:3000/d/2fVmdZAnk/ceph-cluster?orgId=1&refresh=30s, see secreenshot.
I set
sudo ceph dashboard set-grafana-api-ssl-verify False
sudo ceph dashboard set-grafana-api-url http://grafana.theater.net:3000?orgId=1

but I always get a blank iframe, see screenshot.
Do you have any idea howto debug which website the dashboard is calling or where I got some info what URL the dashboard will call to open the embedded grafana page!?

Calling http://grafana.theater.net:3000?orgId=1 as set in ceph dashboard led to my grafana start page. Is this right!? Any ideas or hint for me!? Thanks in advance!

Regards,

Christian
blank_iframe
ceph-cluster grafana
url_set_in_ceph_dashboard

@peters77
Copy link

Ok....it's developing! 👍 Fixed paswordless grafana access and added a self-signed certificate to grafana, now the iframe shows up, but doesn't pick the dedicated dashboard. Instead shows the starting main page!? Any idea!?
embedded_grafana_shows_startpage

@kalaspuffar
Copy link
Author

Hi @peters77

That was strange, as I said in the video https://youtu.be/c8R64LF3JjU

The only thing I had to do is set up all the different Grafana views with the files downloaded from GitHub and then fix the CORS issue that made it not load in the browser and then I was up and running.

Best regards
Daniel

@peters77
Copy link

Ok, I got it working with the provisioning way you showed, the .yaml files are very picky...., but the dedicated dashboards were not picked up by the ceph dashboard....strange...
Dashboard_with-Provisioning
.

@peters77
Copy link

Ok, solved...you can remove all my posts:
I had to set at the end:

sudo ceph dashboard set-grafana-api-url https://grafana.theater.net:3000

without ?orgId=1
Now it picked up only the frame itself!

Thanks for your input!

Regards,

Christian

@kalaspuffar
Copy link
Author

Hi @peters77

If you don't mind I would like to keep them. All resources I can host for people to find solutions to their problems the better.

Your correspondence have a lot of good gotchas that could help someone else that are struggling.

Best regards
Daniel

@peters77
Copy link

peters77 commented Dec 29, 2021

Daniel,

no problem. Maybe it's really useful?

I have again one question: I installed grafana and prometheus on a dedicated VM.
In
/etc/prometheus/prometheus.yml

I set

global:
scrape_interval: 5s

scrape_configs:

  • job_name: 'ceph'
    static_configs:
    • targets: ['ceph1.theater.net:9283', 'ceph2.theater.net:9283', 'ceph3.theater.net:9283']
  • job_name: 'node_exporter'
    static_configs:
    • targets: ['ceph1.theater.net:9100', 'ceph2.theater.net:9100', 'ceph3.theater.net:9100']

Is this ok!? Should I add the prometheus target:

  • job_name: 'prometheus'
    static_configs:
    • targets: ['ceph1.theater.net:9090', 'ceph2.theater.net:9090', 'ceph3.theater.net:9090']

Did I understand it right that the MGRs provide the info in the same way as the node_exporter (installed on all ceph nodes) and the prometheus instance on the extrenal VM scraped the info from all three Ceph nodes!?
Do I have to install prometheus on all three nodes, too!? I don't think so, it's not clear in your video as you provide prometheus und grafana download links in your document as arm64 binarys so I think you did an install of this packages on an extrenal machine, too?

I just wonder if I select the "Host Details" garafan ceph-dashboard page and select e.g. my node1, I get no raw capacity value but if I switch to node2 (where my MGR runs at the moment), I get a value for Raw Capacity!? So maybe something seems to lack on my configuration? Or is this ok?
On OSD Overview I have no Latencies (Read/Writes), too. So something seems not perfekt (datasource?)

...ok....at http://ip:9283 all metrics are provided by the ceph cluster (where ip is the ip of the active MGR). So the active manager gathers all inormation of the cluster nodes...? Maybe I should be pleased with the result, 99% is working. :-)

Regards,

Christian

@kalaspuffar
Copy link
Author

Hi @peters77

Well, the way to understand it is that each node has information to give, some come from managers others are collected by the node exporters. Prometheus is calling each node to fetch the information and put it into a data stream. You could have multiple Prometheus servers if required for the data flow but often only one server is required. Grafana is used to create pretty graphs.

So I install node exporter on all hosts to get OS information and then I connect gather information from other services that provide it.

Best regards
Daniel

@JBlond
Copy link

JBlond commented Feb 28, 2022

The links for the boards have changed ...

wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/ceph-cluster.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/cephfs-overview.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/host-details.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/hosts-overview.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/osd-device-details.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/osds-overview.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/pool-detail.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/pool-overview.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/radosgw-detail.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/radosgw-overview.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/radosgw-overview.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/radosgw-sync-overview.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/rbd-details.json
wget https://raw.githubusercontent.com/ceph/ceph/2b4f3561d2fb49c23dc489475600266f3223232e/monitoring/grafana/dashboards/rbd-overview.json

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment