libvirt + prometheus + grafana

README.md

libvirt + prometheus + grafana

KVM を libvirt でホストしていて、prometheus と grafana で観測したい場合。 exporter の実装は 3 種類みつかる。流儀が異なり、メトリクスが異なるので注意。

• libvirt_exporter (Go)

  • grafana json + prometheus recording rule
  • prometheus の設定ファイルで recording rule を有効化します。
  • job 名は libvirt_exporter

• https://github.com/zhangjianweibj/prometheus-libvirt-exporter

  • 上記の kumina/libvirt_exporter の libvirt ライブラリが pure-go に置き換えられた版。
  • こちらのほうがデプロイは簡単だった。

• libvirt_exporter.py

  • libvirt dashboard
  • job 名は libvirt_exporters
  • fork がたくさんある
    • https://github.com/beylistan/prometheus_libvirt_exporter/network

prometheus_libvirt_rule.yml

groups:
  - name: libvirt_exporter
    rules:
    - record: domain_instance_job_kc_customer:libvirt_domain_info_cpu_time_seconds:rate5m
      expr: rate(libvirt_domain_info_cpu_time_seconds_total[5m])
    - record: domain_instance_job_kc_customer_source_file_target_device:libvirt_domain_block_stats_read_requests:rate5m
      expr: rate(libvirt_domain_block_stats_read_requests_total[5m])
    - record: domain_instance_job_kc_customer_source_file_target_device:libvirt_domain_block_stats_write_requests:rate5m
      expr: rate(libvirt_domain_block_stats_write_requests_total[5m])
    - record: domain_instance_job_kc_customer_source_file_target_device:libvirt_domain_block_stats_read_bytes:rate5m
      expr: rate(libvirt_domain_block_stats_read_bytes_total[5m])
    - record: domain_instance_job_kc_customer_source_file_target_device:libvirt_domain_block_stats_write_bytes:rate5m
      expr: rate(libvirt_domain_block_stats_write_bytes_total[5m])
    - record: domain_instance_job_kc_customer_source_bridge_target_device:libvirt_domain_interface_stats_receive_packets:rate5m
      expr: rate(libvirt_domain_interface_stats_receive_packets_total[5m])
    - record: domain_instance_job_kc_customer_source_bridge_target_device:libvirt_domain_interface_stats_transmit_packets:rate5m
      expr: rate(libvirt_domain_interface_stats_transmit_packets_total[5m])
    - record: domain_instance_job_kc_customer_source_bridge_target_device:libvirt_domain_interface_stats_receive_bytes:rate5m
      expr: rate(libvirt_domain_interface_stats_receive_bytes_total[5m])
    - record: domain_instance_job_kc_customer_source_bridge_target_device:libvirt_domain_interface_stats_transmit_bytes:rate5m
      expr: rate(libvirt_domain_interface_stats_transmit_bytes_total[5m])
    - record: domain_instance_job_kc_customer_source_bridge_target_device:libvirt_domain_interface_stats_receive_errors:rate5m
      expr: rate(libvirt_domain_interface_stats_receive_errors_total[5m])
    - record: domain_instance_job_kc_customer_source_bridge_target_device:libvirt_domain_interface_stats_receive_drops:rate5m
      expr: rate(libvirt_domain_interface_stats_receive_drops_total[5m])
    - record: domain_instance_job_kc_customer_source_bridge_target_device:libvirt_domain_interface_stats_transmit_errors:rate5m
      expr: rate(libvirt_domain_interface_stats_transmit_errors_total[5m])
    - record: domain_instance_job_kc_customer_source_bridge_target_device:libvirt_domain_interface_stats_transmit_drops:rate5m
      expr: rate(libvirt_domain_interface_stats_transmit_drops_total[5m])
    - record: domain_job_kc_customer:libvirt_domain_info_virtual_cpus:count
      expr: sum(libvirt_domain_info_virtual_cpus) by (domain,job,kc_customer)
    - record: instance_job_kc_customer:up:count
      expr: sum(libvirt_up) by (instance,job)
    - record: job_kc_customer_kc_datacenter:up:count
      expr: sum(libvirt_up) by (job)
    - record: job_kc_customer_kc_host:up:count
      expr: sum(libvirt_up) by (job)
    - record: job_kc_customer_source_bridge:libvirt_domain_interface_stats_receive_bytes_total:count
      expr: sum(libvirt_domain_interface_stats_receive_bytes_total) by (job,source_bridge)
    - record: domain_instance_job_kc_customer:libvirt_domain_info_cpu_time_seconds:max_ratio
      expr: max(irate(libvirt_domain_info_cpu_time_seconds_total[5m])) by (domain,instance,job)