Observability in Juju

observability-in-juju.md

juju add-model cos

juju deploy cos-lite --channel edge --trust

Adding metric support

Enable the metrics endpoint in Zinc

                        "environment": {
                            "ZINC_DATA_PATH": "/go/bin/data",
                            "ZINC_FIRST_ADMIN_USER": "admin",
                            "ZINC_FIRST_ADMIN_PASSWORD": self._stored.initial_admin_password,
                            "ZINC_PROMETHEUS_ENABLE": True,
                        },

Deploy the zinc charm

juju deploy \
	./charms/with-metrics-endpoint.charm zinc \
	--resource zinc-image=jnsgruk/zinc:latest

Fetch the scrape library

charmcraft fetch-lib charms.prometheus_k8s.v0.prometheus_scrape

Instantiate MetricsEndpointProvider

from charms.prometheus_k8s.v0.prometheus_scrape import MetricsEndpointProvider

        self._scraping = MetricsEndpointProvider(self, 
            relation_name="metrics-endpoint",
            jobs=[{
                "static_configs": [{"targets": ["*:4080"]
                }]
            }])

Declare the provided relation in metadata.yaml

provides:
  metrics-endpoint:
    interface: prometheus_scrape

Refresh charm

juju refresh zinc --path ./charms/with-scrape-interface.charm

Relate to prometheus

juju relate prometheus zinc

Create src/prometheus_alert_rules/unit_unavailable.rule

alert: ZincUnitIsUnavailable
expr: up < 1
for: 10s
labels:
  severity: critical
annotations:
  summary: Zinc unit {{ $labels.juju_model }}/{{ $labels.juju_unit}} is unavailable
  description: >
    The Zinc unit {{ $labels.juju_model}}/{{ $labels.juju_unit }} has been 
    unavailable for more than 10 seconds.
    LABELS = {{ $labels }}

Refresh charm

juju refresh zinc --path ./charms/with-metrics-alert-rule.charm

Adding logging support

Change the pebble command to tee to a file

    _log_path = "/zinc.logs"

                        "command": "/bin/sh -c \"/go/bin/zinc | tee {}\"".format(self._log_path),

Refresh charm

juju refresh zinc --path ./charms/with-logfile.charm

Tail the log file

juju ssh --container zinc zinc/0 tail /zinc.logs

Fetch the loki push api lib

charmcraft fetch-lib charms.loki_k8s.v0.loki_push_api

Implement the LogProxyConsumer

from charms.loki_k8s.v0.loki_push_api import LogProxyConsumer

        self._logging = LogProxyConsumer(self,
            relation_name="logging",
            log_files=[self._log_path]
        )

requires:
  logging:
    interface: loki_push_api

Refresh charm

juju refresh zinc --path ./charms/with-logproxy.charm

juju relate loki zinc

Adding the Grafana Dashboard

juju run-action grafana/0 get-admin-password --wait

Add panels for metrics

Add Unit Status

sum(up{juju_model="$juju_model", juju_model_uuid="$juju_model_uuid", juju_application="$juju_application"}) by (juju_unit)

Add percentiles

histogram_quantile(
  0.01, 
  sum(
    rate(
      gin_request_duration_seconds_bucket{
        juju_model="$juju_model", 
        juju_model_uuid="$juju_model_uuid",
        juju_application="$juju_application",
        url="/ui/\"
      }[2m]
    )
  ) by (le)
)

Add request rate

sum(
  rate(
    gin_requests_total{
      juju_model="$juju_model", 
      juju_model_uuid="$juju_model_uuid", 
      juju_application="$juju_application"
    }[2m]
  )
)

charmcraft fetch-lib charms.grafana_k8s.v0.grafana_dashboard

from charms.grafana_k8s.v0.grafana_dashboard import GrafanaDashboardProvider

        self._grafana_dashboards = GrafanaDashboardProvider(
            self, 
            relation_name="grafana-dashboard"
        )

  grafana-dashboard:
    interface: grafana_dashboard