slintes/centralized-logging.md

## centralized-logging.md

      
    Raw
  

              centralized-logging.md
            
          
    Centralized logging

While it's easy to get logs from single containers with the kubectl logs command, sometimes it would be nice to have
centralized logging, in order to see logs from different components ordered by time, and searchable.
There are some applications available, which in combination provide this funtionality.
Fluentd

Fluentd is a log aggregator, which gathers logs from different sources, modifies them, and
stores them in different kind of sinks. Here we use Fluentd to gather logs from the pods, enrich them with metadata
(namespace, podname, ...), and store them in Elasticsearch.
Elasticsearch

Elasticsearch stores the logs in a searchable way.
Kibana

Kibana provides an UI for searching and visualization of the Elasticsearch
data.
Installation


NOTE: Because of resource issues, this works better in a cluster with at least 2 nodes

Luckily there is an official Kubernetes addon,
which works with very small notifications on a development cluster, which was started with export KUBEVIRT_NUM_NODES=2 && make cluster up.
In order to install it, follow these steps;

Clone the kubernetes repository, if you don't have it already: git clone git@github.com:kubernetes/kubernetes.git.
Move to the addon's directory: cd kubernetes/cluster/addons/fluentd-elasticsearch.
In fluentd-es-ds.yaml, remove the nodeSelector.


At time of writing this, the specified Fluentd image in fluentd-es-ds.yaml doesn't exist yet.
Replace the v2.2.0 tag with v2.1.0.


In kibana-deployment.yaml, remove the SERVER_BASEPATH env var.


NOTE: If you're running 1 node only, you need to modify es-statefulset.yaml: add a MINIMUM_MASTER_NODES env var
with value "1", and set replicas to 1!


Install everything: kubectl apply -f ..
Wait until all pod are running: kubectl -n kube-system get pods.

Using Kibana

Access

The easiest way to access Kibana, is to setup port forwarding with kubectl:
kubectl port-forward deployment/kibana-logging 5601:5601

Now we can access the Kibana dashboard with a browser on localhost:5601.
Configuration

First you need to setup an index pattern:

Click on the corresponding button in the top right corner.
Define an index pattern by typing log* into the search field, which should select the only available index.
Click "Next step".
Select @timestamp as time filter field.
Click "Create index pattern".

Viewing and searching logs


Click on "Discover" in the left menu.
Under "Available Fields", add the fields you want to see, e.g. kubernetes.host, kubernetes.pod_name and log.
For auto-refresh of the logs, click on the correcponding button on the top and select your refresh interval.
Last but not least you might only want to see kubevirt logs: enter kubernetes.pod_name:virt* in the search field on the top.