Logging trouble shooting steps

Logging-debug-steps.md

Logging trouble shooting steps

Dependence

Check docker log driver type

• Command: docker info
• Confirm logging driver is: json-file

Check docker root

• Command: docker info
• If your docker root is not the default /var/lib/docker, just update the docker root in Cluster Options page

Check whether kubelet create symbol link success

• Command: sudo tail sudo tail /var/log/containers/xxxx.log
• If no files under /var/log/containers, check kubelet log

Logging function

Reachable check

• click Test button under cluster/project level logging page, to check if the logging target is reachable

Check fluentd bind mount path

fluentd should bind mount /var/log/containers, /var/log/pods, docker root

• Simple way to check whether the log mount into the fluentd container, exec into fluentd pod, tail /var/log/containers/xxxx.log

Check fluentd log

• Checking fluentd log whether there are any exception or warning logs
• Could increase the fluentd pod's debug log level by editing the pod's command through UI or yaml, add -v or -vv to the command, -v means enable debug log, -vv means enable trance log.

Reduce influencing factors

If one of the logging target not work, for example, one of the projects logging not work, try to disable others, only enable this project to debug

For collect customise path logs inside container

Need to check what kind of cluster you deployed, we only support rke/gke, because the volume driver need to deploy to specific directory then the kubelet could detect it.