"How do you debug applications running in Kubernetes?"
The strategy to successfully debugging applications in kubernetes is to be consistent with your approach: application to public traffic or public traffic to application.
The question as it is it's very generic, however, let's see the different components we might have to check in our debugging journey:
- Container/Pod
- Service
- Ingress controller (nginx, traefik, istio-ingress...)
- Load Balancer (ELB, GLB...)
To pull logs from the different pods, kubectl logs is your friend:
# list my pods
kubectl get pods -l app=my-app
# get the logs of one of my pods
kubectl logs [POD_NAME]
# If your pod has more than one container:
kubectl logs [POD_NAME] -c my-container
Sometimes you don't deal with only 1 pod, but a few (replicas>1). You can pull logs from a set of pods also:
kubectl logs deployment/my-app -c my-container
# Or using labels
kubectl logs -l app=my-app -c my-container
You can tail, follow or get logs in the last two hours, for example:
# Display only the most recent 50 lines of output
kubectl logs --tail=50 [POD_NAME]
kubectl logs --since=2h [POD_NAME]
kubectl logs [POD_NAME] -f
If logs do not help and you think that you need to "get into" the container of the app, you can use the following command:
kubectl exec -it [POD_NAME] -c [my-container] sh
However, at this point you should have a very good idea of what is in that container and how to interact with it. If you do not, finding it's Dockerfile can help greatly.
Another option to debug applications when there's more of one replica is to map a local port to a remote container port:
kubectl port-foward [PODNAME] local-port:remote-port
By doing this, we can access the container as if the application was running in our machine.
Until now, we have looked at different ways of checking my pods/containers, but what about the other resources. For example, how do we test a kubernetes service? There are 2 ways of doing it:
- Create a pod in the cluster and test it from there.
- Create a local proxy to the service
In the first case you want to run a pod that will allow you to test requests to the service:
kubectl run my-shell --rm -i --tty --image ubuntu -- bash
This creates a pod running ubuntu, you can then install curl and curl your service.
The second case requires to create a local proxy. Conveniently, kubctl provides a way of doing so:
kubectl proxy
This command binds the local port 8001 to the api server. Then, assuming that our service is called myservice and listens to port 80 in the default namespace we can do this:
-> % curl -L localhost:8001/api/v1/proxy/namespaces/default/services/myservice:80
One thing to consider. If your pods are working as expected but your service is not, the first thing you should check are the endpoints of that service. An endpoint is nothing else than the POD IPs.
# this returns our pods and display the POD IP and the node where the pod is running
kubectl get pods -l app=myapp -owide
# describe the service will display the service endpoints.
kubectl describe svc my-service
Make sure that you have endpoints and that they match your pod IPs. If the IPs do not match, probably you have an issue with the service selector.
The last step is to test the ingress. In Kubernetes, the ingress has 2 parts: a controller and a set of rules. First thing, we need to verify that the ingress controller gets our request. Again, the easier way of doing this is treaing the ingress controller like a normal pod, where all the above applies.
One thing to bear in mind is that not all ingresses are the same. Ingress definition is still not finished and implementations are quite different. Usual "gotchas" are not using the right annotation to indicate which ingress should receive our requests and which route has to be applied/followed.