Incorrect limits displayed for multi-container pods

Overview.md

Incorrect limits displayed for multi-container pods

Summary

Pod resource limits are incorrectly calculated in kubectl describe node when all the following conditions are met:

1. The pod has multiple containers (including init containers).
2. At least one container specifies a resource limit.
3. At least one container does not specify a resource limit for a resource type for which another container has specified a resource limit.

Related issues

• kubectl-1385: Describe Node Pod Resource Sum Incorrect/Confusing
• kubectl-1110: Kubectl describe nodes with Multi containers may display error quota [Closed due to lifecycle/rotten]

Example to illustrate the problem

Consider the following example of a manifest that creates a single pod with two containers:

apiVersion: v1
kind: Pod
metadata:
  name: unlimited-container
  namespace: limit-test
spec:
  containers:
  - name: foo
    image: busybox:1.28
    command: ['sh', '-c', 'echo The app is running! && sleep 3600']
    resources:
      limits:
        cpu: 50m
        memory: 200Mi
      requests:
        cpu: 50m
        memory: 200Mi
  - name: bar
    image: busybox:1.28
    command: ['sh', '-c', 'echo The app is running! && sleep 3600']
    resources:
      requests:
        cpu: 100m
        memory: 100Mi

This pod's containers have the following requests and limits set. Notice container bar does not have any limits:

container	cpu requests	memory requests	cpu limits	memory limits
foo	50m	200Mi	50m	200Mi
bar	100m	100Mi	none	none

Based on this configuration, what should be the expected limit for this pod?

Should it show a CPU limit of 50m or unlimited? Should it show a memory limit of 200Mi or unlimited?

Well, let's use kubectl describe node to find out. Below is an excerpt of the output on my cluster:

...

Non-terminated Pods:          (5 in total)
  Namespace                   Name                              CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                              ------------  ----------  ---------------  -------------  ---
  kube-system                 coredns-6bd5b8bf54-bl5sg          100m (2%)     0 (0%)      70Mi (1%)        170Mi (4%)     256d
  kube-system                 kube-flannel-ds-xx9dg             100m (2%)     100m (2%)   50Mi (1%)        50Mi (1%)      310d
  kube-system                 kube-proxy-49gb8                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         231d
  kube-system                 metrics-server-77d575cd7-m5l8g    100m (2%)     0 (0%)      200Mi (5%)       0 (0%)         291d
  limit-test                  unlimited-container               150m (3%)     50m (1%)    300Mi (8%)       200Mi (5%)     64s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                Requests     Limits
  --------                --------     ------
  cpu                     450m (11%)   150m (3%)
  memory                  620Mi (16%)  420Mi (11%)
  
...

It displays the CPU limit for this pod as 50m and memory limit as 200Mi.

Open questions

1. Is 50m the correct CPU limit to display for the limit-test pod?

   1. If not, should it be unlimited instead?
   2. If it should be unlimited, is this a more significant issue requiring an API change?
   3. If 50m is the correct value, should we improve documentation to clarify that setting a limit on one container in a pod determines the overall pod CPU or memory limit (assuming that is the case)?

2. Under Allocated resources, it sums up the CPU limits and shows 150m. Is this misleading?

   1. If there is any unlimited pod, should this not be unlimited?
   2. Should kubectl even attempt to calculate this? kubectl is not the source of node resource allocation, so is it even possible to calculate this correctly?
   3. Is there any API endpoint we can call to get the official node resource allocation?
   4. Should there be a warning indicating that one or more pods are unlimited?

Code references

Here is the code that kubectl describe node uses to calculate the limits:

• func PodRequestsAndLimits(pod *corev1.Pod) (reqs, limits corev1.ResourceList)

Here are the functions used to calculate the requests and limits that are actually applied:

• func PodRequests(pod *v1.Pod, opts PodResourcesOptions) v1.ResourceList
• func PodLimits(pod *v1.Pod, opts PodResourcesOptions) v1.ResourceList

PR references

I tried to fix kubectl issue 1110 back in December 2021 with a PR that attempted to reflect that the limit should be unlimited, but this PR didn't get enough traction and was perhaps too ambitious considering that a change to how resources are interpreted by the cluster is an API change.

• kubernetes/kubernetes#107057