In version 8u131, the JVM became container aware. That means basically two things:
- The JVM will set its max heap space based on the cgroup's memory limit, taking most of that limit for the heap.
- The JVM will set its number of available processors based on the cgroup's settings for
cpu_sharesand/orcpu_quotadepending on the JVM version.
The first point (memory) is somewhat useful, it provides a more reasonable default for max heap space when the JVM is run in a container. Heap space can still be set directly via the -Xmx and/or -Xms flags, and probably should be most of the time.
The second point is somewhat less helpful, and potentially quite harmful, especially in Kubernetes. Kubernetes' pods resource CPU request and limit values control the cpu_shares and cpu_quota respectively (see https://christopher-batey.medium.com/cpu-considerations-for-java-applications-running-in-docker-and-kubernetes-7925865235b7 for more details). For now, it's enough to know that if the CPU limit is unset (as is the generally accepted recommendation: https://www.howtogeek.com/devops/should-you-set-kubernetes-cpu-limits/), the JVM (version < 19) will determine its available processors based on the CPU request. If the CPU request is set low, the JVM will set its available processors to 1. Despite being introduced in JVM version 8u131 and revised in version 11, it doesn't seem right. In fact, version 19 included a change which eliminates the use of cpu_shares (k8s request) from the picture (https://bugs.openjdk.org/browse/JDK-8281571).
How the JVM determines CPU count has changed a few times so a quick summary might be helpful:
| JVM Version | Processor Count |
|---|---|
| v < 8u131 | Host machine's actual number of processor cores |
| 8u131 ≤ v < 11 | Container's cpu_shares divided by 1024, rounded up |
| 11 ≤ v < 19 | Container's cpu_quota, falling back to cpu_shares as described above |
| v ≥ 19 | Container's cpu_quota, falling back to host's physical processor cores |
In some recent load testing, I noticed a couple of things that didn't seem right:
- The CPU utilisation metrics captured by opentelemetry capped out at 1 when 1 whole CPU core was in use, the service could consume more processor time, but the utilisation metric would remain at 1.
- The impact of a major GC on load testing was catastrophic. When the major GC occurred, the service would essentially be unavailable for extended periods.
Investigating the first issue led me to realise that the JVM thought that there was only 1 CPU. When the JVM only has a single CPU, it optimises for a computer which actually only has 1 CPU, so assumes that there is no actual parallelism, and, I believe, uses the serial GC mode. The JVM's processor count is also used to size many thread pools including the internal fork-join pool, Kotlin's default coroutine context, and the thread pools used by many libraries.
The JVM's container support, at least in terms of CPU count detection, really pretty much sucks (until version 19). If we're using a JVM prior to 19, we have to work around it. There are a couple options:
Giving the Kubernetes Pod a CPU limit would cause the JVM to detect the number of CPUs equal to that limit (rounded up). That's pretty good behaviour except that we don't want to limit our service's CPU utilisation.
In development clusters, I tend to give my services a very low CPU request value, this is essentially to allow many services to crowd onto under-provisioned development clusters. In a more production like environment we would certainly want higher CPU requests for most of our services. Setting the CPU request to something greater than 1 CPU would avoid most of the problems described above, however, Kubernetes would refuse to schedule services on under-provisioned nodes, and the original issue of wonky CPU utilisation metrics would remain since the JVM would think we have however many CPUs we request, but might in fact use many more.
A poorly documented JVM flag -XX:ActiveProcessorCount=n allows us to simply tell the JVM to pretend that there are n processors available. This is a reasonable solution, except that we want to set n to the actual number of cores on the host node, which is harder than it sounds since we potentially deploy to many different clusters which may have heterogeneous node pools. Though I'm sure we could find a way to make this work, it sounds like a bit of a pain.
What if 8u131 never happened and the JVM didn't have container support?
- We'd have to set our heap limit manually, which we're already doing anyway, and
- The JVM would use the host node's CPU count rather than inferring it from the
cpu_sharesandcpu_quota, which is just what we're looking for.
To this end, at least until we can run on a JVM >= 19 (probably 21 which is LTS and will be released around Sept. 2023). I'm recommending adding the JVM flag -XX:-UseContainerSupport which simply disables all the fancy stuff described above. The JVM will think it has access to all cores on the host node, which it kind of does; if the OS needs to limit the JVMs CPU time, it will.