To understand the proper memory limits for Prometheus, we first need to observe the memory usage behavior of the deployment. After the increase in memory, we can clearly see the reported usage doesn't decrease but oscillate around new level. This is an unexpected behavior, and why the RSS memory is not released after the increase needs to be investigated. Keep in mind that this pattern was initially observed in only one region.
Current configuration:
- Version 2.18.1
- GoVersion go1.14.2
- Prometheus Helm Chart: v11.5.0
From [4]
Metric name | Metric Type | Description | Units | Option Parameter |
---|---|---|---|---|
container_memory_cache | Gauge | Total page cache memory | bytes | memory |
container_memory_rss | Gauge | Size of RSS | bytes | memory |
container_memory_swap | Gauge | Container swap usage | bytes | memory |
container_memory_usage_bytes | Gauge | Current memory usage, including all memory regardless of when it was accessed | bytes | memory |
container_memory_working_set_bytes | Gauge | Current working set | bytes | memory |
From the graphics it can be seen that with an ever increasing container_memory_usage_bytes, it is not easy to determine a memory limit for this deployment. One potential problem that can arise from ill defined memory limits is the deployment will be OOMKilled unnecessarily.
As a result it is important to understand how the aforementioned container metrics are involved in OOMKill decision. container_memory_working_set_bytes metric is monitored for OOMKill decisions. [2] As a result this metric is used to determine memory limit and related alerting.
From the graph it can be clearly seen that working set bytes is much smaller than the usage bytes metric. Usage byte metrics and RSS memory patterns look very similar, in fact total usage seems to be not going down due to remaining RSS. [3]Explains that memory usage reporting changed after golang 1.12 and the behavior seen in the memory report is not due to a memory leak but instead is a consequence of the change made.
[3] suggests the following
- If you depend on container_memory_usage_bytes switch to container_memory_working_set_bytes metric for closest possible experience to actual usage. It’s not perfect though.
- Use go_memstats_alloc_bytes and others (e.g go_memstats_.*_inuse_bytes) to see actual allocations. Useful when profiling and optimizing your application memory. This helps to filter out the memory that is “cached” and it’s the most accurate from the application perspective.