Checking the Node Stats *.cpu outputs compared to container/system CPU info.
http://localhost:9200/_nodes/stats?filter_path=nodes.*.os.cpu,nodes.*.os.cgroup.cpu,nodes.*.process.cpu.percent
This data backs JSON for CAT Nodes which uses os.cpu per code.
Using Docker to run a local, basic Elasticsearch with Kibana setup:
Docker Compose
# docker-compose -f secure_ek.yaml -p "secure_elastic" up
version: '3'
services:
elasticsearch:
hostname: dotfile_elk
container_name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:7.16.2
environment:
- bootstrap.memory_lock=true
- discovery.seed_hosts=elasticsearch
- discovery.type=single-node
- ELASTIC_PASSWORD=changeme
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- xpack.security.enabled=true
- xpack.license.self_generated.type=trial
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.keystore.path=elastic-stack-ca.p12
- xpack.security.transport.ssl.keystore.type=PKCS12
- xpack.security.transport.ssl.truststore.path=elastic-stack-ca.p12
- xpack.security.transport.ssl.truststore.type=PKCS12
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.watcher.enabled=true
- xpack.ml.enabled=true
volumes:
- /Users/stef/Documents/local_es_certs/elastic-stack-ca.p12:/usr/share/elasticsearch/config/elastic-stack-ca.p12:ro
ports: ["9200:9200"]
networks: ["elastic"]
kibana:
hostname: dotfile_elk
container_name: kibana
image: docker.elastic.co/kibana/kibana:7.16.2
environment:
- ELASTICSEARCH_URL="http://elasticsearch:9200"
- ELASTICSEARCH_USERNAME="elastic"
- ELASTICSEARCH_PASSWORD="changeme"
- XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=something_at_least_32_characters
- XPACK_REPORTING_ENCRYPTIONKEY=something_at_least_32_characters
- XPACK_SECURITY_ENCRYPTIONKEY=something_at_least_32_characters
ports: ["5601:5601"]
depends_on: ["elasticsearch"]
networks: ["elastic"]
networks:
elastic:
driver: bridge
Node Stats outputs
{ "nodes": { "iByh0g1nTxqgSl134uiGnQ": {
"os": {
"timestamp": 1651963197732,
"cpu": {
- "percent": 89,
"load_average": {
"1m": 1.5,
"5m": 0.53,
"15m": 0.2 } },
"mem": {
"total_in_bytes": 4125478912,
"free_in_bytes": 1273294848,
"used_in_bytes": 2852184064,
"free_percent": 31,
"used_percent": 69 },
"swap": {
"total_in_bytes": 1073737728,
"free_in_bytes": 1073737728,
"used_in_bytes": 0 },
"cgroup": {
"cpuacct": {
"control_group": "/",
"usage_nanos": 58722491 },
"cpu": {
+ # percent
"control_group": "/",
"cfs_period_micros": 100000,
"cfs_quota_micros": -1,
"stat": {
"number_of_elapsed_periods": 0,
"number_of_times_throttled": 0,
"time_throttled_nanos": 0 } },
"memory": {
"control_group": "/",
"limit_in_bytes": "max",
"usage_in_bytes": "908759040"} } },
"process": {
"timestamp": 1651963197733,
"open_file_descriptors": 307,
"max_file_descriptors": 1048576,
"cpu": {
- "percent": 67,
"total_in_millis": 49030 },
"mem": {
"total_virtual_in_bytes": 4385533952 } } } } }Simultaneous, running Docker Stats for Elasticsearch shows
$ docker stats --no-stream
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
156767455e75 kibana 4.10% 589.3MiB / 3.842GiB 14.98% 756kB / 579kB 309MB / 0B 12
+ 61c3e8f2feeb elasticsearch 4.72% 956.9MiB / 3.842GiB 24.32% 591kB / 759kB 176MB / 89.7MB 85And Docker exec'ing into the Elasticsearch container and running top shows
Tasks: 5 total, 1 running, 4 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.0 us, 0.2 sy, 0.0 ni, 98.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 3934.4 total, 1098.1 free, 1606.4 used, 1229.8 buff/cache
MiB Swap: 1024.0 total, 1024.0 free, 0.0 used. 1745.1 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
+ 8 elastic+ 20 0 4326920 948348 69252 S 4.0 23.5 0:52.89 java
1 root 20 0 2500 592 528 S 0.0 0.0 0:00.02 tini
362 elastic+ 20 0 108280 7692 7112 S 0.0 0.2 0:00.00 controller
407 root 20 0 4248 3492 2932 S 0.0 0.1 0:00.03 bash
454 root 20 0 6112 3276 2768 R 0.0 0.1 0:00.00 top- Elasticsearch reporting:
os.cpu.percent:89,cgroup.cpu.percent:UNK,process.cpu.percent:67 - System reporting:
docker.stats.cpu:4.72,docker.exec.top.process.cpu:4.0
The Elasticsearch reported metrics are no where near the system reported metrics for CPU.
- Download MacOS of Elasticsearch v7.8.0
- Unzip, navigate to folder, and startup
$ pwd /Users/stef/code/local/elasticsearch-7.8.0 $ bin/elasticsearch
{ "nodes": { "CozBlf2rTBCFTAtoewYj6Q": {
"os": { "cpu": {
+ "percent": 32,
"load_average": {
"1m": 3.13671875 }}},
"process": { "cpu": {
+ "percent": 13 }}}}}
{ "nodes": { "CozBlf2rTBCFTAtoewYj6Q": {
"os": { "cpu": {
+ "percent": 26,
"load_average": {
"1m": 3.0263671875 }}},
"process": { "cpu": {
+ "percent": 0 }}}}}Related process appears to run as [bash,java] in terminal top view. Filtered and simplified, at the same time the system reported:
Processes: 566 total, 3 running, 563 sleeping, 3338 threads
2022/05/07 17:10:47
Load Avg: 3.02, 2.81, 3.03
+ CPU usage: 14.53% user, 18.79% sys, 66.66% idle
MemRegions: 145857 total, 10G resident, 0B private, 1939M shared.
PhysMem: 29G used (3293M wired), 3204M unused.
VM: 19T vsize, 0B framework vsize, 0(0) swapins, 0(0) swapouts.
Networks: packets: 20691907/15G in, 14251825/6650M out.
Disks: 6779145/70G read, 5166605/95G written.
PID COMMAND %CPU TIME PPID STATE
17044 java 0.0 00:45.16 1 sleeping
27795 java 0.1 01:52.26 1 sleeping
28376 java 0.0 01:42.59 1 sleeping
28804 java 0.0 09:15.49 1 sleeping
+ 29799 bash 0.0 00:00.06 29798 sleeping
+ 31012 java 0.6 00:29.68 29799 sleeping- Elasticsearch reporting:
os.cpu.percent:32,process.cpu.percent:13 - System reporting:
all.user.cpu:14.5,all.system.cpu:18.8,sum(bash||java).cpu:0.7
The Elasticsearch reported metrics are closer to but still not near their expected-to-correlate system CPU metrics.
Potentially related to elasticsearch#70200.
- Should system CPU be captured or measured a different way to align closer to what Elasticsearch is pulling?
- Is there an alternative existing way to pull system-relected Elasticsearch CPU via a different Elasticsearch API?