Created
October 17, 2019 10:08
-
-
Save praveenkumar/b5ceab217b57e2fb04606dcb604e0017 to your computer and use it in GitHub Desktop.
Put the details about consumed resources by Openshift Operators.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| This gist thread will be used to find out what all operators are present as part of CRC and what can be done to make low | |
| memory/cpu footprint to the host. |
<crc-vm># crictl stats
CONTAINER CPU % MEM DISK INODES
0898c32069ae1 0.26 130.6MB 0B 0
0c93bf936206d 0.00 42.36MB 0B 0
0f74f973a83ad 0.07 18.28MB 0B 0
15028212f9ee5 2.72 15.36MB 0B 0
15a57925512bc 0.05 20.59MB 0B 0
1ce8918192783 0.05 39.96MB 0B 0
254900f6f1afa 0.50 62.82MB 0B 0
2676b9fca9a48 0.03 59.15MB 0B 0
4102b6fcdc931 0.02 44.46MB 0B 0
4438a68355261 11.10 821.3MB 0B 0
4700700ce52b7 0.80 76.67MB 0B 0
493bdd36d7841 0.00 2.605MB 0B 0
4c0164fd25504 0.00 109.4MB 0B 0
4db73f11add29 0.00 13.1MB 0B 0
4eeb0c4685dd8 0.13 38.35MB 0B 0
55295b2358d22 0.00 28.73MB 0B 0
5a2a6e0525cb9 0.05 17.44MB 0B 0
5e6cf7cf3ebb0 0.29 57.23MB 0B 0
65868a40211f1 8.17 199.5MB 0B 0
6d7004c7a174d 0.07 28.12MB 0B 0
71eae5d5cf72c 0.14 38.65MB 0B 0
72244eaa4a8b2 0.06 46.49MB 0B 0
76301f163745c 0.28 20.21MB 0B 0
800b58fbff97c 0.07 50.03MB 0B 0
81477c95522c3 0.81 192.2MB 0B 0
840e80854c7b1 0.00 38.03MB 0B 0
850b69c9bd393 0.04 41.49MB 0B 0
872957ca7cf3c 0.45 46.39MB 0B 0
8b121f19493bf 0.12 85.2MB 0B 0
957cf074c609f 0.18 89.65MB 0B 0
9a1c8c43c397e 0.11 83.45MB 0B 0
9ddfc6905169e 0.04 32.72MB 0B 0
9f386349c0473 0.17 101.6MB 0B 0
ab7312a2bb95f 0.00 13.19MB 0B 0
ada266c7411ad 2.58 215.6MB 0B 0
b184ad457b820 0.12 24.26MB 0B 0
b60c8b205c0e9 0.78 70.92MB 0B 0
b6c1572dd05f8 0.09 24.05MB 0B 0
c4ac3c806e311 0.12 50.37MB 0B 0
c550e0e388d39 0.09 21.61MB 0B 0
c694ae78e6c55 0.73 90.72MB 0B 0
cc787fb92da6d 0.06 170.7MB 0B 0
cd1c35f7b505c 0.09 25.19MB 0B 0
cf7df591b55c4 1.40 92.32MB 0B 0
cfd0e38de4aca 0.03 48.46MB 0B 0
cfd878e66737a 0.18 17.43MB 0B 0
d55d3064007e9 0.05 64.32MB 0B 0
d8671fba41609 0.15 82.71MB 0B 0
da05e0a14133b 0.00 6.935MB 0B 0
db0f22f17a9eb 0.00 25.28MB 0B 0
db9faa0c39dfd 0.43 81.69MB 0B 0
e81a58f4fadfe 0.00 47.99MB 0B 0
f0224dfb26503 0.09 76.56MB 0B 0
f0b8c77839e9c 0.09 74.69MB 0B 0
f1c90f8e0d49a 0.15 66.73MB 0B 0
f2725f4aa4261 0.00 22.25MB 0B 0
f7f44bf6d5c95 0.57 93.09MB 0B 0
f86ff084ab846 0.14 89.7MB 0B 0
f8dac09bcbbb3 0.00 54.07MB 0B 0
fb0d4e3d36e63 0.07 86.27MB 0B 0
fc0017d724a7d 0.30 104.4MB 0B 0
fd4dd6163e754 0.67 111.4MB 0B 0
For minikube 1.17.0 k8s version
$ sudo crictl stats
CONTAINER CPU % MEM DISK INODES
06d0e4252080d 0.20 46.19MB 0B 0
1c1964ee0bfaa 0.05 65.5MB 0B 0
26f1c832b0b97 1.63 57.72MB 0B 0
2b3ee349f435d 0.00 39.67MB 0B 0
52b8b416afc39 0.18 7.524MB 0B 0
65e73259f88cd 1.03 28.21MB 0B 0
9cbed8996876b 0.00 43.65MB 0B 0
c2326ddf927e9 2.39 271.3MB 0B 0
e9e92bed617c2 0.47 38.87MB 0B 0
$ sudo crictl images
IMAGE TAG IMAGE ID SIZE
docker.io/kubernetesui/dashboard v2.0.0-beta8 eb51a35975256 91.2MB
docker.io/kubernetesui/metrics-scraper v1.0.2 3b08661dc379d 40.1MB
gcr.io/k8s-minikube/storage-provisioner v1.8.1 4689081edb103 80.8MB
k8s.gcr.io/coredns 1.6.5 70f311871ae12 41.7MB
k8s.gcr.io/etcd 3.4.3-0 303ce5db0e90d 290MB
k8s.gcr.io/kube-addon-manager v9.0.2 bd12a212f9dcb 84.7MB
k8s.gcr.io/kube-apiserver v1.17.0 0cae8d5cc64c7 173MB
k8s.gcr.io/kube-controller-manager v1.17.0 5eb3b74868724 162MB
k8s.gcr.io/kube-proxy v1.17.0 7d54289267dc5 118MB
k8s.gcr.io/kube-scheduler v1.17.0 78c190f736b11 96MB
k8s.gcr.io/pause 3.1 da86e6ba6ca19 747kB
Disable all the operators and remove memory from the resources from all the deployments.
oc scale --replicas=0 deployment --all -n openshift-dns-operator
oc scale --replicas=0 deployment --all -n openshift-authentication-operator
oc scale --replicas=0 deployment --all -n openshift-cluster-node-tuning-operator
oc scale --replicas=0 deployment --all -n openshift-console-operator
oc scale --replicas=0 deployment cluster-image-registry-operator -n openshift-image-registry
oc scale --replicas=0 deployment --all -n openshift-ingress-operator
oc scale --replicas=0 deployment marketplace-operator -n openshift-marketplace
oc scale --replicas=0 deployment --all -n openshift-controller-manager-operator
oc scale --replicas=0 deployment --all -n openshift-apiserver-operator
oc scale --replicas=0 deployment --all -n openshift-kube-apiserver-operator
oc scale --replicas=0 deployment --all -n openshift-kube-controller-manager-operator
oc scale --replicas=0 deployment --all -n openshift-kube-scheduler-operator
oc scale --replicas=0 deployment --all -n openshift-machine-api
oc scale --replicas=0 deployment --all -n openshift-machine-config-operator
oc scale --replicas=0 deployment --all -n openshift-monitoring
oc scale --replicas=0 deployment --all -n openshift-service-ca-operator
oc scale --replicas=0 deployment --all -n openshift-service-catalog-apiserver-operator
oc patch deployment cloud-credential-operator -n openshift-cloud-credential-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment machine-approver -n openshift-cluster-machine-approver --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment cluster-node-tuning-operator -n openshift-cluster-node-tuning-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment cluster-storage-operator -n openshift-cluster-storage-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment cluster-version-operator -n openshift-cluster-version --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment console-operator -n openshift-console-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment console -n openshift-console --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment downloads -n openshift-console --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment openshift-controller-manager-operator -n openshift-controller-manager-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment image-registry -n openshift-image-registry --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment router-default -n openshift-ingress --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment insights-operator -n openshift-insights --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment kube-apiserver-operator -n openshift-kube-apiserver-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment kube-controller-manager-operator -n openshift-kube-controller-manager-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment openshift-kube-scheduler-operator -n openshift-kube-scheduler-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment cluster-autoscaler-operator -n openshift-machine-api --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment machine-api-controllers -n openshift-machine-api --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment machine-api-operator -n openshift-machine-api --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment etcd-quorum-guard -n openshift-machine-config-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment machine-config-controller -n openshift-machine-config-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment machine-config-operator -n openshift-machine-config-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment certified-operators -n openshift-marketplace --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment community-operators -n openshift-marketplace --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment marketplace-operator -n openshift-marketplace --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment redhat-operators -n openshift-marketplace --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment cluster-monitoring-operator -n openshift-monitoring --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment grafana -n openshift-monitoring --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment kube-state-metrics -n openshift-monitoring --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment openshift-state-metrics -n openshift-monitoring --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment prometheus-adapter -n openshift-monitoring --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment prometheus-operator -n openshift-monitoring --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment network-operator -n openshift-network-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment catalog-operator -n openshift-operator-lifecycle-manager --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment olm-operator -n openshift-operator-lifecycle-manager --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment packageserver -n openshift-operator-lifecycle-manager --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment service-ca-operator -n openshift-service-ca-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment apiservice-cabundle-injector -n openshift-service-ca --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment configmap-cabundle-injector -n openshift-service-ca --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment service-serving-cert-signer -n openshift-service-ca --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment openshift-service-catalog-controller-manager-operator -n openshift-service-catalog-controller-manager-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
Check all the pods in all the namespaces.
$ oc get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
openshift-apiserver apiserver-6spqp 1/1 Running 0 7d21h
openshift-authentication oauth-openshift-5f47cd5fbd-7xrfg 1/1 Running 0 12h
openshift-authentication oauth-openshift-5f47cd5fbd-cz6jm 1/1 Running 0 12h
openshift-cluster-machine-approver machine-approver-dc7c6f989-gr2fc 1/1 Running 0 11h
openshift-cluster-node-tuning-operator tuned-v8fzh 1/1 Running 0 17h
openshift-cluster-samples-operator cluster-samples-operator-5cff448f4c-8c7l8 1/1 Running 0 7d22h
openshift-cluster-storage-operator cluster-storage-operator-68749dbccb-v7wvt 1/1 Running 0 11h
openshift-console console-59b777fbdd-5lmg8 1/1 Running 0 11h
openshift-controller-manager controller-manager-5c48l 1/1 Running 2 17h
openshift-dns dns-default-s4vmh 2/2 Running 0 7d23h
openshift-etcd etcd-member-crc-rk2fc-master-0 2/2 Running 0 7d23h
openshift-image-registry image-registry-6d8db9f49-kl6z8 1/1 Running 0 11h
openshift-image-registry node-ca-wb4hl 1/1 Running 0 7d22h
openshift-ingress router-default-799b6c477d-hnjc4 1/1 Running 0 11h
openshift-kube-apiserver kube-apiserver-crc-rk2fc-master-0 3/3 Running 11 17h
openshift-kube-controller-manager kube-controller-manager-crc-rk2fc-master-0 2/2 Running 11 7d22h
openshift-kube-scheduler openshift-kube-scheduler-crc-rk2fc-master-0 1/1 Running 11 7d22h
openshift-machine-config-operator machine-config-daemon-z7hlr 1/1 Running 0 7d23h
openshift-machine-config-operator machine-config-server-gq9ww 1/1 Running 0 7d23h
openshift-marketplace certified-operators-6c45d48df5-2qn75 1/1 Running 0 11h
openshift-marketplace community-operators-7ff94d5784-h9xwb 1/1 Running 0 11h
openshift-marketplace redhat-operators-5c559c7bc-8tvvc 1/1 Running 0 11h
openshift-monitoring node-exporter-cnt7b 2/2 Running 0 7d22h
openshift-multus multus-admission-controller-68qmm 1/1 Running 0 7d23h
openshift-multus multus-qjs27 1/1 Running 3 7d23h
openshift-network-operator network-operator-64f87884fb-mgqbf 1/1 Running 0 11h
openshift-operator-lifecycle-manager catalog-operator-84f749ffc-ljttf 1/1 Running 0 11h
openshift-operator-lifecycle-manager olm-operator-7fc87c9c64-fm4c7 1/1 Running 0 11h
openshift-operator-lifecycle-manager packageserver-848764c995-674n7 1/1 Running 0 4m14s
openshift-operator-lifecycle-manager packageserver-848764c995-zs57p 1/1 Running 0 4m
openshift-sdn ovs-42ltn 1/1 Running 0 7d23h
openshift-sdn sdn-controller-wp95c 1/1 Running 1 7d23h
openshift-sdn sdn-jf826 1/1 Running 12 7d23h
openshift-service-ca apiservice-cabundle-injector-6dd5968b98-fgmmw 1/1 Running 0 4m3s
openshift-service-ca configmap-cabundle-injector-5bc9fb4464-bm7ch 1/1 Running 0 4m5s
openshift-service-ca service-serving-cert-signer-55f844f768-d5wgz 1/1 Running 0 4m5s
openshift-service-catalog-controller-manager-operator openshift-service-catalog-controller-manager-operator-68765fwqd 1/1 Running 0 11h
Check crictl stats now, which is much better.
# crictl stats
CONTAINER CPU % MEM DISK INODES
198d7391b4187 0.08 23.19MB 0B 0
203e7292e268e 1.06 151.1MB 0B 0
20b0c496a153b 0.14 49.89MB 0B 0
230126ec6bd44 0.05 17.52MB 0B 0
26019a315efd5 0.04 26.34MB 0B 0
2afc10a0a9ffc 0.13 35.59MB 0B 0
2bb066aa6cb75 0.05 24.15MB 0B 0
2cf3b7299ea4b 0.19 70.73MB 0B 0
318b795029536 0.14 65.71MB 0B 0
3467759e17ea5 0.00 44.22MB 0B 0
39149ade182fe 0.00 28.85MB 0B 0
5e0d102aa9e42 1.09 77.8MB 0B 0
5eb215537672b 0.00 32.52MB 0B 0
610dc0e2d6cbc 3.86 19.36MB 0B 0
6aaf04f44a1a5 0.15 34.22MB 0B 0
7d157acf3f9d2 5.96 417.7MB 0B 0
7d73fd9db859b 0.07 77.99MB 0B 0
7d917841c69b7 0.00 8.802MB 0B 0
915e30ac44a4f 0.02 58.49MB 0B 0
9338c3612c43d 0.08 21.99MB 0B 0
982ac5fbcfe80 0.68 38.44MB 0B 0
9c0b8138ab691 0.00 22.35MB 0B 0
9cd7a685ef01b 0.00 8.684MB 0B 0
9ed0eb3cf5c86 0.00 3.461MB 0B 0
a1bb5331bcd55 2.07 197.4MB 0B 0
a89d53e321f46 1.03 76.6MB 0B 0
a8df1963598e1 1.76 291.4MB 0B 0
ac25188f73d73 0.01 34.94MB 0B 0
af73819bd4f02 0.72 123.2MB 0B 0
b1d82219d1052 0.25 55.63MB 0B 0
b2c2b2e67729e 0.46 79.63MB 0B 0
b75b2c3cbc87c 0.00 8.978MB 0B 0
bb1b6db3b5545 0.12 92.76MB 0B 0
cb45728dfe7b4 0.14 11.26MB 0B 0
cbe6b8d825427 0.27 53.76MB 0B 0
d0e583cf88b7e 1.21 69.3MB 0B 0
d478d4e39655d 0.07 41.65MB 0B 0
d62367c891017 0.00 46.65MB 0B 0
d66a577660254 0.02 233.9MB 0B 0
dd087ff88beb3 5.75 1.917GB 0B 0
df13e81d945f3 0.07 31.79MB 0B 0
e45b2c55a9391 0.03 12.25MB 0B 0
e6b3eee5fd7ba 0.00 31.13MB 0B 0
f375fe4ffbe94 0.20 24.46MB 0B 0
Now try to deploy the knative serving and eventing operator on this cluster.
$ cat sub.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: knative-eventing-operator
namespace: openshift-operators
spec:
channel: alpha
name: knative-eventing-operator
source: community-operators
sourceNamespace: openshift-marketplace
$ cat sub-serving.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: knative-serving-operator
namespace: openshift-operators
spec:
channel: alpha
name: knative-serving-operator
source: community-operators
sourceNamespace: openshift-marketplace
$ oc apply -f sub.yaml
$ oc apply -f sub-serving.yaml
Wait for some time and check the pods (which all are running).
$ oc get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
istio-operator istio-operator-86f7d5ffd8-ssmwq 1/1 Running 0 108m
istio-system cluster-local-gateway-58758f588-wzkm2 1/1 Running 0 106m
istio-system istio-ingressgateway-54968c8854-fx97t 1/1 Running 0 106m
istio-system istio-pilot-577ff6784c-jrjjb 1/1 Running 0 107m
knative-eventing eventing-controller-5d5f979874-96kpb 1/1 Running 0 114m
knative-eventing eventing-webhook-75bcb6d4bb-j5tml 1/1 Running 0 114m
knative-eventing imc-controller-69c54bfdc8-pzxlr 1/1 Running 0 113m
knative-eventing imc-dispatcher-94bc9f6b6-c2kfl 1/1 Running 0 113m
knative-eventing sources-controller-5c6df78ffb-g9qhj 1/1 Running 0 114m
knative-serving activator-6c94b9ff47-9xwlp 1/1 Running 0 106m
knative-serving autoscaler-64c549bcf4-lhfrx 1/1 Running 0 106m
knative-serving controller-564487997f-mh2l2 1/1 Running 0 106m
knative-serving knative-openshift-ingress-c7fd864df-7k6xw 1/1 Running 0 106m
knative-serving networking-certmanager-8466b656dc-ktf9d 1/1 Running 0 106m
knative-serving networking-istio-59f5c789d-9j69g 1/1 Running 0 106m
knative-serving webhook-77ff9bd584-wwz4j 1/1 Running 0 106m
[...]
Try to deploy a sample service and check.
$ cat service.yaml
apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
name: helloworld-go
namespace: default
spec:
template:
spec:
containers:
- image: gcr.io/knative-samples/helloworld-go
env:
- name: TARGET
value: "Go Sample v1"
$ oc get ksvc helloworld-go
NAME URL LATESTCREATED LATESTREADY READY REASON
helloworld-go http://helloworld-go.default.apps-crc.testing helloworld-go-lk6g9 helloworld-go-lk6g9 True
$ curl http://helloworld-go.default.apps-crc.testing
Hello Go Sample v1!
$ oc describe node
[...]
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 3135m (89%) 4700m (134%)
memory 6435496192 (87%) 5140890112 (70%)
ephemeral-storage 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NodeHasSufficientMemory 49m (x37 over 20h) kubelet, crc-rk2fc-master-0 Node crc-rk2fc-master-0 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 49m (x37 over 20h) kubelet, crc-rk2fc-master-0 Node crc-rk2fc-master-0 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 49m (x37 over 20h) kubelet, crc-rk2fc-master-0 Node crc-rk2fc-master-0 status is now: NodeHasSufficientPID
Normal NodeReady 49m (x5 over 20h) kubelet, crc-rk2fc-master-0 Node crc-rk2fc-master-0 status is now: NodeReady
Check the load on the CRC VM.
sh-4.2# chroot /host
sh-4.4# free -h
total used free shared buff/cache available
Mem: 7.4Gi 4.3Gi 885Mi 248Mi 2.3Gi 3.4Gi
Swap: 0B 0B 0B
sh-4.4# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda3 30G 19G 12G 62% /
tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup
devtmpfs 3.7G 0 3.7G 0% /dev
tmpfs 3.8G 84K 3.8G 1% /dev/shm
tmpfs 3.8G 8.6M 3.8G 1% /run
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
On 4.2.0 fresh cluster created using https://github.com/code-ready/snc
$ oc describe node Name: crc-847lc-master-0 Roles: master,worker Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=crc-847lc-master-0 kubernetes.io/os=linux node-role.kubernetes.io/master= node-role.kubernetes.io/worker= node.openshift.io/os_id=rhcos Annotations: machine.openshift.io/machine: openshift-machine-api/crc-847lc-master-0 machineconfiguration.openshift.io/currentConfig: rendered-master-83ff2caf193ce2570b39209d3cabf152 machineconfiguration.openshift.io/desiredConfig: rendered-master-83ff2caf193ce2570b39209d3cabf152 machineconfiguration.openshift.io/reason: unexpected on-disk state validating against rendered-master-83ff2caf193ce2570b39209d3cabf152 machineconfiguration.openshift.io/ssh: accessed machineconfiguration.openshift.io/state: Degraded volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Wed, 16 Oct 2019 06:13:58 -0400 Taints: <none> Unschedulable: false Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Thu, 17 Oct 2019 01:02:31 -0400 Wed, 16 Oct 2019 06:13:57 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Thu, 17 Oct 2019 01:02:31 -0400 Wed, 16 Oct 2019 06:13:57 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Thu, 17 Oct 2019 01:02:31 -0400 Wed, 16 Oct 2019 06:13:57 -0400 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Thu, 17 Oct 2019 01:02:31 -0400 Wed, 16 Oct 2019 06:14:58 -0400 KubeletReady kubelet is posting ready status Addresses: InternalIP: 192.168.126.11 Hostname: crc-847lc-master-0 Capacity: cpu: 4 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 10179028Ki pods: 250 Allocatable: cpu: 3500m hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 9564628Ki pods: 250 System Info: Machine ID: 6688a9e0def54db08ff1f2d3bde07f1e System UUID: 6688a9e0-def5-4db0-8ff1-f2d3bde07f1e Boot ID: 08d4da7d-a9b8-490a-a3ef-9b7df7202746 Kernel Version: 4.18.0-80.11.2.el8_0.x86_64 OS Image: Red Hat Enterprise Linux CoreOS 42.80.20191010.0 (Ootpa) Operating System: linux Architecture: amd64 Container Runtime Version: cri-o://1.14.11-0.23.dev.rhaos4.2.gitc41de67.el8 Kubelet Version: v1.14.6+c07e432da Kube-Proxy Version: v1.14.6+c07e432da Non-terminated Pods: (53 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- openshift-apiserver-operator openshift-apiserver-operator-9bcf9dc8-8nrdp 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-apiserver apiserver-s4t27 150m (4%) 0 (0%) 200Mi (2%) 0 (0%) 17h openshift-authentication-operator authentication-operator-59bd6dffb8-wcch4 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-authentication oauth-openshift-8666b94dbf-6w8wp 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-authentication oauth-openshift-8666b94dbf-p96q4 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-cloud-credential-operator cloud-credential-operator-6b8db4f844-vg6px 10m (0%) 0 (0%) 150Mi (1%) 0 (0%) 18h openshift-cluster-machine-approver machine-approver-557b48d579-5hmb4 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-cluster-node-tuning-operator cluster-node-tuning-operator-bbc964d5f-jbblq 10m (0%) 0 (0%) 20Mi (0%) 0 (0%) 18h openshift-cluster-node-tuning-operator tuned-stwrj 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-cluster-samples-operator cluster-samples-operator-66cdd79f5f-8xzch 10m (0%) 0 (0%) 0 (0%) 0 (0%) 18h openshift-cluster-storage-operator cluster-storage-operator-7547f476d7-dsfl2 10m (0%) 0 (0%) 20Mi (0%) 0 (0%) 18h openshift-console-operator console-operator-566bf94bbd-s8kvx 10m (0%) 0 (0%) 100Mi (1%) 0 (0%) 18h openshift-console console-f87f4d688-96pg9 10m (0%) 0 (0%) 100Mi (1%) 0 (0%) 18h openshift-controller-manager-operator openshift-controller-manager-operator-6d9dccf54-dv79b 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-controller-manager controller-manager-fdrz6 100m (2%) 0 (0%) 100Mi (1%) 0 (0%) 18h openshift-dns-operator dns-operator-75c9d649dd-5j5fj 10m (0%) 0 (0%) 0 (0%) 0 (0%) 18h openshift-dns dns-default-dt2fr 110m (3%) 0 (0%) 70Mi (0%) 512Mi (5%) 18h openshift-etcd etcd-member-crc-847lc-master-0 300m (8%) 0 (0%) 600Mi (6%) 0 (0%) 18h openshift-image-registry cluster-image-registry-operator-74465655b4-c9g94 20m (0%) 0 (0%) 0 (0%) 0 (0%) 18h openshift-image-registry image-registry-6984bcdf68-vpfjt 100m (2%) 0 (0%) 256Mi (2%) 0 (0%) 18h openshift-image-registry node-ca-pg2ph 10m (0%) 0 (0%) 10Mi (0%) 0 (0%) 18h openshift-ingress-operator ingress-operator-6c8cd6cfd8-dlx5b 10m (0%) 0 (0%) 0 (0%) 0 (0%) 18h openshift-ingress router-default-755446d554-22gf6 100m (2%) 0 (0%) 256Mi (2%) 0 (0%) 18h openshift-kube-apiserver-operator kube-apiserver-operator-664f449f96-pcght 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-kube-apiserver kube-apiserver-crc-847lc-master-0 170m (4%) 0 (0%) 1124Mi (12%) 0 (0%) 17h openshift-kube-controller-manager-operator kube-controller-manager-operator-559cb6679b-8w9r7 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-kube-controller-manager kube-controller-manager-crc-847lc-master-0 110m (3%) 0 (0%) 250Mi (2%) 0 (0%) 18h openshift-kube-scheduler-operator openshift-kube-scheduler-operator-566bbd9cf4-fsgxd 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-kube-scheduler openshift-kube-scheduler-crc-847lc-master-0 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-machine-api machine-api-controllers-98bf659c-lzvnm 20m (0%) 0 (0%) 40Mi (0%) 0 (0%) 17h openshift-machine-config-operator machine-config-daemon-965x8 20m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-machine-config-operator machine-config-server-lsm4m 20m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-marketplace certified-operators-76775bfd77-cvsnk 10m (0%) 0 (0%) 100Mi (1%) 0 (0%) 3h38m openshift-marketplace community-operators-7b4658cdb6-44js2 10m (0%) 0 (0%) 100Mi (1%) 0 (0%) 18h openshift-marketplace marketplace-operator-5c846b89cb-z6gfj 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-marketplace redhat-operators-869c657f84-79nlf 10m (0%) 0 (0%) 100Mi (1%) 0 (0%) 7h38m openshift-monitoring node-exporter-7cc8n 10m (0%) 0 (0%) 20Mi (0%) 0 (0%) 18h openshift-multus multus-admission-controller-stf6j 10m (0%) 0 (0%) 0 (0%) 0 (0%) 18h openshift-multus multus-fvh57 10m (0%) 0 (0%) 150Mi (1%) 0 (0%) 18h openshift-network-operator network-operator-75d9ff9f94-z25fq 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-operator-lifecycle-manager catalog-operator-66bbfcb8c6-zf28s 10m (0%) 0 (0%) 80Mi (0%) 0 (0%) 18h openshift-operator-lifecycle-manager olm-operator-56bf56ffcd-v8gpk 10m (0%) 0 (0%) 160Mi (1%) 0 (0%) 18h openshift-operator-lifecycle-manager packageserver-6fd4c798f9-k54tb 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-operator-lifecycle-manager packageserver-6fd4c798f9-r76ff 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-sdn ovs-gf52w 200m (5%) 0 (0%) 400Mi (4%) 0 (0%) 18h openshift-sdn sdn-controller-mq648 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-sdn sdn-mhbb9 100m (2%) 0 (0%) 200Mi (2%) 0 (0%) 18h openshift-service-ca-operator service-ca-operator-7fb8bb4b59-fztp5 10m (0%) 0 (0%) 80Mi (0%) 0 (0%) 18h openshift-service-ca apiservice-cabundle-injector-7c87bcf775-dvznk 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-service-ca configmap-cabundle-injector-f9f4b5fd7-552gr 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-service-ca service-serving-cert-signer-79685fd777-pdrzn 10m (0%) 0 (0%) 120Mi (1%) 0 (0%) 18h openshift-service-catalog-apiserver-operator openshift-service-catalog-apiserver-operator-69d8b48889-dfqtg 0 (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h openshift-service-catalog-controller-manager-operator openshift-service-catalog-controller-manager-operator-6bcdqgfn6 10m (0%) 0 (0%) 50Mi (0%) 0 (0%) 18h Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 1880m (53%) 0 (0%) memory 5906Mi (63%) 512Mi (5%) ephemeral-storage 0 (0%) 0 (0%) Events: <none>