Skip to content

Instantly share code, notes, and snippets.

@praveenkumar
Created October 17, 2019 10:08
Show Gist options
  • Save praveenkumar/b5ceab217b57e2fb04606dcb604e0017 to your computer and use it in GitHub Desktop.
Save praveenkumar/b5ceab217b57e2fb04606dcb604e0017 to your computer and use it in GitHub Desktop.
Put the details about consumed resources by Openshift Operators.
This gist thread will be used to find out what all operators are present as part of CRC and what can be done to make low
memory/cpu footprint to the host.
@praveenkumar
Copy link
Author

Check all the pods in all the namespaces.

$ oc get pods -A
NAMESPACE                                               NAME                                                              READY   STATUS    RESTARTS   AGE
openshift-apiserver                                     apiserver-6spqp                                                   1/1     Running   0          7d21h
openshift-authentication                                oauth-openshift-5f47cd5fbd-7xrfg                                  1/1     Running   0          12h
openshift-authentication                                oauth-openshift-5f47cd5fbd-cz6jm                                  1/1     Running   0          12h
openshift-cluster-machine-approver                      machine-approver-dc7c6f989-gr2fc                                  1/1     Running   0          11h
openshift-cluster-node-tuning-operator                  tuned-v8fzh                                                       1/1     Running   0          17h
openshift-cluster-samples-operator                      cluster-samples-operator-5cff448f4c-8c7l8                         1/1     Running   0          7d22h
openshift-cluster-storage-operator                      cluster-storage-operator-68749dbccb-v7wvt                         1/1     Running   0          11h
openshift-console                                       console-59b777fbdd-5lmg8                                          1/1     Running   0          11h
openshift-controller-manager                            controller-manager-5c48l                                          1/1     Running   2          17h
openshift-dns                                           dns-default-s4vmh                                                 2/2     Running   0          7d23h
openshift-etcd                                          etcd-member-crc-rk2fc-master-0                                    2/2     Running   0          7d23h
openshift-image-registry                                image-registry-6d8db9f49-kl6z8                                    1/1     Running   0          11h
openshift-image-registry                                node-ca-wb4hl                                                     1/1     Running   0          7d22h
openshift-ingress                                       router-default-799b6c477d-hnjc4                                   1/1     Running   0          11h
openshift-kube-apiserver                                kube-apiserver-crc-rk2fc-master-0                                 3/3     Running   11         17h
openshift-kube-controller-manager                       kube-controller-manager-crc-rk2fc-master-0                        2/2     Running   11         7d22h
openshift-kube-scheduler                                openshift-kube-scheduler-crc-rk2fc-master-0                       1/1     Running   11         7d22h
openshift-machine-config-operator                       machine-config-daemon-z7hlr                                       1/1     Running   0          7d23h
openshift-machine-config-operator                       machine-config-server-gq9ww                                       1/1     Running   0          7d23h
openshift-marketplace                                   certified-operators-6c45d48df5-2qn75                              1/1     Running   0          11h
openshift-marketplace                                   community-operators-7ff94d5784-h9xwb                              1/1     Running   0          11h
openshift-marketplace                                   redhat-operators-5c559c7bc-8tvvc                                  1/1     Running   0          11h
openshift-monitoring                                    node-exporter-cnt7b                                               2/2     Running   0          7d22h
openshift-multus                                        multus-admission-controller-68qmm                                 1/1     Running   0          7d23h
openshift-multus                                        multus-qjs27                                                      1/1     Running   3          7d23h
openshift-network-operator                              network-operator-64f87884fb-mgqbf                                 1/1     Running   0          11h
openshift-operator-lifecycle-manager                    catalog-operator-84f749ffc-ljttf                                  1/1     Running   0          11h
openshift-operator-lifecycle-manager                    olm-operator-7fc87c9c64-fm4c7                                     1/1     Running   0          11h
openshift-operator-lifecycle-manager                    packageserver-848764c995-674n7                                    1/1     Running   0          4m14s
openshift-operator-lifecycle-manager                    packageserver-848764c995-zs57p                                    1/1     Running   0          4m
openshift-sdn                                           ovs-42ltn                                                         1/1     Running   0          7d23h
openshift-sdn                                           sdn-controller-wp95c                                              1/1     Running   1          7d23h
openshift-sdn                                           sdn-jf826                                                         1/1     Running   12         7d23h
openshift-service-ca                                    apiservice-cabundle-injector-6dd5968b98-fgmmw                     1/1     Running   0          4m3s
openshift-service-ca                                    configmap-cabundle-injector-5bc9fb4464-bm7ch                      1/1     Running   0          4m5s
openshift-service-ca                                    service-serving-cert-signer-55f844f768-d5wgz                      1/1     Running   0          4m5s
openshift-service-catalog-controller-manager-operator   openshift-service-catalog-controller-manager-operator-68765fwqd   1/1     Running   0          11h

@praveenkumar
Copy link
Author

Check crictl stats now, which is much better.

# crictl stats
CONTAINER           CPU %               MEM                 DISK                INODES
198d7391b4187       0.08                23.19MB             0B                  0
203e7292e268e       1.06                151.1MB             0B                  0
20b0c496a153b       0.14                49.89MB             0B                  0
230126ec6bd44       0.05                17.52MB             0B                  0
26019a315efd5       0.04                26.34MB             0B                  0
2afc10a0a9ffc       0.13                35.59MB             0B                  0
2bb066aa6cb75       0.05                24.15MB             0B                  0
2cf3b7299ea4b       0.19                70.73MB             0B                  0
318b795029536       0.14                65.71MB             0B                  0
3467759e17ea5       0.00                44.22MB             0B                  0
39149ade182fe       0.00                28.85MB             0B                  0
5e0d102aa9e42       1.09                77.8MB              0B                  0
5eb215537672b       0.00                32.52MB             0B                  0
610dc0e2d6cbc       3.86                19.36MB             0B                  0
6aaf04f44a1a5       0.15                34.22MB             0B                  0
7d157acf3f9d2       5.96                417.7MB             0B                  0
7d73fd9db859b       0.07                77.99MB             0B                  0
7d917841c69b7       0.00                8.802MB             0B                  0
915e30ac44a4f       0.02                58.49MB             0B                  0
9338c3612c43d       0.08                21.99MB             0B                  0
982ac5fbcfe80       0.68                38.44MB             0B                  0
9c0b8138ab691       0.00                22.35MB             0B                  0
9cd7a685ef01b       0.00                8.684MB             0B                  0
9ed0eb3cf5c86       0.00                3.461MB             0B                  0
a1bb5331bcd55       2.07                197.4MB             0B                  0
a89d53e321f46       1.03                76.6MB              0B                  0
a8df1963598e1       1.76                291.4MB             0B                  0
ac25188f73d73       0.01                34.94MB             0B                  0
af73819bd4f02       0.72                123.2MB             0B                  0
b1d82219d1052       0.25                55.63MB             0B                  0
b2c2b2e67729e       0.46                79.63MB             0B                  0
b75b2c3cbc87c       0.00                8.978MB             0B                  0
bb1b6db3b5545       0.12                92.76MB             0B                  0
cb45728dfe7b4       0.14                11.26MB             0B                  0
cbe6b8d825427       0.27                53.76MB             0B                  0
d0e583cf88b7e       1.21                69.3MB              0B                  0
d478d4e39655d       0.07                41.65MB             0B                  0
d62367c891017       0.00                46.65MB             0B                  0
d66a577660254       0.02                233.9MB             0B                  0
dd087ff88beb3       5.75                1.917GB             0B                  0
df13e81d945f3       0.07                31.79MB             0B                  0
e45b2c55a9391       0.03                12.25MB             0B                  0
e6b3eee5fd7ba       0.00                31.13MB             0B                  0
f375fe4ffbe94       0.20                24.46MB             0B                  0

@praveenkumar
Copy link
Author

praveenkumar commented Dec 20, 2019

Now try to deploy the knative serving and eventing operator on this cluster.

$ cat sub.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: knative-eventing-operator
  namespace: openshift-operators 
spec:
  channel: alpha
  name: knative-eventing-operator
  source: community-operators
  sourceNamespace: openshift-marketplace 

$ cat sub-serving.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: knative-serving-operator
  namespace: openshift-operators 
spec:
  channel: alpha
  name: knative-serving-operator
  source: community-operators
  sourceNamespace: openshift-marketplace 

$ oc apply -f sub.yaml
$ oc apply -f sub-serving.yaml

Wait for some time and check the pods (which all are running).

$ oc get pods -A
NAMESPACE                                               NAME                                                              READY   STATUS    RESTARTS   AGE
istio-operator                                          istio-operator-86f7d5ffd8-ssmwq                                   1/1     Running   0          108m
istio-system                                            cluster-local-gateway-58758f588-wzkm2                             1/1     Running   0          106m
istio-system                                            istio-ingressgateway-54968c8854-fx97t                             1/1     Running   0          106m
istio-system                                            istio-pilot-577ff6784c-jrjjb                                      1/1     Running   0          107m
knative-eventing                                        eventing-controller-5d5f979874-96kpb                              1/1     Running   0          114m
knative-eventing                                        eventing-webhook-75bcb6d4bb-j5tml                                 1/1     Running   0          114m
knative-eventing                                        imc-controller-69c54bfdc8-pzxlr                                   1/1     Running   0          113m
knative-eventing                                        imc-dispatcher-94bc9f6b6-c2kfl                                    1/1     Running   0          113m
knative-eventing                                        sources-controller-5c6df78ffb-g9qhj                               1/1     Running   0          114m
knative-serving                                         activator-6c94b9ff47-9xwlp                                        1/1     Running   0          106m
knative-serving                                         autoscaler-64c549bcf4-lhfrx                                       1/1     Running   0          106m
knative-serving                                         controller-564487997f-mh2l2                                       1/1     Running   0          106m
knative-serving                                         knative-openshift-ingress-c7fd864df-7k6xw                         1/1     Running   0          106m
knative-serving                                         networking-certmanager-8466b656dc-ktf9d                           1/1     Running   0          106m
knative-serving                                         networking-istio-59f5c789d-9j69g                                  1/1     Running   0          106m
knative-serving                                         webhook-77ff9bd584-wwz4j                                          1/1     Running   0          106m
[...]

Try to deploy a sample service and check.

$ cat service.yaml 
apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
  name: helloworld-go
  namespace: default
spec:
  template:
    spec:
      containers:
        - image: gcr.io/knative-samples/helloworld-go
          env:
            - name: TARGET
              value: "Go Sample v1"

$ oc get ksvc helloworld-go
NAME            URL                                             LATESTCREATED         LATESTREADY           READY   REASON
helloworld-go   http://helloworld-go.default.apps-crc.testing   helloworld-go-lk6g9   helloworld-go-lk6g9   True    

$ curl http://helloworld-go.default.apps-crc.testing
Hello Go Sample v1!

$ oc describe node
[...]
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests          Limits
  --------           --------          ------
  cpu                3135m (89%)       4700m (134%)
  memory             6435496192 (87%)  5140890112 (70%)
  ephemeral-storage  0 (0%)            0 (0%)
Events:
  Type    Reason                   Age                 From                         Message
  ----    ------                   ----                ----                         -------
  Normal  NodeHasSufficientMemory  49m (x37 over 20h)  kubelet, crc-rk2fc-master-0  Node crc-rk2fc-master-0 status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    49m (x37 over 20h)  kubelet, crc-rk2fc-master-0  Node crc-rk2fc-master-0 status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID     49m (x37 over 20h)  kubelet, crc-rk2fc-master-0  Node crc-rk2fc-master-0 status is now: NodeHasSufficientPID
  Normal  NodeReady                49m (x5 over 20h)   kubelet, crc-rk2fc-master-0  Node crc-rk2fc-master-0 status is now: NodeReady

Check the load on the CRC VM.

sh-4.2# chroot /host 
sh-4.4# free -h
              total        used        free      shared  buff/cache   available
Mem:          7.4Gi       4.3Gi       885Mi       248Mi       2.3Gi       3.4Gi
Swap:            0B          0B          0B

sh-4.4# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda3        30G   19G   12G  62% /
tmpfs           3.8G     0  3.8G   0% /sys/fs/cgroup
devtmpfs        3.7G     0  3.7G   0% /dev
tmpfs           3.8G   84K  3.8G   1% /dev/shm
tmpfs           3.8G  8.6M  3.8G   1% /run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment