canhnt

```
awk 'function hextodec(str,ret,n,i,k,c){
    ret = 0
    n = length(str)
    for (i = 1; i <= n; i++) {
        c = tolower(substr(str, i, 1))
        k = index("123456789abcdef", c)
        ret = ret * 16 + k
    }
    return ret

canhnt

Grok pattern for glog

E%{MONTHNUM:month}%{MONTHDAY:day} %{TIME}\s*%{NUMBER:threadid} (?<file_source>%{WORD}\.%{WORD}):(?<file_line>%{NUMBER})\] (?<message>.*)

Example:

E0423 11:41:45.456532       1 reconcile.go:30] Upsert role 'xxx' failed: Failed to upsert ES role, expect StatusCode=200, but has 400

canhnt

Access K8s cluster directly

Get kubeconfig from secret

CLUSTERID='c-1234'
kubectl get secret c-$CLUSTERID -n cattle-system -o json | jq -r .data.cluster | base64 -D | jq -r .metadata.state > kubeconfig-direct

Change endpoint to https://localhost:6443

canhnt

Keybase proof

I hereby claim:

• I am canhnt on github.
• I am cngo (https://keybase.io/cngo) on keybase.
• I have a public key ASDceW9Aeeknx07KzDaju7gKF6jnOVHYswzSrkcG59jV7go

To claim this, I am signing this object:

canhnt

Tips & tricks on k8s

Get leader

kubectl get --raw=/api/v1/namespaces/kube-system/endpoints/kube-controller-manager | jq .metadata.annotations

For kubectl via Rancher:

canhnt

Container killed by OOM

When a container exit with code 137, it means the process is killed by OOM Killer

docker container ls -a
"task: non-zero exit (137)"

Check OOM in journal log

journalctl -k | grep -i -e memory -e oom

canhnt

Check free storage in gluster nodes

for i in $(seq 1 9); do node="829k8xlstorage$i.company.org"; echo "$node checking:"; ssh $node 'df -h'; done 2>/dev/null | grep -E '829k8xlstorage|centos-root'

If any node free is lower than 18%, delete logs:

rm -f /var/log/glusterfs/gluster*
rm -f /var/log/glusterfs/bricks/*

canhnt

Get current cluster.yaml

kc get cm cluster-state -n kube-system -o jsonpath='{.data.cluster-state}'

Get user-addon settings:

kc get cm  -n kube-system rke-user-addon -o jsonpath='{.data}'

canhnt

Check numbers of templatecontent custom resource objects in the etcd DB:

etcdctl get  /registry/management.cattle.io/templatecontents --keys-only --prefix | sed '/^\s*$/d' | wc -l

A bug in Rancher prior v2.0.8 and v2.1.0 added a lot of html files in templatecontent objects, which fills up etcd db and may cause K8s master nodes unavailable.

Delete all templatecontent custom resource objects:

etcdctl del /registry/management.cattle.io/templatecontents/ --prefix

canhnt

# Ref https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/maintenance.md#space-quota
export ETCDCTL_API=3
etcdctl endpoint status --write-out="json"
etcdctl endpoint status --write-out="table"

etcdctl compact $rev
etcdctl defrag

etcdctl endpoint status --write-out="table"
etcdctl alarm list