personal CKS notes

cksnotes.md

external tools

• CIS benchmark:
  • check inproper configs of a k8s node
  • kube-bench run --targets=master
• Falco: fields
  • monitor container activities
  • crictl: container id -> pod id -> pod name -> deploy name
  • write to falco_rules.local.yaml
  • memorize the following techniques

 - macro: user_name_exists
   condition: (user.name!="<NA>")
 - rule  
  condition: spawned_process and container and user_name_exists
  output: myq6 %evt.time,%user.name,%proc.name,%container.id

tail -f /var/log/syslog > /tmp/aa.log #open another for 30secs
grep myq6 /tmp/aa.log|cut -d ' ' -f9

• OPA(open policy agent): OPA gatekeeper
  • deny pods image from a bad container registry, deny pods without certain labels
  • k get crd, k get constraints
  • k edit requiredlabels xxxx
  • k edit constraintstemplate requiredlabels # count ==1 to count > 0
  • rego syntax

cluster maintain

• ETCD: etcdctl secret
  • you can using etcdctl to read a secret on a node instead of throught k8s
  • cacert -> ca.crt, cert-> server.crt, key-> server.key
  • get /register/secrets//
• Dashboard
  • change configs
  • k edit deploy/svc kubernetes-dashboard
• master worker node config: nodeport or cluster
• Replace ingress TLS: TLS secrets and edit ingress
  • as title said
  • k get ing(get the ingress's host name)
  • curl -kv hostname:31443/api (check certificate)
  • k create secret & k edit ing
  • check again
• Audit log policy: audit policy, audit log backends
  • see what/when/who happened
  • there must be a - level: None at end #don't log otherwise requests
• trivy:
  • just remember its simple syntax: trivy nginx:1.16.1-alpine

Security Context

• apparmor:
  • deny all file writes
  • install by apparmor_parser -q
  • enabled in metadata.annotations
  • schedule node by spec.nodeSelector or spec.nodeName
  • container.apparmor.security.beta.kubernetes.io/: localhost/
• gvisor: runtime class
  • with purpose
  • created by RuntimeClass resource manifest. handler is runsc. gvisor is the name
  • enabled by pod.spec.runtimeClassName: gvisor
  • check by dmesg in pod
• POD Security Admission: enforce pod security standards with namespace label
  • deny pods with bad mount
  • apply on namespace -- metadata.labels

network

• network policy: network policy, Default deny all ingress traffic
  • be familiar with NP
  • podSelector: {} means apply to all pods in namespace
  • curl inside a pod as check

pod security

• secret leak: API from a Pod, config service account for pods
  • find out what secrets have been leaked
  • possible causes: env, volume mount, service account -- a sa bind with over-powered roles
  • set ServiceAccount.automountServiceAccountToken to false
• find the pod that calls a certain syscall
  • container is actually process. it's not VM.
  • Pod id, node id, container id, process name, pid, strace
  • crictl inspect
• immutable filesystem: readOnlyRootFilesystem, emptyDir
  • spec.containers[0].securityContext.readOnlyRootFilesystem = true
  • emptyDir: {}
• Investigate Break-in via Audit Log: auditing
  • grep p.auster audit.log to get the secret's name and ns
  • change password by echo newpass | base64 and k edit secret vault-token

RBAC

• an user's permission on namespaces: RBAC
  • create a clusterrole and rolebind it on namespaces
  • since it's cross multiple namespaces -- system wide, we use clusterrole instead of role.
  • --resource= could be multiple while creating clusterrole

MISC

• Docker Image Attack Surface
  • as title
  • modify Dockerfile
  • build -t -> run -> push
• Common dockerfile security problems
  • notice all the keywords in the Dockerfile env, password, secrets`
  • notice the create and then delete pattern
• verify platform binary: sha512sum and uniq
• kill mining process
  • netstat -plnt | grep
  • cat /proc//exec # tell where the executable is
  • kill -9 & rm -f

tips

• k replace -f

Self practice

• CIS benchmark:
• Falco:
• OPA(open policy agent)
• ETCD
• Dashboard
• Replace ingress TLS:
• Audit log policy
• trivy:
• apparmor:
• gvisor
• POD Security Admission
• network policy
• serviceaccount secret leak:
• find the pod that calls a certain syscall
• immutable filesystem
• Investigate Break-in via Audit Log
• an user's permission on namespaces: RBAC
• Docker Image Attack Surface
• Common dockerfile security problems
• verify platform binary:
• killing mining process

• CIS benchmark:
  • kube-bench --targets=master
• Falco:
  • crictl ps --id, crictl pods --id
  • supported fields
• OPA(open policy agent):
  • k get constraints
  • k get requiredlabels
  • k get blacklistimges
  • k edit contrainstemplate # edit rego syntax
• ETCD, etcdctl
  • etcdctl get /registry/secrets --cacert --cert --key
• Dashboard
  • just change config
  • k edit svc/deployment Kubernetes-dashboard
• Replace ingress TLS: ingress
  • k create secret tls xxx #import
  • k edit ing
  • check by curl -kv
• Audit log policy: audit policy
  • define what to log
  • config is constructed with level, group, resource
  • use level:none to catch all
• trivy:
  • trivy <image: tag> | grep CVE

• apparmor: apparmor
  • apparmor -q <profile>
  • check its name
  • metadata. Annotations: xxx/: localhost/
• gvisor: runtime class
  • create a resource kind: RuntimeClass. metadata.name: gvisor
  • create a pod with spec.runtimeClassName: gvisor
• POD Security Admission: Enforce Pod Security Standards with Namespace Labels
  • applied to a ns: namespace.metadata.labels
  • predefined policies: xxxx
• network policy: Network Policies
  • networkpolicy.spec.podselector: what pods to apply. podSelector: {} means all
  • block all but ip by cidr: 0.0.0.0/0 except ip/32
• serviceaccount secret leak: serviceaccount
  • env,volume,serviceaccount
  • automountServiceAccountToken
• find the pod that calls a certain syscall
  • for each pod : -> -> -> -> -> strace
• immutable filesystem: readOnlyRootFilesystem, emptyDir
  • thats all
• Investigate Break-in via Audit Log: `auditing
  • grep audit.log
  • base64
  • k edit secret
• an user's permission on namespaces: RBAC
  • specified ns: Role, all namespaces: 'ClusterRole`
• Docker Image Attack Surface
  • fix the dockerfile then build -t, run(test), push
• Common dockerfile security problems
  • password, secret, copy/create and then delete
• verify platform binary:
  • sha512sum + uniq
• killing mining process
  • netstat -nlps -> port number -> -> /proc//exec -> command line

be careful

• trivy: search report by trailing number ex. CVE-2020-10878, grep 10878 q21_*.txt

• immutable filesysem:

  • emptyDir
  • securityContext.readonlyfilesystem should be inside pod.spec.containers

• audit log: the basic format of audit.log

  • one event per line and each line is in format json
  • "kind": "Event",
    "verb":"get",
    "user": { "username":"p.auster"},
    "responseObject": { "kind": "Secret"}
  • grep p.auster audit.log | grep -i Secret | grep get | wc -l

• it's tricky to make /etc/kubernetes/audit/policy.yaml take effect

move kube-apiserver.yaml ..
watch crictl ps
mv ../kube-apiserver.yaml .
watch crictl ps

• it's tricky to verify audit.log entries. It's kind of ensuring number of "non-rule" entries are zero.

# shows Secret entries
cat audit.log | grep '"resource":"secrets"' | wc -l
# confirms Secret entries are only of level Metadata
cat audit.log | grep '"resource":"secrets"' | grep -v '"level":"Metadata"' | wc -l
# shows RequestResponse level entries
cat audit.log | grep -v '"level":"RequestResponse"' | wc -l
# shows RequestResponse level entries are only for system:nodes
cat audit.log | grep '"level":"RequestResponse"' | grep -v "system:nodes" | wc -l

• how to create tls secret
  k create secret tls -h

• ingress.spec.tls there is 'hosts'. It is the original HOSTS in ingress

• crictl pods --name collector1

• the faster way to check the main executable in a pod is k exec -it pod/<pod id> -- ps

• network policy. deny all but allow

- ipBlock: 0.0.0.0
  except:
  - 192.168.x.x/32

• automountServiceAccountToken

• etcdctl options should consult cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd

• https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/

• apparmor: pod.metadata.annotations xxxx/: localhost/

• k8s dashboard: no reference on k8sdoc. please memorize authentication-mode=token

• opa: blacklistimages is under contrainttemplate so to edit it
  k get contrainttemplates blacklistimages -o yaml

• how to see OPA violates messages? `k describe blacklistimages

• CIS: you can't get kube-bench run --targets=master by kube-bench -h

• CIS: they might ask you to check something already pass

• apiserver:

  • get apiserver's mode by k get svc
  • you can always check apiserver's arguments by ps aux on master node

• pod security standard: it's applied to a namespace and just write labels of a namespace. https://k
  ubernetes.io/docs/tutorials/security/ns-level-pss/