BSidesDC Training

bsidesdctraining.md

Pick a cluster - https://docs.google.com/spreadsheets/d/1iLt6dAw3JlfP9EK5sp8z1CtNYnEvEEfvGOBtELpI6mA/edit?usp=sharing

Lab 1

Login and initialize

1. https://ou.apps.IP.nip.io/
2. Login with the username / password - k8s-lab/$tart123
3. Logout
4. SSH to your server, user name root and this ssh key:

-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABFwAAAAdzc2gtcn
NhAAAAAwEAAQAAAQEAwO4emjnsCc56OTYkNuYeJJFmBRSy33UgHEDgwIhTLQU7+C3npYJm
kaJP1VbeOzJGQH03La9KsDrZLDx5XsAIAgN91A8OCts2hKAO4ZKg+Y8XGKvN9Pqsz3uqXg
UFiGDEv4TDxRtk+sFtTnac+o6rUe8I5PmKZlCpLykYEmcV4evUii4/A1xDs5U82sruyrgi
am9EhVVJt7Nm7cP4ZGfjyzE9lBF7H4Suw4HKEM9whS8QQmGItxnhfmwjNeJ07Jw9GtZ5HV
Dbfvo8bVpoAqrVKcTH0iPnoBYqQ2UEqL0d2AjJhXVWkouSP7RE8BVrSA3mt2UB1O2a1Nz0
E3zw+EZRawAAA9jkwZMM5MGTDAAAAAdzc2gtcnNhAAABAQDA7h6aOewJzno5NiQ25h4kkW
YFFLLfdSAcQODAiFMtBTv4LeelgmaRok/VVt47MkZAfTctr0qwOtksPHlewAgCA33UDw4K
2zaEoA7hkqD5jxcYq830+qzPe6peBQWIYMS/hMPFG2T6wW1Odpz6jqtR7wjk+YpmUKkvKR
gSZxXh69SKLj8DXEOzlTzayu7KuCJqb0SFVUm3s2btw/hkZ+PLMT2UEXsfhK7DgcoQz3CF
LxBCYYi3GeF+bCM14nTsnD0a1nkdUNt++jxtWmgCqtUpxMfSI+egFipDZQSovR3YCMmFdV
aSi5I/tETwFWtIDea3ZQHU7ZrU3PQTfPD4RlFrAAAAAwEAAQAAAQAXMcl+Ey6cczVggXDT
JNFE5jIUtEtY8BTfMLGUlA8j33g5OxOZY1b/dTsR0/K9vSqTADrNBED/dOO9HLwetwl/bH
oJL5Ipnfcs9K01U1KFDj3hmm4VYLm77AssaIAkyDV/LO/4V3XJa82/XCU9Pj+8hvdTx+hq
qpHceJ1LQRdxZJo6s1Ivm4moVGE8Mi/FmQiiDj6VfoZbOpJoyO8PYNdcAs0/CnGTtSf90Z
f7vR/PsAAtl5e/0UNpe9l+TkBoQctaUGFK/Tj/q9oB8uRYvvaaI4pR2aYipSwvpHyAob1r
unB/fzV+1WrRK3EFXU6kDe7NQp7jsWfzSPEEkndaFLYhAAAAgQDHdRp+xiGR65ZnoCZ8hL
yDp9DIPhwUDyH6HlAExtHy64pTcnpGxcSqBEiLclh4/fLYu9zmcRqGm5zYSa+4TczMYxjf
KpVLw+w0Nbki0ZRO8KDvCTjw/TvWkkDP05o8fEFkxMT9ECBq0zRG7ZgXONML/C7BR0HZ6u
ZAa0aIgquvFgAAAIEA8KFRiLLRaW9o+t/GV4Vzw867yTwb/mVgI2f9mRt3TGL0tWZrFVa4
LzkWKPv2BbJk73esYnTWyzvaZ08bjO0SvV+VB0/3eABZDYAtgTDsOSa4Rj9wixrIRvnFFT
/2Lqqg6S6Np1IoP0ku2lkwxmWsoUbOVKF0qNsvMKUtgFWHZDEAAACBAM1A1MFlgiLjkxt1
oHxf6paWifggaTlUTz8f5zGPMIK3Bd986BuRq2YyvVGxrpNg+mors7VvbLvHlKixqWnRQ0
Pg9FKZLwUs6yTs84MdaDQRiRA68Tm+zdg2ylTw8Ni16CUojJyAqKwRYheMMVsulhMSYdcz
yapfzYe6OiDJV/RbAAAAG21sYkBNYXJjcy1NYWNCb29rLVByby5sb2NhbAECAwQFBgc=
-----END OPENSSH PRIVATE KEY-----

PuTTY-User-Key-File-2: ssh-rsa
Encryption: none
Comment: mlb@Marcs-MacBook-Pro.local
Public-Lines: 6
AAAAB3NzaC1yc2EAAAADAQABAAABAQDA7h6aOewJzno5NiQ25h4kkWYFFLLfdSAc
QODAiFMtBTv4LeelgmaRok/VVt47MkZAfTctr0qwOtksPHlewAgCA33UDw4K2zaE
oA7hkqD5jxcYq830+qzPe6peBQWIYMS/hMPFG2T6wW1Odpz6jqtR7wjk+YpmUKkv
KRgSZxXh69SKLj8DXEOzlTzayu7KuCJqb0SFVUm3s2btw/hkZ+PLMT2UEXsfhK7D
gcoQz3CFLxBCYYi3GeF+bCM14nTsnD0a1nkdUNt++jxtWmgCqtUpxMfSI+egFipD
ZQSovR3YCMmFdVaSi5I/tETwFWtIDea3ZQHU7ZrU3PQTfPD4RlFr
Private-Lines: 14
AAABABcxyX4TLpxzNWCBcNMk0UTmMhS0S1jwFN8wsZSUDyPfeDk7E5ljVv91OxHT
8r29KpMAOs0EQP90470cvB63CX9segkvkimd9yz0rTVTUoUOPeGabhVgubvsCyxo
gCTINX8s7/hXdclrzb9cJT0+P7yG91PH6Gqqkdx4nUtBF3FkmjqzUi+biahUYTwy
L8WZCKIOPpV+hls6kmjI7w9g11wCzT8KcZO1J/3Rl/u9H8+wAC2Xl7/RQ2l72X5O
QGhBy1pQYUr9OP+r2gHy5Fi+9pojilHZpiKlLC+kfIChvWu6cH9/NX7VatErcQVd
TqQN7s1CnuOxZ/NI8QSSd1oUtiEAAACBAPChUYiy0WlvaPrfxleFc8POu8k8G/5l
YCNn/Zkbd0xi9LVmaxVWuC85Fij79gWyZO93rGJ01ss72mdPG4ztEr1flQdP93gA
WQ2ALYEw7DkmuEY/cIsayEb5xRU/9i6qoOkujadSKD9JLtpZMMZlrKFGzlShdKjb
LzClLYBVh2QxAAAAgQDNQNTBZYIi45MbdaB8X+qWlon4IGk5VE8/H+cxjzCCtwXf
fOgbkatmMr1Rsa6TYPpqK7O1b2y7x5Sosalp0UND4PRSmS8FLOsk7PODHWg0EYkQ
OvE5vs3YNspU8PDYteglKIycgKisEWIXjDFbLpYTEmHXM8mqX82HujogyVf0WwAA
AIEAx3UafsYhkeuWZ6AmfIS8g6fQyD4cFA8h+h5QBMbR8uuKU3J6RsXEqgRIi3JY
eP3y2Lvc5nEahpuc2EmvuE3MzGMY3yqVS8PsNDW5ItGUTvCg7wk48P071pJAz9Oa
PHxBZMTE/RAgatM0Ru2YFzjTC/wuwUdB2ermQGtGiIKrrxY=
Private-MAC: 0c1d4ff803a676cb0b0fbd25030a24370f39cd8b

5. Make yourself an administrator /usr/bin/mysql -u root -h $(/usr/bin/kubectl get svc -n mariadb -o json | /snap/bin/jq -r .items[0].spec.clusterIP) --password=start123 -e "insert into userGroups (userId,groupId) values (2,1);" unison
6. Make yourself a cluster administrator /usr/bin/mysql -u root -h $(/usr/bin/kubectl get svc -n mariadb -o json | /snap/bin/jq -r .items[0].spec.clusterIP) --password=start123 -e "insert into userGroups (userId,groupId) values (2,2);" unison
7. Log back in
8. Click on Kubernetes Dashboard

Enable SSO

1. SSH to your server
2. Get api server parameter flags kubectl describe configmap api-server-config -n openunison
3. Export CA certificate kubectl get secret ou-tls-certificate -n openunison -o json | jq -r '.data."tls.crt"' | base64 -d > /etc/kubernetes/pki/ou-ca.pem
4. Update /etc/kubernetes/manifests/kube-apiserver.yaml with output of #2
5. clear your k8s config rm /root/.kube/config
6. kubectl get pods --all-namespaces
7. Load token
8. kubectl get pods --all-namespaces
9. Logout of openunison
10. watch kubectl get pods --all-namespaces

Lab 2

Create Namespace

1. Login to your openunison with the user makens and the password $tart123

2. Setup kubectl using your token

3. Try to create a NS kubectl create ns mynewns, it will fail

4. Enable audit logging:

   • mkdir /var/log/k8s
   • mkdir /etc/kubernetes/audit
   • cp k8s-audit-policy.yaml /etc/kubernetes/audit
   • Edit /etc/kubernetes/manifests/kube-apiserver.yaml
     • add to command

   - --audit-log-path=/var/log/k8s/audit.log
   - --audit-log-maxage=1
   - --audit-log-maxbackup=10
   - --audit-log-maxsize=10
   - --audit-policy-file=/etc/kubernetes/audit/k8s-audit-policy.yaml
   

   • add:

   - mountPath: /var/log/k8s
     name: var-log-k8s
     readOnly: false
   - mountPath: /etc/kubernetes/audit
     name: etc-kubernetes-audit
     readOnly: true
   

   to volumeMounts section

   • add:

   - hostPath:
       path: /var/log/k8s
       type: DirectoryOrCreate
     name: var-log-k8s
   - hostPath:
       path: /etc/kubernetes/audit
       type: DirectoryOrCreate
     name: etc-kubernetes-audit
   

   to volumes

5. Once the api server is running again, login as makens again and try creating a namespace again kubectl create ns mynewns, it will fail

6. Look for the audit logs message grep makens /var/log/k8s/audit.log

7. Generate RBAC rules from audit2rbac, replace IP with the IP of your cluster ./audit2rbac --filename=/var/log/k8s/audit.log --user=https://ou.apps.IP.nip.io/auth/idp/k8sIdp#makens > newrbac.yaml

8. Set your context to admin export KUBECONFIG=/root/.kube/config-admin 9 . Import the RBAC kubectl create -f ./newrbac.yaml

9. Unset your kubeconfig to go back to your default export KUBECONFIG=

10. kubectl create ns mynewns, SUCCESS!

Lab 3

Enable Pod Security Policies

1. Create the policies - kubectl create -f ./podsecuritypolicies.yaml
2. Edit /etc/kubernetes/manifests/kube-apiserver.yaml, change --enable-admission-plugins=NodeRestriction to --enable-admission-plugins=PodSecurityPolicy,NodeRestriction
3. Save
4. Delete all your pods kubectl delete pods --all-namespaces --all
5. Once done, check if OpenUnison is running and what policy its running under kubectl describe pods -l application=openunison-orchestra -n openunison
6. Chcek if tthe ingress pod is running kubectl get pods -n ingress-nginx
7. Check if mariadb is running kubectl get pods -n mariadb
8. Look at the events for both the mariadb and ingress-nginx namespace - kubectl get events -n mariadb / kubectl get events -n ingress-nginx
9. Why isn't it running? :

kubectl create -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: privileged-psp
subjects:
# For the kubeadm kube-system nodes
- kind: ServiceAccount
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
EOF

8. Update the ingress-nginx Deployment to force a redeploy - kubectl edit deployment nginx-ingress-controller -n ingress-nginx
9. Fix mariadb:

kubectl create -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: mariadb
  namespace: mariadb
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: privileged-psp
subjects:
# For the kubeadm kube-system nodes
- kind: ServiceAccount
  name: default
  namespace: mariadb
EOF