Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
k8s daily and notes

Getting Started

Notes

Application troubleshooting

  • POD:
    • Describe pod -- ready or running / pending(Enough resources) or waiting (Image pull issue) or unhealthy/crashing (Check Logs)
      • Pending : kubectl describe pod POD-UID
      • Check events
      • ImagePullBackOff Or ErrImagePull : kubectl describe pod POD-UID
        • Check events
        • Check that the image name is correct and try pulling the image manually on the host using docker pull docker pull <image_name>
        • PVC Status Show Pending: kubectl get pvc --> kubectl describe pvc pvc-UID
          • Check events
    • Pod running but not working:
      • Validate - Check the above steps
      • Validate yaml file kubectl apply --validate -f pod.yml
      • It could be due to a failed process in the container or insufficient storage:
        • Login in to the container and check the running processes
          kubectl exec -ti POD-UID -- /bin/bash
          > ps ax
    • List Containers inside the pod
      kubectl get pod <pod_name> -o jsonpath='{.spec.containers[*].name}' -n <namespace>
      
  • Unable To Look Up Or Resolve Kubernetes Service Names Using DNS:
    • Check if the service name you are using is correct.
    • Run these commands to check if the service is registered and the pods selected:
      kubectl get svc
      kubectl get endpoints
      
    • If the service is registered run the below command to check the DNS resolution is working or not.
      kubectl exec -it POD-UID nslookup SERVICE-NAME
      

Cluster Troubleshooting:

  • Cluster health

    $ kubectl get cs # Component status
    NAME                 STATUS    MESSAGE              ERROR
    scheduler            Healthy   ok
    controller-manager   Healthy   ok
    etcd-0               Healthy   {"health": "true"}
  • Check nodes are all registered correctly.

    kubectl get nodes
    
  • Check the logs

    • Master:

      • /var/log/kube-apiserver.log - API Server, responsible for serving the API
      • /var/log/kube-scheduler.log - Scheduler, responsible for making scheduling decisions
      • /var/log/kube-controller-manager.log - Controller that manages replication controllers
    • Node:

      • /var/log/kubelet.log - Kubelet, responsible for running containers on the node
      • /var/log/kube-proxy.log - Kube Proxy, responsible for service load balancing
  • Kubectl Unable To Find Your Nodes:

    • Error: the server doesn't have a resource type "nodes" Solution: This occurs because the authentication credentials are not correctly set. To resolve this, copy the configuration file /etc/kubernetes/admin.conf to ~/.kube/config in a regular user account (with sudo if necessary) and try again. This command should not be performed in the root user account.

Docker Space issue

systemctl stop kubelet
systemctl stop docker
  cd /var/lib/docker
  tar -czvf /tmp/docker.tar.gz *
  cd /tmp
  ls -latr
  tar -zxvf /tmp/docker.tar.gz -C /docker
  rm -rf /var/lib/docker
  ln -s /docker /var/lib/docker
  ll /var/lib/docker
  cd /var/lib/docker
  du -sh *
systemctl start docker
systemctl start kubelet
  docker ps
  journalctl -u kubelet.service -f
  docker ps

Controller-Manager:

  • Node Controller: Responsible for noticing and responding when nodes go down.
  • Replication Controller: Responsible for maintaining the correct number of pods for every replication controller object in the system.
  • Endpoints Controller: Populates the Endpoints object (that is, joins Services & Pods).
  • Service Account & Token Controllers: Create default accounts and API access tokens for new namespaces.

Cloud Controller-Manager

  • Node Controller: For checking the cloud provider to determine if a node has been deleted in the cloud after it stops responding
  • Route Controller: For setting up routes in the underlying cloud infrastructure
  • Service Controller: For creating, updating and deleting cloud provider load balancers
  • Volume Controller: For creating, attaching, and mounting volumes, and interacting with the cloud provider to orchestrate volumes

TAINT:

  • Node restrictions like labels

Difference between Replication Controller, Replica Sets and Deployements.

  • The major difference is that the rolling-update command works with Replication Controllers, but won’t work with a Replica Set. This is because Replica Sets are meant to be used as the backend for Deployments.

  • Replication Controller: Will make sure the desired number of pods are running as per the spec.

  • Replica Sets: Same as replication controller but have few more feature like you can deploy pods using matchlabel. Example:

    spec:
       replicas: 3
       selector:
         matchExpressions:
          - {key: app, operator: In, values: [soaktestrs, soaktestrs, soaktest]}
          - {key: teir, operator: NotIn, values: [production]}
      template:
         metadata:
    
    • The app label must be soaktestrc, soaktestrs, or soaktest
    • The tier label (if it exists) must not be production
  • Deployments: Deployments are intended to replace Replication Controllers. They provide the same replication functions (through Replica Sets) and also the ability to rollout changes and roll them back if necessary.

  • Services:

    • Abstraction layer which make sure the traffic is flowed through it in a stateless application
    • Abstraction defines a logical set of pods and enable external traffic exposure
      1. LB/External IP: Accessible from outside the cluster
      2. ClusterIP: Accessible from inside the cluster
      3. NodePort: ServiceDNS and Pod DNS
      • Service DNS: <SERVICE_NAME>.<namespace>.<service>.<cluster>.local
      • POD DNS: <POD_IP>.<namespace>.<service>.<cluster>.local

ETCD:

  • Backup ETCD
  • Side effects we get when we retrive etcd cluster from back up

Canary Updates

  • Pattern for reducing risk involved with releasing new software versions.
  • Can be done using percentage basis rollout.
  • We can use Ingress to split the traffic of the Canary deployment to be accessible on a subdomain.
    example.com        --|                 |-> app-service 80
                         |   Ingress IP    |
    canary.example.com --|                 |-> app-canary-service 80
    

Garbage Collection

  • Used to delete objects which doesnot have owners and dependents
  • Garbage collector deletes objects in 2 ways
    • Foreground cascading deletion
    • Background cascading deletion

Click here for reference

Varnish-Ingress

Click here for reference

Secret Types

SecretType = "Opaque"                                 // Opaque (arbitrary data; default)
SecretType = "kubernetes.io/service-account-token"    // Kubernetes auth token
SecretType = "kubernetes.io/dockercfg"                // Docker registry auth
SecretType = "kubernetes.io/dockerconfigjson"         // Latest Docker registry auth
SecretTypeTLS SecretType = "kubernetes.io/tls"
SecretTypeBasicAuth SecretType = "kubernetes.io/basic-auth"
SecretTypeSSHAuth SecretType = "kubernetes.io/ssh-auth"
SecretTypeTLS SecretType = "kubernetes.io/tls"
SecretTypeBootstrapToken SecretType = "bootstrap.kubernetes.io/token"

For reference of the Source code list click here

Kubernetes Pentest Methodology

Kubernetes hyperkube

- Is an all-in-one binary for the Kubernetes server components.
- Built for multiple architectures and the image is pushed automatically on every release.

Identify Component Leader

```
kubectl get ep <component-name> -n kube-system -o yaml
```

example:

```
kubectl get ep kube-controller-manager -n kube-system -o yaml
kubectl get ep kube-scheduler -n kube-system -o yaml
```
  • Look for holderIdentity field of the ...kubernetes.io/leader annotation

Query kubernetes api from inside the pod

  • Spin up a alpine container
    kubectl run -it --rm --restart=Never --image=alpine:latest alpine
    
    / # apk add --update curl
    
  • Now run the below commands
    # Get the ServiceAccount token from within the Pod's container
    / # TOKEN=$(cat /run/secrets/kubernetes.io/serviceaccount/token)
    
    #  Get information from the API server
    / # curl -H "Authorization: Bearer $TOKEN" https://kubernetes/api/v1/ --insecure
    
    # Call an API Server's endpoint (using the ClusterIP kubernetes service) to get all the Pods running in the default namespace
    / # curl -H "Authorization: Bearer $TOKEN" https://kubernetes/api/v1/namespaces/default/pods/ --insecure
    
  • If you want to query the kubernetes api from a pod which in a namespace apart from default
    # Get the ServiceAccount token from within the Pod's container
    / # TOKEN=$(cat /run/secrets/kubernetes.io/serviceaccount/token)
    
    #  Get information from the API server
    / # curl -H "Authorization: Bearer $TOKEN" https://kubernetes.default.svc.cluster.local/api/v1/ --insecure
    
    # Call an API Server's endpoint (using the ClusterIP kubernetes service) to get all the Pods running in the default namespace
    / # curl -H "Authorization: Bearer $TOKEN" https://kubernetes.default.svc.cluster.local/api/v1/namespaces/default/pods/ --insecure
    

Usage

  1. Create a shortcut to switch namespaces:

    $ vi usr/local/bin/kubens
    
    $ kubectl config set-context $(kubectl config current-context) --namespace=$1
    Run:
    $ kubens <name_space>
    
  2. Scaling:

    kubectl scale --replicas=3 rs/foo                                 # Scale a replicaset named 'foo' to 3
    kubectl scale --replicas=3 -f foo.yaml                            # Scale a resource specified in "foo.yaml" to 3
    kubectl scale --current-replicas=2 --replicas=3 deployment/mysql  # If the deployment named mysql's current size is 2, scale mysql to 3
    kubectl scale --replicas=5 rc/foo rc/bar rc/baz                   # Scale multiple replication controllers
  3. Node Metrics:

    kubectl top nodes

    Pod Metrics:

    kubectl top pod POD_NAME --containers               # Show metrics for a given pod and its containers

    Sort Pod Metrics:

    kubectl top pod --all-namespaces | sort --reverse --key 3 --numeric | head -10
    • sort --reverse --key 3 --numeric: Sorts the input according to the numeric values found in column 3 in descending order.
    • head -10 Prints only the top 10 lines.
  4. Get events from namespace:

    kubectl get events -w -n <NAME_SPACE>
    
  5. Extract .crt file from already created tls secret.

    kubectl get secret example-tls -n dev -o yaml | grep tls.crt | awk '{print $2}' | base64 -d
    • See the output in a text format
    kubectl get secret example-tls -n dev -o yaml | grep tls.crt | awk '{print $2}' | base64 -d| openssl x509 -noout -text
    
  6. Create a dockerconfig secret

    kubectl create secret docker-registry docker-hub --docker-server=https://index.docker.io/v1/ --docker-username=xxxxx --docker-password=xxxxx --docker-email=xxxxx@email.com
  7. Verify the above created dockerconfig secret

    kubectl get secrets docker-hub -o yaml | grep .dockerconfigjson: | awk '{print $2}' | base64  -D
  8. Force delete a pod

    kubectl delete pod --grace-period=0 --force --namespace <NAMESPACE> <PODNAME>
  9. Create busybox/dnstools pod to check dns resolution using nslookup

    kubectl run -it --rm --restart=Never --image=busybox busybox
    kubectl run -it --rm --restart=Never --image=infoblox/dnstools:latest dnstools
  10. Apply yaml file through shell

    kubectl apply -f - <<EOF
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: build-robot
    EOF
  11. Kubectl Context and Configuration

    kubectl config view -o jsonpath='{.users[].name}'    # display the first user
    kubectl config view -o jsonpath='{.users[*].name}'   # get a list of users
    kubectl config get-contexts                          # display list of contexts 
    kubectl config current-context                       # display the current-context
    kubectl config use-context my-cluster-name           # set the default context to my-cluster-name
    
    # add a new cluster to your kubeconf that supports basic auth
    kubectl config set-credentials kubeuser/foo.kubernetes.com --username=kubeuser --password=kubepassword
    
    # permanently save the namespace for all subsequent kubectl commands in that context.
    kubectl config set-context --current --namespace=ggckad-s2
    
    # set a context utilizing a specific username and namespace.
    kubectl config set-context gce --user=cluster-admin --namespace=foo && kubectl config use-context gce
    
    kubectl config unset users.foo                       # delete user foo
  12. To view node and volume mount information where pods are running

    • View node and volume info:
    kubectl describe pod <pod_name> | egrep '^Node:|^Status:|^    Mounts:|^      \/'
    
    • View Node and its status
    kubectl describe pod <pod_name> | egrep '^Node:|^Status:'
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment