- Notes
- Application troubleshooting
- Cluster Troubleshooting
- Docker Space issue
- Controller-Manager
- Cloud Controller-Manager
- TAINT
- Difference between Replication Controller, Replica Sets and Deployements
- ETCD
- Canary Updates
- Garbage Collection
- Varnish-Ingress
- Secret Types
- Kubernetes Pentest Methodology
- hyperkube
- Identify Components Leader
- Query kubernetes api from inside the pod
- k8s network troubleshooting
- Usage
- POD:
- Describe pod -- ready or running / pending(Enough resources) or waiting (Image pull issue) or unhealthy/crashing (Check Logs)
- Pending :
kubectl describe pod POD-UID
- Check events
- ImagePullBackOff Or ErrImagePull :
kubectl describe pod POD-UID
- Check events
- Check that the image name is correct and try pulling the image manually on the host using docker pull
docker pull <image_name>
- PVC Status Show Pending:
kubectl get pvc
-->kubectl describe pvc pvc-UID
- Check events
- Pending :
- Pod running but not working:
- Validate - Check the above steps
- Validate yaml file
kubectl apply --validate -f pod.yml
- It could be due to a failed process in the container or insufficient storage:
- Login in to the container and check the running processes
kubectl exec -ti POD-UID -- /bin/bash > ps ax
- Login in to the container and check the running processes
- List Containers inside the pod
kubectl get pod <pod_name> -o jsonpath='{.spec.containers[*].name}' -n <namespace>
- Describe pod -- ready or running / pending(Enough resources) or waiting (Image pull issue) or unhealthy/crashing (Check Logs)
- Unable To Look Up Or Resolve Kubernetes Service Names Using DNS:
- Check if the service name you are using is correct.
- Run these commands to check if the service is registered and the pods selected:
kubectl get svc kubectl get endpoints
- If the service is registered run the below command to check the DNS resolution is working or not.
kubectl exec -it POD-UID nslookup SERVICE-NAME
-
Cluster health
$ kubectl get cs # Component status NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-0 Healthy {"health": "true"}
-
Check nodes are all registered correctly.
kubectl get nodes
-
Check the logs
-
Master:
/var/log/kube-apiserver.log
- API Server, responsible for serving the API/var/log/kube-scheduler.log
- Scheduler, responsible for making scheduling decisions/var/log/kube-controller-manager.log
- Controller that manages replication controllers
-
Node:
/var/log/kubelet.log
- Kubelet, responsible for running containers on the node/var/log/kube-proxy.log
- Kube Proxy, responsible for service load balancing
-
-
Kubectl Unable To Find Your Nodes:
- Error:
the server doesn't have a resource type "nodes"
Solution: This occurs because the authentication credentials are not correctly set. To resolve this, copy the configuration file /etc/kubernetes/admin.conf to ~/.kube/config in a regular user account (with sudo if necessary) and try again. This command should not be performed in the root user account.
- Error:
systemctl stop kubelet
systemctl stop docker
cd /var/lib/docker
tar -czvf /tmp/docker.tar.gz *
cd /tmp
ls -latr
tar -zxvf /tmp/docker.tar.gz -C /docker
rm -rf /var/lib/docker
ln -s /docker /var/lib/docker
ll /var/lib/docker
cd /var/lib/docker
du -sh *
systemctl start docker
systemctl start kubelet
docker ps
journalctl -u kubelet.service -f
docker ps
- Node Controller: Responsible for noticing and responding when nodes go down.
- Replication Controller: Responsible for maintaining the correct number of pods for every replication controller object in the system.
- Endpoints Controller: Populates the Endpoints object (that is, joins Services & Pods).
- Service Account & Token Controllers: Create default accounts and API access tokens for new namespaces.
- Node Controller: For checking the cloud provider to determine if a node has been deleted in the cloud after it stops responding
- Route Controller: For setting up routes in the underlying cloud infrastructure
- Service Controller: For creating, updating and deleting cloud provider load balancers
- Volume Controller: For creating, attaching, and mounting volumes, and interacting with the cloud provider to orchestrate volumes
- Node restrictions like labels
-
The major difference is that the rolling-update command works with Replication Controllers, but won’t work with a Replica Set. This is because Replica Sets are meant to be used as the backend for Deployments.
-
Replication Controller: Will make sure the desired number of pods are running as per the spec.
-
Replica Sets: Same as replication controller but have few more feature like you can deploy pods using matchlabel. Example:
spec: replicas: 3 selector: matchExpressions: - {key: app, operator: In, values: [soaktestrs, soaktestrs, soaktest]} - {key: teir, operator: NotIn, values: [production]} template: metadata:
- The app label must be soaktestrc, soaktestrs, or soaktest
- The tier label (if it exists) must not be production
-
Deployments: Deployments are intended to replace Replication Controllers. They provide the same replication functions (through Replica Sets) and also the ability to rollout changes and roll them back if necessary.
-
Services:
- Abstraction layer which make sure the traffic is flowed through it in a stateless application
- Abstraction defines a logical set of pods and enable external traffic exposure
- LB/External IP: Accessible from outside the cluster
- ClusterIP: Accessible from inside the cluster
- NodePort: ServiceDNS and Pod DNS
- Service DNS:
<SERVICE_NAME>.<namespace>.<service>.<cluster>.local
- POD DNS:
<POD_IP>.<namespace>.<service>.<cluster>.local
- Backup ETCD
- Side effects we get when we retrive etcd cluster from back up
- Pattern for reducing risk involved with releasing new software versions.
- Can be done using percentage basis rollout.
- We can use Ingress to split the traffic of the Canary deployment to be accessible on a subdomain.
example.com --| |-> app-service 80 | Ingress IP | canary.example.com --| |-> app-canary-service 80
- Used to delete objects which doesnot have owners and dependents
- Garbage collector deletes objects in 2 ways
- Foreground cascading deletion
- Background cascading deletion
Click here for reference
Click here for reference
SecretType = "Opaque" // Opaque (arbitrary data; default)
SecretType = "kubernetes.io/service-account-token" // Kubernetes auth token
SecretType = "kubernetes.io/dockercfg" // Docker registry auth
SecretType = "kubernetes.io/dockerconfigjson" // Latest Docker registry auth
SecretTypeTLS SecretType = "kubernetes.io/tls"
SecretTypeBasicAuth SecretType = "kubernetes.io/basic-auth"
SecretTypeSSHAuth SecretType = "kubernetes.io/ssh-auth"
SecretTypeTLS SecretType = "kubernetes.io/tls"
SecretTypeBootstrapToken SecretType = "bootstrap.kubernetes.io/token"
For reference of the Source code list click here
- Is an all-in-one binary for the Kubernetes server components.
- Built for multiple architectures and the image is pushed automatically on every release.
```
kubectl get ep <component-name> -n kube-system -o yaml
```
example:
```
kubectl get ep kube-controller-manager -n kube-system -o yaml
kubectl get ep kube-scheduler -n kube-system -o yaml
```
- Look for
holderIdentity
field of the...kubernetes.io/leader
annotation
- Spin up a alpine container
kubectl run -it --rm --restart=Never --image=alpine:latest alpine / # apk add --update curl
- Now run the below commands
# Get the ServiceAccount token from within the Pod's container / # TOKEN=$(cat /run/secrets/kubernetes.io/serviceaccount/token) # Get information from the API server / # curl -H "Authorization: Bearer $TOKEN" https://kubernetes/api/v1/ --insecure # Call an API Server's endpoint (using the ClusterIP kubernetes service) to get all the Pods running in the default namespace / # curl -H "Authorization: Bearer $TOKEN" https://kubernetes/api/v1/namespaces/default/pods/ --insecure
- If you want to query the kubernetes api from a pod which in a namespace apart from
default
# Get the ServiceAccount token from within the Pod's container / # TOKEN=$(cat /run/secrets/kubernetes.io/serviceaccount/token) # Get information from the API server / # curl -H "Authorization: Bearer $TOKEN" https://kubernetes.default.svc.cluster.local/api/v1/ --insecure # Call an API Server's endpoint (using the ClusterIP kubernetes service) to get all the Pods running in the default namespace / # curl -H "Authorization: Bearer $TOKEN" https://kubernetes.default.svc.cluster.local/api/v1/namespaces/default/pods/ --insecure
-
Create a shortcut to switch namespaces:
$ vi usr/local/bin/kubens $ kubectl config set-context $(kubectl config current-context) --namespace=$1
Run: $ kubens <name_space>
-
Scaling:
kubectl scale --replicas=3 rs/foo # Scale a replicaset named 'foo' to 3 kubectl scale --replicas=3 -f foo.yaml # Scale a resource specified in "foo.yaml" to 3 kubectl scale --current-replicas=2 --replicas=3 deployment/mysql # If the deployment named mysql's current size is 2, scale mysql to 3 kubectl scale --replicas=5 rc/foo rc/bar rc/baz # Scale multiple replication controllers
-
Node Metrics:
kubectl top nodes
Pod Metrics:
kubectl top pod POD_NAME --containers # Show metrics for a given pod and its containers
Sort Pod Metrics:
kubectl top pod --all-namespaces | sort --reverse --key 3 --numeric | head -10
sort --reverse --key 3 --numeric
: Sorts the input according to the numeric values found in column 3 in descending order.head -10
Prints only the top 10 lines.
-
Get events from namespace:
kubectl get events -w -n <NAME_SPACE>
-
Extract .crt file from already created tls secret.
kubectl get secret example-tls -n dev -o yaml | grep tls.crt | awk '{print $2}' | base64 -d
- See the output in a text format
kubectl get secret example-tls -n dev -o yaml | grep tls.crt | awk '{print $2}' | base64 -d| openssl x509 -noout -text
-
Create a dockerconfig secret
kubectl create secret docker-registry docker-hub --docker-server=https://index.docker.io/v1/ --docker-username=xxxxx --docker-password=xxxxx --docker-email=xxxxx@email.com
-
Verify the above created dockerconfig secret
kubectl get secrets docker-hub -o yaml | grep .dockerconfigjson: | awk '{print $2}' | base64 -D
-
Force delete a pod
kubectl delete pod --grace-period=0 --force --namespace <NAMESPACE> <PODNAME>
-
Create busybox/dnstools pod to check dns resolution using
nslookup
kubectl run -it --rm --restart=Never --image=busybox busybox kubectl run -it --rm --restart=Never --image=infoblox/dnstools:latest dnstools
-
Apply yaml file through shell
kubectl apply -f - <<EOF apiVersion: v1 kind: ServiceAccount metadata: name: build-robot EOF
OR
cat <<EOF | kubectl create -f - apiVersion: v1 kind: ServiceAccount metadata: name: build-robot EOF
-
Kubectl Context and Configuration
kubectl config view -o jsonpath='{.users[].name}' # display the first user kubectl config view -o jsonpath='{.users[*].name}' # get a list of users kubectl config get-contexts # display list of contexts kubectl config current-context # display the current-context kubectl config use-context my-cluster-name # set the default context to my-cluster-name # add a new cluster to your kubeconf that supports basic auth kubectl config set-credentials kubeuser/foo.kubernetes.com --username=kubeuser --password=kubepassword # permanently save the namespace for all subsequent kubectl commands in that context. kubectl config set-context --current --namespace=ggckad-s2 # set a context utilizing a specific username and namespace. kubectl config set-context gce --user=cluster-admin --namespace=foo && kubectl config use-context gce kubectl config unset users.foo # delete user foo
-
To view node and volume mount information where pods are running
- View node and volume info:
kubectl describe pod <pod_name> | egrep '^Node:|^Status:|^ Mounts:|^ \/'
- View Node and its status
kubectl describe pod <pod_name> | egrep '^Node:|^Status:'