herpiko/kubedingdong.md

## kubedingdong.md

      
    Raw
  

              kubedingdong.md
            
          
    This notes applied to Kubernetes version 1.9.1-00 on Ubuntu Xenial.
Basic concept

TO BE WRITTEN
Installation and starting up 🏁

Installation


Swap should be disabled (see /etc/fstab)
Docker, apt-get install -y docker.io
Add Kubernetes repository, deb http://apt.kubernetes.io/ kubernetes-xenial main to /etc/apt/sources.list.d/kubernetes.list
Add the gpg key of the repo, curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
Install Kubernetes tools (kubeadm, kubelet), apt-get update && apt-get install -y kubeadm=1.9.1-00 kubelet=1.9.1-00
To get the autocomplete feature on bash, source <(kubectl completion bash)

Starting (master node)


Initialize, kubeadm init --apiserver-advertise-address=172.20.10.8 --pod-network-cidr 10.244.0.0/16. 172.20.10.8 is the specific public IP that will be used as apiserver endpoint. 10.244.0.0 is the internal network that used by pods to communicates.
Follow the instruction to run kubernetes as regular user, or just mkdir -p ~/.kube && sudo cp /etc/kubernetes/admin.conf ~/.kube/config && sudo chown $(id -u):$(id -g) ~/.kube/config
At this point you are in regular user and the node should be NotReady (see kubectl get node)
Prepare the network configuration file, for instance use Flannel
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Set the Network on the configuration, example : set it to 10.244.0.0 as intialized on kubeadm init before.
kubectl apply -f kube-flannel.yaml
At this point, regular user should be able to use kubectl command seamessly, but kubeadm still need to be executed from root user.
Check frequently with kubectl get node
Check the node's detail, kubectl describe node nodename
Check pods, kubectl get pods --all-namespaces
Tada!

Starting (other node)

Preparation on the master node


Get token value from sudo kubeadm token list
Get the Discovery Token CA cert hash from openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex

On the regular node


kubeadm join --token a37497.be623ec61a8122dd 10.0.2.15:6443 --discovery-token-ca-cert-hash sha256:6581dc2a13c1848943b4297c0ffc121cfb6b894a083321673aeac6e9a374ee48
In case the master node refused to connect (getsockopt : connection refused error), look at the result of kubectl get svc

Simple deployment and scaling 👶


kubectl run omama --image nginx
kubectl get deployments
kubectl get deployment omama -o yaml > omama.yaml
Edit the file and add these line below under container's name line :

...
ports:
- containerPort: 80
...


Delete the existing deployment, kubectl delete deployment omama
Apply the new deploy conf, kubectl apply -f omama.yaml
Scaling, kubectl scale deployment omama --replicas=3, the scale count represented on pods' count.
Deleting a pod should recreate the new pod, up to the replication conf, kubectl delete pod omama-xxxxx, the new pod will be created with initial status ContainerCreating
Expose the port, kubectl expose deployment/omama
Look at the services and endpoint, kubectl get svc,ep
Log into the child node, try to curl the nginx webserver default page, curl 10.244.1.12:80
Deleting the deployment, remove deployment, svc, and ep by the deployment name, example : kubectl delete deployment omama

Explosing the service outside the cluster


See the pods, kubectl get pods
Remove the related service, kubectl delete svc omama
Recreate the service with LoadBalancer type, kubectl expose deployment omama --type=LoadBalancer
See the port on services, kubectl get svc omama

NAME      TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
omama     LoadBalancer   10.106.16.55   <pending>     80:32576/TCP   2m


Access the cluster service with the specific port through the browser : http://10.106.16.5:32576 🚀

Stopping


For non maste node : kubectl drain --force nodename
Unschedule the node : kubectl cordon nodename
Stop the service (or turn of the machine) : systemctl stop kubelet

Reset (master)


Remove virtual interface, ip link delete virtualinterfacename

kubeadm reset
systemctl stop kubelet
systemctl stop docker
rm -rf /var/lib/cni/
rm -rf /var/lib/kubelet/*
rm -rf /etc/cni/
ifconfig cni0 down
ifconfig flannel.1 down
ifconfig docker0 down
ip link delete cni0
ip link delete flannel.1
sudo rm -rf /etc/kubernetes/manifests/*
sudo rm -rf /var/lib/etcd/*

Miscellaneous

Setup a virtual network interface based on existing physical interface

root@x1carbon:/home/herpiko/src/lfs258# ifconfig wlp4s0:0
wlp4s0:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 60:57:18:0c:77:db  txqueuelen 1000  (Ethernet)

root@x1carbon:/home/herpiko/src/lfs258# ifconfig wlp4s0:0 192.168.1.1
root@x1carbon:/home/herpiko/src/lfs258# ping 192.168.1.101

Another way to get as shell access of pod's container ❓


You may face this issue, kubernetes-retired/kubernetes-anywhere#337
Inspect the pod with describe
You will see the Node and Container ID values.
Enter the node and use docker to access the container.

Operation

Resource limits


kubectl run hog --image vish/stress
kubectl get deployment hog -o yaml > hog.yaml
Add the resource limit, like this :

resources:
  limits:
    memory: 3Gi
  request:
    memory: 2500Mi
  terminationmessagePath: /dev/termination-log
  terminationMessagePolicy: File


Replace the current deployment, kubecl replace -f hog.yaml
Confirm the replacement, kubectl get deployment hog -o yaml
Modify the yaml again (don't forget to remove status and other unique ID, we'll apply this instead of replace)

        resources:
          limits:
            cpu : 1
            memory: 4Gi
          requests:
            cpu: 0.5
            memory: 2500Mi
        args:
        - -cpus
        - "2"
        - -mem-total
        - "950Mi"
        - -mem-alloc-size
        - "100Mi"
        - -mem-alloc-sleep
        - "1s"


kubectl delete deployment hog
kubectl apply -f hog.yaml
The cpu usage on node 1 should be increased.
Modify the resource again, set the CPU (limits, requests) to more than the resource that you have, re-apply.
See pods, the deployment should be failing.
See the log with kubectl logs hog-xxxxx to see what's going wrong.

Resource limits on namespace


Create new namespace, kubectl create namespace low-usage-limit
Check the namespaces available, kubectl get namespaces
Create a LimitRange configuration on an yaml file, low-resource-range.yaml

apiVersion: v1
kind: LimitRange
metadata:
  name: low-resource-range
spec:
  limits:
  - default:
      cpu: 1
      memory: 500Mi
    defaultRequest: 
      cpu: 0.5
      memory: 100Mi
    type: Container


Create the LimitRange, kubectl create -f low-resource-range.yaml --namespace=low-usage-limit
Check the available LimitRange, kubectl get LimitRange --all-namespaces. Not specifying any namespace will return no resources.
Run the hog again on the low-usage-limit namespace, kubectl run limited-hog --image vish/stress -n low-usage-limit
Check the result, kubectl get deploy,pods -n low-usage-limit
Delete the deployment
Redeploy a normal hog, export the yaml, add the namespace low-usage-limit, then apply. The deployment should inherite the resource limit from the namespace / LimitRange conf.

API

TLS access


Isolate the ca, cert and key from ~/.kube/config to txt file, then decode it, cat ca.txt | base64 -d > ca.pem. Now you should've ca.pem, cert.pem and key.pem on your filesystem.
See the apiserver address on these config file too.
Try to access the apiserver, curl --cert cert.pem --key key.pem --cacert ca.pem https://192.168.1.100:6443/ap/v1/pods. You should get the expected result.
You can explore more about the API here, ~/.kube/cache/discovery/192.168.1.100_6443
These API consumed by kubectl command through openat, see strace kubectl get node

RESTful API access


kubectl config view to find the server entry, will get c5c82d8aca5e1a312d077506f50a720a000eb7cbe1b0f862c8b3339887664767IP address and port.
kubectl get secrets --all-namespaces to get the bearer-token.
Lets take a look to default-token-xxx, kubectl describe secret default-token-xxx
Copy the token to access the cluster API, curl https://192.168.1.101/apis --header "Authorization: Bearer inserttokenstringhere" -k
The cert also available inside the pod's container. ❓

Or using a proxy


kubectl proxy --api-prefix=/
Then try to curl without any auth, curl localhost:8001/api/v1/namespaces

Cronjob ⏰


Create the yaml file, cron-job.yaml

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: date
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: dateperminute
            image: busybox
            args:
            - /bin/sh
            - -c
            - date; sleep 30
          restartPolicy: OnFailure


Create it, kubectl create -f cron-job.yaml
Check, kubectl get cronjob
Watch,kubectl get jobs --watch
The jobs will done in it's own pods, kubectl get pods -a
Delete the cronjob if you wish, kubectl delete cronjob date

ReplicaSets 🗿 🗿 🗿


Create the ReplicaSet configuration, named rs.yaml

apiVersion: extions/v1beta1
kind: ReplicaSet
metadata:
  name: rs-one
spec:
  replicas: 2
  template:
    metadata:
      labels:
        system: ReplicaOne
      spec:
        containers:
        - name: nginx
          image: nginx:1.7.9
          ports:
          - containerPort: 80


Fire! kubectl create -f rs.yaml
Inspect, kubectl get rs, kubectl decribe rs rs-one, kubectl get pods
Delete ReplicaSet instance without aspecting pods, kubectl delete rs/rs-one --cascade=false
If we create the ReplicaSet again, the new ReplicaSet will take the ownership of the current pods, kubectl create -f rs.yaml
We can isolate a pod from it's ReplicaSet, kubectl edit pod rs-one-xxxxx
Change the system value to IsolatedPod then save the file.
The pod will be excluded from the ReplicaOne and a new pod will be created to fulfill the ReplicaOne's replication count.
Lets delete the ReplicaOne, kubectl delete rs rs-one then check the pods again.
All the pods will be flushed except the IsolatedPod's pod
It's also possble to delete pod by system label, kubectl delete po -l system=IsolatedPod

DaemonSet


DaomenSet will ensure the the pods will be deployed to each node. ❓ Need more understanding
Copy the rs.yaml to ds.yaml, change Replica string to Daemon and remove the replicas:2 line.
Create the DaemonSet, kubectl create -f ds.yaml
Try to join a new node to the cluster, see if the new pod get deployed on these particular node.

Rolling Updates and Rollbacks


Inspect the previous DaemontSet, specific to update strategy, kubectl get ds ds-one -o yaml | grep -A 1 Strategy, the result will be like this :

  updateStrategy:
    type: OnDelete


Which means the container will be upgraded when the predecessor is deleted. Lets try it.
Update the nginx image on ds-one, kubectl set image ds ds-one nginx=nginx:1.8.1-alpine
Check the current pod, kubectl describe po/ds-one-xxxx | grep image:, should be nginx:1.7.9
Delete the current pod and inspect the new one, the nginx version should be upgraded to 1.8.1-alpine. Pooh!
Any changes on DaemonSet will be recorded, see it with kubectl rollout history ds/ds-one
Inspect the specific revision, kubectl rollout history ds/ds-one --revision=1
To rollback to revision 1, kubectl rollout undo ds/ds-one --to-revision=1, just delete the current pod. The new pod will be deployed using the nginx 1.7.9 back.
There is also another strategy named RollingUpdate. Lets try it.
Copy the current ds, kubectl get ds ds-one -o yaml > ds-two.yaml
herpiko@ubuntu-node-master:~$ kubectl get node --show-labels
NAME                 STATUS    ROLES     AGE       VERSION   LABELS
Change OnDelete string to RollingUpdate on ds-two.yaml, then deploy it.
Inspect the ds-two pods, it's on nginx 1.7.9.
Edit the ds-two directly, kubectl edit ds/ds-two, change the image to nginx:1.8.1-alpine, save it.
Inspect the pods again, it should on nginx 1.8.1-alpine.
Deleting ds is just like deleting another type of objects, kubectl delete ds ds-one,ds-two, the pods will be deleted too.

NodeSelector and labels (8.1)


Cerate a nginx-one.yaml, please pay attention to the nodeSelector value.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-one
  labels:
    system: secondary
  namespace: accounting
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:1.7.9
        imagePullPolicy: Always
        name: nginx
        ports:
        - containerPort: 8080
          protocol: TCP  
      nodeSelector:
        system: secondOne


If you tried to create the object, it'll fail since there is no namespace named accounting. Create the namespace first.
If you check the pods, no pod are got deployed (it stay Pending) since there are no nodes that labeled with secondOne
Check the labels of the nodes, kubectl get nodes --show-labels
Lets add a label to one of the node, kubectl label node ubuntu-node-1 system=secondOne
Check the labels of the nodes again, then check the pods. They should got deployed properly.
If you want to remove label from a node, use minus sign, kubectl label onde ubuntu-node-1 system-
It's also possible to assign pods to master node using labels. For example, some deployment need more resources and master node has it. You also can check the deployed container within docker to make sure the pods are deployed on the correct node.

NodePort


Delete the previous deployment (nginx-one), then redeploy again.
Expose with NodePort type service, kubectl expose deployment nginx-one --type=NodePort --name=service-lab -n accounting
Inspect the service, kubectl describe svc -n accounting, see the NodePort value.
Try to access the public IP of the node with the NodePort's value as port. ❓ The exercise lab said that it could be accessible from master node's IP too

ConfigMap


Try to create some files :

$ mkdir primary
$ echo c > primary/cyan
$ echo m > primary/magenta
$ echo y > primary/yellow
$ echo k > primary/black
$ echo "known as key" >> primary/black
$ echo blue > favorite


These files (and it's content) will be mapped to a ConfigMap, lets create it.
kubectl create configmap colors --from-literal=text=black --from-file=./favorite --from-file=./primary/
See the configmap resources, kubectl get configmaps
See the mapped values, kubectl get configmap colors -o yaml
Lets create a pod that use the values from colors

apiVersion: v1
kind: Pod
metadata:
  name: shell-demo
spec:
  containers:
  - name: nginx
    image: nginx
    env:
    - name: ilike
      valueFrom:
        configMapKeyRef:
          name: colors
          key: favourite


After deployed, you can see that the blue value can be fetched inside the container. ❓ 1.9.1 has issue with kubectl exec

Persistent NFS Volume (PV)

On master node


Install nfs server, sudo apt-get install -y nfs-kernel-server
Create the dir to be shared, sudo mkdir /opt/sfw && sudo chmod 1777 /opt/sfw && sudo echo "software" > /opt/sfw/hello.txt
Add this line to /etc/exports

/opt/sfw/ *(rw,sync_no_root_squash,subtree_check)


Reread the conf, sudo exportfs -ra

On child node


Install the nfs client, sudo apt-get install -y nfs-common
Take a look to the mountpoints of master node, showmount -e 192.168.1.100
Try to mount, sudo mount 192.168.1.100:/opt/sfw /mnt
Check it, ls -l /mnt/ && cat /mnt/hello.txt

Back to the master node to create pv object


Write this file pvol.yaml then fire it.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pvvol-1
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  nfs:
    path: /opt/sfw
    server: 192.168.1.100
    readOnly: false


Check the pv resources, kubectl get pv

Persistent NFS Volume Claim (PVC)


Create the object file, pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-one
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 200Mi


Create it, kubectl create -f pvc.yaml
The pvc shoud be bound to pv pvvol-1, kubectl get pvc,pv
Lets create deployment that use this pv, nfs-pod.yaml,

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: 2018-03-04T17:14:36Z
  generation: 1
  labels:
    run: nginx
  name: omama
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        run: nginx
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: nginx
        volumeMounts:
        - name: nfs-vol
          mountPath: /opt
        ports:
        - containerPort: 80
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      volumes:
      - name: nfs-vol
        persistentVolumeClaim:
          claimName: pvc-one
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30


After got deployed, you can take a sneak peek to the container to make sure the NFS is mounted to /opt

ResourceQuota to Limit PVC Count and Usage


Remove any previous deployment, pv and pvc.
Create ResourceQUota object, named storage-quota.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: storagequota
spec:
  hard:
    persistentvolumeclaims: "10"
    requests.storage: "500Mi"


Create a new namespace named small
Deploy the pv, pvc and storagequota (sequentially) on these namespace
Inspect the namespace, you'll see that there are Resource Quotas.
Remove namespace line from nfs-pod.yaml to allow us to assign another namespace, deploy it on small namespace.
After the pod got deployed, check again the Resource Quotas on namespace's detail.
Create any empty file that has 300mb size on /opt/sfw then check again the Resource Quoatas on small namespace.
Lets simulate what happens when a deployment requests more than the quota. Remove any deployment and check the ns's detail. The storage did not get cleaned when pod was shutdown.
Delete the pvc, pv STATUS should be released.
Delete pv, change PersistentVolumeReclaimPolicy to Delete, the create again.
The current quota on ns should be zero.
Create pvc again.
Remove the current resourcequota, change 500Mi to 100Mi, create it again.
Verify the current quota on ns, the hard limit has already been exceeded.
Create deployment, no error seen, check if the pods actually running.
Remove the deployment then check the pv. Status should be Failed since there's no deleter volume plugin for NFS.
Delete any pv, pvc and deployments, the change pv's persistentVolumeReclaimPolicy value to Recycle
Apply the low-resource-range.yaml to the small namespace
Create the pv again (now on Recycled mode)
If we tried to create the pvc, it should return an error because of exceeded quota. Change the requests.storage to 500Mi with kubectl edit resourcequota -n small then create pvc again. It should work this time.
Create deployment again, then inspect ns.
Delete pvc, the STATUS of pv should be Released then Available
❓ Need more understanding of Retain, Delete, and Recycle mode difference

Taints ❓ Need a beast machine


You can control the pods spreads on nodes with taints. There are 3 modes : NoSchedule, PreferNoSchedule, NoExecute. They will be applied to the next deployment but only NoExecute will force to move the pods to another nodes in instance.
Set taint to a node, kubectl taint nodes ubuntu-node-2 bubba=value:NoSchedule. I dunno why the key is bubba or is it required to be bubba
Check the taint on the specific node, kubectl describe node ubuntu-node-2 | grep Taint
To remove taint from node, kubectl taint nodes ubuntu-node-2 bubba-
You can try redeloy again and again (using simple deployment with replica count more than 8) to see the effect of each taint mode on each node.

Logs


Logs are in :

journalctl -u kubelet
sudo find /var/log -name "*apiserver*log"
kubectl logs podID
kubectl get events


Custom Resource Definition


Helm and Charts


TLS


Auth


Adminission controllers