Skip to content

Instantly share code, notes, and snippets.

@jeremy-donson

jeremy-donson/00-README.md

Last active October 28, 2022 08:32
Show Gist options
  • Save jeremy-donson/fd587fa5ea6e1c80e8ed82c3f1686181 to your computer and use it in GitHub Desktop.
Save jeremy-donson/fd587fa5ea6e1c80e8ed82c3f1686181 to your computer and use it in GitHub Desktop.
Download ZIP
Tech Cert Syllabi 2022
Raw
01-gist-tools.md

gist + tools

  • Public and private gists are simply ad-hoc github repositories.

  • We will track gists and repos.

  • https://pypi.org/project/python-gist/

  • Search All User Gists: content + metadata

  • Search All User Repos: content + metadata

  • Gist API

  • Actions API

  • gists => github => gitpod

  • Dependencies

  • Track doc+test+code release sites for beta/alpha/stable changes

Raw
02-certified-kubernetes-admin.md

Certified Kubernetes Administrator

Table of Contents

  • Public Repo Tour
  • Test Matrix: container flavors + k8s implementations + sample apps + app testing + app scaling
  • Prerequisites: bash
  • Syllabus

Syllabus

  • Section 1: Introduction (21m)

    • Lecture 1: Course Intro

      • Structure: Lecture + Demos + Quizzes + Practice Questions + Slack Channel + Q&A
      • Prerequisites: Docker + K8s Basics + YAML + Lab Setups Using VirtualBox

    • Lecture 2: Certification

      • K8s is at the center of cloud app serving. The CKA cert is in high-demand.
      • Cloud Native Foundation (CNF) Certified Kubernetes Administrator (CKA) Exam Home
      • Performance-based exam. Cost = $300 + One Free Retake. Duration = Two hours max.
      • K8s docs site is available during exam, so we need to be fluent with those docs.

    • Lecture 3: Certification Details

      • In the video, 3 hour exam duration is noted. With the latest version of the exam, it is now only 2 hours. The contents of this course has been updated with the changes required for the latest version of the exam(Q3,2022).
      • Stay current as exam evolves via these references:

    • Lecture 4: Course Release Notes

      • Course updates are ongoing, and recorded here.

    • Lecture 5: The Kubernetes Trilogy

      • Basic K8s Training

      • CKA Training

        • ha deployments
        • logging + monitoring
        • maintenance + troubleshooting + demos
        • scheduler
        • app lifecycle
        • security
        • storage
        • core concepts
        • coding exercises

      • CKAD Training

    • Lecture 6: Slack Support Channel

      • If you run into issues while working with Coding Exercises the best way to reach us is through the slack channel.

      • The slack channel is a collaborative learning and sharing environment for all students enrolled in this course. Feel free to ask your doubts here and request any assistance related to the topic. For those who have completed the exam: please remember to adhere to your NDA agreement. Others, please respect the NDA agreement of others and refrain from asking questions about the exam. Have Fun!

      • Dos:

        • Ask questions and get your doubts cleared about topics discussed in the course
        • Get help on solving lab activities
        • Connect with other students and study as a group

      • Donts:

        • Ask questions about the exam and questions in the exam, such as:

        • What are the questions in the real exam?

        • Is there a question on a specific topic in the exam?

        • What kind of questions come in the exam on a specific topic?

        • What is the answer to a question in the exam?

        • How are the exams scored? Is there partial scoring or full scoring?

        • Questions about the exam environment that is not given in the CNCF FAQ

        • Is a particular tool/utility available within the environment?

        • Are we asked to perform a specific action during the exam?

        • These questions violate Non-Disclosure Agreement of the exam and of those who already took the exam. In case you are not sure what information can be shared, refer to the FAQ here. Anything that's not here shouldn't be shared.

    • KodeKloud Workspace Slack Group Invitation

    • Lecture 7: Course Curriculum Notes

      • One of the questions we frequently get asked is: "Is this course sufficient to prepare for the CKA exam? Do we need to go through any other materials." The answer is: Yes! This course is sufficient to prepare and clear the CKA exam. We have carefully crafted this course with just the topics required for the CKA exam and any pre-requisite knowledge you may need to understand them. And we have practice tests after each concept so you don't just learn theory, but gain enough practice to appear for the exam. In addition, we have multiple mock exams that will help you test your skills, so you shouldn't have to rely on any additional training materials elsewhere. This is all you need!

      • As I said, our first priority is to cover all the concepts part of the CKA exam curriculum. However, we plan to cover additional topics that are not part of the CKA exam curriculum around the following very soon:

        • Auto scaling a cluster
        • Horizontal POD autoscalers
        • Stateful Sets
        • Kubernetes Federation
        • Admission Controllers

      • If you'd like any other topics to be covered please raise an issue here and tag it "Lecture Request" and "CKA".

    • Lecture 8: Reference Notes For Labs + Lectures

      • We have created a repository with notes, links to documentation and answers to practice questions here. Please make sure to go through these as you progress through the course.

  • Section 2: Core Concepts (200m)

    • Lecture 9: Core Concepts Section Introduction

      • Cluster Architecture + API Primitives + Services + Other Network Primitives

    • Lecture 10: Download Presentation Decks For This Section

    • Lecture 11: Cluster Architecture

      • Diagrams:

        • app dev code => github=>Dockerfile=>DockerImage=>DockerHub=>HelmChart=>K8s Runs Dockerized Containers In Pods => Logs
        • lecture-11-cluster-arch-control-plane.png
        • lecture-11-cluster-arch-worker-nodes.png
        • lecture-11-k8s-api-endpoints.png
        • k8s Architecture: Three Core Diagrams

      • K8s Components Architecture - Control Plane

        • Reference Docs Diagram
        • kube-api server: highest level of control over cluster controllers
        • kube-scheduler: ??
        • Controller-Manager: takes orders from kube-api
        • Node-Controller: manages node health
        • Replication-Controller: maintains steady node count
        • Docker: container runtime deployed on workers; negotiates directly with kube-scheduler
          • kubulet: manages nodes
          • kube-proxy: supports inter-worker communications, eg [ app server <=> db server ]

      • ETCD For Beginners

  • Sample tabular formatted data: custom columns
Name Age Location
Jim Hicks 45 New York
Dave Smith 48 New York
Aryan Kumar 50 New York
Aryan Kumar 50 New York
Aryan Kumar 50 New York
--- --- ---
  • key-value-store: [ key : value ]
Key Value
Name Jim Hicks
Age 48
Location New York
Salary 50000
--- ---

  • NOTE: KV Triples Do Not Allow For Duplicate Keys!!

  • ETCD Docs

  • Installation:

    • Download binaries: $curl -L LATEST_BINARY_URL -o $LATEST_BINARY wget -q --https-only
    • Extract: $ tar xzvf $LATEST_BINARY
    • Test Run Service: $ ./etcd => Listens@Port=2379 By Default
    • Test client: $ ./etcdctl set key1 value1; ./etcdctl get key1
    • Client Built-In Help: $ ./etcdctl

  • Lecture 12: ETCD For Beginners

    • ETCD in K8s
    • Kube-API Server
    • Controller Managers
    • Kube Scheduler
    • Kubelet
    • Kube Proxy
    • EXTRA CREDIT: [Kubeflow]
    • Helpful etcd Article

  • Lecture 13: ETCD in Kubernetes

    • Two installation Modes:
      • Manual Setup => advertise-client-urls https://${INTERNAL_IP}:2379
      • kubeadm Setup => Prepackaged ETCD Container Deployed2=> K8s
        • Kube Admin Help: $ kubeadm --help
        • List all pods in a namespace: $ kubectl get pods -n kube-system
        • List only etcd keys in etcd-master node: $ kubectl exec etcd-master -n kube-system etcdctl get / --prefix -keys-only
        • HA => Multiple master nodes => multiple etcd service instances running and listening on port 2379
$ ExecStart=/usr/local/bin/etcd  --initial-cluster controller-0=https://${CONTROLLER0_IP}:2380, \\
                           controller-1=https://${CONTROLLER1_IP}:2380, \\
                           controller-2=https://${CONTROLLER2_IP}:2380

    - Registry/{minions,pods,replicasets,deployments,roles,secrets}/

  • Lecture 14: ETCD - Commands (Optional)

    • Additional information about ETCDCTL Utility
    • ETCDCTL is the CLI tool used to interact with ETCD.
    • ETCDCTL can interact with ETCD Server using 2 API versions - Version 2 and Version 3.
      • By default its set to use Version 2. Each version has different sets of commands.

  • For example ETCDCTL version 2 supports the following commands:

    • etcdctl backup
    • etcdctl cluster-health
    • etcdctl mk
    • etcdctl mkdir
    • etcdctl set

  • Whereas the commands are different in version 3

    • etcdctl snapshot save
    • etcdctl endpoint health
    • etcdctl get
    • etcdctl put

  • To set the right version of API set the environment variable ETCDCTL_API command: export ETCDCTL_API=3

  • When API version is not set, it is assumed to be set to version 2. And version 3 commands listed above don't work.

  • When API version is set to version 3, version 2 commands listed above don't work.

  • Apart from that, you must also specify path to certificate files so that ETCDCTL can authenticate to the ETCD API Server.

  • The certificate files are available in the etcd-master at the following path.

  • We discuss more about certificates in the security section of this course, so don't worry if this looks complex:

--cacert /etc/kubernetes/pki/etcd/ca.crt     
--cert /etc/kubernetes/pki/etcd/server.crt     
--key /etc/kubernetes/pki/etcd/server.key
  • For commands shown in previous video to work, we must specify ETCDCTL API version and path to certificate files.
    • Below is the final form:
$ PARAMS_STRING="ETCDCTL_API=3 etcdctl get / --prefix --keys-only --limit=10  --cacert /etc/kubernetes/pki/etcd/ca.crt \
     --cert /etc/kubernetes/pki/etcd/server.crt  --key   /etc/kubernetes/pki/etcd/server.key"
$ kubectl exec etcd-master -n kube-system -- sh -c "${PARAMS_STRING}"
  • Note that we have several k8s api client utilities/tools:

    • kubectl

    • kubeadm

    • kubulet

    • kubens

    • [kubectx] kubectx is a tool to switch between contexts (clusters) on kubectl faster. kubens is a tool to switch between Kubernetes namespaces (and configure them for kubectl) easily.

    • IBM: Installing kube{adm,ulet,ctl}

    • Updating Tools

    • Lecture 15: Kube-API Server

      • K8s API Docs:kube-api server
      • kube-api server request cycle steps via binary client + HTTPS POST calls:
        • Remember that we might be going through endpoints + load balancers + proxy services, making return trip callback required.
        • Request k8s cluster access via authentication from kube-apiserver.
        • Request validation by kube-apiserver.
        • Route request to ETCD cluster. GET:read as POST:write
        • Retrieve data and return to client who issued request.
        • POST requests write to ETCD and state changes must be executed by k8s api component: kube-scheduler

$ curl -X POST /api/v1/namespaces/default/pods ...

  • The kube-scheduler process constantly checks the etcd cluster database for pod/node state changes.

    • New pods are created accordingly, tasked by the kube-scheduler api component, on a given node.
    • kube-api server is responsible for:
      • auth user
      • validate request
      • retrieve data
      • update etcd => Only done by kube-api server process!!!
      • scheduler
      • kubulet tasks + report back

  • Installation of kube-api server:

    • Kubeadm auto-install: deploys many k8s api components to namespace = 'kube-system' as 'kube-apiserver-master'
      • After we are autenticated and registered, we can issue: $ kubectl get pods -n kubesystem
    • Install kube-apiserver binary manually:
$ wget https://storage.googleapis.com/kubernetes-release/release/v${K8S_VERSION}/bin/linux/amd64/kube-apiserver
$ ExecStart=/usr/local/bin/kube-apiserver ...
  • View api-server options: $ cat /etc/kubernetes/manifests/kube-apiserver.yaml
  • View api-server options at play: $ cat /etc/systemd/system/kube-apiserver.service
  • See running process: ps -aux | grep kube-apiserver

  • Lecture 16: Kube Controller Manager

    • Controller managers are responsible for managing controllers.
    • Controllers resolve any disparities between state db/file changes and system states.
      • Node-Controllers => Node Monitor Period = 5s, Node Monitor Grace Period = 40s, Pod Eviction Timeout = 5m.
      • Other controller types packaged into kube-controller-manager:
        • Deployment-Controller
        • Namespace-Controller
        • Endpoint-Controller
        • CronJob
        • Stateful-Set
        • Node-Controller
        • Job-Controller
        • Replication Controller
        • ReplicaSet
        • PV-Protection-Controller
        • PV-Binder-Controller
        • We can create custom controller configs and custom controllers.

  • Installation of kube-controller-manager services:

    • Install kube-controller-manager binary manually:
$ wget https://storage.googleapis.com/kubernetes-release/release/v${K8S_VERSION}/bin/linux/amd64/kube-controller-manager
$ ExecStart=/usr/local/bin/kube-controller-manager ...
  --node-monitor-period=5s --node-monitor-grace-period=40s --pod-eviction-timeout=5m0s # defaults shown
  --controllers stringSlice   Default: [*] # defaults =  all controllers

  • View api-server options after kubeadm auto-install: $ kubectl get pods -n kube-system | grep -i -n kube-controller-manager-master

  • View kube-controller-manager options at play after kubeadm auto-install: $ cat /etc/kubernetes/manifests/kube-controller-manager.yaml

  • View kube-controller-manager options available: $ cat /etc/systemd/system/kube-controller-manager.service

  • kube-controller-manager docs

  • Lecture 17: Kube Scheduler

    • Assigns pods to nodes. Kubulet executes those tasks on respective nodes.
    • Decisions about which pods will be created on which node are complex as well as configurable.
      • Filter out nodes to ignore.
      • Rank candidate nodes. Often this is about most available node capacity.

  • Kube Scheduler topics are many:

    • Labels + Selectors
    • Resource Limits
    • Manual Scheduling
    • Daemon Sets
    • Multiple Schedulers
    • Scheduler Events
    • Configure K8s Scheduler

  • Installation of kube-scheduler services:

$ wget https://storage.googleapis.com/kubernetes-release/release/v${K8S_VERSION}/bin/linux/amd64/kube-scheduler
# Start service...
$ ExecStart=/usr/local/bin/kube-scheduler --config=/etc/kubernetes/config/kube-scheduler.yaml --v=2

  • View kube-scheduler options: $ cat /etc/kubernetes/manifests/kube-scheduler.yaml

  • View running kube-scheduler process: $ ps -aux | grep kube-scheduler

  • Lecture 18: Kubelet

    • Kubulet listens to requests from kube-apiserver, which is informed of tasks by kube-scheduler.
    • All node work is executed by kubulet.
    • Register Node With Container Runtime (usually docker)
    • Create PODs
    • Monitor Nods + PODs

  • Installation of kubelet services on each worker node is required, since NOT installed by kubeadm: $ wget https://storage.googleapis.com/kubernetes-release/release/v${K8S_VERSION}/bin/linux/amd64/kubulet

  • Run kubulet.service

      --container-runtime-endpoint=unix:///var/run/containerd.sock \\
      --image-pull-progress-deadline=2m  --kubeconfig=/var/lib/kubelet/kubeconfig \\
      --network-plugin=cni --register-node=true --v=2
  • View running kubelet process on workers: $ ps -aux | grep kubelet
  • Lecture 19: Kube Proxy

    • All cluster pods can inter-communicate via kube-proxy.

    • Since pods can move between nodes, pod networking feature gets superseded by providing persistent named cluster service.

    • Simplest example is virtual database service deployed on static ip. Could be file server, etc.

    • kube-proxy worker agents are responsible for cluster-wide service registry routing those requests to proper services.

    • One kubelet method for routing requests is via iptables rules, based upon rules kept on each worker by kubelet.

    • Installation of kube-proxy on each worker node: $ wget https://storage.googleapis.com/kubernetes-release/release/v${K8S_VERSION}/bin/linux/amd64/kube-proxy

    • Start kube-proxy service from worker node CLI:

ExecStart=/usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/kube-proxy-config.yaml Restart=on-failure RestartSec=5
  • kubeadm deploys kube-proxy on each worker node as DaemonSet: $ kubectl get pods -n kube-system | grep proxy $ kubectl get daemonset -n kube-system
  • Lecture 20: Recap - PODs
    • Pods are assigned to nodes. Deployed containers are assigned to pods.
    • Additional app instances require additional pods, unless there are node resource constraints => additional node(s).
    • Some pods can have multiple heterogenous containers, like a pod-local 2nd 'helper container'.
    • If there are already multiple identical pods of app, then each container may need their own sidecar or helper container.
    • Networking and load balancing details are omitted above for topic simplicity.
    • kubectl commands:
      • $ kubctl run $NAME --image $IMAGE_NAME
      • $ kubctl get pods
  • Lecture 21: PODs with YAML
Kind Version
pod v1
Service v1
ReplicaSet apps/v1
Deployment apps/v1
--- --- 
--- pod-definition.yaml
apiVersion:
kind:
metadata:
  name: "${IDENTIFIER}"-pod
  labels: # labels = yaml dict
    app: "${IDENTIFIER}"
    tier: front-end # filters = [ name, env, stacklayer ]
    env: dev-local
    
spec:
  containers:
  - name: nginx
    image: nginx
#  - name: busybox
#    image: busybox
---

$ kubectl get pods
$ kubectl apply -f pod-definition.yaml
$ kubectl get pods
$ kubectl describe pod "${IDENTIFIER}"-pod
  • Lecture 22: Demo - PODs with YA
YAML_FILE='pods.yaml'; INSTANCE_NAME='nginx'; TIER='front-end'
CONTAINER_NAME_01="${INSTANCE_NAME}"; CONTAINER_NAME_02='busybox'
METADATA="metadata:\n  name: ${INSTANCE_NAME}\n  labels:\n    app: ${INSTANCE_NAME}\n    tier: ${TIER}\n"
SPEC="spec:\n  containers:\n  - name: ${CONTAINER_NAME_01}\n    image: ${CONTAINER_NAME_01}\n"
SPEC_EXTENDED="  - name: ${CONTAINER_NAME_02}\n    image: ${CONTAINER_NAME_02}\n"
STRING2FILE="apiVersion: v1\nkind: Pod\n${METADATA}${SPEC}"
echo -e "${STRING2FILE}" > ${YAML_FILE}
cat ${YAML_FILE}
kubectl apply -f pod.yaml
kubectl get pods
kubectl describe pod "${INSTANCE_NAME}
  • yaml is case-sensitive
  • yaml linters verify elements are properly indented, ie, multiples of 2 spaces.
  • Lecture 23: Practice Test Introduction
    • Practice Test Link
    • Laptops and desktops only. No mobile. 5-15 questions per test.
    • Udemy course students get that kodekloud service for free: couponcode = udemystudent151113
    • Practice Test Starts Here
    • Quiz portal on the left is accompanied by running terminal on the right. Tabs support multiple concurrent terminals.
    • Questions can request: view info + configure tasks + scenario-based multiple choice questions.
    • The provided text file editor is vi, so that is an expected skill.
    • Spec files must match exactly!!
    • NOTE: Skipping questions does not allow rollback.
    • NOTE: Actual exam uses a different interface. This is custom tool for practice.
  • Lecture 25: Accessing the Labs
    • Work locally using docker first: $ $ brew install --cask docker
    • Docker Desktop Docs
    • Essential Docker Commands
    • Udemy course access supports access to corresponding kodekloud practice labs via coupon code.
    • Each practice test below has its own url and coupon code.
    • Direct support questions regarding kodekloud services to email: support@kodekloud.com
  • Lecture 27: Practice Test - Solution (Optional)
    • How many pods running in default namespace: $ kubectl get pods
    • Get kubectl run help: $ kubectl run --help
    • Create new pod with nginx image: $ kubectl run xyz --image=nginx
    • Get more info about running pod:
$ POD_NAME='xyz'
$ mkdir -p ~/tmp/; OUT_FILE="describe-pod-${POD_NAME}-output.yaml"
$ kubectl describe pod ${POD_NAME} > ~/tmp/${OUT_FILE}
$ NODE_NAME=$( grep -i Node: ${OUT_FILE} | rev | cut -d' ' -f1 | rev )
$ CONTAINERS_LINE_NUM=$(grep -n '^Containers:' ~/tmp/${OUT_FILE})
$ CONDITIONS_LINE_NUM=$(grep -n '^Conditions:' ~/tmp/${OUT_FILE})
$ sed -n ${CONTAINERS_LINE_NUM},${CONDITIONS_LINE_NUM}p ~/tmp/${OUT_FILE} | grep Image:
$ rm ~/tmp/${OUT_FILE}
$ kubectl delete pod ${POD_NAME}
  • Create new pod with name redis and image redis123 (which does not exist).
$ INSTANCE_NAME=redis ; IMAGE_NAME=redis123 ; OUT_FILE=redis-tests.yaml
$ kubectl run $INSTANCE_NAME --image=$IMAGE_NAME --dry-run=client -o yaml > ~/tmp/${OUT_FILE}
$ cat ${OUT_FILE}
$ kubectl create -f ${OUT_FILE}
$ kubectl get pods
$ sed -i 's/redis123/redis/g' ${OUT_FILE}
$ cat ${OUT_FILE}
$ kubectl apply -f ${OUT_FILE} # warning will be explained later
$ kubectl get pods | grep '^redis '

  • use yq to parse elements from yaml.

  • Wider output includes node names: $ kubectl get pods -o wide

  • Lecture 28: Recap - ReplicaSets
    • Replication controllers are needed to maintain multiple instances of a single pod, providing high availability.
    • The replication controller is in control of multi-pod and multi-node instance resource management.
    • Even a single pod can be maintained by replication controller.
    • Multiple pods can also share the load (load-balancer). That could be on one node, or across multiple nodes.
    • Replication Controller is legacy. Replica Set is current, and requires selector element under spec.
YAML_FILE_NAME='rc-definition.yml'; APP_NAME='myapp'; INSTANCE_NAME="${APP_NAME}-rc"; TIER='front-end';
KIND='ReplicationController'; IMAGE='nginx'; REPLICA_COUNT='3'
METADATA="metadata:\n  name: ${INSTANCE_NAME}\n  labels:\n    app: ${APP_NAME}\n    tier: ${TIER}\n"
SPEC_TEMPLATE="  template:\n\n    metadata:\n     name: ${APP_NAME}-pod\n     \
labels:\n        app: ${APP_NAME}\n        type: ${TYPE}\n    spec:\n      \
containers:\n      - name: ${IMAGE}-container\n        image: ${IMAGE}\n\n  replicas: ${REPLICA_COUNT}\n"
SPEC="spec:\n${SPEC_TEMPLATE}"
STRING2FILE="apiVersion: v1\nkind: ${KIND}\n${METADATA}${SPEC}"
echo -e "${STRING2FILE}" > ${YAML_FILE_NAME}
cat ${YAML_FILE_NAME}
# lint+test yaml via dry-run=client? Use apply rather than create?
kubectl get replicationcontroller
kubectl apply -f ${YAML_FILE_NAME}
kubectl get replicationcontroller
kubectl get pods
kubectl describe pod ${??}
# kubectl create pod
# kubectl get pods
# kubectl kill pod
# kubectl get pods | grep ${INSTANCE_NAME}
  • replicaset-definition.yml => bears similarity to replication control yaml file
YAML_FILE_NAME='replicaset-definition.yml'
APP_NAME='myapp'; INSTANCE_NAME="${APP_NAME}-replicaset"; TIER='front-end';
KIND='ReplicaSet'; IMAGE='nginx'; REPLICA_COUNT='3'
REPLICASET_HEADER="apiVersion: apps/v1\nkind: ${KIND}\n"
METADATA="metadata:\n  name: ${INSTANCE_NAME}\n  labels:\n    app: ${APP_NAME}\n    tier: ${TIER}\n"
SPEC+="  selector:\n     matchLabels:\n        tier: ${TYPE}\n"
STRING2FILE="${REPLICASET_HEADER}${METADATA}${SPEC}"
echo -e "${STRING2FILE}" > ${YAML_FILE_NAME}
cat ${YAML_FILE_NAME}
  • Create, then query replicaset and pods:
$ kubectl get replicaset
$ kubectl create -f ${YAML_FILE_NAME}
$ kubectl get replicaset
$ kubectl get pods

  • Labels vs Selectors

    • replicationController will use label
    • replicaSet requires explicit selector, which points to element named tier with value = 'front-end'
    • Selectors in replicaset-definition.yml file reference labels in order to filter pods created via pod-definition.yml file.
    • Spec template supports corrections of pod failures, so even if no more deployments, we still want the spec template.

  • ReplicaSet scaling

git status
sed -i 's/' 3'/' 6'/g' ${YAML_FILE_NAME} > ${YAML_FILE_NAME}
git status; git add . ; git commit -m "Updated replicas from 3 to 6 in ${YAML_FILE_NAME}."
kubectl get replicaset
kubectl replace -f ${YAML_FILE_NAME}
kubectl get replicaset
# alt: $ kubectl scale --replicas=6 ${YAML_FILE_NAME} # Assumes yaml file has been updated.
# alt: $ KIND_LC=$(echo "${KIND}" | tr '[:upper:]' '[:lower:]'); kubectl scale --replicas=6 ${KIND_LC} ${INSTANCE_NAME}
  • Command Review: ReplicaSet
kubectl create -f replicaset.yml
kubectl get replicaset
kubectl delete replicas ${INSTANCE_NAME} # deletes all underlying pods
kubectl replace -f ${YAML_FILE_NAME} # assumes file has been modified
# CLI Scale: kubectl scale replicas=6 ${YAML_FILE_NAME} # no file mods
  • Other commonly used kubectl statements can be found here
  • Lecture 29: Practice Test - ReplicaSets
  • Lecture 30: Practice Test - ReplicaSets - Solution (Optional)
$ kubectl get namespace
$ kubectl get pods
$ kubectl -A get pods
$ kubectl describe pod ${POD_NAME} # see events of type warning for exact error
  • replicaset version was wrong: $ kubectl explain replicaset
$ kubectl create -f ${YAML_FILE_NAME}
$ cat ${YAML_FILE_NAME}
$ sed -i 's/' v1'/' apps/v1'/g' ${YAML_FILE_NAME} > ${YAML_FILE_NAME} # git
$ kubectl create -f ${YAML_FILE_NAME}
$ kubectl get replicaset
$ YAML_FILE_NAME_2='replicaset-definition-2.yaml'
$ kubectl create -f ${YAML_FILE_NAME_2} # error: selector does not match template labels
$ cat ${YAML_FILE_NAME_2} # selector tier and template tier values must match
$ sed -i 's/' frontend'/' nginx'/g' ${YAML_FILE_NAME_2} > ${YAML_FILE_NAME_2} 
$ cat ${YAML_FILE_NAME_2}
$ kubectl create -f ${YAML_FILE_NAME_2}
# Delete replicasets explicitly by name.
$ kubectl get replicaset
$ REPLICASETS=$( kubectl get replicaset | awk 'NR!=1' '{ print $1 }' )
$ for RSET in "${REPLICASETS[@]}"; do kubectl delete replicaset $RSET ; done
$ kubectl get replicaset
  • Fix original replicaset error with correct busybox image name.
kubectl get rs
kubectl apply -f ${YAML_FILE_NAME_0}
kubectl edit rs new-replica-set
# sed -i 's/' busybox777'/' busybox'/g' ${YAML_FILE_NAME_0} > ${YAML_FILE_NAME_0}
kubectl get pods
kubectl apply -f ${YAML_FILE_NAME_0}
kubectl get pods
PODS=$( kubectl get pods | awk 'NR!=1' '{ print $1 }' )
kubectl delete pod "${PODS}"
kubectl get pods
kubectl get rs. # READY = 4
  • Scale to 5 pods
kubectl scale rs new-replica-set --replicas=5 ; kubectl get pods
# OR
kubectl edit rs new-replica-set # set to 5
  • Lecture 31: Deployments
    • Web server scaled out across cluster.
    • Update docker file in registry => test before cluster update.
    • Deployments allow for code changes to be updated.
DEPLOY_SPEC_FILE='deployment-definition.yml'
REPSET_FILE_TPL='replicaset-definition.yml.tpl'
KIND=Deployment # Override single element.
kubectl get deployments
kubectl get pods
kubectl get rs
kubectl get all
kubectl get services
kubectl create -f ${DEPLOY_SPEC_FILE} 
kubectl get deployments
  • Lecture 32: Certification Tip!

    • As you might have seen already, it is a bit difficult to create and edit YAML files. Especially in the CLI. During the exam, you might find it difficult to copy and paste YAML files from browser to terminal. Using the kubectl run command can help in generating a YAML template. And sometimes, you can even get away with just the kubectl run command without having to create a YAML file at all. For example, if you were asked to create a pod or deployment with specific name and image you can simply run the kubectl run command.

    • Use the below set of commands and try the previous practice tests again, but this time try to use the below commands instead of YAML files. Try to use these as much as you can going forward in all exercises

    • Command Reference

      • Create an NGINX Pod: $ kubectl run nginx --image=nginx

      • Generate POD Manifest YAML file (-o yaml). Don't create it(--dry-run)

kubectl run nginx --image=nginx --dry-run=client -o yaml

- Create a deployment: ``` $ kubectl create deployment --image=nginx nginx ```
- Generate Deployment YAML file (-o yaml). Don't create it(--dry-run)

kubectl create deployment --image=nginx nginx --dry-run=client -o yaml

- Generate Deployment YAML file (-o yaml). Don't create it(--dry-run) with 4 Replicas (--replicas=4)

kubectl create deployment --image=nginx nginx --dry-run=client -o yaml > nginx-deployment.yaml

- Save it to a file, make necessary changes to the file (for example, adding more replicas) and then create the deployment.

kubectl create -f nginx-deployment.yaml

OR

In k8s version 1.19+, we can specify the --replicas option to create a deployment with 4 replicas.

kubectl create deployment --image=nginx nginx --replicas=4 --dry-run=client -o yaml > nginx-deployment.yaml
  • Lecture 34: Solution - Deployments (optional)

  • Lecture 35: Services

    • Back end and front end pods connect via services.
    • Connect application tiers and allow user access via services.

  • Example web access via ssh into node:

    • Internal cluster communication network: 10.244.0.x
    • Internal pod ip: 10.244.0.2
    • External node network: 192.168.1.x
    • External node ip communication: 192.168.1.2
    • External client communication: 192.168.1.10 => ssh into node => $ curl http://10.244.0.2 # Hello World!
    • We want the equivalent, but without ssh. Something must help us map requests from external to internal network.
    • That is where a SERVICE listens at a given NODE IP port to route requests to the 10.244.0.x network.
    • $ curl http://192.168.1.2:30008 # Hello World!
    • This is called a 'NodePort service'.

  • Service Types:

    • NodePort
    • ClusterIP
    • LoadBalancer

  • NodePort Service Relies Upon Three Ports

    • TargetPort = 10.244.0.2:80 = pod ip:port
    • ServicePort = 10.106.1.12:80 = Cluster IP of service
    • NodePort @ listener, range = 30000 - 32767
SERVICE_DEF_FILE_NAME='service-definition.yml'
FILE_HEADER="apiVersion: v1\nkind: Service\nmetadata:\n.   name: myapp-service\n\nspec:\n"
SPEC="    type: NodePort\n    ports:\n     - targetPort: 80\n       port: 80\n       nodePort: 30008\n"
SELECTOR="    selector:\n       app: myapp\n       type: front-end\n"
echo -e "${FILE_HEADER}${SPEC}" > ${SERVICE_DEF_FILE_NAME}
# From 192.168.1.1: $ curl http://192.168.1.2:30008
kubectl create -f ${SERVICE_DEF_FILE_NAME}
kubectl get services
# From 192.168.1.1: $ curl http://192.168.1.2:30008 # This should work on any cluster node, provided correct node ips.
  • HA and LB: Selectors Target Labels
  • Lecture 36: Services Cluster IP
    • ClusterIP architectures support microservice-per-tier so that scaling does not change service architectures.
    • Each service gets network name and virtual ip address.
--- service-definition-02.yml
apiVersion: v1
kind: Service
metadata:
    name: back-end

spec:
    type: ClusterIP
    ports:
     - targetPort: 80
       port: 80

    selector:
       app: myapp
       type: back-end
---
kubectl create -f service-definition-02.yml
kubectl get services
  • Lecture 37: Services - Loadbalancer

    • voting-app => 2 dedicated nodes for several node pods 192.168.56.{70,71} => service @ 30035

    • result-app => 2 dedicated nodes for several node pods 192.168.56.{72,73} => service @ 31061

    • Now we want to map single url to two paths per service. Four exist, but only 2 will be used.

    • Load balancers from cloud providers can be implemented. Also can be done locally with vbox.

--- service-definition-lb.yml
apiVersion: v1
kind: Service
metadata:
    name: myapp-service

spec:
    type: LoadBalancer
   ports:
     - targetPort: 80
       port: 80

    selector:
       app: myapp
       type: back-end
---
  • Lecture 39: Solution - Services (optional)
    • Endpoints map from service to underlying pods, as pod selectors target labels.
    • If service selector and target labels dont align, then service will fail.
    • K8s Service Docs
  • Lecture 40: Namespaces
    • Namespaces support aggregations of services, and isolation between services.
    • The namespaces k8s creates at startup: kube-system + default + kube-public
    • Each namespace gets its own resource limits and policies.
    • Service names are supported by local dns service.
    • Namespaces can aggregate or segregate services.
    • Aggregated services can be called by short name: mysql.connect("db-service")
    • Segregated services call each other by long name for specificity: mysql.connect("db-service.dev.svc.cluster.local")
      • service name: db-service
      • namespace: dev
      • service: svc
      • domain: cluster.local
kubectl get namespaces -o wide
kubectl get pods -A
kubectl get pods --namespace=kube-system -o wide
kubectl get pods --namespace=default -o wide
kubectl create -f pod-definition.yml --namespace=dev # Alt add 'namespace: dev' to pod-definition.yml
kubectl create namespace dev-ns
kubectl create deployment --image=redis redis-deploy --replicas=2 --namespace=dev-ns
kubectl get pods --namespace=dev-ns -o wide

--- namespace-dev.yml
apiVersion: v1
kind: Namespace
metadata:
    name: dev
---
kubectl create -f namespace-dev.yml # Alt: $ kubetl create namespace dev
kubectl get namespaces
  • Set a namespace to be user default:
$ kubectl get pods
$ kubectl get pods --namespace=dev
$ kubectl config set-context $(kubectl config current-context) --namespace=dev
$ kubectl get pods
$ kubectl get pods --namespace=default
  • Create resource quota for a namespace.
---compute-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
    name: compute-quote
    namespace: dev
spec:
  hard:
    pods: "10"
    requests.cpu: "4"
    requests.memory: 5Gi
    limits.cpu: "10"
    limits.memory: 10Gi
---
kubectl create -f compute-quota.yaml
  • Lecture 41: Practice Test - Namespaces
    • Practice Test Link
      • $ NS_COUNT=$( kubectl get namespaces --no-headers | sed -n '$=' )
      • $ kubectl run redis --image=redis -n=finance
      • $ kubectl get pods -A | grep blue
      • The practice test provides a web ui for testing service host and port connectivity.

  • Lecture 42: Solution - Namespaces (optional)

  • Lecture 43: Imperative vs Declarative

    • K8s Object Management Methods
    • Imperatively Managed Infrastructure: Instructions How To Deliver Service (what + how)
    • Declaratively Managed Infrastructure: Request For New/Change Service OR Check Exists

  • Imperative Methods:

# create objects
kubectl run --image=nginx nginx
kubectl create deployment --image=nginx nginx
kubectl expose deployment nginx --port 80

# update objects
kubectl edit deployment nginx # edits copy of running configs => NOT The same as config file!!!!
kubectl scale deployment nginx replicas=5
kubectl set image deployment nginx nginx=nginx:1.18

# use manifests to support imperative
kubectl create -f nginx.yaml # assumes pod of that name does not exist
kubectl replace -f nginx.yaml # fails if object does not exist
kubectl delete -f nginx.yaml # fails if object does not exist
# change management become increasingly impossible

Declarative Methods:

# apply command is smarter than create, as it expects single yaml file to reflect state reqs.
kubectl apply -f nginx.yaml
# BATCH: $ kubectl apply -f /path/to/config/files/
# Use imperative calls for simple one-offs.  Use declarative apply call for complex integrations.

  • Lecture 44: Certification Tips

    • Imperative Commands with Kubectl
    • While you would be working mostly the declarative way - using definition files, imperative commands can help in getting one time tasks done quickly, as well as generate a definition template easily. This would help save considerable amount of time during your exams.

  • Before we begin, familiarize with the two options that can come in handy while working with the below commands:

    --dry-run: By default as soon as the command is run, the resource will be created. If you simply want to test your command , use the --dry-run=client option. This will not create the resource, instead, tell you whether the resource can be created and if your command is right.

    -o yaml: This will output the resource definition in YAML format on screen.

  • Use the above two in combination to generate a resource definition file quickly, that you can then modify and create resources as required, instead of creating the files from scratch.

  • Create an NGINX Pod: $ kubectl run nginx --image=nginx

    • Generate POD Manifest YAML file (-o yaml). Don't create it(--dry-run)
$ kubectl run nginx --image=nginx --dry-run=client -o yaml

$ kubectl run custom-nginx --image=nginx --port=8080 --dry-run=client -o yaml
  • Create a deployment: $ kubectl create deployment --image=nginx nginx
    • Generate Deployment YAML file (-o yaml). Don't create it(--dry-run)
$ kubectl create deployment --image=nginx nginx --dry-run=client -o yaml
  • Generate Deployment with 4 Replicas
$ kubectl create deployment nginx --image=nginx --replicas=4

$ kubectl create deployment --image=kodekloud/webapp-color  webapp --replicas=3 --dry-run=client \
  -o yaml  > deployment-kloud-image.yml
  • You can also scale a deployment using the kubectl scale command.
$ kubectl scale deployment nginx --replicas=4
  • Another way to do this is to save the YAML definition to a file and modify
$ kubectl create deployment nginx --image=nginx --dry-run=client -o yaml > nginx-deployment.yaml

  • You can then update the YAML file with the replicas or any other field before creating the deployment.

  • Create a Service named redis-service of type ClusterIP to expose pod redis on port 6379

kubectl expose pod redis --port=6379 --name redis-service --dry-run=client -o yaml
  • This will automatically use the pod's labels as selectors, OR
kubectl create service clusterip redis --tcp=6379:6379 --dry-run=client -o yaml

  • This will not use the pods labels as selectors, instead it will assume selectors as app=redis. You cannot pass in selectors as an option. So it does not work very well if your pod has a different label set. So generate the file and modify the selectors before creating the service.)

  • Create a Service named nginx of type NodePort to expose pod nginx's port 80 on port 30080 on the nodes:

kubectl expose pod nginx --type=NodePort --port=80 --name=nginx-service --dry-run=client -o yaml
  • This will automatically use the pod's labels as selectors, but you cannot specify the node port. You have to generate a definition file and then add the node port in manually before creating the service with the pod, OR
kubectl create service nodeport nginx --tcp=80:80 --node-port=30080 --dry-run=client -o yaml

  • This will not use the pods labels as selectors, but it does allow for designating node-port

  • Both the above commands have their own challenges. While one of it cannot accept a selector the other cannot accept a node port. I would recommend going with the kubectl expose command. If you need to specify a node port, generate a definition file using the same command and manually input the nodeport before creating the service.

  • Reference:

kubectl run redis --image=redis:alpine --labels="tier=db"--dry-run=client -oyaml > redis-pod.yaml
cat  redis-pod.yaml # verify label set
kubectl apply -f  redis-pod.yaml

kubectl create deployment --image=kodekloud/webapp-color  webapp --replicas=3 --dry-run=client -o yaml > deployment-kloud-image.yml

kubectl expose pod httpd --port=80 --name httpd --dry-run=client -o yaml > expose-httpd-svc.yml
kubectl apply -f expose-httpd-svc.yml
kubectl get services
kubectl describe service redis-service # check lables + endpoint port
  • faster method
kubectl run httpd --image=httpd:alpine --port=80 --expose=true
kubectl get pods
kubectl get svc
kubectl describe svc httpd
  • Lecture 46: Solution - Imperative Commands (optional)
    • $ kubectl run --help
  • Lecture 47: Kubectl Apply Command
    • Try catch features allow us to use files to drive the object states.
    • Apply command supports config state tracking via: local file configs + last applied configs + live object configs
    • yaml => json => compare to last applied config => compare to k8s system state
    • Merging State Changes
    • Only the apply command tracks last applied configurations in annotations section.
    • Use apply workflow and stick to it.
  • Lecture 48: Here's some inspiration to keep going
  • Section 3: Scheduling (124m)
  • Lecture 50: Download Presentation Deck for this section
  • Lecture 51: Manual Scheduling
    • pod-definition.yaml file gets nodeName field appended automatically.
    • scheduler will assign pod to node in that field, but what about without a scheduler?
    • kubectl get pods -n kube-system | grep sched
    • manual appending of nodeName field and value in pod-definition.yaml
    • To mimic scheduler when nodes are already assigned, we can also use pod-bind-definition.yaml file to send bind request.
    • Next we convert that yaml file to json and reference it in a curl call:
$ curl --header "Content-Type:application/json" --request POST --data pod-bind-definition.json http://$SERVER/api/v1/namespaces/default/pods/$PODNAME/binding/

$ kubectl replace --force -f nginx.yaml
$ kubectl get pods --watch

  • Lecture 53: Solution - Manual Scheduling (optional)

  • Lecture 54: Labels and Selectors

    • Labels are standard grouping of pods and pod sets for filtering.
    • Labels are key:value pairs.
    • Selectors target label:value matching to filter.
    • K8s objects need to be grouped by app or type or tier or env or etc.
      • $ kubectl get pods --selector tier=dev
      • Labels can be hierarchical or divisional or ??

  • Sometimes we will have multiple label sets in a definiton file, but:

    • top labels are for replicaset
    • bottom labels in templates are for pods

  • Service definitions depend upon replicaset definitions.

    • Annotations record other key vaue pairs.
  • Lecture 55: Practice Test - Labels and Selectors
    • Labels + Selectors Practice Test Link
      • $ kubectl get pods --selector env=prod --no-headers | wc -l
      • $ kubectl get all --selector env=prod --no-headers | wc -l
      • $ kubectl get all --selector env=prod,bu=finance,tier=frontend
  • Lecture 56: Solution : Labels and Selectors : (Optional)

  • Lecture 57: Taints and Tolerations

    • Tainted nodes are usually avoided (node avoidance), but pod tolerance can create or increase node affinity.
    • Nodes 1,2,3 and pods A,B,C,D
    • We taint Node 1 with taint named blue, and now all pods will reject assgnment to Node 1.
    • Pods A+B+C will reject assignment to node 1, and will be assigned to other available nodes.
    • If we add a toleration to pod D for blue taints, then pod D will NOT reject attempt to assign it to the first node.

  • Taints are for nodes: taint-effect options = [ NoSchedule PreferNoSchedule NoExecute ] $ kubectl taint nodes node-name key=value:taint-effect

  • Tolerances are for pods

  tolerations:
  - key: "app"
  operator: "Equal"
  value: "blue"
  effect: "NoSchedule"
  • The result is either pod eviction, else simply no further scheduling on that node for that pod.
  • NoExecute would evict a pod that was already running on a given node.
  • Master node is tainted by default: $ kubectl describe node kubemaster | grep Taint
  • To untaint: ``` $ kubectl taint nodes controlplane node-role.kubernetes.io/master:NoSchedule-
  • Lecture 59: Solution - Taints and Tolerations (Optional)
  • Lecture 60: Node Selectors
    • This is easier method for limiting pod to run on a particular node.
    • nodeSelector:\n size: Large # assumes node is labeled accordingly: size=large
    • Labeling nodes: "$ kubectl label nodes ="
    • $ kubectl label nodes node-1 size=Large
    • If other more complex logic is required, like not small or lg+med... we use node (anti-)affinity.

  • Lecture 61: Node Affinity

    • This ensures pods are hosted on partcular sets of nodes.
    • nodeSelector option is simple.
    • affinity section of pod spec is relatively verbose, but it supports complex matches
    • Operator choices: Exists + NotIn + In + Equal (?? Docs)

  • Node Affinity Types

    • Available => during scheduling (required OR preferred) + ignored during execution
    • Planned => requiredDuringSchedulingRequiredDuringExecution + required during execution
  • Lecture 63: Solution - Node Affinity (Optional)
  • Lecture 64: Taints and Tolerations vs Node Affinity
    • Here we have 3 pods which we would like to run on respectively labeled three nodes.
    • However, we want limit our pod deployments to ONLY the correspondingly labeled nodes.
    • We can combine taints +tolerations with node affinity to achieve desired results.

  • Lecture 65: Resource Requirements and Limits

    • Three node sample cluster, each node with: [ cpu • memory • disk ]
    • Scheduler distributes resource load across nodes as per most available resources.
    • Min hardware requests for running container: 0.5 cpu + 256Mi mem
    • We can specify those resources for pod in spec:resources:requests section.
    • 1 cpu = 1 hyperthread, etc

  • memory resource request units

1 G (Gigabyte) = 1,000,000,000 bytes
1 M (Megabytes) = 1,000,000 bytes
1 K (Kilobytes) = 1,000 bytes

1 Gi (Gibibyte) = 1,073,741,824 bytes
1 Mi (Mebibyte) = 1.048,576 bytes
1 Ki (kebibyte) = 1,024 bytes
  • Note on default resource requirements and limits
    • In the previous lecture, I said - "When a pod is created the containers are assigned a default CPU request of .5 and memory of 256Mi". For the POD to pick up those defaults you must have first set those as default values for request and limit by creating a LimitRange in that namespace.
apiVersion: v1
kind: LimitRange
metadata:
  name: mem-limit-range
spec:
  limits:
  - default:
      memory: 512Mi
    defaultRequest:
      memory: 256Mi
    type: Container

https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/

apiVersion: v1
kind: LimitRange
metadata:
  name: cpu-limit-range
spec:
  limits:
  - default:
      cpu: 1
    defaultRequest:
      cpu: 0.5
    type: Container

https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/cpu-default-namespace/ https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource

  • Default cpu limits on k8s = 1vcpu per container.
  • Default memory limits on k8s = 512 Mi per container.
    resources:
      requests:
        memory: "1Gi"
        cpu: 1
      limits:
        memory: "2Gi"
        cpu: 2
  • CPU is throttled, but container can consume more memory than its limit, until it is killed.

  • Lecture 66: Note on default resource requirements and limits

  • Lecture 67: A quick note on editing PODs and Deployments A quick note on editing PODs and Deployments Remember, you CANNOT edit specifications of an existing POD other than the below.

  • spec.containers[*].image

  • spec.initContainers[*].image

  • spec.activeDeadlineSeconds

  • spec.tolerations

  • For example you cannot edit the environment variables, service accounts, resource limits (all of which we will discuss later) of a running pod. But if you really want to, you have 2 options:

      1. Run the kubectl edit pod command. This will open the pod specification in an editor (vi editor). Then edit the required properties. When you try to save it, you will be denied. This is because you are attempting to edit a field on the pod that is not editable.

    • A copy of the file with your changes is saved in a temporary location as shown above.

    • You can then delete the existing pod by running the command:

    • kubectl delete pod webapp

    • Then create a new pod with your changes using the temporary file

    • kubectl create -f /tmp/kubectl-edit-ccvrq.yaml

      1. The second option is to extract the pod definition in YAML format to a file using the command

    • kubectl get pod webapp -o yaml > my-new-pod.yaml

    • Then make the changes to the exported file using an editor (vi editor). Save the changes

    • vi my-new-pod.yaml

    • Then delete the existing pod: kubectl delete pod webapp

    • Then create a new pod with the edited file: kubectl create -f my-new-pod.yaml

    • Edit Deployments

      • With Deployments you can easily edit any field/property of the POD template. Since the pod template is a child of the deployment specification, with every change the deployment will automatically delete and create a new pod with the new changes. So if you are asked to edit a property of a POD part of a deployment you may do that simply by running the command
kubectl edit deployment my-deployment
  • Lecture 69: Solution: Resource Limits : (Optional)
    • $ kubectl get po elephant -o yaml > elephant.yaml ; kubectl delete pod elephant
    • $ kubectl edit pod elephant
    • $ kubectl replace --force -f /tmp/kubectl-edit-234567.yaml
  • Lecture 70: DaemonSets
    • DaemonSets configure pods across nodes. Min 1 pod per node in set.
    • Use cases include monitoring solution / log viewer.
    • kube-proxy can be deployed as daemonset since it is a one per node setup.
    • Networking agents also can be required to run in min 1 pod per node.
    • DaemonSet creation is like ReplicaSet creation in terms of spec structure.
    • Kind: DaemonSet => Ensure that selectors match labels in pod template.
    • kubectl create deployment elasticsearch --image=k8s.gcr.io/fluentd-elasticsearch:1.20 -n kube-system --dry-run=client -o yaml > fluentd.yaml # remove replicas and change type
    • kubectl get daemonsets
    • kubectl describe daemonset xyz
    • NodeAffinity and default scheduler are used since v 1.12
  • Lecture 71: Practice Test - DaemonSets

  • Lecture 72: Solution - DaemonSets (optional)

  • Lecture 73: Static Pods

    • Even a single solo worker can run pods, but they must be configured in local dir: /etc/Kubernetes/manifests

    • Pods will be created and sustained despite crashes. These are called static pods, managed by kubelet.

    • kubelet will first run static pods, mirroroed by k8s api, and then take requests from k8s api.

    • Two config options:

      • kubelet.service: --pod-manifest-path=/etc/Kubernetes/manifests => kubelet service config
      • or --config=kubeconfig.yaml => kubeconfig.yaml: staticPodPath: /etc/kubernetes/manifests
      • View running pods: $ docker ps
      • Static pod use cases do not depend upon k8s api.
        • Controller-manager + apiserve + etcd can all be deployed this way.

  • Both of the following are ignored by the kube-scheduler.

    • Static pods: Created by kubelet.
    • DaemonSets: Created by Kube-API server daemonset controller. Deploy monitoring and logging agents on the nodes.
  • Lecture 74: Practice Test - Static Pods 
kubectl run --restart=Never --image=busybox static-busybox --dry-run=client -o yaml --command -- sleep 1000 > /etc/kubernetes/manifests/static-busybox.yaml
  • Change image running in static container.
kubectl run --restart=Never --image=busybox:1.28.4 static-busybox --dry-run=client -o yaml --command -- sleep 1000 > /etc/kubernetes/manifests/static-busybox.yaml
  • Find and delete static pod on node:
kubectl get pods -A -o wide
ssh root@IP
cat /var/lib/kubelet/config.yaml | grep  static
mv /file/in/question/ ~ # this will stop that static pod activity on that node
exit
kubectl get pods -A --watch
  • Lecture 75: Solution - Static Pods (Optional)
    • Static pod yaml owner = Node.
--- kube-scheduler.service
ExecStart=/usr/local/bin/kube-scheduler \\
  --config=/etc/kubernetes/config/kube-scheduler.yaml \\
  --scheduler-name=default-scheduler

--- my-custom-scheduler.service
ExecStart=/usr/local/bin/kube-scheduler \\
  --config=/etc/kubernetes/config/kube-scheduler.yaml \\
  --scheduler-name=my-custom-scheduler
  • kubeadm tool deploys additional schedulers as a static pod in manifests folder.
  • We can copy /etc/kubernetes/manifests/kube-scheduler.yml and create our own static pod scheduler.
    • append below command section: schedulerName: my-scheduler
    • leader-elect option chooses leader when 2+ schdulers are running
      • set to false when we dont have multiple masters
      • with multiple masters: - --lock-object-name=my-custom-scheduler
  • Then we check to see that both scheduler pods are running: kubectl get pods --namespace=kube-system
  • Now we want to create a pod-definition.yaml with schedulerName: my-custom-scheduler
  • Verify new scheduler is at work using $ kubectl get events -A -owide
  • Lecture 77: Practice Test - Multiple Schedulers

    • Practice Test Link

    • kubectl get serviceaccount -n kube-system

    • kubectl get clusterrolebinding

    • kubectl create -n kube-system configmap my-scheduler-config --from-file=/root/my-scheduler-config.yaml

    • kubectl get configmaps -A

  • Lecture 78: Solution - Practice Test - Multiple Schedulers : (Optional)
  • Lecture 80: References
  • Section 4: Logging + Monitoring (14m)
  • Lecture 81: Logging and Monitoring Section Introduction
    • Monitor Cluster Components
    • Monitor Cluster Components Log
    • Monitor Applications
    • Application Logs
  • Lecture 82: Download Presentation Deck
    • pdf
  • Lecture 83: Monitor Cluster Components
    • Monitor of node and pod stats for analytics.
    • Metrics Server is in-memory solution without logs, so logging solution is still needed.
    • Kubulet cAdvisor sends stats to metrics server object.
    • minikube: $ minikube addons enable metrics-server
    • git clone https://github.com/kubernetes-incubator/metrics-serve ; cd metrics-serve
    • kubectl create -f . # let time elapse
    • kubectl top node
    • kubectl top pod
  • Lecture 84: Practice Test - Monitoring
    • Practice Test
    • Remember to let at least 5m elapse to allow collect and report time to the cluster monitor.
kubectl get --help
kubectl api-resources
kubectl get pods --help

  • Lecture 85: Solution: Monitor Cluster Components : (Optional)

  • Lecture 86: Managing Application Logs
  • Image = kodekloud/event-simulator => generates web server event log
  • $ docker run kodekloud/event-simulator ; docker ps # proc in fg is default
  • Run in bg: $ docker -d run kodekloud/event-simulator ; docker ps # docker logs -f 
--- event-simulator.yaml
apiVersion: v1
kind: Pod
metadata:
  name: event-simulator-pod
spec:
  containers:
  - name: event-simulator
    image: kodekloud/event-simulator
---
  • Kubernetes logs follow similar bg pattern:
$ kubectl create -f event-simulator.yaml
$ kubectl logs -f event-simulator-pod

# When pod supports 2+ containers, then we must specify log by container name
$ kubectl edit -f event-simulator.yaml # echo "  - name: image-processor\n    some-image=processor\n"
$ kubectl apply -f event-simulator.yaml
$ kubectl get pods -A --no-headers | wc -l
$ kubectl describe pod event-simulator-pod # Count 2 Containers?
$ kubectl logs -f event-simulator-pod event-simulator
  • Lecture 88: Solution: Logging : (Optional)

  • Section 5: App Life Cycle Mgt (100m)

  • Lecture 89: Application Lifecycle Management - Section Introduction

    • Rolling Updates and Rollbacks in Deployments
    • Configure Applications
    • Scale Applications
    • Self-Healing Applications
  • Lecture 90: Download Slide Deck
  • Lecture 91: Rolling Updates and Rollbacks
    • Rollout + Versioning => Revision 1 => 2
$ kubectl rollout status deployment/myapp-deployment # are we rolled out ok?
$ kubectl rollout history deployment/myapp-deployment # what was changed?
  • Rolling Updates is Default Deployment Strategy
kubectl create deployment deployTest01 -oyaml --dry-run=client > deployment-definition.yml
cat deployment-definition.yml | grep -i strategy # Recreate | RollingUpdate
kubectl apply -f deployment-definition.yml
kubectl get deployments
kubectl describe deployment xyz
  • deployment => replica set created => pods created and containers deployed => rollback
kubectl get resplicasets
kubectl rollout undo deployment/myapp-deployment
kubectl get resplicasets
  • Deployment command summary:
    • kubectl create -f deployment-definition.yml # CREATE
    • kubectl get deployments # GET
    • kubectl apply -f deployments-definition.yml # UPDATE
    • kubectl set image deployment/myapp-deployment nginx=nginx:1.9.1 # UPDATE
    • kubectl edit deployment frontend # edit using vi
    • kubectl rollout status deployment/myapp-deployment # STATUS
    • kubectl rollout history deployment/myapp-deployment # STATUS
    • kubectl rollout undo deployment/myapp-deployment # ROLLBACK
  • Lecture 92: Practice Test - Rolling Updates and Rollbacks
    • Practice Test Link
    • Replacing container version did not set both containers into action, as suggested. ??
  • Lecture 93: Solution: Rolling update : (Optional)
  • Lecture 94: Configure Applications
    • Configuring applications comprises of understanding the following concepts:
    • Configuring Command and Arguments on applications
    • Configuring Environment Variables
    • Configuring Secrets

  • Lecture 95: Commands

    • RUN # install
    • VOLUME # define mountable dirs
    • WORKDIR # define working dir
    • CMD # define default command

  • Terminal exits when bash STDIN listener finds no terminal input.

    • We could attach a terminal to a container, if desired, but that is not the default action.
    • $ docker run ubuntu sleep 5
    • $ CMD sleep 5 # or $ CMD ["sleep", "5"]
    • $ docker build -t ubuntu-sleeper . && docker run $_

  • We can override on CLI, but we want suntax that passes new sleep value alone:

$ docker run ubuntu-sleeper 10 # instead of $ docker run ubuntu-sleeper sleep 10

  • For this we use the ENTRYPOINT instruction: ENTRYPOINT ["sleep"]

  • CMD gets replaced entirely including base executable, yet ENTRYPOINT accepts value only.

  • We also want to configure 5 as default value which can be overriden as above:

  • For this we cobine both the ENTRYPOINT and CMD instructions ENTRYPOINT ["sleep"] CMD ["5"]

$ docker run ubuntu-sleeper
# OR
$ docker run ubuntu-sleeper 10
# OR override entrypoint itself
$ docker run --entrypoint sleep2.0 ubuntu-sleeper 10
  • Lecture 96: Commands and Arguments
$ docker run --name ubuntu-sleeper ubuntu-sleeper
$ docker run --name ubuntu-sleeper ubuntu-sleeper 10

--- k8s pod-definition.yml
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-sleeper-pod
spec:
  containers
    - name: ubuntu-sleeper
      image: ubuntu-sleeper
      command: ["sleep2.0"]   # corresponds to ENTRYPOINT field (!CMD in Dockerfile)
      args: ["10"]             # corresponds to CMD field
---
  • Lecture 98: Solution - Commands and Arguments (Optional)
  • Lecture 99: Configure Environment Variables in Applications
    • Three ways to set env values:
      • Plain Key Values: explicit
---
env:
  - name: APP_COLOR
    value: pink
---
  • ConfigMap => by reference valueFrom: configMapKeyRef:
  • Secrets => by reference valueFrom: secretKeyRef
  • Lecture 100: Configuring ConfigMaps in Applications
    • We can aggregate key-value env var pairs infor multiple pods into a ConfigMap spec.
kubectl get configmaps; kubectl describe configmaps

kubectl create configmap app-config --from-file=app_config.properties --dry-run=client -oyaml
kubectl create configmap app-config --from-file=app_config.properties --dry-run=client -oyaml > config-map.yml
vi config-map.yml
---
...
    envFrom:
      - configMapRef:
            name: app-config
---
kubectl create pod simple-webapp-color --dry-run=client -oyaml > pod-definition.yaml
vi pod-definition.yaml
  • ConfigMaps in pods
    • envFrom:
    • env:
    • volumes:
  • Lecture 102: Solution - Environment Variables (Optional)
    • kubectl create configmap --help # --from-literal
  • Lecture 103: Configure Secrets in Applications
    • If app.py includes database login host+username+password, then that needs to be configured as a SECRET.
    • This is again a configmap by reference, and a hash is used:
    • kubectl create secret generic --help # look for example using --from-literal
    • kubectl create secret generic db-secret --from-literal DBHOST=sql01 --from-literal DBUser=root --from-literal DB_Password=password123
    • kubectl get secrets
    • kubectl get pod webapp-pod -oyaml > webapp-pod.yaml ; vi webapp-pod.yaml # EnvFrom
    • kubectl replace --force -f webapp-pod.yaml
    • Check ui is now working....
set +o history
encode() { echo -n "${1}" | base64 ; }
decode() { echo -n "${1}" | base64 --decode ; }
DB_HOST_ENCODED=$(encode mysql); DB_USER_ENCODED=$(encode root); DB_PASS_ENCODED=$(encode 'mypass123');
APPEND_LINES_SECRET="data:\n  DB_Host: ${DB_HOST}\n  DB_User: ${DB_USER}\n  DB_Password: ${DB_PASS}\n"
kubectl create secret secret-xyz -oyaml > secret-data.yaml
echo -e "${APPEND_LINES}" >>  secret-data.yaml
kubectl create secret generic secret-xyz --from-file=secret-data.yaml # no -f ?
kubectl get secrets
kubectl describe secret secret-xyz
kubectl get secret secret-xyz -oyaml
kubectl create pod simple-webapp-color --dry-run=client -oyaml > pod-def.yaml
APPEND_LINES_POD="    envFrom:\n      - secretRef:\n         name: secret-xyz"
echo -e ${APPEND_LINES_POD} >> pod-def.yaml
kubectl create -f pod-def.yaml
set -o history
  • Above we see injecting secrets as environment variables into pods. (envFrom)
  • We can also inject secrets as single env vars.
  • We can also inject secrets as a volume.
    • If we were to mount the secret as a volume in a pod, each attribute becomes file with secret as contents.

  • Lecture 104: A note about Secrets!

    • A note about Secrets!
    • Remember that secrets encode data in base64 format. Anyone with the base64 encoded secret can easily decode it. As such the secrets can be considered as not very safe. The concept of safety of the Secrets is a bit confusing in Kubernetes. The kubernetes documentation page and a lot of blogs out there refer to secrets as a "safer option" to store sensitive data. They are safer than storing in plain text as they reduce the risk of accidentally exposing passwords and other sensitive data. In my opinion it's not the secret itself that is safe, it is the practices around it.

  • Secrets are not encrypted, so it is not safer in that sense. However, some best practices around using secrets make it safer. As in best practices like:

    • Not checking-in secret object definition files to source code repositories.
    • Enabling Encryption at Rest for Secrets so they are stored encrypted in ETCD.

  • Also the way kubernetes handles secrets. Such as:

    • A secret is only sent to a node if a pod on that node requires it.
    • Kubelet stores the secret into a tmpfs so that the secret is not written to disk storage.
    • Once the Pod that depends on the secret is deleted, kubelet will delete its local copy of the secret data as well.

  • Read about the protections and risks of using secrets. Having said that, there are other better ways of handling sensitive data like passwords in Kubernetes, such as using tools like Helm Secrets, HashiCorp Vault. I hope to make a lecture on these in the future.

  • Lecture 105: Practice Test - Secrets
    • Practice Test
    • We first create the secret. Then we configure the pod to use that secret.
  • Lecture 106: Solution - Secrets (Optional)
  • Lecture 107: Scale Applications
    • See Deployment + Rolling Updates + Rollback Sections
  • Lecture 108: Multi Container PODs
    • Web server and logging agent service per pod => created and managed together from net to storage thru lifecycle
    • add second container to running pod...
kubectl edit pod xyz
APPEND_STRING_LOG_AGENT="  - name: log-agent\n    image: log-agent"
echo "${APPEND_STRING_LOG_AGENT}" >> /tmp/45678.yaml
kubectl apply -f /tmp/45678.yaml --force
kubectl describe pod xyz

$ cat yellow.yaml 
---
apiVersion: v1
kind: Pod
metadata:
  name: yellow
spec:
  containers:
  - name: lemon
    image: busybox
    command:
      - sleep
      - "1000"
  - name: gold
    image: redis
---
$ kubectl create -f yellow.yaml
$ kubectl describe pod yellow
  • elastic-stack ns
kubectl get namespaces
kubectl describe ns elastic-stack
kubectl get pods --namespace=elastic-stack 
kubectl -n elastic-stack logs kibana # view logs
kubectl -n elastic-stack exec -it app -- cat /log/app.log # inspect pod log file
kubectl edit pod app --namespace=elastic-stack
vi /tmp/xyz.yml # add container to pod definition
kubectl replace -force -f /tmp/xyz.yml
  • Lecture 110: Solution - Multi-Container Pods (Optional)

    • kubectl edit pod app --namespace=elastic-stack

    • kubectl delete pod app --namespace

    • kubectl create -f pod-2-containers.yaml --namespace=elastic-stack

    • Kibana UI depends upon creation of index pattern

  • Lecture 111: Multi-container PODs Design Patterns
    • Multi-container PODs Design Patterns
  • There are 3 common patterns, when it comes to designing multi-container PODs. The first and what we just saw with the logging service example is known as a sidecar pattern. The others are the adapter and the ambassador pattern. But these fall under the CKAD curriculum and are not required for the CKA exam. So we will be discuss these in more detail in the CKAD course.
  • sidecar
  • ambassador
  • adapter

  • Lecture 112: InitContainers

    • InitContainers: In a multi-container pod, each container is expected to run a process that stays alive as long as the POD's lifecycle. For example in the multi-container pod that we talked about earlier that has a web application and logging agent, both the containers are expected to stay alive at all times. The process running in the log agent container is expected to stay alive as long as the web application is running. If any of them fails, the POD restarts.

    • But at times you may want to run a process that runs to completion in a container. For example a process that pulls a code or binary from a repository that will be used by the main web application. That is a task that will be run only one time when the pod is first created. Or a process that waits for an external service or database to be up before the actual application starts. That's where initContainers comes in.

  • An initContainer is configured in a pod like all other containers, except that it is specified inside a initContainers section, like this:

---
apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
spec:
  containers:
  - name: myapp-container
    image: busybox:1.28
    command: ['sh', '-c', 'echo The app is running! && sleep 3600']
  initContainers:
  - name: init-myservice
    image: busybox
    command: ['sh', '-c', 'git clone <some-repository-that-will-be-used-by-application> ; done;']
---

  • When a POD is first created the initContainer is run, and the process in the initContainer must run to a completion before the real container hosting the application starts. You can configure multiple such initContainers as well, like how we did for multi-pod containers. In that case each init container is run one at a time in sequential order.

  • If any of the initContainers fail to complete, Kubernetes restarts the Pod repeatedly until the Init Container succeeds.

---
apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
spec:
  containers:
  - name: myapp-container
    image: busybox:1.28
    command: ['sh', '-c', 'echo The app is running! && sleep 3600']
  initContainers:
  - name: init-myservice
    image: busybox:1.28
    command: ['sh', '-c', 'until nslookup myservice; do echo waiting for myservice; sleep 2; done;']
  - name: init-mydb
    image: busybox:1.28
    command: ['sh', '-c', 'until nslookup mydb; do echo waiting for mydb; sleep 2; done;']
---
  • Lecture 114: Solution - Init Containers (Optional)
kubectl get pods # hint: one pod with 2 containers
kubectl describe pod # describes all in default namespace
kubectl describe pod xyz | grep -in initc
kubectl edit pod red -oyaml
kubectl replace --force -f /tmp/kubectl-edit-???.yaml
...
kubectl logs orange
kubectl logs orange -c $INIT_CONTAINER_NAME # initContainer
kubectl edit pod orange -oyaml
kubectl replace --force -f /tmp/kubectl-edit-???.yaml
  • Lecture 115: Self Healing Applications

    • Kubernetes supports self-healing applications through ReplicaSets and Replication Controllers. The replication controller helps in ensuring that a POD is re-created automatically when the application within the POD crashes. It helps in ensuring enough replicas of the application are running at all times.

    • Kubernetes provides additional support to check the health of applications running within PODs and take necessary actions through Liveness and Readiness Probes. However these are not required for the CKA exam and as such they are not covered here. These are topics for the Certified Kubernetes Application Developers (CKAD) exam and are covered in the CKAD course.

  • Lecture 116: If you like it, Share it!
  • Section 6: Custer Maintenance (105m)
  • Lecture 117: Cluster Maintenance - Section Introduction
    • Cluster Upgrade Process
    • Operating System Upgrades
    • Backup + Restore Methodologies
  • Lecture 118: Download Presentation Deck
    • Cluster Maintenance pdf
  • Lecture 119: OS Upgrades
    • We need to safely drain node of workloads: $ kubectl drain node-1 --ignore-daemonsets
    • When it comes back online, we release those constraints: $ kubectl uncordon node-1
    • We can use cordon to prevent new pods from being scheduled on a node (no evictions): $ kubectl cordon node-1
  • Lecture 121: Solution - OS Upgrades (optional)
    • kubectl describe node controlplane | grep -i taint

  • Lecture 122: Kubernetes Software Versions

    • kubectl get nodes # VERSION column
    • Github Repo: k8s
    • Versioning: major + minor + patch(bugfixes) # Every few months produces new minor version releases with new features and functionalities.
    • First release 1.0: 07/2015
    • Alpha and beta releases are also available prior to minor release.
      • alpha => new features are disabled and buggy
      • beta => well-tested and enabled
      • main stable release

  • Downloaded package has several k8s api and control plane components, all at same release version:

    • kube-apiserver
    • Controller-manager
    • kube-scheduler
    • kubulet
    • kube-proxy
    • kubectl

  • Other controller components are external projects, and thereby set their own release streams:

  • Only last 3 minor k8s versions are supported.

  • SOME disparity in component versions is ok, BUT all must be versioned below kube-apiserver component!!

    • kube-apiserver = v1.x
    • Controller-manager + kube-scheduer = 1.(x-1)
    • kubelet + kube-proxy = 1.(x-2)
    • kubectl can be supported when ahead one minor version or behind one minor version.
    • upgrading one minor version at a time is recommended method (incremental updates vs leap updates vs clean install)
    • For a managed cloud service, this is a web ui option.
    • For kubeadm installations: $ kubeadm upgrade plan ; kubeadm upgrade apply

  • Lecture 124: Cluster Upgrade Process

  • Sample cluster:

    • k8s Master @ v1.10 => master is upgraded first => k8s Master @ v1.11
      • Which does NOT bring down workers, though no kubectl functions are available, including self-healing.
    • k8s Worker 01 @ v1.10 => rolling worker node upgrades follow after master is upgraded
    • k8s Worker 02 @ v1.10
    • k8s Worker 03 @ v1.10

  • It makes great sense to back up and clone a k8s cluster prior to attempting k8s cluster upgrades.

  • We could also add newer nodes with newer versions + migrate workloads between old and newer nodes.

  • First we review kubeadm upgrade plan # then we manually upgrade kubelets on each node

  • Upgrades start with upgrade to kubeadm binary:

    • $ kubeadm --version
    • $ apt-get upgrade -y kubeadm=1.12.0-00
    • $ kubeadm --version
    • $ kubeadm upgrade apply v1.12.0.
    • $ kubeadm --version

  • We end by upgrading kubelet code on worker nodes:

    • $ kubectl get nodes # Only shows nodes where kubelet is installed!! (usually NOT on master)
    • $ apt-get upgrade -y kubelet=1.12.0-00
    • $ systemctl restart kubelet
    • $ kubectl get nodes # Version should reflect master node kubelet upgrade only.
    • Now we can follow through with rolling worker kubelet upgrades.
    • $ kubectl get nodes # Version should now reflect upgrades on all workers too.

  • kubeadm upgrade process for each worker node:

kubectl drain node-1
apt-get upgrade -y kubeadm=1.12.0-00
kubeadm upgrade node config --kubelet-version v1.12.0
systemctl restart kubelet
kubectl uncordon node-1

  • Lecture 125: Demo - Cluster upgrade

    • Kubeadm cluster upgrade docs
    • $ kubectl get nodes
    • $ kubeadm token list
    • $ kubectl version
    • $ kubeadm version
    • $ kubectl get pods -A -owide
    • We will ONLY upgrade control plane and worker nodes.
    • $ watch kubectl get nodes
    • Verify OS linux platform specs: $ cat /etc/*release*/
    • Ubuntu: apt update ; apt-cache madison kubeadm | grep kubeadm
    • Good rule of thumb: Find latest patch to latest minor version.

  • First we upgrade control plane nodes.

    • $ kubadm version Verify GitVersion...
    • Then we upgrade worker nodes, and starting a rolling upgrade log is wise.
    • $ kubadm version | tee $FILE
    • $ sudo kubeadm upgrade plan | tee $FILE
    • $ sudo kubeadm upgrade v1.??.? | tee $FILE
    • $ sudo kubeadm upgrade plan | tee $FILE # No upgrade suggestions now?

  • Control plane kubectl + kubelet upgrades are next:

    • $ kubectl drain $CONTROL_PLANE_NODE --ignore-daemonsets
    • Copy commands from docs to upgrade kubelet and kubectl to proper minor version.
    • Append the following to watch and log output: CMDS | tee $FILE
    • Complete upgrade by reloading those procs from new binaries, then making control plane node schedulable:
      • kubectl get nodes # see VERSION column
      • kubectl version
      • sudo systemctl daemon-reload
      • sudo systemctl restart kubelet
      • kubectl uncordon $CONTROL_PLANE_NODE
      • kubectl get nodes # see VERSION column
      • kubectl version

  • Now we can move to worker node upgrades:

    • ssh into worker node: WORKER_NODE='node01'; ssh $WORKER_NODE ;

    • $ UTIMESTAMP=$( date +'%s'); HOST=$(hostname -s); FILE="k8s-upgrade-plan-${HOST}-${UTIMESTAMP}-worker-${WORKER_NODE}.log"

    • Node 01: upgrade kubeadm on worker nodes (copy/paste)

    • Node 01: $ sudo kubeadm upgrade node

    • ON CONTROL PLANE NODE: $ kubectl drain $WORKER --ignore-daemonsets

    • ON WORKER: Upgrade kubelet and kubectl on all worker nodes. (copy/paste)

    • ON WORKER:

      • $ kubelet --version
      • $ sudo systemctl daemon-reload; sudo systemctl restart kubelet
      • $ kubelet --version

    • ON CONTROL PLANE NODE:

      • $ kubectl get nodes
      • $ kubectl uncordon $WORKER_NODE
      • $ kubectl get nodes

  • Lecture 126: Practice Test - Cluster Upgrade

  • Control plane first: NODE_NAME='controlplane'

    • $ kubeadm version ; kubelet --version
    • $ apt update && apt install kubeadm=1.20.0-00 && kubeadm upgrade apply v1.20.0
    • $ apt install kubelet=1.20.0-00 && systemctl restart kubelet
    • $ kubeadm version ; kubelet --version
    • $ kubectl uncordon controlplane

  • Workers next, in rolling fashion: NODE_NAME='node01'

    • kubectl drain node01 --ignore-daemonsets
    • FROM MASTER: $ ssh node01; kubeadm --version; apt update; apt install kubeadm=1.20.0-00; kubeadm --version; exit
    • FROM MASTER: $ kubeadm upgrade node # This upgrades node01!!
    • FROM WORKER: apt install kubelet=1.20.0-00 && systemctl restart kubelet && kubelet --version

  • Lecture 127: Solution: Cluster Upgrade

  • Lecture 128: Backup and Restore Methods

  • Components to backup:

  • k8s resource config files

    • Use of kubectl apply -f supports complete resource file backup option in one-liner.
      • Github repo supports team management of those resource config file sets across clusters + environments.
      • Alternatively: kubectl get all --allnamespaces -oyaml > all-deploy-services.yaml

  • etcd cluster

    • Backup cluster via binary copy of data-dir directive: $ grep data-dir etcd.service
    • Alt backup via etcd backup command: $ ETCDCTL_API=3 etcdctl snapshot save snapshot.db # writes file to current dir
      • Check snapshot status: $ ETCDCTL_API=3 etcdctl snapshot status snapshot.db
      • Restore method:
        • $ service kube-apiserver stop
        • $ ETCDCTL_API=3 etcdctl snapshot restore snapshot.db --data-dir /var/lib/etcd-from-backup
--endpoints=https://[127.0.0.1]:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \


  - Configure etcd.service to use new restored data-dir
  - $ systemctl daemon-reload
  - $ service etcd restart
  - $ service kube-apiserver start
  • NOTE: for all etcdctl calls, append...
    • --endpoints=https://127.0.0.1:2379 --cacert=/etc/etcd/ca.cert --cert=/etc/etcd/etc-server.crt
      --key=/etc/etcd/etc-server.key
etcdctl --endpoints=https://[127.0.0.1]:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/snapshot-pre-boot.db

etcdctl snapshot status /opt/snapshot-pre-boot.db

etcdctl snapshot restore --data-dir=/var/lib/etcd-from-backup /opt/snapshot-pre-boot.db
ls /var/lib/etcd-from-backup/

vi /etc/kubernetes/manifests/etcd.yaml # ∆ etcd-data.hostpath.path: /var/lib/etcd-from-backup
kubectl get pods -n kube-system --watch
kubectl describe pod etcd-controlplane -n kube-system | grep -i fail
# as needed: kubectl delete pod etcd-controlplane -n kube-system
kubectl get pods -n kube-system 
kubectl get deploy
kubectl get services
  • persistent volumes
    • Use

  • Lecture 129: Working with ETCDCTL

  • Working with ETCDCTL. etcdctl is a command line client for etcd.

  • In all our Kubernetes Hands-on labs, the ETCD key-value database is deployed as a static pod on the master. The version used is v3. To make use of etcdctl for tasks such as back up and restore, make sure that you set the ETCDCTL_API to 3. You can do this by exporting the variable ETCDCTL_API prior to using the etcdctl client. This can be done as follows:

  • On the Master Node:

etcdctl version
export ETCDCTL_API=3
etcdctl version

  • To see all the options for a specific sub-command, make use of the -h or --help flag.

  • For example, if you want to take a snapshot of etcd, use:

    • etcdctl snapshot save -h and keep a note of the mandatory global options.

  • Since our ETCD database is TLS-Enabled, the following options are mandatory:

    --cacert verify certificates of TLS-enabled secure servers using this CA bundle

    --cert identify secure client using this TLS certificate file

    --endpoints=[127.0.0.1:2379] This is the default as ETCD is running on master node and exposed on localhost 2379.

    --key identify secure client using this TLS key file

  • Similarly use the help option for snapshot restore to see all available options for restoring the backup.

    • etcdctl snapshot restore -h

  • For a detailed explanation on how to make use of the etcdctl command line tool and work with the -h flags, check out the solution video for the Backup and Restore Lab.

  • Lecture 130: Practice Test - Backup and Restore Methods
    • Backup + Restore Practice Lab
    • kubectl describe pod etcd-controlplane -n kube-system | grep -i image
    • kubectl logs etcd-controlplane -n kube-system
    • kubectl describe pod etcd-controlplane -n kube-system | grep -i listen
    • kubectl describe pod etcd-controlplane -n kube-system | grep -i cert
    • kubectl describe pod etcd-controlplane -n kube-system | grep -i path
    • cat /etc/kubernetes/manifests/etcd.yaml
    • export ETCDCTL_API=3 # proven with help support

  • Lecture 131: Solution - Backup and Restore

  • Lecture 132: Practice Test Backup and Restore Methods 2
    • Practice Test Link
    • Multiple k8s clusters at play in this lab: $ kubectl config get-contexts
    • ∆ClusterContext: kubectl config use-context cluster1
    • kubectl describe pod -n=kube-system kube-apiserver-cluster2-controlplane | grep -i etcd # see if externally referenced service
  • Lecture 133: Certification Exam Tip!
    • Here's a quick tip. In the exam, you won't know if what you did is correct or not as in the practice tests in this course. You must verify your work yourself. For example, if the question is to create a pod with a specific image, you must run the the kubectl describe pod command to verify the pod is created with the correct name and correct image.

  • Section 7: Security (218m)

  • Lecture 135: Security - Section Introduction

    • k8s security primitives
    • secure persistent key-value store
    • authentication
    • authorization
    • security contexts
    • TSL certificates for cluster components
    • images securely
    • network policies

  • Sops is a proven way to handle secrets securely and scalably

  • Lecture 136: Download Presentation Deck

  • Lecture 137: Kubernetes Security Primitives

    • We assume hosts are secure, ie, ssh key auth only.
    • First line of k8s security defense: kube-apiserver
    • who can access => authorizations
    • what can they do? => authentications

  • Authentication methods:

    • files - username and passwords
    • files - username and tokens
    • certificates
    • ext auth services like ldap
    • service accounts
    • admin-only accounts

  • authorization

    • RBAC authorization
    • ABAC authorization
    • Node authorization
    • Webhook Mode

  • tls certificates are used to secure commuication between:

    • kube-apiserver
    • etcd cluster
    • kube scheduler
    • kube controller manager
    • kube proxy
    • kubelet

  • Network policies allow pods to intercommunicate across cluster nodes.

  • Lecture 138: Authentication

  • User types:

    • admins
    • devs
    • end users => handled by application itself
    • 3rd party clients + service accounts => $ kubectl create serviceaccount sa1

  • Admin and dev requests hit same target: kube-apiserver

    • kubectl
    • curl

  • Mechanisms of authentication

    • static password file
    • static token file
    • certificates
    • 3rd party identity services

  • Auth Mechanisms - basic

    • user-details.csv: password,login,uid
    • vi /etc/kubernetes/manifests/kube-apiserver.yaml # - command:\n - --basic-auth-file=user-details.csv
    • test login: $ curl -v -k https://master-node-ip:6443/api/v1/pods -u "User1:Password1"
    • we can add groups to user-detais.csv: password,login,uid,gid

  • Auth Mechanisms - static token file (not recommended)

  • for kubeadm setups, consider volume mount whle providing auth file

  • setup rbac for new users

  • Lecture 139: Article on Setting up Basic Authentication

    • Setup basic authentication on Kubernetes (Deprecated in 1.19)
    • Note: This is not recommended in a production environment. This is only for learning purposes. Also note that this approach is deprecated in Kubernetes version 1.19 and is no longer available in later releases

  • Follow the below instructions to configure basic authentication in a kubeadm setup.

  • Create a file with user details locally at /tmp/users/user-details.csv

# User File Contents
echo 'password123,user1,u0001
password123,user2,u0002
password123,user3,u0003
password123,user4,u0004
password123,user5,u0005' > /tmp/users/user-details.csv

  • Edit the kube-apiserver static pod configured by kubeadm to pass in the user details. The file is located at /etc/kubernetes/manifests/kube-apiserver.yaml

  • Modify the kube-apiserver startup options to include the basic-auth file

... containers:

  • command:

    • kube-apiserver
    • --authorization-mode=Node,RBAC
    • --basic-auth-file=/tmp/users/user-details.csv

  • Create the necessary roles and role bindings for these users:

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
 
---
# This role binding allows "jane" to read pods in the "default" namespace.
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: User
  name: user1 # Name is case sensitive
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role #this must be Role or ClusterRole
  name: pod-reader # this must match the name of the Role or ClusterRole you wish to bind to
  apiGroup: rbac.authorization.k8s.io
  • Lecture 141: TLS Introduction
    • Basic tls certs knowledge is fundamental to understand k8s.
    • What are TLS certificates?
    • How does kubernetes use certificates?
    • How to generate them?
    • How to configure them?
    • How to view them?
    • How to troubleshoot cert-related issues?

  • Lecture 142: TLS Basics

    • Encryption keys prevents plain text from being sent. Server needs key to decrypt.
    • Assymmetric encryption allows for private key and public lock.
      • On local host: $ ssh-keygen # creates both files ~/.ssh/{id_rsa,id_rsa.pub)
      • On remote host: $ cat ~/.ssh/authorized_keys # ssh-rsa $STRING $USERNAME

  • To prevent user ssh private key from being shared insecurely:

    • $ openssl genrsa -out my-bank.key 1024 # creates my-bank.key, used in next step
    • $ openssl rsa -in my-bank.key -pubout > mybank.pem
    • server public key is provided to https users => symmetric key
      • hacker has server public key and encrypted data, but not server private key
      • now we can send user public ssh keys thru that channel
      • now we have exchanged user ssh public key securely

  • hacker now hosts website and tries to get login info from redirected user login attempt

    • Certificate authorities are able to verify key certificate as authentic.
    • Cert issuer and alt names can be faked. Self-signed fake cert warning?
    • CA will sign and validate certificate signing request (CSR)
    • Hacker fake certs will fail CA verification.
    • Browsers all have all CA public keys to validate certs.
    • private sites require private CA services, either self-hosted or 3rd-party-hosted
    • server can request signed cert from valid ca from any client to verify

  • cert mgt = Public Key Infrastructure = PKI

  • private keys usually have the word "key" in the file name

  • public keys: *.crt *.pem

  • private keys: *.key *.key.pem

  • Lecture 143: TLS in Kubernetes

    • Three kinds of certificates:
      • root certificates are at certificate authorities
      • server certificates are generated by servers using openssl or similar tool
      • client certificates are generated from clients public keys

  • TLS Wikipedia Entry

  • Certificates guarantee two way trust between two hosts.

  • Outbound client data is encrypted and sent to target server. Server also needs private key.

    • Keeping that symmetric leaves us vulnerable to man-in-middle, solved by asymmetric encryption.
    • "private key : public lock" reduces ambiguity of components

  • K8s cluster nodes must be able to intercommunicate securely.

  • The same is true for all kube-apiserver components: all server communications must be 2-way secure.

  • Servers use server certificates:

    • CONTROL PLANE: kube-apiserver => apiserver.crt (cert) + apiserver.key (private key)
    • CONTROL PLANE: etcd server => etcdserver.crt + etcdserver.key
    • WORKERS: kubelet server => kubelet.crt + kubelet.key

  • Kube api server has clients with their own client certificates + private keys:

    • kubectl => admin.crt + admin.key
    • kube-scheduler => scheduler.crt + scheduler.key
    • kube-controller-manager => controller-manager.crt + controller-manager.key
    • kube-proxy => kube-proxy.crt + kube-proxy.key

  • Kube api server is also client to both etcd server and kubelet server on workers.

    • That could reuse existing kube-apiserver client certificate and key pair.
    • That could also use new client cert+key dedicated for this client-connection purpose:
      • apiserver-etcd-client.crt (cert)+ apiserver.key (private key)
      • apiserver-kubelet-client.crt (cert)+ apiserver-kubelet-client.key (private key)

  • We can have min one or several certs for a cluster.

  • Multiple certs can support multiple CAs.

  • Remember that CA root components = ca.crt + ca.key

  • .pem file background

  • Lecture 144: TLS in Kubernetes - Certificate Creation

    • We can have min one or several certs for a cluster.
    • This demo uses openssl, but others are available: easyrsa + cfssl

  • Server certs

# ca.key => ca.csr => ca.crt
openssl genrsa -out ca.key 2048 && \
  openssl req -new ca.key -subj "/CN=KUBERNETES-CA" -out ca.csr && \
    openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt
ls
  • Admin user client certs request CA signing...
# kube-admin-client.key => admin.csr => admin.crt
openssl genrsa -out kube-admin-client.key 2048 && \
  openssl req -new -key kube-admin-client.key -subj "/CN=kube-admin/O=system:masters" -out admin.csr && \
    openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -out admin.crt
ls

  • kube-scheduler + kube-controller-manager + kube-proxy => all must be prefixed with SYSTEM: in certificate.

  • apiserver and kubelet client certs will be created later

--- kube-config.yaml
apiVersion: v1
clusters:
- cluster:
    certificate-authority: ca.crt
    server: https://kube-apiserver:6443
  name: kubernetes
kind: Config
users:
- name: kubernetes-admin
  user:
    client-certificate: admin.crt
    client-key: admin.key
---

  • all clients will need CA root certificates like all browsers do...

    • client certs
    • server certs
    • ca certs

  • kube apiserver server certificate

  • kubeapiserver has many names and ip's and all must be explicitly written to kubeapi server certificate

openssl genrsa -out apiserver.key 2048
#  edit openssl.conf with all possible names to embed in cert, as well as ip numbers
--- openssl.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation,
subjectAltName = @alt_names
[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
IP.1 = 10.96.0.1
IP.2 = 172.17.0.87
---
# apiserver.csr => apiserver.crt + ca.crt => apiserver.crt
openssl req -new -key apiserver.key -subj "/CN=kube-apiserver" -out apiserver.csr --config openssl.conf
openssl x509 -req -in apiserver.csr -CA ca.crt -CAkey ca.key -out apiserver.crt
kubectl get pod kube-apiserver -n kube-system -oyaml | grep 'pem \| crt \| key'
# => group: SYSTEM:MASTERS

  • kubelet ??client certs and keys must be configured per worker node: kubelet-config.yaml

    • authentication:x509:clientCAFile: "/var/lib/kubernetes/ca.pem"
    • tlsCertFile: "/var/lib/kubelet/kubelet-node01.crt"
    • tlsPrivateKeyFile: "/var/lib/kubelet/kubelet-node01.key"

  • kubelet nodes client cert => group: SYSTEM:NODES

    • system:node:

  • If ETCD service has HA config with multiple peers, then those will also need client certs.

$ cat etcd.yaml | grep 'pem \| crt \| key'
  • Kubeapi server configs: grep 'etcd | client | tls | kubelet'
    • --etcd-{cafile,certfile,keyfile}
    • --client-ca-file
    • tls-{cert,private-key}-file
    • --kubelet

  • Lecture 145: View Certificate Details

  • Two methods:

    • Method 1: raw systemd config files: $ cat /etc/systemd/system/kube-apiserver.service
    • Method 2: kubeadm will auto-provision all certs: $ cat /etc/kubernetes/manifests/kube-apiserver.yaml

  • K8s Docs: Managing Cluster TLS

  • Certificate Details = Component + Type + Certificate Path + CN Name + ALT NAmes + Org + Issues + Expiration

  • Method 1: Reference kube-apiserver service startup configs.

    • Decode each cert: $ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout > apiserver-crt-decoded.txt
    • Parse decoded cert: $ grep 'Issuer | Subject | After: | DNS:' apiserver-crt-decoded.txt
    • Verify: cn names + alt names + ips + subject + org + issuer + expiration date
    • Best Certificate Practices

  • Check OS service logs with: $ journalctl -u etcd.service -l | grep -i 'embed | Fail | tls | error'

  • Method 2

    • kubeadm setups support kubectl logs: $ kubectl logs etcd-master | grep -i 'embed | Fail | tls | error'

  • When kubeapiserver or etcd server are down, we cannot use kubectl so we fall back to docker ps + logs:

$ kubectl get nodes # should fail with error
$ KUBEAPI_CONTAINER_ID=$( docker ps -a | grep kube-apiserver | cut -d' ' -f1 )
$ ETCD_CONTAINER_NAME=$( docker ps -a | grep etcd | cut -d' ' -f1 )
$ docker logs $KUBEAPI_CONTAINER_NAME | grep -i 'err \| fail'
$ docker logs $ETCD_CONTAINER_NAME | grep -i 'err \| fail'
  • Remember etcd server port = 2379
  • Reference correct crt file under /etc/kubernetes/pki/etcd/
  • Lecture 146: Resource: Download Kubernetes Certificate Health Check Spreadsheet

  • Lecture 147: Practice Test - View Certificates

    • Practice Test Link

    • openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text | grep -i subj

    • openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text | grep -i issuer # kubeapiserver server cert

    • openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -text | grep -i issues # etcd server cert

    • openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -text | grep -i subj # etcd server cert

    • openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -text | grep -i not # etcd server cert

    • openssl x509 -in /etc/kubernetes/pki/ca.crt -text | grep -i not # root ca

    • fix kubectl lost access to control plane: $ vi /etc/kubernetes/manifests/etcd.yaml

  • ETCD has its own CA, so this file needs an edit: /etc/kubernetes/manifests/kube-apiserver.yaml

    • check kubeapiserver: $ crictl ps -a | grep kube-apiserver ; crictl logs --tail=2 $POD_NAME
$ ls -al /etc/kubernetes/
$ tree -d /etc/kubernetes/pki/
$ ls /etc/kubernetes/manifests/
$ cat /etc/kubernetes/manifests/kube-apiserver.yaml
$ cat /etc/kubernetes/manifests/etcd.yaml

  • Lecture 148: Solution - View Certificate Details

    • Overall cluster physical components
    • Overall cluster logical components.
    • Overall cluster access.
    • Overall cluster health.
    • Server + CA (remote root ca) + Client Certificate validations.
    • Simplify all above prior to suggesting custom local CA service.
    • Add new admin
      • The default directory for SSH keys is ~/.ssh , private key named id_rsa , public key named id_rsa.pub
      • NEW ADMIN LOCAL: ssh-keygen => id_rsa.pub + id_rsa => certificate authority signing request (csr) => current admin
      • CURRENT ADMIN: csr => ca server private key + root cert => valid [private+public] key pair + overlapping expirations
      • CA: center of certificate security

  • Certificates API supports better flow of certificate management.

      1. Create Object: CertificateSigningRequest
      1. Request Reviews
      1. Approve Requests
      1. Share Signed Certs to Users

  • Lecture 149: Certificates API

      1. Create Object: CertificateSigningRequest
      1. Request Reviews
      1. Approve Requests
      1. Share Signed Certs to Users

  • New user creates their own CSR...

openssl genrsa -out jane.key 2048
openssl req -new -key jane.key -subj "/CN=jane" -out jane.csr # provide this file to current k8s admin
  • K8s admin creates yaml spec file for Jane's csr:
--- jane.csr.yml
apiVersion: certificates.k8s.io/v1beta1
kind: certificateSigningRequest
metadata:
  name: jane
spec:
  groups:
  - system:authenticated
  usages:
  - digital signature
  - key encipherment
  - server auth
  request:
    $( cat jane.csr | base64 | tr -d "\n" )
---
# NOTE: conroller-manager includes csr-approving + csr-signing
# $ cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep 'cluster-signing'
$ kubectl get csr
$ kubectl certificate approve jane
$ kubectl get csr
$ kubectl get csr jane -oyaml
$ echo "${CERT_STRING}" | base64 --decode # This is what we share with end user.
  • deny request
kubectl get csr agent-smith -o yaml
kubectl certificate deny agent-smith
kubectl delete csr agent-smith
  • Lecture 150: Practice Test - Certificates API 
--- akshay.yaml
apiVersion: certificates.k8s.io/v1
kind: certificateSigningRequest
metadata:
  name: akshay
spec:
  request: $( cat akshay.csr | base64 -w 0 )
  signerName: kubernetes.io/kube-apiserver-client
  usages:
  -- client auth
---
kubectl create -f akshay.yml
kubectl get csr
  • Lecture 151: Solution Certificates API

  • Lecture 152: KubeConfig

  • Default location of kubeconfig file: /$HOME/.kube/config

  • Config file will define and map clusters and users:

    • clusters
    • contexts => map users@clusters, with optional default namespace configured
    • users

  • Kube Config File

---
apiVersion: v1
kind: Config
current-context: my-kube-admin@my-kube-playground

clusters:
- name: my-kube-playground
  cluster:
    certificate-authority: ca.crt
    server: https://my-kube-playground:6443

contexts:
- name: my-kube-admin@my-kube-playground
  context:
    cluster: my-kube-playground
    user: my-kube-admin
    namespace: finance

users:
- name: my-kube-admin
  user:
    client-certificate: admin.crt
    client-key: admin.key
---

  • Additional clusters + users + context mappings can be added. Config file gets used automatically if in default path.

  • kubectl config commands

kubectl config view # see current config
kubectl config view --kubeconfig=my-custom-config
kubectl get contexts
kubectl config use-context x@y # assumes this is an available context in config
kubectl config -h
  • Certs in KubeConfig:
    • use full file paths instead of file name
    • swap certificate-authority field reference to file to embed encoded data from file
      • certificate-authority-data: $( cat ca.crt | base64 )
  • Lecture 153: Practice Test - KubeConfig
    • Practice Test Link
    • Set context from custom config: $ kubectl config --kubeconfig=/root/my-kube-config use-context research
    • Get current context, run the command: $ kubectl config --kubeconfig=/root/my-kube-config current-context
  • Lecture 154: Solution KubeConfig
    • Default path: $HOME/.kube/config
    • We test custom config file before setting it to default path.
  • Lecture 155: Persistent Key/Value Store
    • We have already seen how to secure the ETCD key/value store using TLS certificates in the previous videos. Towards the end of this course, when we setup an actual cluster from scratch we will see this in action.

  • Lecture 156: API Groups

  • Core api group includes many resources, each with many resource actions

    • api/v1/{namespaces,pods,rc,events,endpoints,nodes,bindings,PV,PVC,configmaps,secrets, services}

  • Core api group

    • apis/v1
      • deployments: actions = [ list, get, create, delete, update, watch ]
      • replicasets
      • statefulsets
    • extensions
    • networking.k8s.io/v1
      • /networkingpolicies
    • storage.k8s.io
    • authentication.k8s.io
    • certificates.k8s.io/v1
      • /certificatesigningrequests
curl http://localhost:6443 -k --key admin.key --cert admimn.crt --cacert ca.crt
=> returns path array
curl http://localhost:6443/apis -k --key admin.key --cert admimn.crt --cacert ca.crt

  • Use kubectl proxy to pass those 3 configs and use proxy port: $ kubectl proxy

  • Avoid confusing kube proxy wth kubectl proxy.

  • Lecture 157: Authorization

    • First admin needs to pave the road for:
      • other admins
      • devs
      • bots

  • There are four major authorization mechanisms:

    • Node
    • ABAC
    • RBAC
    • Webhook

  • Node Authorization: permissioned by node, where kubelet is a good example...

    • This method uses kubelet client node certificate: system:node:, group=SYSTEM:NODES
    • kubelet reads from kubeAPI: services, endpoints, nodes, pods
    • kubelet writes to kubeAPI: node status, pod status, events

  • Attribute Based Access Control

    • User or user group is given external access to api with permission set:
      • json policy file mgt: statically linked to kubeserver binary {"kind": "Policy", "spec": ... }
      • the policy is a CSR for privs for user or user group
      • project managers can request CSRs
      • security managers can view and approve CSRs

  • Role Based Access Control

    • Role Provides Abstraction Layer For Group Of Privileges
    • First roles are cluster-admins and security-managers.
    • Then we create a project-manager role for project managers.
    • Then we create a developer role for all devs.
    • Roles map users to privs. Security manager can view and approve CSRs.
    • Security Manager approves Project Manager access.
    • Project Manager prompts new Dev for CSR.
    • Project Manager requests Dev CSR approval from Security Manager.
    • RBAC modeling and testing are crucial for keeping security enforcement as simple as possible.

  • Webhook Access Control

    • The webhook model uses external service to approve privilege requests per user.
    • Open Policy Agent

  • Two other authorization modes exist:

    • AlwaysAllow => default kube-apiserver setting for --authorization-mode config flag
    • AlwaysDeny

  • We can also set multiple concurrent modes (chain): --authorization-mode=Node,RBAC,Webhook

    • Requests are made in that order. Denials are forwarded as per config settings.

  • Lecture 158: Role Based Access Controls

  • Lecture 159: Practice Test - RBAC
    • Intended privs: [ create+view+delete pods ] + create ConfigMaps
--- developer-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["list", "get", "create", "update", "delete"]

- apiGroups: [""]
  resources: ["ConfigMap"]
  verbs: ["create"]
---
$ kubectl create -f developer-role.yaml
$ kubectl get roles
$ kubectl describe role developer
  • Next step is to create role binding where we assign role to user named dev-user.
--- devuser-developer-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: dev-user-binding
subjects:
- kind: User
  name: dev-user
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: developer
  apiGroup: rbac.authorization.k8s.io
---
$ kubectl create -f devuser-developer-binding.yaml
$ kubectl get rolebindings
$ kubectl describe rolebinding dev-user-binding
User Show Privs: $ kubectl auth can-i create deployments
User Show Privs: $ kubectl auth can-i delete nodes
Admin Masquerades As User: $ kubectl auth can-i create deployments --as dev-user
Admin Masquerades As User: $ kubectl auth can-i delete nodes --as dev-user
Admin Masquerades As User: $ kubectl auth can-i create pods --as dev-user
Admin Masquerades As User: $ kubectl auth can-i create pods --namespace test --as dev-user
  • We can also specify namespace if other than default namespace. (wider or diverted access scope)
  • We can also speciify specific pod names (narrow access scope) with role rule append:
    • resourceNames: ["blue", "orange"]

  • Lecture 160: Solution Role Based Access Controls

    • Practice Test Link

    • kubectl get rolebindings -n kube-system

    • kubectl describe rolebinding kube-proxy -n kube-system

    • kubectl auth can-i list pods --as dev-user

    • kubectl get pods --as dev-user

    • kubectl edit role developer -n blue

    • kubectl describe pod kube-apiserver -n kube-system

    • cat /etc/kubernetes/manifests/kube-apiserver.yaml

    • ps -aux | grep authorization-mode

    • kubectl describe rolebinding kubeproxy -n kube-system

  • kubectl create role developer --verb=list,create,delete --resource=pods

  • kubectl describe role developer

  • kubectl create rolebinding dev-user-binding --role=developer --user=dev-user

  • kubectl describe rolebinding dev-user-binding

  • kubectl get pods --as dev-user

    • Confirm reported failure: kubectl -as dev-user get pods dark-blue-app -n blue

    • kubectl edit role developer -n blue

    • Confirm reported failure: kubectl -as dev-user create deployment nginx --image=nginx -n blue

    • Be careful of singular and plural references to objects: $ kubectl api-resources

  • Lecture 161: Cluster Roles and Role Bindings

    • All roles are namespace scoped, with default namespace as implicit scope.

      • Other namespace-scoped objects: pods replicasets jobs deployments services secrets roles rolebindings configmaps PVC

    • Other objects are CLUSTER-scoped resources:

      • nodes
      • PV
      • clusterroles
      • clusterrolebindings
      • certificatesigningreuests
      • namespaces

  • clusterroles + clusterrolebindings support cluster-wide access privileges

    • clusteradmin: node view+create+delete
    • storageadmin: PV view+create, PVC delete

  • clusterrole definition file

--- cluster-admin-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cluster-administrator
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list", "get", "cerate", "delete"]
---
$ kubectl create -f cluster-admin-role.yaml
$ kubectl get clusterroles
$ kubectl describe clusterrole cluster-administrator
  • clusterrolebinding definition file links cluster-admin user to role cluster-administrator
--- cluster-admin-role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-admin-role-binding
subjects:
- kind: User
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-administrator
  apiGroup: rbac.authorization.k8s.io
---
$ kubectl create -f cluster-admin-role-binding.yaml
$ kubectl get clusterrolebindings
$ kubectl describe clusterrolebinding cluster-admin-role-binding
  • Clusterroles can be for namespaces as well. ??
  • Many clusterroles are setup by default.
  • Lecture 162: Practice Test - Cluster Roles and Role Bindings

    • Practice Test Link

    • kubectl get clusterroles --no-headers | wc -l

    • kubectl get clusterrolebindings --no-headers | wc -l

    • kubectl describe clusterrole cluster-admin

    • kubectl describe clusterrolebindings cluster-admin

    • kubectl describe clusterrole node-admin

    • kubectl create clusterrolebinding

    • kubectl get nodes --as michelle

  • Lecture 163: Solution Cluster Roles
$ kubectl api-resources
$ kubectl create clusterrole michelle-role --verb=get,list,watch --resource=nodes
$ kubectl describe clusterrole michelle-role
$ kubectl create clusterrolebinding michelle-role-binding --clusterrole=michelle-role --user=michelle
$ kubectl describe clusterrolebinding michelle-role-binding
$ kubectl edit clusterrole michelle-role
$ kubectl edit clusterrolebdingin michelle-role-binding
$ kubectl apply -f michelle-role.yaml
$ kubectl apply -f michelle-role-binding.yaml
  • ClusterRole template
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ...

---
$ kubectl get storageclass --as michelle # fails
$ kubectl create clusterrole storage-admin --resource=persistentvolumes,storageclasses --verb=list,create,get,watch
$ kubectl describe clusterrole storage-admin -oyaml
$ kubectl create clusterrolebinding michelle-storage-admin  --clusterrole=storage-admin --user=michelle
$ kubectl describe clusterrolebinding michelle-storage-admin -o yaml
$ kubectl get storageclass --as michelle
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: michelle-storage-admin
subjects:
- kind: User
  name: michelle
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: storage-admin
  apiGroup: rbac.authorization.k8s.io
---
  • Lecture 164: Service Accounts
    • Service accounts is related to other security concepts. User accounts are something different.
    • Third party services might want a service account. Token is generated for service account access.
    • This token is encrypted as secret, stored inside secret object and referenced by name.
kubectl create serviceaccount dashboard-sa
kubectl describe serviceaccount  dashboard-sa | grep -i token | cut -d' ' -f2
kubectl describe secret dashboard-sa-token-kbbdm
token test:  curl https://192.168.56.70:6443/api -insecure --header "authorization: Bearer $TOKEN_STRING"

  • We will configure our app with same token for same type of access.

  • We can embed secret by prompting volume mount by pod with token secret file in volume.

    • This happens automatically for default serviceaccount: $ kubectl get serviceaccount
    • $ kubectl describe pod xyz # see sections: Mounts Volumes
    • $ kubectl exec -it xyz ls /var/run/secrets/kubernetes.io/serviceaccount
    • $ kubectl exec -it xyz cat /var/run/secrets/kubernetes.io/serviceaccount/token

  • Pods must be destroyed and recreated to assign them to Serviceaccounts.

    • Default behavior is to automount token file.
    • In pod definition spec, explicit override is optional: automountServiceAccountToken: false

  • Lecture 165: Practice Test Service Accounts

    • As of version 1.22, serviceaccount methods have changed: TokenRequestAPI (audience-bound,time-bound,object-bound)
    • Before: Every namespace has default serviceaccount, and that token file is accessed by pod upon creation.
    • SWITCHED: projected volume with dedicated token including expiration date deprecates secret.
    • SWITCHED: secret is NOT autocreated upon serviceaccount creation!!!
      • Additional statement is now required: $ kubectl create token

  • Check encoded token for expiration date:

$ jq -R 'split(".") | select(length > 0) | .[0],.[1] | @base63d | fromjson' <<< $ENCODED_TOKEN | grep exp
  • Note that legacy method is still supported, and assumes serviceaccount exists:
--- secret-definition.yml
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: mysecretname
  annotations:
    kubernetes.io/service-account.name: dashboard-sa
---
  • This is only advisable when TokenRequest API is not available and non-expiring token exposure is acceptable.
  • k8s secrets docs
  • Practice Test Link
    • Which service account for a pod: $ kubectl get pods -oyaml | grep -i service
    • kubectl describe pod web-dashboard-767bc588bc-lfs2m | grep -A6 volume
    • kubectl edit deployment web-dashboard #

  • Lecture 166: Image Security

    • Image reference conventions follow docker conventions: //
    • image: nginx => image: docker.io/library/nginx, since library is default docker hub registry username.
    • A custom image from my own repo: image: github.com/jeremy-donson:jd-app1
    • internal private registries support better control of container quality

  • Docker private registry access process:

docker login <private-registry-name>
docker run <private-registry-name>/apps:internal-app
  • k8s approach
kubectl create secret docker-registry --help
kubectl create secret docker-registry regcred --docker-server= --docker-username= --docker-password= --docker-email=
In Pod spec: "  image: <private-registry-name>/apps/internal-app\n. imagePullSecrets:\n. - name: regcred"

kubectl create secret docker-registry private-reg-cred --docker-username=dock_user --docker-password=dock_password --docker-server=myprivateregistry.com:5000 --docker-email=dock_user@myprivateregistry.com
  • Seems we have two deployments to show how diff options work in parallel.

  • Lecture 168: Solution Image Security

  • Lecture 169: Security Contexts

    • Docker security concepts are prerequisite to k8s security concepts.
    • Local docker run os, docker, etc
    • $ docker run ubuntu sleep 3600 # root user is implicit
    • Containers and host share the same kernel.
    • Process isolation in k8s is achieved via namespaces.
    • OS only uses its own namespace.
    • Docker container can only see its own docker namespace, as container root is constrained.
    • See all linux root user capabilities listed here: $ cat /usr/include/linux/capability.h
    • Add or drop additional calls: $ docker run --cap-{add,drop} KILL ubuntu
      • Provide all sudo privs: $ docker run --privileged ubuntu
    • From Outer Host: $ ps aux # we see docker subproc
    • Docker runs procs by default as container user: root
    • $ docker run --user=1000 ubuntu sleep 3600 # non-root user is explicit

  • This could also be preset in Dockerfile:

echo "FROM ubuntu\nUSER 1000" > my-ubuntu-image
docker build -t my-ubuntu-image
docker run my-ubuntu-image sleep 3600
ps aux

  • Lecture 170: Practice Test - Security Contexts

  • We saw container uid (--user=1001) and container root capabilities ( --privileged ) modified in last lecture for docker.

    • Here we do the same in k8s...

  • In k8s, we secure things at pod level or at container level.

    • pod level: applies to all containers on pod
    • container level: will override pod security for a given container

  • Here we see it set for both pod and container, so container config will rule.

    • Note: capabilities can ONLY be set at container level and NOT at pod level.
---
apiVersion: v1
kind: Pod
metadata:
  name: web-pod
spec:
  securityContext:
    runAsUser: 1000
  containers:
  - name: ubuntu
    image: ubuntu
    securityContext:
      runAsUser: 1001
      capabilities:
        add: ["MAC_ADMIN"]
  • Who is user running procs in that pod?: $ kubectl exec ubuntu-sleeper -- whoami
  • Lecture 171: Solution Security Contexts
    • kubectl exec ubuntu-sleeper -- whoami
    • kubectl edit pod ubuntu-sleeper
    • kubectl replace --force -f sleeper.yaml
  • Lecture 173: Practice Test Solutions
    • ??

  • Lecture 174: Network Policy

    • Simple traffic to web-app:80 <=> APIServer:5000 <=> dbserver:3306

      • ingress: inbound to web app from client user
      • egress: outbound to api

    • api traffic

      • inbound from web app server
      • egress: outbound to db server

  • Total network policy rule count = 5

      1. ingress:80
      1. egress:5000
      1. ingress:5000
      1. egress:3306
      1. ingress:3306

  • Cluster Network Security:

    • All nodes can intercommunicate across cluster.
    • All pods can intercommunicate across cluster.

  • Default NetworkPolicy = "All Allow"

  • If we want to disable direct calls from web pod to db pod, then we create a custom NetworkPolicy

    • Database pod NetworkPolicy: Allow DBServer:3306 Ingress Traffic From API Pod

  • This depends upon similar syntax as selectors and labels:

    • db pod spec: "labels:\n role: db"
    • network policy spec: "podSelector:\n matchLabels:\n role: db"

  • network-policy.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-policy
spec:
  podSelector:
    matchLabels:
      role: db
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          name: api-pod
    ports:
    - protocol: TCP
      port: 3306
---
Note: This spec is ingress only (policyTypes).  Egress is not constrained!!!!
  • Some solutions do support k8s Network Policies.
  • Some solutions do NOT support k8s Network Policies, and no errors will be reported.

  • Lecture 175: Developing network policies

    • Restrict db pod to access via 3306 and only from api pod.
    • Ingress policy type supports implicit return trip access for same route.

  • network-policy-db.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-policy

spec:
  podSelector:
    matchLabels:
      role: db
    policyTypes:
    - Ingress
    - Egress
    ingress:

    - from:
      - podSelector:
            matchLabels:
              name: api-pod
    ports:
    - protocol: TCP
      port: 3306

    egress:

    - to:
      - ipBlock:
          cidr: 192.168.5.10/32
    ports:
    - protocol: TCP
      port: 80

  • We can also have network policy that bridges pods across namespaces: - from: - namespaceSelector

  • ingress from+to options correspond have some tricky syntax:

  • podSelector

    • namespaceSelector

    • OR

    • ipBlock

    • podSelector

    • OR

    • namespaceSelector

    • OR

    • ipBlock

  • Consider how this supports both AND + OR logic.

  • Lecture 176: Practice Test - Network Policy

    • Practice Test Link
    • $ kubectl get all # --no-headers | grep wc -l
    • $ kubectl get all -A --no-headers | grep wc -l
    • $ kubectl get networkpolicies # netpol
    • $ kubectl descrbe netpol payroll-policy # only ingress has networkpolicy, so egress stays unaffected

  • Consider two service types across 5 services:

    • kubernetes + db-service => ClusterIP
    • {payroll,external,internal}-service => NodePort

  • Network Policy Spec File

--- netpolicy-from-internal-to-payroll-and-db.yaml (internal-policy.yaml)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: internal-policy
  namespace: default
spec:
  podSelector:
    matchLabels:
      name: internal
  policyTypes:
  - Egress
  - Ingress
  ingress:
    - {}
  egress:
  - to:
    - podSelector:
        matchLabels:
          name: mysql
    ports:
    - protocol: TCP
      port: 3306

  - to:
    - podSelector:
        matchLabels:
          name: payroll
    ports:
    - protocol: TCP
      port: 8080
---
$ kubectl create -f netpolicy-from-internal-to-payroll-and-db.yaml
$ kubectl describe netpol internal-policy

  • Lecture 177: Kubectx and Kubens – Command line Utilities

  • Through out the course, you have had to work on several different namespaces in the practice lab environments. In some labs, you also had to switch between several contexts. While this is excellent for hands-on practice, in a real “live” kubernetes cluster implemented for production, there could be a possibility of often switching between a large number of namespaces and clusters. This can quickly become and confusing and overwhelming task if you had to rely on kubectl alone.

  • This is where command line tools such as kubectx and kubens come in to picture.

  • Reference

  • Kubectx:

  • With this tool, you don't have to make use of lengthy “kubectl config” commands to switch between contexts. This tool is particularly useful to switch context between clusters in a multi-cluster environment.

  • Installation:

sudo git clone https://github.com/ahmetb/kubectx /opt/kubectx
sudo ln -s /opt/kubectx/kubectx /usr/local/bin/kubectx

  • Syntax:

    • To list all contexts: $ kubectx
    • To switch to a new context: $ kubectx <context_name>
    • To switch back to previous context: $ kubectx -
    • To see current context: $ kubectx -c

  • Kubens: This tool allows users to switch between namespaces quickly with a simple command.

  • Installation:

sudo git clone https://github.com/ahmetb/kubectx /opt/kubectx
sudo ln -s /opt/kubectx/kubens /usr/local/bin/kubens
  • Syntax:
    • To switch to a new namespace: $ kubens <new_namespace>
    • To switch back to previous namespace: $ kubens -

  • Lecture 178: Network Policy Test Solutions

  • Section 8: Storage (75m)

  • Lecture 179: Storage - Section Introduction

    • Persistent Volumes
    • Persistent Volume Claims
    • Configure Applications With Persistent Storage
    • Access Modes For Volumes
    • Kubernetes Storage Object

  • Lecture 180: Introduction to Docker Storage

    • Storage Drivers => physical storage
    • Volume Drivers => logical storage

  • k8s storage is derived from docker storage concepts, so learn about docker storage first.

  • Lecture 181: Storage in Docker
    • Docker stores all objects under: /var/lib/docker/{aufs,containers,image,volumes}/
    • Docker uses a layered container image file architecture which leverages docker local cache.
--- Dockerfile
FROM Ubuntu
RUN apt-get update && apt-get -y install python
RUN pip install flask flask-mysql
COPY . /opt/source-code
ENTRYPOINT FLASK_APP=/opt/source-code/app.py flask run
---
$ docker build Dockerfile -t jeremy-donson/my-custom-app

  • Some concerns are evident:

    • Docker cache resources grow stale. Pin vs chase?
    • Dockerfile design can reduce container image file size.
    • The container service may need its own user account and group?
    • How does this overall approach compare with systemd and control groups?
    • Compare base linux images: alpine/ubuntu/rhel
    • Reducing build time and images size are great, but quality precludes speed.
    • All network storage architectures and stateful challenges are to be compared for trade-offs.

  • Consider an inverted ladder of dependency:

    • DOCKER RUN => container object layer is the only writeable temp filesystem.
    • DOCKER BUILD => image file layers are read-only: ubuntu later + apt + pip + source + entrypoint
    • Dockerfile Templates => Dockerfile

  • Tooling Notes:

    • Remember that we can use ansible+py3 to manage both vms and docker containers.
    • We often use terraform to stand up network infrastructure.
    • In parallel, we often use consul to manage network services.
    • Sops is a great open source tool for security done simple.

  • Editing of image text file in running container creates copy-on-write newer text file version in container layer.

    • We can start and stop the container to test manual source code changes.
    • However, when we kill or delete container/pod, the rw filesystem is gone.
    • Now we update source code + rebuild image from updated Dockerfile.
      • How do we avoid using cached version?

  • To create database service with data storage that persists:

# volume mounting
docker volume create data_volume                # Create persistent volume: data_volume
docker run -v data_volume:/var/lib/mysql mysql  # We implement storage volume under service: mysql
docker ps # mysqld with mounted local volume... [ /var/lib/mysql:/var/lib/docker/volumes/data_volume ]
ls -al /var/lib/docker/volumes
docker run -v data_volume2:/var/lib/mysql mysql  # Docker will auto-create new volume in default location.

# bind mounting method supports custom storage
docker run -v /data:/var/lib/mysql mysql         # Here we associate mysql data dir with custom local storage path.
ls -al /var/lib/docker/volumes
docker ps

# Current supported bind method syntax is more explicit:
docker run --mount type=bind,source=/data/mysql,target=/var/lib/mysql mysql
  • Storage Drivers depend upon underlying OS, auto-selected by Docker:
    • AUFS
    • ZFS
    • BTRFS
    • Device Mapper
    • Overlay
    • Overlay2

  • Lecture 182: Volume Driver Plugins in Docker

  • Volume Drivers Depend Upon Storage Drivers

    • Local + AzureFS + Portworx + VMware vSphere Storage + 3rd party etc
    • Volume drivers can support saving our data in the cloud:
    • $ docker run -it --name mysql --volume-driver rexray/ebs --mount src=ebs-vol,target=/var/lib/mysql mysql
    • rexray: Enterprise-grade storage plugins for the Container Storage Interface (CSI).

  • Lecture 183: Container Storage Interface (CSI)

    • Container Runtime Interface (CRI) is where k8s supports docker + any containerization platforms via plugins.
    • Container Network Interface (CNI) extended the same courtesy to network tool vendors via plugins.
    • Container Storage Interface (CSI) extended the same courtesy to storage tool vendors via plugins.
    • In each case above, drivers can be customized for a given network service provider type.

  • CSI Model: ^

    • rpc
    • CreateVolume: Provision New Volume => new volume on storage
    • DeleteVolume: Delete Existing Volume => decom volume on storage
    • ControllerPublishVolume: Assign workload to node by using volume. => make volume available on a node

  • CSI Main Github Repo

  • Lecture 184: Download Slide Deck
    • pdf
  • Lecture 185: Volumes
    • Exporting and persisting container data requires volume be mounted to persistent storage.
    • Each docker container can have its own persistent directory just below storage mount points.
    • There we write container logs, etc, and analyze container behavior.
    • Pods can come and go, but pod storage claims can persist from pod to pod.
--- pod-stores-number.yaml
apiVersion: v1
kind: Pod
metadata:
  name: random-number-generator
spec:
  containers:
  - image: alpine
    name: alpine
    command: ["/bin/sh","-c"]
    args: ["shuf -i 0-100 -n 1 >> /opt/number.out;"]
    volumeMounts:
    - mountPath: /opt
      name: data-volume

  volumes:
  - name: data-volume
    hostPath:
      path: /data
      type: Directory
---
# NEW BINDING: [container:/opt]:[host:/data]

$ kubectl get pods
$ kubectl create -f pod-stores-number.yaml
$ kubectl get pods
  • Makes sense for single node architecture to write locally, but what about pods using multiple nodes?
    • No. Multi-node cluster architecture usually has dedicated cluster storage in addition to worker nodes.
      • For example: ceph, nfs, glusterfs, flocker, scaleio, etc
      • Eg: for aws ebs volume.... hostPath section is swapped for:
volumes
- name: data-volume
  awsElasticBlockStore:
    volumeId: <volume-id>
    fsType: ext4

  • Lecture 186: Persistent Volumes

  • When we have a lot of pods, configuring storage across pods is done by selectors and labels again.

  • Persistent Volume Claims (PVCs) depend upon Persistent Volumes (PVs).

    • This way we have an inventory of dedicated storage for pods to claim.

  • We can govern access from PV by access mode:

    • ReadOnlyMany
    • ReadWriteOnce
    • ReadWriteMany

  • Create persistent volume: pv-definition.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-vol-01
spec:
  accessModes:
    - ReadWriteOnce
  capacity
    storage: 1Gi
  hostPath:
    path: /tmp/data

---
# Local hostpath is not for production!!
$ kubectl create -f pv-definition.yaml
$ kubectl get persistentvolume # pv
$ kubectl describe pv pv-vol-01

  • Lecture 187: Persistent Volume Claims

    • PVC remains in Pending state until bound to PV.
    • PVC:PV = 1:1, so claiming a pv makes it dedicated to pvc.

  • Criteria for binding PVC to PV:

    • Sufficient Capacity
    • Access Modes
    • Volume Modes
    • Storage Class

  • If we wish to explicitly target a specific PV, labels and selectors can be utilized.

  • Persistent Volume Claim (PVC) Spec Sample

$ kubectl get persistentvolumeclaim
--- pvc-definition.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
   name: mypvclaim
spec:
  accessModes:
    - ReadWriteOnce

  resources:
    requests:
      storage: 500Mi
---
$ kubectl create -f pvc-definition.yaml
$ kubectl get persistentvolumeclaim
# K8s will compare pvc to unclaimed pv definitions in an attempt to select proper PV to bind PVC.
$ kubectl get persistentvolumeclaim
$ kubectl describe persistentvolumeclaim mypvclaim
$ kubectl delete persistentvolumeclaim mypvclaim
# default requires manual PV deletion by admin... persistentVolumeReclaimPolicy: Retain => but not made available
# Other pvrp options:  Delete or Recycle, which will scrub data prior to new PVC...
  • Lecture 188: Using PVCs in PODs
    • Once you create a PVC use it in a POD definition file by specifying the PVC Claim name under persistentVolumeClaim section in the volumes section like this:
apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
    - name: myfrontend
      image: nginx
      volumeMounts:
      - mountPath: "/var/www/html"
        name: mypd
  volumes:
    - name: mypd
      persistentVolumeClaim:
        claimName: myclaim

  • The same is true for ReplicaSets or Deployments. Add this to the pod template section of a Deployment on ReplicaSet.

  • Reference URL

  • Lecture 189: Practice Test - Persistent Volumes and Persistent Volume Claims

  • pod-pv.yaml

apiVersion: v1
kind: Pod
metadata:
  name: webapp
spec:
  containers:
  - name: event-simulator
    image: kodekloud/event-simulator
    env:
    - name: LOG_HANDLERS
      value: file

    volumeMounts:
    - mountPath: /log
      name: log-volume

  volumes:
  - name: log-volume
    hostPath:
      # directory location on host
      path: /var/log/webapp
      # this field is optional
      type: Directory
---
$ kubectl create -f pod-pv.yaml # $ kubectl replace -f pod-pv.yaml --force
  • Now we have separate persistent volume:
-- pv-1.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-log
spec:
  persistentVolumeReclaimPolicy: Retain
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 100Mi
  hostPath:
    path: /pv/log
---
$ kubectl create -f pv-1.yaml
$ kubectl get persistentvolumes
  • PVC
--- pvc-1.yaml
---
kubectl replace -f pvc-1.yaml --force
  • If we delete pvc used by a pod, it stays in terminating state until released.
    • pod : claim : volume : storage
  • Lecture 190: Solution - Persistent Volumes and Persistent Volume Claims
  • Lecture 191: Application Configuration
  • Lecture 192: Additional Topics

  • Lecture 193: Storage Class

    • We learned how pod uses pvc, which is a claim on pre-existing pv.
    • {pv,pvc,pod}-definition.yaml
    • static provisioning from cloud storage would require that cloud storage exists first
    • dynamic provisioning uses storage class objects to autocreate PV
      • In PVC spec we now simply specify storage class name: storageClassName: google-storage
      • Now PVC and PV can be auto-provisioned....

  • gcp example follows:

--- sc-definition.yaml
apiVersion: storage.k8s.io/v1
kind: storageClass
metadata:
  name: google-storage
provisioner: kubernetes.io/gce-pd
---
$ kubectl create -f sc-definition.yaml
  • Consider additional gcp storage parameters.
parameters:
  type: pd-standard
  replication-type: none

  • We could create three storage hardware options: silver + gold + platinum, using three storage class spec files...

    • silver-sc.yaml: pd-standard
    • gold-sc.yaml: pd-ssd
    • platinum-sc.yaml: pd-ssd,regional-pd

  • Now pvc can reference storage class desired.

  • Lecture 194: Practice Test - Storage Class
    • Practice Test Link
    • $ kubectl get storageclasses
      • see /no-provisioner for which does not support dynamic allocation
    • $ kubectl describe storageclasses local-storage
--- local-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: local-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: local-storage
  resources:
    requests:
      storage: 500Mi
---
$ kubectl -create -f local-pvc.yaml

--- delayed-volume-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: delayed-volume-sc
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
$ kubectl -create -f delayed-volume-sc.yaml
$ kubectl get sc

  • Lecture 195: Solution - Storage Class

    • $ kubectl get storageclass # sc
    • $ kubectl describe sc xyz

  • $ kubectl run nginx --image=nginx --dry-run=client -oyaml > nginx-alpine.yaml

  • $ vi nginx-alpine.yaml # add volume mount to container spec + volume section below

  • $ kubectl ceate -f nginx-alpine.yaml

  • Section 9: Networking (201m)

  • Lecture 196: Networking - Section Introduction

    • Prerequisites:

      • switching
      • routing
      • default gateway
      • coreDNS
      • Tools
      • DNS
      • CNI
      • Docker Networking

    • Core Networking Topics

      • Networking Configuration On Cluster Nodes
      • Pod networking concepts
      • CNI in K8s
      • Service Networking
      • ClusterDNS
      • Network Loadbalancer
      • Ingress

  • Lecture 197: Download Presentation Deck

  • Lecture 198: Prerequisite - Switching + Routing

  • Network 1 Switch assumes hub + 2 hosts:

    • hub: 192.168.1.0
      • host A: 192.168.1.10
      • host B: 192.168.1.11
Detect Network Interface On Both: $ ip link
Set IP Address Locally On A: $ ip addr add 192.168.1.10/24 eth0
Set IP Address Locally On B: $ ip addr add 192.168.1.11/24 eth0
Ping from A to B: $ ping 192.168.1.11

  • Network 2 Switch assumes hub + 2 hosts:

    • hub: 192.168.2.0
      • host C: 192.168.2.10
      • host D: 192.168.2.11

  • Router is required to bridge these two networks:

    • network 1: 192.168.1.1
    • network 2: 192.168.2.1
View routing table: $ route
Create new route: $ ip route add 192.168.2.0/24 via 192.168.1.1
$ route
Set default gateway: $ ip route add default via 192.168.2.1
$ route

  • When we have multiple routers, then this gets more complex:

    • one router for local LAN access, which already has its own entry from above.
    • one router for external internet access: $ ip route add 192.168.1.0/24 via 192.168.2.2

  • Linux host as software router is instructive:

  • Center B host must have min two network interfaces:

    • A 192.168.1.5 <=> 192.168.1.0 <=> 192.168.1.6:[B]:192.168.2.6 <=> 192.168.2.0 <=> 192.168.2.5 C

  • On host A: $ ping 192.168.2.5 # fails

    • $ ip route add 192.168.2.0/24 via 192.168.1.6
    • $ route ; ping 192.168.2.5

  • On host C: $ ping 192.168.1.5 # fails

    • $ ip route add 192.168.1.0/24 via 192.168.2.6
    • $ route; $ ping 192.168.1.5

  • Now we need to configure port forwarding on host B between networks:

    • $ cat /proc/sys/net/ipv4/ip_forward # 0
    • echo 1 > /proc/sys/net/ipv4/ip_forward # pings between networks now work
    • Configure as permanent on reboot in /etc/sysctl.conf: net.ipv4.ip_forward = 1

  • Summary:

    • $ ip link # see network interfaces

    • $ ip addr # see ip addresses to network interfaces

    • $ ip addr add 192.168.1.10/24 eth0

    • $ ip route

    • Configure as permanent on reboot in /etc/network-interfaces

    • $ ip route

    • $ ip route add 192.168.1.0/24 via 192.168.2.1

    • $ route

    • $ cat /proc/sys/net/ipv4/ip_forward # 0

    • echo 1 > /proc/sys/net/ipv4/ip_forward # pings between networks now work

    • Configure as permanent on reboot in /etc/sysctl.conf: net.ipv4.ip_forward = 1

  • Lecture 199: Prerequisite - DNS

    • app@192.168.1.10
    • db@192.168.1.11

  • from app@192.168.1.10

ping db
echo '192.168.1.11   db' >> /etc/hosts # This overrides DNS name resolution locally.
ping db

  • With many hosts on a network, we centralize this feature to DNS to make it scalable.

    • Let's add a dedicated DNS host at 192.168.1.100. How does 192.168.1.10 know to use that remote service?
    • $ echo 'nameserver 192.168.1.100' >> /etc/resolv.conf
    • We can also have multiple local hostnames+ host aliases + nameservers at play.
    • Note that /etc/hosts and local DNS can both be at play, with the former being checked first.
      • We can change order of precedence in /etc/nsswitch.conf

  • DNS can be configured to resolve some and then Forward All to 8.8.8.8, for example:

$ echo "nameserver   192.168.1.100\n nameserver  8.8.8.8" >  /etc/resolv.conf
# OR
$ echo 'Forward All to 8.8.8.8' >> /etc/hosts
# To add local host aliases by subdomain to make the following work: $ ping web
$ echo "search   acme.com  prod.acme.com" >>  /etc/resolv.conf

  • Domain names:

    • . => root
    • .com => top-level domains
    • google => domain
    • www => subdomain

  • Name resolution traverses the namespace:

    • We either have a temp cached ip address in local dns, else we traverse the namespace hierarchy...
    • apps.google.com
    • . => com => google => apps => google DNS => IP returned + optionally temp cached

  • Subdomains can represent top level org depts or service types or ?

  • DNS Record Types

    • Hostname:IP = A => web-server <=> 192.168.1.1
    • Hostname:Hostalias = CNAME => food.web-server <=> eat.web-server,hungry.web-server
    • Hostname:IPv6 = AAAA => web-server <=> 2001:0987654456789876567898765

  • We query dns using nslookup:

  • Lecture 200: Prerequisite - CoreDNS
    • In the previous lecture we saw why you need a DNS server and how it can help manage name resolution in large environments with many hostnames and Ips and how you can configure your hosts to point to a DNS server. In this article we will see how to configure a host as a DNS server. We are given a server dedicated as the DNS server, and a set of Ips to configure as entries in the server. There are many DNS server solutions out there, in this lecture we will focus on a particular one – CoreDNS. So how do you get core dns? CoreDNS binaries can be downloaded from their Github releases page or as a docker image. Let’s go the traditional route. Download the binary using curl or wget. And extract it. You get the coredns executable.
$ wget https://github.com/coredns/coredns/releases/download/v${CORE_DNS_VERSION}/coredns_${CORE_DNS_VERSION}_linux_amd.tgz
$ tar -xzvf coredns_${CORE_DNS_VERSION}_linux_amd.tgz
$ ./coredns --version
$ ./coredns

  • Run the executable to start a DNS server. It by default listens on port 53, which is the default port for a DNS server.

  • Now we haven’t specified the IP to hostname mappings. For that you need to provide some configurations. There are multiple ways to do that. We will look at one. First we put all of the entries into the DNS servers /etc/hosts file. And then we configure CoreDNS to use that file. CoreDNS loads it’s configuration from a file named Corefile. Here is a simple configuration that instructs CoreDNS to fetch the IP to hostname mappings from the file /etc/hosts. When the DNS server is run, it now picks the Ips and names from the /etc/hosts file on the server.

$ cat /etc/hosts
$ echo ". {\n       hosts /etc/hosts\n}" > Corefile
$ ./coredns

  • Lecture 201: Prerequisite - Network Namespaces

    • Network namespaces provide process isolation in linux.
    • Container process is isolated by namespace to see only its own processes and proc id metadata.
      • Container output of ps is a subset of host ps outut.
      • Container also has its own vip + routing + arp tables, separate from host.

  • Network namespaces can be created:

$ ip netns
$ arp
$ route
$ ip netns add blue
$ ip netns add red
$ ip netns
$ ip link
# We speak to that namespace via
$ ip netns exec red ip link
# OR
$ ip -n red link
# Confirm eth0 invisible to red namespace.
$ ip -n red arp
# Confirm namespaced arp call returns nothing.
$ ip -n red route
# Confirm namespaced route call sees no host routes.
  • We can connect two local namespaces:
$ ip link add veth-red type veth peer name veth-blue # link names
$ ip link set veth-red netns red    # assign name
$ ip link set veth-blue netns blue  # assign name
$ ip -n red addr 192.168.15.1 dev veth-red # assign ip
$ ip -n blue addr 192.168.15.2 dev veth-blue
$ ip -n red set veth-red up
$ ip -n blue set veth-blue up
$ ip netns exec red ping 192.168.15.2
# OR
$ ip -n red ping 192.168.15.2
# compare 3 outputs:
$ ip netns exec red arp
$ ip netns exec blue arp
$ arp
# delete peer2peer links:
$ ip -n red link del veth-red
# AUTO_RUN DUE TO PAIR: $ ip -n blue link del veth-blue
$ ip ns
$ ip -n red link
$ ip -n blue link
$ ip link
  • Virtual switch supports 3+ host network:
    • Linux Bridge and open vswitch are both options. We use the former here.
# create + bring up virtual switch / network:
$ ip link
$ ip link add v-net-0 type bridge
$ ip link | grep -i down
$ ip link set dev v-net-0 up
$ ip link | grep -i up
$ ip link add veth-red type veth peer name veth-red-br # red:vnet
$ ip link
$ ip link add veth-blue type veth peer name veth-blue-br # blue:vnet
$ ip link
# create virtual cable connecting those 2 namespace interfaces
$ ip link set veth-red netns red
$ ip link set veth-red-br master v-net-0
$ ip link set veth-blue netns blue
$ ip link set veth-blue-br master v-net-0
# reassign ip addresses we used before
ip -n red addr add 192.168.15.1 dev veth-red
ip -n blue addr add 192.168.15.2 dev veth-blue
# enable virtual ports
ip -n red link set veth-red up
ip -n blue link set veth-blue up
ip link | grep -i 'up\|down'
# does ping work yet?
$ ping 192.168.15.1 # not yet!!
# Network switch needs ip assigned
$ ip addr add 192.168.15.5/24 dev v-net-0
$ ping 192.168.15.1 # should work!!
  • Assuming we are clear thus far, we will add an external node network and host.
$ ip netns exec blue ping 192.168.1.3 # fails
$ ip netns exec blue route # no route for 192.168.1.0 network
# Add new route via vswitch to network 192.168.1.0/24
$ ip netns exec blue ip route add 192.168.1.0/24 via 192.168.15.5
$ ip netns exec blue route
$ ip netns exec blue ping 192.168.1.3 # should work, but no response yet...
  • Diagram Yaml Spec 0.001
apiVersion: 0.001
Type: Diagram
Kind: Network
Metadata:
  Name: Local Container Namespaces
Spec:
  Diagram Elements:
  - gateway@localhost
  - host@192.168.1.2
  - containers@192.168.15.[red@1,blue@2,orange@3,black@4]
  - vswitch@192.168.15.5
  - LAN@192.168.1.0
  - LAN HOST@192.168.1.3 => no ping response, but pingable
  Details:
  - vlink exists: vport
  - vlink connected: vcable
  - vlink addressed: vip
  - vlink up
  • In order for ping to work on LAN hosts, we need NAT enabled on localhost gateway...
# Now responses to LAN calls will look like they are from LAN members only:
$ iptables -t nat -A POSTROUTING -s 192.168.15.0/24 -j MASQUERADE
$ ip netns exec blue ping 192.168.1.3
  • If the internet is available network service on LAN, then we will need to create a default gateway...
ip netns exec blue ping 8.8.8.8
ip netns exec blue route
ip netns exec blue ip route add default via 192.168.15.5
ip netns exec blue route
ip netns exec blue ping 8.8.8.8
  • We can bridge LAN hosts back to namespaced containers on first host
    • $ iptables -t nat -A PREROUTING --dport 80 --to-destination 192.168.15.2:80 -j DNAT
    • Now any LAN host ping to 192.168.15.2 will be forwarded to port 80 via localhost gateway.
  • Lecture 202: FAQ
    • While testing the Network Namespaces, if you come across issues where you can't ping one namespace from the other, make sure you set the NETMASK while setting IP Address. ie: 192.168.1.10/24
ip -n red addr add 192.168.1.10/24 dev veth-red
  • Another thing to check is FirewallD/IP Table rules. Either add rules to IP Tables to allow traffic from one namespace to another. Or disable IP Tables all together (Only in a learning environment).
  • Lecture 203: Prerequisite - Docker Networking
$ docker run --network none nginx

  • network flag options

    • none: running container totally isolated AND multiple containers cannot intercommunicate!!
      • $ docker run --network none nginx && docker run --network none nginx
    • host: ports autoforwarded
      • $ docker run --network host nginx && docker run --network host nginx # fails due to default port 80 in use
    • bridge: internal private network@default = 172.17.0.0
      • Docker installs BRIDGE by default: $ docker network ls
      • This very same resource is named docker0 in ip link output: $ ip link & ip addr && ip netns
      • $ docker inspect 342567894
      • We need to bring that link up:

  • Here we query both sides of the bridge:

IP_NETNS=$(ip netns)
echo $IP_NETNS
ip link
ip -n $IP_NETNS link
  • We can also query docker container ips:
ip -n $IP_NETNS addr

  • Docker repeats this process for each new container:

    • Create bridge vport
    • Create container namespace+vport
    • Connect container to bridge: vethxxx@ifN
      • Odd:Even pairs as bridge:container interface NUMBERS
    • Now all containers can communicate with each other.

  • Docker can also expose container ip:port to external access:

    • From localhost: $ curl 172.17.0.3:80 # success
    • From external host: $ curl 172.17.0.3:80 # fails

  • $ docker run -p 8080:80 nginx

    • From external host: $ curl 192.168.1.10:8080 # Succeeds

  • This is auto-achieved by docker via iptables NAT rule:

iptables -nvL -t nat
iptables -t nat -A DOCKER -j DNAT --dport 8080 --to-destination 80
iptables -nvL -t nat

  • Network Namespace Process:

    • create network namespace
    • create bridge network and interface
    • create veth pairs (vcable)
    • attach vEth to Namespace
    • attach other vEth to bridge
    • assign ip addresses
    • bring interfaces up
    • enable NAT - IP Masquerade

  • Docker Namespace Process:

    • similar, with some naming pattern variants

  • rkt or mesos

    • again: similar, with some naming pattern variants

  • CNI aggregates this into a single standard

    • Allows any container runtime to run with any plugin...
    • BRIDGE is plugin for CKI
    • BRIDGE is achieved via $ bridge add

  • CNI Requirements For Container Runtime:

    • Container runtime must create network namespace.
    • Identify network to which container must be attached.
    • Container runtime to invoke Network Plugin bridge when container is ADDed
    • Container runtime to invoke Network Plugin bridge when container is DELeted
    • JSON format of the Network Configuration

  • CNI Plugin Reqs:

    • Must support CLI args ADD-DEL-CHECK
    • Must support parameters: cid + netns, etc
    • Must manage ip address assignment to pods
    • Must return results in a specific format.

Example PLUGINS:

  • BRIDGE

  • VLAN

  • IPVLAN

  • MACVLAN

  • WINDOWS

  • DHCP

  • host-local

  • VMWare NSX

  • Lecture 204: Prerequisite - CNI
  • Docker supports Container Network Model, NOT CNI, so the workaroudns are myriad.
    • Instead of: $ docker run --network=cni-bridge nginx
    • Try this: $ docker run --network=none nginx ; bridge add 2eiu3gd2 /var/run/netns/2eiu3gd2
    • Since k8s chases docker, k8s workarounds also exist where CNM and CNI are misaligned.

  • Lecture 205: Cluster Networking

    • K8s Docs: Required Ports

    • K8s Master and worker nodes require some basic networking setups:

      • Network: 192.168.1.0
      • Master: 192.168.1.10 master-01 $MAC_ADDR_MASTER
      • Workers: 192.168.1.1{1,2} worker-0{1,2} $MAC_ADDR_WORKER_{1,2}

  • Node Ports to Open:

    • Master: kube-api:6443, accessed by...
      • Locally: kubectl + kubelet:10250 + kube-scheduler:10251 + kube-controller-manager:10252 + etcd:2379
      • Remotely: worker kubelete + API clients
    • Worker:
      • Worker kubelet listens:10250 for master kubeapi calls from kube-scheduler.
    • Worker listens for services on port range: 30000 - 32767

  • For multiple master configurations, we will need this done for all master nodes.

    • Multiple etcd instances on masters all intercommunicate via port 2380.

  • Handy commands:

ip link
ip addr
ip addr add 192.168.1.10/24 dev eth0
ip route
ip route add 192.168.1.0/24 via 192.168.2.1
route

cat /proc/sys/net/ipv4/ip_forward # 1
arp
netstat -plnt

In the CKA exam, for a question that requires you to deploy a network addon, unless specifically directed, you may use any of the solutions described in the link above.

  • Lecture 207: Practice Test - Explore Kubernetes Environment
    • get node ip
kubectl get nodes -owide
ip a | grep -B2 $NODE_IP
  • get mac: $ip link show eth0
  • get scheduler listener port: $ netstat -nplt | grep -i sched
  • get etcd connection counts: $ netstat -anp | grep etcd | grep LISTEN
  • get etcd connection counts: $ netstat -anp | grep etcd | grep -v 2379 # 2380 is only for etcd peer2peer connectivity.

  • Lecture 208: Solution - Explore Environment (optional)

    • $ kubectl get nodes -owide

  • Three ways to look at local network interfaces...

    • $ if config -a # see them all
    • $ ifconfig ens3 # see details of one
    • $ cat /etc/network/interfaces
    • $ ip link | grep ens3

  • node01 can be evaluated via ssh directly: $ ssh node01 ifconfig ens3

    • else via $ kubectl get nodes -owide

  • docker vhost: $ ssh node01 ifconfig -a | grep -i docker

  • docker vhost: $ ssh node01 ip link | grep -i docker

  • ip r = route

  • $ netstat -natulp | grep kube-sched

  • Lecture 209: Pod Networking

  • K8s does not have its own solution for:

    • Pod networking within cluster.
    • Pod networking from external hosts.

  • Networking Model:

    • Every pod must have an ip address.
    • Every pod on node must be able to intercommunicate.
    • Every pod across cluster nodes must be able to intercommunicate without NAT.

  • By implementing the equivalent topology on linux, we pave the road for how this works on K8s.

    • Three node cluster:
      • LAN@192.168.1.0
      • node{1,2,3}@192.168.1.1{1,2,3}

  • Tasks Per Host (3):

    • $ ip link set dev v-net-0 up
    • $ ip addr add 10.244.1.{1,2,3}/24 dev v-net-0

  • net-script.sh

echo "#!/bin/bash -c
NS=$2
HOST=$1
ip link add        # create veth pair
ip link set        # attach veth pair
ip link set
ip -n $NS link set" > net-script.sh
# Good ansible playbook use-case.

  • Better approach is to add those routes to router so that each bridge has its own local gateway.

    • Network : Gateway => 10.244.{1,2,3}.0/24 : 192.168.1.1{1,2,3}

  • Now the three bridge 10.244.x.x networks are aggregated into single network@10.244.0.0/16 via gateways.

  • We now update script to adhere to CNI standards:

  • net-script.sh

ADD)
# Create veth pair
# Attach veth pair
# Assign IP Address
# Bring Up Interface
ip -n <namespace> link set ...

DEL)
# delete veth pair
ip link del ...
  • We can make this script available to kubectl:
    • --cni-conf-dir=/etc/cni/net.d
    • --cni-bin-dir=/etc/cni/bin
    • ./netscript add

  • Lecture 210: CNI in kubernetes

  • Prerequisites:

    • Linux Network Namespaces
    • Networking in Docker
    • CNI Essentials
    • CNI Plugins

  • Here we look at how k8s does the same....

  • CNI Requirements For Container Runtime:

    • Container runtime must create network namespace.
    • Identify network to which container must be attached.
    • Container runtime to invoke Network Plugin bridge when container is ADDed
    • Container runtime to invoke Network Plugin bridge when container is DELeted
    • JSON format of the Network Configuration

  • kubelet.service configs:

    • --network-plugin=cni
    • --cni-bin-dir=/opt/cni/bin
    • --cni-conf-dir=/etc/cni/net.d

  • Look at live kubeclet options: $ ps -aux | grep kubelet

  • All CNI plugins: $ ls /opt/cni/bin

  • CNI config dir: $ ls /etc/cni/net.d

    • $ cat /etc/cni/net.d/10-bridge.conf

  • Referenced binary exec to run: $ cat /etc/cni/net.d/10-flannel.conflist | grep -i type

  • Lecture 211: CNI weave

    • WeaveWorks => we use net-script.sh for mock hundred+ node use-case
    • Route maintenance must stay secure and dynamically flexible.
    • WeaveWorks agent is installed on each node => they intercommunicate via weaveworks network service

  • Simplest weave install is where weave is installed as a pod.

  • Lecture 212: Practice Test - Explore CNI Weave
  • Lecture 213: Solution - Explore CNI Weave (optional)
curl -L https://github.com/weaveworks/weave/releases/download/latest_release/weave-daemonset-k8s-1.11.yaml > weave-setup.yaml
kubectl -f weave-setup.yaml
OR
kubectl apply -f "http://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

  • Lecture 215: Solution - Deploy Network Solution (optional)

  • Lecture 216: IP Address Management (IPAM) - Weave

    • This is NOT about node host ip addressing.
    • This IS about virtual network + subnet + ip address mgt.

  • Managing available virtual ip addresses per node with separate local file per node: ip-list.txt

IP     STATUS    POD
10.244.1.2 ASSIGNED BLUE
10.244.1.3 ASSIGNED RED
10.244.1.4 ASSIGNED ORANGE
10.244.1.5 FREE
  • We can implement this by following host-local plugin standards:
# invoke IPAM host-local plugin
ip = get_free_ip_from_host_local()

# assign ip address
ip -n <namespace> addr add
ip -n <namespace> route add

# look at ipam section to see if there is a default set:
cat /etc/cni/net.d/net-script.conf
  • Weave defaults:
    • Network: 10.32.0.0/12 = [ 10.32.0.1 => 10.47.255.254 ] ~ 10^6+ unique ip addresses
      • Subnets: 10.32.0.1,10.38.0.0,10.44.0.0
ps aux | grep kubelet | grep network # plugin is CNI, s consult those dirs
ls /etc/cni/net.d/
cat /etc/cni/net.d/*
kubectl get pods -n kube-system | grep weave | wc -l
ip addr show weave
ssh node01 ip route
  • Lecture 218: Solution - Networking Weave (optional)
  • default gateway => we want to see which node gets pod...
kubectl run busybox --image=busybox --dry-run=client -oyaml  -- sleep 1000 > pod-gateway.yaml
kubectl create -f pod-gateway.yaml
kubectl get pods -owide
k exec busybox -- route -n
k exec busybox -- ip route

  • Lecture 219: Service Networking

    • Despite what we have learned, proper connection of two services is how most pods intercommunicate.
    • If two cluster pods on same host want to intercommunicate, same deal: services

  • Two pods on node 1 of 3 want to interact within a cluster:

    • blue@10.244.1.2
    • orange@10.244.1.3 => orange-service@10.99.13.178 of type ClusterIP => available across cluster

  • To instead make a pod accessible outside of cluster => service of type NodePort

    • Nodeport also provides external port exposed on each node.

  • There are many service types, and each gets ip assigned by a range and

  • Sample cluster with 3 worker nodes: NODE{1,2,3}

    • Each worker combines efforts from kubelet and kube-proxy to reach clusterIP service:
      • new clusterip service ip@name is targeted, which actually forwards requests to underlying service pod
      • iptarget:ipforward pair underpin rules that are managed by kube-proxy

  • Kube-proxy has several methods available for managing these rules:

    • iptables=default, else: $ kube-proxy --proxy-mode [userspace | iptables | ipvs]
    • ipvs
    • userspace

  • When we create a service from running pod(s), it is assigned an ip from a configurable range of ip numbers:

    • SET: kube-api-server --service-cluster-ip-range ipNet => default = 10.0.0.0/24
    • GET: $ ps -aux | grep kube-apiserver

  • CIDR range = IP range, and they must NOT overlap.

    • 10.96.0.0/12 => [10.96.0.0 <=> 10.111.255.255] = service ip cidr
    • 10.244.0.0/16 => [10.244.0.0 <=> 10.244.255.255] = pod ip cidr
    • Node ip cidrs + pod ip cidrs + service ip cidrs => names@ips
    • pod@10.244.1.2=>node1 vlan
    • cluster-ip@10.103.132.104 cluster-wide
    • pod and service are NEVER assigned same IP

  • $ kubectl get services

  • We use iptables to query ClusterIP services: $ iptables -L -t nat | grep db-service

  • Kube-proxy log can be helpful: $ cat /var/log/kube-proxy.log. # default location

    • kube-proxy keeps a log with correct verbosity setting @ specified file path if not default path

  • Lecture 220: Practice Test - Service Networking

    • Practice Test Link
    • $ kubectl get nodes -owide
    • $ kubectl get pods -A -owide
    • $ kubectl logs weave -n kube-system # seeking pod cidr for cluster
    • $ cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep cluster-ip-range # seeking services cidr for cluster

  • How does this Kubernetes cluster ensure that a kube-proxy pod runs on all nodes in the cluster?

    • Inspect the kube-proxy pods and try to identify how they are deployed:
      • $ kubectl get ds -n kube-system; kubectl describe ds kube-proxy -n kube-system

  • Lecture 221: Practice Test Solutions- Service Networking

    • ip ranges for the following should not overlap: nodes + pods + services

  • Query different objects for ip ranges:

    • nodes: $ kubectl get nodes -owide
    • pods: $ kubectl -n kube-system logs weave-net-ytf -c weave | grep ipalloc
    • services: $ cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep 'service-cluster-ip-range'

  • Which routing table solution is kube-proxy using?

    • $ kubectl get pods -A | grep kube-proxy
    • $ kubectl -n kube-system logs kube-proxy-uy7 | grep mode

  • Kube-proxy usually runs as a DaemonSet:

    • $ kubectl -n kube-system get ds
    • $ kubectl -n kube-system describe ds xyz;

  • Lecture 222: DNS in K8s

  • Prerequisites

    • What is DNS?
    • Host/NS lookup => dig utility
    • Recorded host names: A, CNAME, etc
    • Domain hierarchy.

  • Objectives:

    • Object naming.
    • Service DNS records
    • POD DNS records

  • DNS resolution Nodes <=> LAN is not in scope here, as that is handled by external DNS service.

    • K8s manages all pods and services to be clusterwide-accessible.

  • Sample k8s Cluster:

    • workers: node{1,2,3}
    • pods:
      • test@10.244.1.5 on "node1"
      • web@10.244.2.5 on "node2"
    • services:
      • internal: web-service@10.107.37.188
      • external:

  • Call web-service from test pod in SAME namespace: $ curl http://web-service

  • Call web-service from test pod to web-service in DIFF namespace "apps": $ curl http://web-service.apps

    • The cross-namespace aspect is specified via subdomain created by dns per service.
    • ALL of that is aggregated under namespace = svc, so this should work too: $ curl http://web-service.apps.svc

  • KubeDNS resolves cluster service name with service ip.

  • Pod ips can also be tracked in DNS, optionally available...

  • Lecture 223: CoreDNS in K8s

  • Manual Scenario:

    • pod: test@10.244.1.5: echo 'web 10.244.2.5' >> /etc/hosts
    • pod2: web@10.244.2.5: echo 'test 10.244.1.5' >> /etc/hosts

  • Central DNS service entries

    • service: dns@10.96.0.10=(test@10.244.1.5,web@10.244.1.5)
      • pod1: dns@10.96.0.10: $ echo 'nameserver 10.96.0.10' >> /etc/resolve.conf
      • pod2: dns@10.96.0.10: $ echo 'nameserver 10.96.0.10' >> /etc/resolve.conf

  • Third such pod gets same treatment from kubelet+kube-proxy:

    • entry created in DNS for this host:ip
    • nameserver reference set in new-pod:/etc/resolve.conf

  • K8s works similarly for services, BUT not for pods.

    • Pod names are derived from pod ipaddresses
    • CoreDNS has superseded kubedns in k8s v1.12.x

  • CoreDNS

    • Deployed as pod of replicaset in kube-system namespace.
    • Requires config file: $ cat /etc/coredns/Corefile | grep -A5 kubernetes
    • CoreDNS is also as configmap named coredns in namespace kube-system.
    • Every new pod or service created will be given DNS nameserver info + given central DNS entry.
    • The nameserver value is mapped to the service: $ kubectl get services -n kube-system # get CLUSTER_IP
    • $ cat /etc/resolve.conf # This is done by kubelet!!
    • $ cat /var/lib/kubelet/config.yaml | grep -A2 clusterDNS
    • $ host web-service # returns fqdn + ip
    • $ cat /etc/resolv.conf | grep search
      • $ host web-service # supports services, but NOT pods, where full pod fqdn is needed...
      • $ host 10-244-2-5.default.pod.cluster.local

  • Lecture 224: Explore DNS Practice Test

    • Practice Test Link
    • kubectl get services -n kube-system
    • CoreDNS Config File: $ kubectl -n kube-system describe deployments.apps coredns | grep -A2 Args | grep Corefile
    • kubectl get configmaps -A
    • kubectl describe configmap coredns -n kube-system | grep -A 3 kubernetes

  • $ kubectl exec hr -- nslookup mysql # fail

  • $ kubectl edit deploy webapp # qualify host with namespace

  • Use nslookup the mysql service from hr pod to verify db service namespace is set:

    • $ kubectl exec hr -- nslookup mysql # fail
    • $ kubectl exec hr -- nslookup mysql.payroll # works

  • Lecture 225: Explore DNS Practice Test Solution

  • ConfigMap for coreDNS is configured under spec section = volumes:

    • $ kubectl get pod coredns-654tfv876-87 -n kube-system -oyaml | grep -A8 volumes:

  • Root domain/zone configured for k8s cluster:

    • $ kubectl describe configmap coredns -n kube-system | grep -A2 kubernetes

  • General exercise of how to name xyz web service from application abc ?

    • $ kubectl get svc; kubectl describe svc web-service | grep -i selector # call to web-service targets pod hr
    • Curl call into pod: $ NAMED_STRING='web-service.payroll' ; kubectl exec -it test -- curl http://${NAMED_STRING}
    • We also are given a UI to test the same connectivity, but call source remains unclear.

  • Lecture 226: Ingress

  • Scenario

    • www.my-online-store.com
    • wear-service via NodePort: svc ext
    • wear pod/app deployment: deployment
    • mysql-service via ClusterIP: svc int

  • Now we can reach service at ANY node: $ curl http://${NODE_IP}:38080

    • Adding pods with traffic => service handles traffic across pods.

  • We also need to support http://www.my-online-store.com:38080

    • This requires DNS entry.

  • We can further smooth url to drop port specifier via proxy-server, serving externally at port 80: http://www.my-online-store.com

  • From gcp, we can request service of type LoadBalancer, which autoprovisions both NodePort + Proxy services,

    • Now dns entry maps fqdn to gcp loadbalancer (proxy ext) ip address.

  • As any web site evolves, the url branches and paths will change:

    • online-store.com => online-store.com/wear (buy clothes service)
    • new branch => online-store.com/watch (video streaming service)

  • Initially we might be drawn to request a loadbalancer per service.

    • We DO need two separate services on this cluster.
      • That would equire new LB configs
      • That would require new LB dns entry

  • Ingress manages all of this for us by externally exposing services@ports and routing those requests cluster-wide.

    • We still need to publish exposed ingress service port via NodePort or loadbalancer, yet that is now a ONE TIME effort.
    • From there forward, we will push all of that work to the ingress controller.
      • authentications + SSL + loadbalancing + routing

  • K8s Ingress supports many reverse proxy or loadbalancing solutions, like haproxy and nginx

    • Deploy Ingress Controller
    • Configure Ingress Resources

  • Ingress controller services ARE NOT deployed by default with ingress controllers, so dont expect just a spec to work.

    • Ingress Controller Deployments require selecting ingress controller service:
      • gcp https LB (GCE)
      • nginx

  • This has intelligence for new resources and nginx configs:

$ kubectl create configmap --name nginx-configuration # used below!!
--- deployment-nginx-ingress-controller.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
spec:
  replicas: 1
  selector:
    matchLabels:
      name: nginx-ingress
  template:
    metadata:
      labels:
        name: nginx-ingress
    spec:
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0

      args:
        - /nginx-ingress-controller/
        - --configmap=$(POD_NAMESPACE)/nginx-configuration

      env:
        - name: POD_NAME
          value_from:
            fieldRef:
              fieldPath: metadata.name

        - name: POD_NAMESPACE
          value_from:
            fieldRef:
              fieldPath: metadata.namespace

      ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
---

  • The above auto-configures nginx for err-log-path + keep-alive + ssl-protocols

  • The below links service to deployment:

--- nginx-ingress-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
spec:
  type: NodePort
  ports
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  - port: 443
    targetPort: 443
    protocol: TCP
    name: https
  selector:
    name: nginx-ingress
---
  • Finally, we will need a service account to deploy it:
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount

  • Summary of above:

    • deployment
    • service
    • configmap
    • auth via service user:
      • Roles
      • ClusterRoles
      • RoleBindings

  • Next We Configure Ingress Resources (externally accessible + expressable targets)

    • www.online-store.com => wear app
      • /wear => wear app
      • wear.online-store.com
      • /watch => VID app
      • watch.online-store.com

  • File: ingress-wear.yaml => single app

apiversion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-wear
spec:
  backend:
    serviceName: wear-service
    servicePort: 80
---
$ kubectl create -f ingress-wear.yaml
$ kubectl get ingress
  • We are explicitly mapping namespaced urls to rule paths. Spec must support rules.
  • Ingress rules each handle URL traffic to allow or deny, and different route targets can be used.
  • File: ingress-wear-watch.yaml => 2+ apps
apiversion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-wear-watch
spec:
  rules:
  - http:
      paths:

      - path: /wear
        backend:
         serviceName: wear-service
         servicePort: 80

      - path: /watch
        backend:
         serviceName: watch-service
         servicePort: 80
---
$ kubectl create -f ingress-wear-watch.yaml
$ kubectl get ingress
$ kubectl describe ingress ingress-wear-watch # default backend is listed BUT not depoyed yet!!!
  • Third type of configuration is splitting by domain name.
    • wear.online-store.com
    • watch.online-store.com
...
spec:
  rules:
  - host: wear.online-store.com
    http:
      paths:
      - backend:
          serviceName: wear-service
          servicePort: 80

  - host: watch.online-store.com
    http:
      paths:
      - backend:
          serviceName: watch-service
          servicePort: 80
---
$ kubectl create -f ingress-wear-watch.yaml
$ kubectl get ingress
$ kubectl describe ingress ingress-wear-watch
  • Comparing splitting
    • by path
    • by subdomain

  • Lecture 227: Ingress Article

    • As we already discussed Ingress in our previous lecture. Here is an update. In this article, we will see what changes have been made in previous and current versions in Ingress. Like in apiVersion, serviceName and servicePort etc.

  • Now, in k8s version 1.20+ we can create an Ingress resource from the imperative way like this:

    • Format: $ kubectl create ingress --rule="host/path=service:port"
    • Example: $ kubectl create ingress ingress-test --rule="wear.my-online-store.com/wear*=wear-service:80"

  • Find more information + examples

  • References:

  • Lecture 228: Ingress 1 Practice Test
    • Here we have resources already deployed, and we will need to fix reported ingress issues.
    • Practice Test Link
    • kubectl describe ingress ingress-wear-watch -n app-space
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  namespace: critical-space
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - http:
      paths:
      - path: /pay
        pathType: Prefix
        backend:
          service:
           name: pay-service
           port:
            number: 8282

  • Lecture 229: Ingress 1 Practice Test Solution

    • $ kubectl get deploy -n app-space
    • $ kubectl get svc -n app-space # verify food-service@port

  • kubectl output sorting options

kubectl get deploy -n critical-space
kubectl get svc -n critical-space
kubectl create ingress new-ingress -n critical-space --rule="/pay=pay-service:8282"
kubectl get ingress -n critical-space
  • This still does not work, because url is being forwarded as is, and app does not support /pay url...
    • So we do a rewrite
kubectl get pods -n critical-space
kubectl logs $POD_NAME -n critical-space
kubectl edit ingress ingress-pay -n critical-space
# '    annotations:\n    nginx.ingress.kubernetes.io/rewite-target: /'

curl again and success

  • Lecture 230: Ingress - Annotations and rewrite-target

    • Different ingress controllers have different options that can be used to customise the way it works. NGINX Ingress controller has many options that can be seen here. I would like to explain one such option that we will use in our labs: Rewrite target option.

  • Our watch app displays the video streaming webpage at http://:/

  • Our wear app displays the apparel webpage at http://:/

  • We must configure Ingress to achieve the below. When user visits the URL on the left, his request should be forwarded internally to the URL on the right. Note that the /watch and /wear URL path are what we configure on the ingress controller so we can forwarded users to the appropriate application in the backend. The applications don't have this URL/Path configured on them:

    • http://:/watch --> http://:/
    • http://:/wear --> http://:/

  • Without the rewrite-target option, this is what would happen:

    • http://:/watch --> http://:/watch
    • http://:/wear --> http://:/wear

  • Notice watch and wear at the end of the target URLs. The target applications are not configured with /watch or /wear paths. They are different applications built specifically for their purpose, so they don't expect /watch or /wear in the URLs. And as such the requests would fail and throw a 404 not found error.

  • To fix that we want to "ReWrite" the URL when the request is passed on to the watch or wear applications. We don't want to pass in the same path that user typed in. So we specify the rewrite-target option. This rewrites the URL by replacing whatever is under rules->http->paths->path which happens to be /pay in this case with the value in rewrite-target. This works just like a search and replace function.

    • For example: replace(path, rewrite-target)
    • In our case: replace("/path","/")
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  namespace: critical-space
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - http:
      paths:
      - path: /pay
        backend:
          serviceName: pay-service
          servicePort: 8282

In another example given here, this could also be:

replace("/something(/|$)(.*)", "/$2")

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  name: rewrite
  namespace: default
spec:
  rules:
  - host: rewrite.bar.com
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /something(/|$)(.*)
  • Lecture 231: Practice Test - Ingress - 2
  • Here we deploy resources ourselves.
  • Practice Test Link
  • Lecture 232: Solution - Ingress Networking - 2
kubectl create namespace ingress-space
kubectl get namespaces
kubectl create configmap nginx-configuration --namespace ingress-space
kubectl get configmaps -A
kubectl create serviceaccount ingress-serviceaccount --namespace ingress-space
kubectl get serviceaccount
...
kubectl get roles
kubectl get rolebindings
  • Ingress Troubleshooting
    • We create configmap before we create ingress controller
    • kubectl expose deploy ingress-cotroller -n ingress-space -name ingress --port=80 --target-port=800 --type NodePort
    • kubectl edit svc ingress -n ingress-space
    • kubectl create ingress ingress-wear-watch -n app-space --rule="/wear=wear-service:8080" --rule="/watch=video-service:8080"
$ kubectl edit ingress ingress-wear-watch -n app-space
---
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "false"

  • Section 10: Design + Install a Kubernetes Cluster (32m)

  • Installatiom + Configuration + Validation

    • Design K8s Cluster: purpose + hosted where + workloads + ...

    • Choose k8s infra config

    • Choose network solution

    • HA K8s Cluster

    • Provision Infrastructure

    • Secure cluster commuication.

    • Kubernetes Release Binaries

    • Install Kubernetes Master Nodes

    • Install Kubernetes Worker Nodes

    • TLS Bootstrapping Nodes

    • Node end-to-end tests

    • Run & Analyze End 2 End Tests

  • Lecture 233: Download Presentation Deck

  • Lecture 234: Design a Kubernetes Cluster

    • Hosting prod apps:
      • HA Multi-node cluster with mutiple master nodes.
      • kubeadm or gcp etc
      • Up to 5k nodes.
      • Up to 150k pods.
      • Up to 300k ttal containers.
      • Up to 100 pods per node.

  • Multiple master nodes usually aggregate dns to external etcd service (usually multi-node)...

  • The following questions drive choices in physical cluster design.

    • Intended Purpose: [ edu + dev + prod ]
    • Footprint: Cloud Services + On Prem
    • Workloads: types + counts + kind (web vs data) + app resource reqs + traffic (heavy + burst)

  • Edu starts with single node clusters: minikube or single node vm with kubeadm/gcp/aws

  • Dev + testing:

    • multi-node-cluster with single master and multiple workers
    • kubeadm on managed cloud environments

  • Hosting Production Applications

    • High Availability multi-node cluster with multiple master nodes.
    • kubeadm or cloud scaling

  • Up to 5k nodes.

  • Up to 150k pods.

  • Up to 300k ttal containers.

  • Up to 100 pods per node.

  • Cluster node count drives default choice of cloud vm server class sizes.

  • Cloud vs on-prem

  • storage

    • high performance => ssd
    • Multiple concurrent connections => network based storage
    • Persistent shared volumes for shared access across multiple pods.
    • Label nodes with specific disk types.
    • Use node selectors to assign apps to nodes with specific disk types.

  • nodes:

    • vm or physical
    • minimum of 4 node cluster is common
    • master vs worker nodes
    • linux X86_64 Architecture.

  • Multiple masters are usually accompanied by dedicated physical etcd cluster.

  • Lecture 235: Choosing Kubernetes Infrastructure

    • Local laptop:
      • minikube: deploys local vm
      • kubeadm: requires pre-provisioned vms

  • turnkey solutions

    • provision vms
    • configure vms
    • use scripts to deploy cluster
    • maintain vms
    • eg: k8s on aws via kops
    • Other options: openshift + cloud foundry container runtime

  • managed or hosted solutions

    • k8s-aas
    • provider provisions vms
    • provider installs k8s
    • vmware cloud
    • vagrant
    • GCP: GKE
    • RedHat: Openshift Online
    • Azure: Kubernetes Service
    • AWS: EKS

  • VirtualBox implementation: 1 master + 2 workers

  • Lecture 236: Configure High Availability

    • HA assumes k8s can support multiple masters to avoid single-points-of-failure.
    • We can elect to have 2+ masters configured to support active:active mode.
    • We can send requests to either master, so loadbalancer is commonly employed between active masters

  • To support active:standby modes, there is a cluster leader election process:

  • The kube-controller-manager endpoint decides which master is active via first one wins.

$ kube-controller-manager --leader-elect true --leader-elect-lease-duration 15s  --leader-elect-retry-period 10s \
   --leader-elect-retry-period 2s

  • Scheduler has same approach with same cli options.

  • Is this a way to test-compare new k8s releases??

  • Four main components from highest level:

    • Controller Manager
    • Scheduler
    • API Server
    • etcd:
      • stacked topology = min hardware + simplest setup + simplest mgt + failureProne@port=2379
      • external etcd topology = less risk to cluster health

  • kube-apiserver has config for etcd service(s): --etcd-servers=URL1:PORT,URL:PORT2

    • $ cat /etc/systemd/system/kube-apiserver.service | grep etcd-servers

  • Final design for first target student cluster:

      1. Master1 + ETCD
      1. Master2 + ETCD
      1. Load Balancer Node For Multiple Active Masters
      1. Worker1
      1. Worker2

  • Lecture 237: ETCD in HA

  • What is:

    • etcd: distributed reliable key-value store that is simple + secure + fast
    • key-value store: non-tablular in sets of files => [ yaml <=> json ]
    • distributed system: available at multiple nodes at same port for reads + writes (single leader + int replication)
      • write is only considered complete when all hosts confirm: quorum = N(/2) + 1, rounded down
      • 2 nodes with one down => single node which is NOT enough for quorom!! Second instance still not enough!!
      • total nodes - fault tolerance = sufficient for quorum; prefer odd totals!!
      • table = instances:quorum:fault tolerance
    • RAFT protocol: Leader is elected based upon request of all 3 => winner requests from peers to leader
      • vanished leader prompts new election process from remaining nodes
    • how to operate etcd
    • etcd standard operation patterns
    • best practices on number of nodes

  • To install etcd on a server:

ETCD_VERSION=??
wget -q --https-only "https://github.com/coreos/etcd/releases/download/v${ETCD_VERSION}/etcd-v3.3.9-linux-amd64.tar.gz"
tar -xvf etcd-v${ETCD_VERSION}-linux-amd64.tar.gz
mv etcd-v${ETCD_VERSION}-linux-amd64 /usr/local/bin/
mkdir -p /etc/etcd /var/lib/etcd
cp ca.pem kubernetes-key.pem kubernees.pem /etc/etcd/
  • etcd service configure peers bootstrap: --initial-cluster
etcdctl --help
export ETCDCTL_API=3
etcdctl --help
etcdctl put name john
etcdctl get / --prefix --keys-only
etcdctl get name
  • etcd itself needs to be designed to withstand node outages, so we would prefer to use 3 nodes for distributed etcd service.
    • however, we have 2 masters here and no dedicated etcd nodes, so we will use the 2 master nodes to run etcd

  • Lecture 238: Important Update: Kubernetes the Hard Way

  • Installing Kubernetes the hard way can help you gain a better understanding of putting together the different components manually. An optional series on this is available at our youtube channel here.

  • The GIT Repo for this tutorial can be found here

  • Section 11: Install 'kubernetes the kubeadm way' (39m)

  • Lecture 239: Introduction to Deployment with Kubeadm

  • kubeadm steps:

    • provision 3 nodes
    • designate one node as master and 2 as workers
    • install container runtime on every node, which will be docker
    • install kubeadm tool on every nodes
    • initialize master server
    • test pod networking
    • join two worker nodes to cluster

  • Lecture 241: Deploy with Kubeadm - Provision VMs with Vagrant

  • Steps:

    • Clone git repo: $ git clone $URL; cd $REPO_NAME
    • Read Vagrantfile from git repo.
    • $ ( watch free || memory_pressure ) || echo 'Are you not running linux nor osx?'
    • Install virtualbox + vagrant.
    • $ vagrant status
    • $ time vagrant up 2>> vagrant.log # process times are of interest for vm provisioning and reboots
    • $ vagrant status
    • $ vagrant ssh kubemaster; ssh kubenode02 sudo bash free; exit; exit
    • $ vagrant ssh kubenode01; ssh kubenode02 sudo bash free; exit; exit
    • $ ip link && sudo cat /sys/class/dmi/id/product_uuid

  • Lecture 242: Demo - Deployment with Kubeadm

  • Now we have three nodes provisioned ready for k8s installation.

  • Review first tasks in docs

    • Having multiple network adapters would require ip routes be added to choose local correct adapter.
    • On all three nodes, let iptables see bridged traffic after ensuring br_netfilter module is loaded...
sudo bash
lsmod | grep br_netfilter || modprobe br_netfilter
lsmod | grep br_netfilter || echo 'Exiting due to modeprobe issues.' && exit
echo  "net.bridge.bridge-nf-call-ipv6tables = 1\nnet.bridge.bridge-nf-call-iptables = 1" > tee /etc/sysctl.d/k8sconf
sysctl --system
exit

  • Install container runtimes on all nodes (we use docker).

  • Compare sudo methods

    • This is also a good time to reflect on ansible playbooks.
    • Test-driven ansible is a wise move...

  • We could do all from master node via ssh, or three terminal windows to admin each node separately.

  • Test success: $ sysctl daemon-reload; sysctl restart docker; systemctl status docker.service

  • Install all three of these packages on all three machines:

    • kubeadm: command to bootstrap cluster
    • kubelet: takes orders from kubeapiserver + manages pods and containers locally
    • kubectl: command line utility to talk to clusters

  • These three tools must be kept in lockstep version-wise, and that needs to be done manually.

    • Same goes for kubens and kubectx!!
    • kubelet and api versions must match too.
    • $ systemctl daemon-reload; systemctl restart kubelet ; systemctl status kubelet
    • $ {kubeadm,kubectl,kubelet} --version

  • Next section regards cgroup driver, which is auto-setup by docker....

  • k8s docs: kubeadm setup

  • k8s docs: kubeadm HA

  • Kubeview is a nice open source visualization tool for k8s deps.

  • Initialize control-plane node:

      1. To support ease-of-upgrade to HA designs, --control-plane-endpoint alows us to set shared endpoint for all control plane nodes.
      1. Choose pod network add-on; verify whether it requires any args to be passed to kubeadm init, like --pod-network-cidr
        • remember that cluster node network and pod network are quite different.
      1. If we have non-docker container runtime: --cri-socket
      1. kubeadmin uses default gateway to set advertise address control plane api => alt: --api-server-address=
      1. From each node verify connectivity to gcr.io container image registry: $ kubeadm config images pull

  • $ kubeadm init --help

  • $ kubeadm init # ONLY on master node

  • Prep kubeadm init args string: $ # kubeadm init --pod-network-cidr 10.244.0.0/16 --apiserver-advertise-address=192.168.56.2

    • alt: echo command string to screen

  • Verify local ip with ifconfig: $ ifconfig | grep inet

  • Run kubeadm command: $ kubeadm init --pod-network-cidr 10.244.0.0/16 --apiserver-advertise-address=192.168.56.2 > init.log

  • Start cluster as regular user:

kubectl get nodes # check STATUS
logout; whoami;
kubectl get nodes # check STATUS
mkdir -p $HOME/.kube/
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# deploy pod network chosen from [here](https://kubernetes.io/docs/concepts/cluster-administration/addons/)
kubectl apply -f [podnetwork].yaml
kubectl get nodes # check STATUS
kubectl get pods -A -owide
watch kubectl nodes
  • We add nodes from the given node to have it join k8s cluster network:
$ kubeadm join 192.168.56.2:6443 --token 51x3ii.lzsu8haydiaqekji --discovery-token-ca-cert-hash sha256:$SHA_256_HASH
$ kubectl get nodes # check STATUS

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
# kubelet now restarts every few seconds, waiting in a crashloop for kubeadm instructions

  • From here:

    • prepare kubectl init arg string
    • run init command

  • To make kubectl run for non-root user, run these commands:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
  • To config kubectl root user, run these commands:
export KUBECONFIG=/etc/kubernetes/admin.conf

  • Add kubeadm users

  • Warning: Kubeadm signs the certificate in the admin.conf to have Subject: O = system:masters, CN = kubernetes-admin. system:masters is a break-glass, super user group that bypasses the authorization layer (e.g. RBAC). Do not share the admin.conf file with anyone and instead grant users custom permissions by generating them a kubeconfig file using the kubeadm kubeconfig user command.

  • Make a record of the kubeadm join command that kubeadm init outputs. You need this command to join nodes to your cluster.

  • The token is used for mutual authentication between the control-plane node and the joining nodes. The token included here is secret. Keep it safe, because anyone with this token can add authenticated nodes to your cluster. These tokens can be listed, created, and deleted with the kubeadm token command. See the kubeadm reference guide.

  • Check network host discovery: $ nmap -sP 192.168.2.1/24

    • Why does this not work all the time?
  • Lecture 244: Solution - Deploy a Kubernetes Cluster using kubeadm : (Optional)
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet=1.24.0-00 kubeadm=1.24.0-00 kubectl=1.24.0-00
sudo apt-mark hold kubelet kubeadm kubectl
$ kubectl get nodes
error: the server doesn't have a resource type "nodes"
  • Run kubeadm init
ifconfig eth0 # verify ip
kubeadm init --apiserver-cert-extra-sans=controlplane --apiserver-advertise-address 10.2.223.3 --pod-network-cidr=10.244.0.0/16 | tee -o init.log
  • setup kubeconfig file
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl get nodes
kubectl get pods -A

  • Section 12: End to End Tests On a Kubernetes Cluster

  • Lecture 245: Important Update: End to End Section

    • As per the CKA exam changes (effective September 2020), End to End tests is no longer part of the exam and hence it has been removed from the course. If you are still interested to learn this, please check out the complete tutorial and demos in our YouTube playlist

  • Section 13: Troubleshooting (65m)

  • Lecture 246: Troubleshooting - Section Introduction

    • Application Failure
    • Control PLane Failure
    • Worker Node Failure
    • Networking
  • Lecture 247: Download Presentation Deck

  • Lecture 248: Application Failure

  • Top-Down Approach

    • user: $ curl http://web-service-ip:node-Port
    • web service: $ kubectl describe service web-service # focus on endpoint + selector (svc2pod discovery)
    • web app pods: $ kubectl get pods -A -owide; kubectl describe pod webapp; kubectl logs webapp -f --previous
    • db service: $ kubectl describe service db # focus on endpoint + selector (svc2pod discovery)
    • db pods: $ kubectl get pods -A -owide

  • k8s docs: App Troubleshooting

  • Lecture 249: Practice Test - Application Failure
  • Lecture 250: Solution - Application Failure : (Optional)
    • Config current context with namespace: $ kubectl config set-config --current --namespace=beta; kubectl get pods
    • Look at ui app error. Then go top down if troubleshooting is not yet targeted.
    • Examine ui app container code.
    • mysql:3306 => mysql-service:3306 => webapp-mysql:8080 => web service:8080 => nodeport:30081
    • Set namespace alpha in current context: $ kubectl config set-context --current --namespace=alpha
    • $ kubectl get pods; kubectl get deploy; kubectl get svc # describe deploy
    • UI App Equiv: $ curl http://localhost:30081
  • Lecture 251: Control Plane Failure

    • kubeadm installation deploys controlplane as pods in kube-system namespace.

    • Controlplane could also be depoyed as services: $ service kube-{apiserver,controller-manager,scheduler} status

    • Check kubelet and kube-proxy as well: $ service kubelet status; service kube-proxy status

    • Check service logs: $ kubectl logs kube-apiserver-master -n kube-system

    • alt: $ sudo journalctl -u kube-apiserver

    • k8s docs: debugging clusters

  • Lecture 252: Practice Test - Control Plane Failure
    • Practice Test Link
    • Enable kubectl autocomplete $ source <(kubectl completion bash)
    • Make permanent: $ echo "source <(kubectl completion bash)" >> ~/.bashrc
    • Make alias for kubectl that supports autocomplete: $ alias k=kubecyl ; complete -F __start_kubectl k

  • Lecture 253: Solution - Control Plane Failure : (Optional)

    • Pod pending but unassigned to node = scheduler error: $ k describe pod kube-scheduler -n kube-system # events? nodeip?
    • $ kubectl get pods -n kube-system --watch

  • $ k logs kube-controller-manager-controlplane -n kube-system

  • Lecture 254: Worker Node Failure
    • $ kubectl get nodes -owide
    • $ kubectl describe node worker-1 # down nodes show status and lastHeartbestTime
    • Is node in list of cluster nodes? Is it running? Can we ssh in?
    • Check certificate not expired: $ openssl x509 -in /var/lib/kubelet/worker-1.crt -text | grep -i after
    • $ df -h; service kubelet status; sudo journalctl -u kubelet; top

  • Lecture 255: Practice Test - Worker Node Failure

  • $ ssh node01; systemctl status kubelet | grep -i active

    • $ service kubelet start ; service kubelet status | grep -i active
    • $ sudo journalctl -u kubelet | tail -f
    • $ cat /var/lib/kubelet/config.yaml
    • $ cat /etc/kubernetes/kubelet.conf | grep -i server
    • $ service kubelet start ; service kubelet status
    • $ exit; kubectl get nodes

  • Lecture 256: Solution - Worker Node Failure : (Optional)

  • Lecture 257: Network Troubleshooting

    • Network Plugin in kubernetes

  • Kubernetes uses CNI plugins to setup network. The kubelet is responsible for executing plugins as we mention the following parameters in kubelet configuration.

    • cni-bin-dir: Kubelet probes this directory for plugins on startup
    • network-plugin: The network plugin to use from cni-bin-dir; must match the name reported by a plugin probed from plugin directory.

  • There are several plugins available and these are some.

    1. Weave Net:
    1. Flannel:
    1. Calico:

  • Calico is said to have most advanced cni network plugin.

  • In CKA and CKAD exam, you won't be asked to install the CNI plugin

  • If asked you will be provided with the exact URL to install it. If not, you can install weave net from the documentation:

  • Note: If there are multiple CNI configuration files in the directory, the kubelet uses the configuration file that comes first by name in lexicographic order.

  • DNS in Kubernetes

    • Kubernetes uses CoreDNS. CoreDNS is a flexible, extensible DNS server that can serve as the Kubernetes cluster DNS.

  • Memory and Pods

    • In large scale Kubernetes clusters, CoreDNS's memory usage is predominantly affected by the number of Pods and Services in the cluster. Other factors include the size of the filled DNS answer cache, and the rate of queries received (QPS) per CoreDNS instance.

  • Kubernetes resources for coreDNS are:

    • a service account named coredns
    • cluster-roles named coredns and kube-dns
    • clusterrolebindings named coredns and kube-dns,
    • a deployment named coredns,
    • a configmap named coredns
    • service named kube-dns.

  • While analyzing the coreDNS deployment you can see that the the Corefile plugin consists of important configuration which is defined as a configmap. Port 53 is used for for DNS resolution.

    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
  • This is the backend to k8s for cluster.local and reverse domains.
proxy . /etc/resolv.conf

  • Forward out of cluster domains directly to right authoritative DNS server.

  • Troubleshooting issues related to coreDNS

      1. If you find CoreDNS pods in pending state first check network plugin is installed.
      1. coredns pods have CrashLoopBackOff or Error state
    • If you have nodes that are running SELinux with an older version of Docker you might experience a scenario where the coredns pods are not starting. To solve that you can try one of the following options:
      • a) Upgrade to a newer version of Docker.
      • b) Disable SELinux.
      • c) Modify the coredns deployment to set allowPrivilegeEscalation to true:
kubectl -n kube-system get deployment coredns -o yaml | \
  sed 's/allowPrivilegeEscalation: false/allowPrivilegeEscalation: true/g' | \
  kubectl apply -f -

  • d) Another cause for CoreDNS to have CrashLoopBackOff is when a CoreDNS Pod deployed in Kubernetes detects a loop.

  • There are many ways to work around this issue, some are listed here:

    • Add the following to your kubelet config yaml: resolvConf: This flag tells kubelet to pass an alternate resolv.conf to Pods. For systems using systemd-resolved, /run/systemd/resolve/resolv.conf is typically the location of the "real" resolv.conf, although this can be different depending on your distribution.

    • Disable the local DNS cache on host nodes, and restore /etc/resolv.conf to the original.

    • A quick fix is to edit your Corefile, replacing forward . /etc/resolv.conf with the IP address of your upstream DNS, for example forward . 8.8.8.8. But this only fixes the issue for CoreDNS, kubelet will continue to forward the invalid resolv.conf to all default dnsPolicy Pods, leaving them unable to resolve DNS.

      1. If CoreDNS pods and the kube-dns service is working fine, check the kube-dns service has valid endpoints.
kubectl -n kube-system get ep kube-dns

  • If there are no endpoints for the service, inspect the service and make sure it uses the correct selectors and ports.

  • Kube-Proxy

    • kube-proxy is a network proxy that runs on each node in the cluster. kube-proxy maintains network rules on nodes. These network rules allow network communication to the Pods from network sessions inside or outside of the cluster.

  • In a cluster configured with kubeadm, you can find kube-proxy as a daemonset.

  • kubeproxy is responsible for watching services and endpoint associated with each service. When the client is going to connect to the service using the virtual IP the kubeproxy is responsible for sending traffic to actual pods.

  • kube-proxy binary runs with following command inside the kube-proxy container:

$ kubectl describe ds kube-proxy -n kube-system
    Command:
      /usr/local/bin/kube-proxy
      --config=/var/lib/kube-proxy/config.conf
      --hostname-override=$(NODE_NAME)

  • So it fetches the configuration from a configuration file ie, /var/lib/kube-proxy/config.conf and we can override the hostname with the node name of at which the pod is running.

  • In the config file we define the clusterCIDR, kubeproxy mode, ipvs, iptables, bindaddress, kube-config etc.

  • Troubleshooting issues related to kube-proxy

      1. Check kube-proxy pod in the kube-system namespace is running.
      1. Check kube-proxy logs.
      1. Check configmap is correctly defined and the config file for running kube-proxy binary is correct.
      1. kube-config is defined in the config map.
      1. check kube-proxy is running inside the container
# netstat -plan | grep kube-proxy
tcp        0      0 0.0.0.0:30081           0.0.0.0:*               LISTEN      1/kube-proxy
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      1/kube-proxy
tcp        0      0 172.17.0.12:33706       172.17.0.12:6443        ESTABLISHED 1/kube-proxy
tcp6       0      0 :::10256                :::*                    LISTEN      1/kube-proxy

  • Section 14: Other Topics (12m)

  • Lecture 259: Pre-Requisites - JSON PATH

    • In the upcoming lecture we will explore some advanced commands with kubectl utility. But that requires JSON PATH. If you are new to JSON PATH queries get introduced to it first by going through the lectures and practice tests available here.

  • Afterwards, review two more sets of JSON PATH exercises with Kubernetes Data Objects.

  • Lecture 260: Practice Test - JSON PATH

  • Practice Test Link

  • xml: sample.xml

  • json: sample.json

  • yaml: sample.yaml

  • yaml

    • Key Value Pair: Fruit: Apple
    • Arrays/Lists: ``` Fruits:\n- Orange
    • Dictionary/Map: Banana:\n Calories: 105\n Fat: 0.4 g\n Carbs: s7 g

  • Elements of arrays and maps MUST align, or sibling items will be misinterpreted as sub-items.

  • We can combine arrays and maps.

    • Array (list) of maps (dicts)
    • Maps (dicts) of arrays (lists)

  • Model is a dict in a dict in car-example.yaml

Color: Blue
Model:
  Name: Corvette
  Year: 1995
Transmission: Manual
Price: $20,000
  • A yaml array supports multiple cars in single file: cars-example.yaml
- Blue Corvette
- Grey Corvette
- Red Corvette
- Green Corvette
- Blue Corvette
- Black Corvette
  • To have single file support all data above, replace each array element with entire corresponding dict entry.
    • Dict vs list vs list of dicts....
- Color Blue
  Model:
    Name: Corvette
    Model : 1995
  Transmission: Manual
  Price: $20,000

- Color Grey
  Model:
    Name: Corvette
    Model : 1995
  Transmission: Manual
  Price: $22,000

- Color Blue
  Model:
    Name: Corvette
    Model : 1995
  Transmission: Manual
  Price: $24,000
# This is a comment:  Array elements need not be unique, yet dict element keys must be unique!!
...
---
  • Data Structure Fundamentals
    • dictionary (map) is not ordered, so properties can be in any order and dicts are still equal
    • not true for list (array), which is ordered
employee:
  name: john
  gender: male
  age: 24
  address:
    city: edison
    state: 'new jersey'
    country: 'united states'
  payslips:
  -
    month: june
    amount: 1400
  -
    month: july
    amount: 2400
  -
    month: august
    amount: 3400

  • Lecture 261: Advanced Kubectl Commands

    • Using jsonpath suggests that kubectl may report on 100s of nodes and thousands of pods, deployments, replicasets...
    • How do we ensure that big responses are processed most efficiently?
    • If we get json data, then how to convert to yaml?
    • How to parse that output?

  • kubectl converts kubeapiserver responses into human-readable formats, usually filtered down.

    • see more details with command option: -owide
    • $ kubectl get nodes -owide # still quite incomplete
    • jsonpath format allows us to filter and format output as desired.
    • $ kubectl get pods -o json => Formulate json path query by eye: .items[0].spec.containers[0].image
    • $ kubectl get pods -o=jsonpath='{ .items[0].spec.containers[0].image }'
    • $ kubectl get pods -o=jsonpath="${JSON_PATH_QUERY_STRING}"
# get all item names
JSON_PATH_QUERY_STRING1='{.items[*].metadata.name}'; kubectl get pods -o=jsonpath="${JSON_PATH_QUERY_STRING1}"
# get all node architectures
JSON_PATH_QUERY_STRING2='{.items[*].status.nodeInfo.architecture}'; kubectl get pods -o=jsonpath="${JSON_PATH_QUERY_STRING2}"
# get all node cpu counts
JSON_PATH_QUERY_STRING3='{.items[*].status.capacity.cpu}'; kubectl get pods -o=jsonpath="${JSON_PATH_QUERY_STRING3}"
# combine queries into single call
JSON_PATH_QUERY_STRING4="${JSON_PATH_QUERY_STRING2}${JSON_PATH_QUERY_STRING3}"
kubectl get pods -o=jsonpath="${JSON_PATH_QUERY_STRING4}"

  • This raw output can be formatted using newlines and tabs and labels:

    • JSON_PATH_QUERY_STRING4="Node Names: ${JSON_PATH_QUERY_STRING2}\tCPUs: ${JSON_PATH_QUERY_STRING3}"
    • We would prefer to flip this output counter-clockwise, achieved via loops.

  • Loops - Range

QUERY_LOOP_STRING='{range .items[*]}{.metadata.name\ {"\t"} {.status.capacity.cpu} {"\n"}{end}'
kubectl get pods -o=jsonpath="${QUERY_LOOP_STRING}"

  • Custom Columns Add Column Headings + Cleaner :)

    • Pattern: $ kubectl get nodes -o=custom-columns=:
    • $ kubectl get nodes -o=custom-columns=NODE:.metadata.name,CPUs:.status.capacity.cpu

  • Sorting is also a supported option in jsonpath

    • $ kubectl get nodes --sort-by= .metadata.name

  • Sorting reversal and sorting by age are also nice.

  • Lecture 262: Practice Test - Advanced Kubectl Commands
    • Practice Test Link
    • $ kubectl get nodes -ojson > /opt/outputs/nodes.json
    • $ kubectl get node node01 -o json > /opt/outputs/node01.json
    • $ kubectl get nodes -o=jsonpath='{.items[*].metadata.name}'
    • $ kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.osImage}'
    • $ kubectl get pv --sort-by="{.spec.capacity.storage}"
    • $ kubectl config view --kubeconfig=my-kube-config -o jsonpath="{.contexts[?(@.context.user=='aws-user')].name}"

  • Section 15: Lightning Labs (1m)

  • Lecture 263: Lightning Lab Introduction

    • Welcome to the KodeKloud CKA Lightning Labs! This section has been created to give you hands-on practice in solving questions of mixed difficulty in a short period of time. This environment is valid for 60 minutes, challenge yourself and try to complete all 5-8 questions within 30 minutes. You can toggle between the questions but make sure that you click on END EXAM before the timer runs out. To pass, you need to secure 80%. Good Luck!!!

  • Disclaimers: Please note that this exam is not a replica of the actual exam. Please note that the questions in these exams are not the same as in the actual exam. Please note that the interface is not the same as in the actual exam. Please note that the scoring system may not be the same as in the actual exam. Please note that the difficulty level may not be the same as in the actual exam.

  • Lecture 254: Lightning Lab - 1

  • 1/7:

    • Upgrade the current version of kubernetes from 1.23.0 to 1.24.0 exactly using the kubeadm utility. Make sure that the upgrade is carried out one node at a time starting with the controlplane node. To minimize downtime, the deployment gold-nginx should be rescheduled on an alternate node before upgrading each node.

    • Verify two node cluster.

    • Cluster component versioning checklist.

    • This is tricky because we want to toggle pod placement while upgrading nodes in a rolling fashion.

      • Upgrade controlplane node first and next drain node node01 before upgrading it.
      • Pods for gold-nginx should run on the controlplane node subsequently.
      • Cluster Upgraded? How to verify?
      • Pods 'gold-nginx' running on controlplane?

  • Scenario:

    • kubectl get pods -owide; kubectl get nodes -owide; kubectl get deploy
    • We must shuffle deployment from node targeted for kubeadm upgrade to other node.
    • We must drain node. We must run kubeadm upgrade on node.

  • Steps:

    • check versions
    • backup cluster + test clone-restore
    • migrate deployment from controlplane to node01
    • drain controlplane node
    • upgrade controlplane node
    • verify controlplane upgrade + test + uncordon
    • migrate deployment from node01 to controlplane
    • drain node01
    • upgrade node01
    • verify node01 upgrade
  • 2/7
  • Print the names of all deployments in the admin2406 namespace in the following format:
DEPLOYMENT CONTAINER_IMAGE READY_REPLICAS NAMESPACE
<deployment name> <container image used> <ready replica count> <Namespace>

  • The data should be sorted by the increasing order of the deployment name.

  • Write the result to the file /opt/admin2406_data.

  • Example:

DEPLOYMENT CONTAINER_IMAGE READY_REPLICAS NAMESPACE
deploy0 nginx:alpine 1 admin2406
  • Here we report on deployments in a specific namespace:
kubectl get deploy -n admin2406 -oyaml > deploys.yaml
cat deploys.yaml
kubectl get deploy -n admin2406 -o=custom-columns=DEPLOYMENT:.metadata.name,\
CONTAINER_IMAGE:??,\
READY_REPLICAS:??,\
NAMESPACE:??\
 --sort-by=.spec.metadata.name > /opt/admin2406_data
  • ??

  • 3/7

    • A kubeconfig file called admin.kubeconfig has been created in /root/CKA
    • There is something wrong with the configuration. Troubleshoot and fix it.
    • Fix /root/CKA/admin.kubeconfig

  • ??

  • 4/7
    • Create a new deployment called nginx-deploy, with image nginx:1.16 and 1 replica. Next upgrade the deployment to version 1.17 using rolling update. Image: nginx:1.16. Task: Upgrade the image version of the deployment to 1:17.
    • $ kubectl create deployment nginx-deploy --image=nginx:1.16 --replicas=1
    • ??
  • 5/7
    • A new deployment called alpha-mysql has been deployed in the alpha namespace. However, the pods are not running. Troubleshoot and fix the issue. The deployment should make use of the persistent volume alpha-pv to be mounted at /var/lib/mysql and should use the environment variable MYSQL_ALLOW_EMPTY_PASSWORD=1 to make use of an empty root password.
    • Important: Do not alter the persistent volume.
    • Troubleshoot and fix the issues
kubectl get pods -n alpha
kubectl get deploy alpha-mysql -n alpha -oyaml > alpha-mysql-deploy.yaml
vi alpha-mysql-deploy.yaml
kubectl get pvc -n alpha
  • ??
  • ??
  • 7/7
    • Create a pod called secret-1401 in the admin1401 namespace using the busybox image. The container within the pod should be called secret-admin and should sleep for 4800 seconds. The container should mount a read-only secret volume called secret-volume at the path /etc/secret-volume. The secret being mounted has already been created for you and is called dotfile-secret. Pod created correctly?
$ kubectl get secrets -n admin1401
$ kubectl get volumes -n admin1401
$ kubectl get pods -n admin1401
$ kubectl create pod secret-1401 -n admin1401 --image=busybox -- sleep 4800
$ kubectl edit pod secret-1401 -n admin1401
$ kubectl replace --force -f /tmp/${XYZ}.yaml
  • ??

  • Section 16: Mock Exams (103m)

  • Lecture 265: Mock Exam - 1

  • 1/12.

    • Deploy a pod named nginx-pod using the nginx:alpine image.

    • Once done, click on the Next Question button in the top right corner of this panel. You may navigate back and forth freely between all questions. Once done with all questions, click on End Exam. Your work will be validated at the end and score shown. Good Luck!

    • Name: nginx-pod

    • Image: nginx:alpine

  • 2/12.

    • Deploy a messaging pod using the redis:alpine image with the labels set to tier=msg.
    • Pod Name: messaging
    • Image: redis:alpine
    • Labels: tier=msg

  • $ kubectl run messaging --image=redis:alpine label tier=msg

  • 3/12.
    • Create a namespace named apx-x9984574. Namespace: apx-x9984574
    • $ kubectl create namespace apx-x9984574; kubectl get ns
  • 4/12.
  • Get the list of nodes in JSON format and store it in a file at /opt/outputs/nodes-z3444kd9.json
  • $ kubectl get nodes -ojson > /opt/outputs/nodes-z3444kd9.json

  • 5/12: Create a service messaging-service to expose the messaging application within the cluster on port 6379.

    • Use imperative commands.
    • Service: messaging-service
    • Port: 6379
    • Type: ClusterIp
    • Use the right labels

  • $ kubectl expose --help

    • How does expose compare with create svc??

  • 6/12.

  • Create a deployment named hr-web-app using the image kodekloud/webapp-color with 2 replicas.

    • Name: hr-web-app
    • Image: kodekloud/webapp-color
    • Replicas: 2

  • $ kubectl create deployment hr-web-app --image=kodekloud/webapp-color --replicas=2

  • 7/12.
    • Create a static pod named static-busybox on the controlplane node that uses the busybox image and the command sleep 1000.
    • Name: static-busybox
    • Image: busybox
# Static pod needs to be made manually.
kubectl create pod static-busybox --image=busybox -oyaml -- sleep 1000 > pod-static-busybox.yaml
  • This is correct approach
service kubelet status; mkdir -p /etc/kubernetes/manifests/
cat /etc/kubernetes/manifests/static-web.yaml
STRING='FILE_TEXT_BELOW'; echo "${STRING}" > /etc/kubernetes/manifests/static-web.yaml
KUBELET_CONFIG_FILE='/etc/kubernetes/kubelet.conf'
echo  ' --pod-manifest-path=/etc/kubernetes/manifests/' >> ${KUBELET_CONFIG_FILE}
service kubelet restart
  • FILE TEXT BELOW
apiVersion: v1
kind: Pod
metadata:
  name: static-web
  labels:
    role: myrole
spec:
  containers:
    - name: web
      image: nginx
      ports:
        - name: web
          containerPort: 80
          protocol: TCP

  • 8/12.

    • Create a POD in the finance namespace named temp-bus with the image redis:alpine.
    • Name: temp-bus
    • Image Name: redis:alpine

  • $ kubectl run temp-bus -n finance --image=redis:alpine

  • 9/12. Fix orange pod crash.
$ kubectl edit pod orange
$ kubectl replace --force -f /tmp/kubectl-edit-1558520976.yaml
  • 10/12:
    • Expose the hr-web-app as service hr-web-app-service application on port 30082 on the nodes on the cluster.
      • The web application listens on port 8080.
      • Name: hr-web-app-service
      • Type: NodePort
      • Endpoints: 2
      • Port: 8080
      • NodePort: 30082

  • 11/12:

    • Use JSON PATH query to retrieve the osImages of all the nodes and store it in a file /opt/outputs/nodes_os_x43kj56.txt.
    • The osImages are under the nodeInfo section under status of each node.

  • $ JSON_PATH_STRING="{??.status.nodeInfo.osImages}"; kubectl get nodes -o=jsonpath="$JSON_PATH_STRING"

  • 12/12: Create a Persistent Volume with the given specification.
    • Volume Name: pv-analytics
    • Storage: 100Mi
    • Access modes: ReadWriteMany
    • Host Path: /pv/data-analytics
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-analytics
spec:
  capacity:
   storage: 100Mi
  accessModes:
   - ReadWriteMany
  hostPath:
    path: "/pv/data-analytics"
---
$ kubectl create -f pv-yaml

echo "alias reload='source ~/.bash_profile'\nalias kc=kubectl\ncomplete -F __start_kubectl kc\alias ka=kubeadmin\nalias kcx=kubectx\nalias kcn=kubens" >> ~/.bash_profile ; source ~/.bash_profile
source <(kubectl completion bash) # Assumes bash completion is enabled
echo "source <(kubectl completion bash)" >> ~/.bashrc # add autocomplete permanently to bash shell

  • First test cli command alias. Then test alias autocompletion...

    • Solution 1/12: Deploy a pod named nginx-pod using nginx:alpine image.

      • $ kubectl run nginx-pod --image=nginx:alpine; kubectl get pods -A; kubectl describe pod nginx-pod # | grep -i image

    • Solution 2/12. Deploy a messaging pod using the redis:alpine image with labels set to tier=msg.

      • $ kubectl run --help
      • $ kubectl run messaging --image=redis:alpine --labels="tier=msg"
      • $ kubectl get pods -A; kubectl describe pod messaging # | grep -i 'image|label'

    • Solution 3/12. Create a namespace of name apx-x9984574

      • $ kubectl create namespace apx-x9984574 ; kubectl get namespaces

    • Solution 4/12. Get a list of nodes in json format and store output in file at /opt/oututs/nodes-z3444kd9.json

      • $ kubectl get nodes -ojson > /opt/oututs/nodes-z3444kd9.json

    • Solution 5/12. Create a service messaging-service to expose the messaging app withi cluster on port 6379.

      • $ kubectl get pods -A; kubectl get services -A
      • $ kubectl create service --help
      • $ kubectl expose --help
      • $ kubectl expose pod messaging --port=6379 --name=messaging-service
      • $ kubectl get services; kubectl describe service messaging-service # | grep -i 'endpoint|label'
      • $ kubectl get pods -owide

    • Solution 6/12. Create a deployent named hr-web-app using image kodekloud/webapp-color with 2 replicas.

      • $ kubectl create deployment hr-web-app --image=kodekloud/webapp-color replicas=2
      • $ kubectl describe deployment hr-web-app

    • Solution 7/12. Create static pod named static-busybox on controlplane node using busybox image and command sleep 1000.

      • $ kubectl get nodes -owide
      • $ kubectl get pods -owide -A
      • $ kubectl run static-busybox --dry-run=client -oyaml --command -- sleep 1000 > static-pod.yaml
      • $ mv static-pod.yaml /etc/kubernetes/manifests/
      • $ kubectl describe pod static-busybox # | grep -i 'state:'
      • $ kubectl get pods -owide -A

    • Solution 8/12. Create pod in finance namespace named temp-bus with image redis:alpine.

      • $ kubectl run temp-bus -n finance --image=redis:alpine
      • $ kubectl get pods -n finance; kubectl describe pod temp-bus -n finance | grep -i image

    • Solution 9/12. A new app named orange is deployed. There is something wrong with it. Identify problem and fix.

      • $ kubectl get pods | grep -i orange # Init: suggests initContainer issue
      • $ kubectl describe pod orange # see Init Container = init-myservice
      • $ kubectl logs orange init-myservice
      • $ kubectl edit pod orange

  • Improved yaml editing with vi ??

EDITS_DIR='~/edits-yml/'; POD_NAME='orange'; OUT_FILE="pod-${POD_NAME}.yaml"
SESSION_SUBDIR="edits-subdir-session-$(date +'%s')/"
OUT_FILE_PATH="${EDITS_DIR}${SESSION_SUBDIR}${OUT_FILE}"
mkdir $EDITS_DIR; cd $EDITS_DIR; git init
kubectl get pod $POD_NAME -oyaml > $OUT_FILE_PATH
git add . ; git commit -m 'Raw file 
OLD_STRING=sleeeep
NEW_STRING=sleep
cat $OUT_FILE_PATH | grep -i ${OLD_STRING}
vi $OUT_FILE_PATH +":%s/${OLD_STRING}/${NEW_STRING}/g" # ':wq!' => autosave under new name?

kubectl replace --force -f $OUT_FILE_PATH
  • Solution 10/12. Expose the hr-web-app as service hr-web-app-service application on port 30082 on cluster nodes.
    • The web app listens on port 8080.
kubectl get pods -A
kubectl get deploy -A ; kubectl describe deploy hr-web-app
kubectl get services
kubectl expose --help
OUTFILE='hr-web-app-service.yaml'
kubectl expose deploy hr-web-app --type=NodePort --name=hr-web-app-service --port=8080 --dry-run=client -oyaml > $OUTFILE
kubectl describe deployment hr-web-app-service # | grep -i 'nodeport\|endp'
# edit NodePort=30082 OR $ kubectl edit service hr-web-app-service
OLD_LINE=$(grep -i nodeport $OUTFILE)
NEW_LINE='  - nodePort: 30082'
vi $OUTFILE +":%s/${OLD_STRING}/${NEW_STRING}/g" # redirect to file makes this angry!  :( ??
kubectl apply -f $OUTFILE
kubectl get deployment
kubectl get svc
kubectl get pods
kubectl describe deployment hr-web-app-service | grep -i 'nodeport\|endpoint'
  • Solution 11/12. Use jsonpath query method to retrieve osImage items of all nodes and store that output in file /opt/outputs/nodes_os_x43kj56.txt
    • hint: .items[*].status.nodeInfo.osImages
kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.osImage}' > /opt/outputs/nodes_os_x43kj56.txt
kubectl get node docker-desktop -o jsonpath='{.status.nodeInfo.osImage}'
cat /opt/outputs/nodes_os_x43kj56.txt

  • Solution 12/12. Create persistent Volume with the given spec:

    • volume name: pv-analytics
    • storage: 100Mi
    • access modes: ReadWriteMany
    • Host Path: /pv/data-analytics

  • k8s pod storage doc

  • k8s Storage Doc

  • pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-analytics
spec:
  capacity:
    storage: 100Mi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: /pv/data-analytics
---
$ kubectl get pv
$ kubectl apply -f pv.yaml
$ kubectl get pv; kubectl describe pv pv-analytics

ETCDCTL_API=3 etcdctl snapshot save --help
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=<trusted-ca-file> --cert=<cert-file> --key=<key-file> snapshot save <backup-file-location>
  • backup process
BKP_FILE=/opt/etcd-backup.db
ls $BKP_FILE || echo "Expected Absent: $BKP_FILE"
( ETCDCTL_API=3 etcdctl -h  || apt-get etcd ) || brew install etcd ; ETCDCTL_API=3 etcdctl -h
kubectl get pods -A | grep -i etcd
kubectl describe pod etcd-controlplane -n kube-system | grep '\-\-key-file\|\-\-cert-file\|\-\-trusted-ca-file'
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key  --cacert=/etc/kubernetes/pki/etcd/ca.crt snapshot save /opt/etcd-backup.db
du /opt/etcd-backup.db
read -p 'Can we confirm the file exists and has data?' X
  • 2/8. Create pod 'redis-storage' image=redis:alpine with volume of type emptyDir that lasts for the life of the pod.

    • $ kubectl create pod redis-storage --image=redis:alpine --dry-run=client -oyaml > pod-with-volume-redis-storage.yaml

    • pod-with-volume-redis-storage.yaml

apiVersion: v1
kind: Pod
metadata:
  name: redis-storage
spec:
  containers:
  - name: redis
    image: redis:alpine
    volumeMounts:
    - name: redis-volume
      mountPath: /data/redis
  volumes:
  - name: redis-volume
    emptyDir: {}

---
$ kubectl apply -f pod-with-volume-redis-storage.yaml
$ kubectl get pods
$ kubectl describe pod redis-storage

  • 3/8. Create a new pod called super-user-pod with image busybox:1.28. Allow the pod to be able to set system_time.

    • The container should sleep for 4800 seconds.
    • Pod: super-user-pod
    • Container Image: busybox:1.28
    • SYS_TIME capabilities for the container?

  • $ kubectl run super-user-pod --image=busybox:1.28 --dry-run=client -oyaml --command -- sleep 4800 > super-user-pod.yaml

  • super-user-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: super-user-pod
  namespace: default
spec:
  containers:
  - command:
    - sleep
    - "4800"
    image: busybox:1.28
    securityContext:
     capabilities:
        add: ["SYS_TIME"]
    name: busybox
---
$ kubectl apply -f  super-user-pod.yaml
$ kubectl describe pod super-user-pod
  • 4/8. A pod definition file is created at /root/CKA/use-pv.yaml.

    • Make use of this manifest file and mount the persistent volume called pv-1.

    • Ensure the pod is running and the PV is bound.

    • mountPath: /data

    • persistentVolumeClaim Name: my-pvc

    • persistentVolume Claim configured correctly

    • pod using the correct mountPath

    • pod using the persistent volume claim?

cat /root/CKA/use-pv.yaml # pod is to mount pv-1
kubectl get pv
kubectl get pvc
kubectl create -f /root/CKA/use-pv.yaml
kubectl describe pv pv-1
vi /root/CKA/use-pv.yaml
kubectl apply -f /root/CKA/use-pv.yaml
  • sample-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv
spec:
  storageClassName: ""
  claimRef:
    name: foo-pvc
    namespace: foo
  • pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: foo-pvc
  namespace: foo
spec:
  storageClassName: "" # Empty string must be explicitly set otherwise default StorageClass will be set
  volumeName: foo-pv
  • 5/8. Create a new deployment called nginx-deploy, with image nginx:1.16 and 1 replica.

    • Next upgrade the deployment to version 1.17 using rolling update.

    • Deployment : nginx-deploy

    • Image: nginx:1.16

    • $kubectl create deployment nginx-deploy --image=nginx:1.16 replicas=1

    • Task: Upgrade the version of the deployment to 1:17

    • Task: Record the changes for the image upgrade

  • 6/8. Create a new user called john. Grant him access to the cluster. John should have permission to create, list, get, update and delete pods in the development namespace . The private key exists in the location: /root/CKA/john.key and csr at /root/CKA/john.csr.

    • Important Note: As of kubernetes 1.19, the CertificateSigningRequest object expects a signerName.

    • Please refer the documentation to see an example. The documentation tab is available at the top right of terminal.

    • CSR: john-developer Status:Approved

    • Role Name: developer, namespace: development, Resource: Pods

    • Access: User 'john' has appropriate permissions

  • 7/8. Create a nginx pod called nginx-resolver using image nginx, expose it internally with a service called nginx-resolver-service. Test that you are able to look up the service and pod names from within the cluster. Use the image: busybox:1.28 for dns lookup. Record results in /root/CKA/nginx.svc and /root/CKA/nginx.pod

    • Pod: nginx-resolver created
    • Service DNS Resolution recorded correctly
    • Pod DNS resolution recorded correctly
  • 8/8. Create a static pod on node01 called nginx-critical with image nginx
    • Make sure that it is recreated/restarted automatically in case of a failure.
    • Use /etc/kubernetes/manifests as the Static Pod path for example.
    • static pod configured under /etc/kubernetes/manifests ?
    • Pod nginx-critical-node01 is up and running

  • Lecture 268: Mock Exam - 2 - Solution (Optional)

    • kubectl configured with alias + auto-complete
source <(kubectl completion bash); echo 'source <(kubectl completion bash)' >> ~/.bashrc
alias kcl=kubectl; complete -F __start_kubectl kcl
kcl get p[TAB] # test
  • 1/8.
ETCDCTL_API=3 etcdctl snapshot save -h
# need keys + certs + endpoints
cat /etc/kubernetes/manifests/etcd.yaml | grep 'cert\|key\|listen\-client\-urls'
ETCDCTL_API=3 etcdctl snapshot save /opt/etcd-backup.db --ca-cert=/etc/kubernetes/pki/etcd.ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key
stat /opt/etcd-backup.db
  • 2/8.
kubectl redis-storage --image=redis:alpine --dry-run=client -oyaml > pod-redis-storage.yaml
cat pod-redis-storage.yaml
# https://kubernetes.io/docs/concepts/storage/volumes/#emptydir
# Add volume to spec + add volumeMount to spec.containers
vi pod-redis-storage.yaml
kubectl create -f pod-redis-storage.yaml
kubectl get pods --watch
kubectl describe pod redis-storage | grep -A4 'Volumes:'
  • 3/8.
kubectl run super-user-pod --image=busybox:1.28 --dry-run=client -oyaml command -- sleep 4800 > super-user-pod.yaml
kubectl run --help | grep secur # security context supported ??
vi super-user-pod.yaml

cat /root/CKA/use-pv.yaml
kubectl get pv
kubectl get pvc
kubectl get pods
vi /root/CKA/use-pv.yaml
kubectl create -f use-pv.yaml
kubectl get pods --watch
kubectl describe pod use-pv
  • 5/8.
kubectl create deploy nginx-deploy --image=nginx:1.16 replicas=1
kubectl set image --help
kubectl describe deploy nginx-deploy
kubectl set image deployment/nginx-deploy nginx:nginx1.9.1
kubectl describe deploy nginx-deploy
# kubectl edit deploy nginx-deploy
# kubectl replace --force -f /tmp/
# kubectl get deploy nginx-deploy | grep -i image
  • 6/8.
ls ~/CKA
cat /root/CKA/john.key # private key
cat /root/CKA/john.csr # csr

  • Approach

    • create csr request from .csr + .key => user in system
    • create developer role => grant privs on pods --namespace=development
    • create rolebinding so that user is attached to that role

  • Generate CSR from k8s:

--- john-csr.yaml
apiVersion: certificates.k8s.io.v1
kind: CertificateSigningRequest
metadata:
  name: john-developer
spec:
  request: OIDUGHJVDKJHGDYJHGCVDBILHKJD...
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds:
  - client auth
  • Steps

    • $ cat john.csr | base64 | tr -n '\n'

    • $ vi john-csr.yaml

    • $ kubectl create -f john-csr.yaml

    • $ kubectl get csr

    • $ kubectl certificate approve john-developer

    • $ kubectl get csr

    • $ kubectl create role --help ...

    • $ kubectl create role foo --verb=create,get,list,update,delete -resource=pods -n development

    • $ kubectl describe role -n development

    • $ kubectl auth --help

    • $ kubectl auth can-i {get,create} pods --namespace=development --as john # expected to fail due to no rolebinding

    • $ kubectl create rolebinding --help

    • $ kubectl create rolebinding john-developer --role=developer --user=john -n development

    • $ kubectl describe rolebinding -n development

    • $ kubectl auth can-i {get,create} pods --namespace=development --as john # expected to succeed

    • Now we prompt jonathan to give it a shot from his desktop.

    • 7/8.

kubernetes run nginx-resolver --image=nginx
kubernetes get pods
kubernetes expose pod nginx-resolver --name=nginx-resolver-service --port=80
kubectl describe service nginx-resolver-service | grep -i endpoint
kubectl run busybox --image=busybox:1.28 -- sleep 4000
kubectl exec busybox -- nslookup nginx-resolver-service > /root/CKA/nginx.svc
kubectl get pods -owide # get ip
kubectl exec busybox -- nslookup 10-50-192-4.default.pod.cluster.local > /root/CKA/nginx.pod

kubectl get nodes -owide
ssh node01
ls /etc/kubernetes/manifests
kubectl run --help | grep restart # Default=Always
kubectl run nginx-critical --image=nginx --dry-run=client -oyaml static-pod-nginx-critical.yaml
mv static-pod-nginx-critical.yaml /etc/kubernetes/manifests
exit
kubectl get pods
kubectl describe pod nginx-critical-node01

  • Lecture 269: Mock Exam - 3

  • 1/9. Create a new service account with the name pvviewer. Grant this Service account access to list all PersistentVolumes in the cluster by creating an appropriate cluster role called pvviewer-role and ClusterRoleBinding called pvviewer-role-binding. Next, create a pod called pvviewer with the image: redis and serviceAccount: pvviewer in the default namespace.

    • ServiceAccount: pvviewer
    • ClusterRole: pvviewer-role
    • ClusterRoleBinding: pvviewer-role-binding
    • Pod: pvviewer
    • Pod configured to use ServiceAccount pvviewer ?
  • 2/9. List the InternalIP of all nodes of the cluster. Save the result to a file /root/CKA/node_ips.
    • Answer should be in the format: InternalIP of controlplaneInternalIP of node01 (in a single line)
  • 3/9. Create a pod called multi-pod with two containers.

    • Container 1, name: alpha, image: nginx

    • Container 2: name: beta, image: busybox, command: sleep 4800

    • Environment Variables:

      • container 1: name: alpha
      • container 2: name: beta

    • Pod Name: multi-pod

    • Container 1: alpha

    • Container 2: beta

    • Container beta commands set correctly?

    • Container 1 Environment Value Set

    • Container 2 Environment Value Set

  • 4/9. Create a Pod called non-root-pod , image: redis:alpine
    • runAsUser: 1000
    • fsGroup: 2000
    • Pod non-root-pod fsGroup configured
    • Pod non-root-pod runAsUser configured

  • 5/9. We have deployed a new pod called np-test-1 and a service called np-test-service. Incoming connections to this service are not working. Troubleshoot and fix it. Create NetworkPolicy, by the name ingress-to-nptest that allows incoming connections to the service over port 80.

  • Important: Don't delete any current objects deployed.

  • Important: Don't Alter Existing Objects!

  • NetworkPolicy: Applied to All sources (Incoming traffic from all pods)?

  • NetWorkPolicy: Correct Port?

  • NetWorkPolicy: Applied to correct Pod?

  • 6/9. Taint the worker node node01 to be Unschedulable. Once done, create a pod called dev-redis, image redis:alpine, to ensure workloads are not scheduled to this worker node. Finally, create a new pod called prod-redis and image: redis:alpine with toleration to be scheduled on node01.

    • key: env_type, value: production, operator: Equal and effect: NoSchedule

    • Key = env_type

    • Value = production

    • Effect = NoSchedule

    • pod 'dev-redis' (no tolerations) is not scheduled on node01?

    • Create a pod 'prod-redis' to run on node01

  • 7/9. Create a pod called hr-pod in hr namespace belonging to the production environment and frontend tier.
    • image: redis:alpine
    • Use appropriate labels and create all the required objects if it does not exist in the system already.
    • hr-pod labeled with environment production?
    • hr-pod labeled with tier frontend?

  • 8/9. A kubeconfig file called super.kubeconfig has been created under /root/CKA. There is something wrong with the configuration. Troubleshoot and fix it.

    • Fix /root/CKA/super.kubeconfig

  • 9/9. We have created a new deployment called nginx-deploy. scale the deployment to 3 replicas. Has the replica's increased? Troubleshoot the issue and fix it.

    • deployment has 3 replicas
    • $ kubectl edit deploy nginx-deploy
kubectl get pods
kubectl create serviceaccount --help
kubectl create serviceaccount pvviewer
kubectl create clusterrole --help
kubectl create clusterrolebinding --help
kubectl create clusterrole pvviewer-role --verb=list --resource=persistentVolumes
kubectl describe clusterrole pvviewer-role
kubectl create clusterrolebinding pvviewer-role-binding --clusterrole=pvviewer-role --serviceaccount=default:pvviewer
kubectl describe clusterrolebinding pvviewer-role-binding
kubectl run pvviewer --image=redis --serviceacount=pvviewer
kubectl get pods --watch
kubectl get pod pvviewer -oyaml | grep -i serviceaccount
  • 2/9
kubectl get nodes -o=json | jq -c 'paths' # shows how to reference elements.
kubectl get nodes -o=json | grep InternalIP -B4 -A4
kubectl get nodes -o=json | grep type | grep -v metadata
kubectl get nodes -o=jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}' > /root/cka/node_ips
cat /root/cka/node_ips
  • 3/9
kubectl run --help
kubectl run multi-pod --image=busybox --dry-run=client -oyaml --command -- sleep 4800 > multi-pod.yaml # container 2 first
vi multi-pod.yaml
---
spec:
  containers:
  - name: alpha
    image=nginx
  - command:
    - sleep
    - "4800"
    env:
    - name: name
    - value: alpha
  - name : beta
    image=busybox
    env:
    - name: name
    - value: beta
---
$ kubectl -f vi multi-pod.yaml
$ kubectl get pods
$ kubectl describe pod multi-pod

kubectl run non-root-pod --image=redis:alpine --dry-run=client -oyaml > non-root-pod.yaml
vi non-root-pod.yaml # securityContext: runAsUser: 1000 fsGroup
kubectl create -f non-root-pod.yaml
kubectl describe pod non-root-pod


kubectl get pods
kubectl get deploy
kubect get svc
k run curl --image=alpine/curl --rm -it -- sh
curl np-test-service # expected fail

kubectl get pod np-test-1 -oyaml | grep -A3 labels # matchLabels:\n run: np-test-1
echo '  ingress:\n    -\n     ports:\n    - protocol: TCP\n      port: 80' >> np.yaml
kubectl apply -f np.yaml
kubectl run alpine/curl --rm -it -- sh
curl np-test-service
  • 6/9
$ kubectl get pods -owide
$ kubectl taint node --help # key=value:effect
$ kubectl taint node node01 env_type=production:NoSchedule
$ kubectl describe node node01 | grep -i taint
$ kubectl run prod-redis --image=redis:alpine --dry-run=client -oyaml > prod-redis-pod.yaml
$ echo 'tolerations:\n- key: "env_type"\n  operator: "Exists"\n  value: "production"\n    effect: "NoSchedule" >> prod-redis-pod.yaml
$ kubectl create -f > prod-redis-pod.yaml
$ kubectl get pods -owide
$ kubectl describe pod prod-redis -oyaml | grep -i toleration -A3


kubectl run hr-pod --image=redis:alpine --labels="environment=production,tier=frontend"  -n hr --dry-run=client -oyaml 
kubectl describe pod hr-pod
  • 8/9
ls /root/CKA
cat /root/CKA/super.kubeconfig | grep server
cat ~/.kube/config | grep server
kubectl get nodes --kubeconfig/root/CKA/super.kubeconfig # fails with port error
vi /root/CKA/super.kubeconfig
kubectl get nodes --kubeconfig/root/CKA/super.kubeconfig
  • 9/9
kubectl get deploy
kubectl scale deployment nginx-deploy --replicas=3
kubectl get deploy # only 1 replica of 3 nodes
kubectl get pods -n kube-system
kubectl describe pod -n kube-system <angry-pod>
's/control1er/controller/g'
kubectl get pods
kubectl get deploy --watch

  • Section 17: Course Conclusion (1m)

  • Lecture 271: Frequently Asked Questions!

  • Q. Am I ready for the real exam?

    • A. If you have completed all lectures, labs and mock exams in this course, you are almost ready for the exam. To be sure, randomly different labs or mock exams and see how you perform. If you can breeze through those without having to peek at the hints or answer files, consider yourself ready. Remember, you only need 66% to clear the real exam, and you also have a free retake. So go for it!

  • Use the code - DEVOPS15 - while registering for the CKA or CKAD exams at Linux Foundation to get a 15% discount.

  • Q. How much time does it take to get results after the exam?

    • A. Results will be emailed within 48 hours.

  • Q. Is this course and mock exams sufficient for the exam?

    • A. Yes! The course covers all the topics required for the exam. If you practice the practice tests and mock exams enough times and make peace with the Kubernetes documentation pages, you should be good.

  • Q. Are the difficulty level of the mock exams similar to the ones in the exam?

    • A. We have created 2 new mock exams (2 & 3) that have a difficulty level more similar to the ones in the exam. So please check them out.

  • Q. Is auto-completion available in the exam environment?

    • A. Yes.

  • Q. How do I switch between clusters/environments in the exam environment?

    • A. The command to switch to the right cluster/context will be given at the top of each question. Make sure you run that always. Even if you think you are on the right cluster/context!

  • More FAQs

  • More questions

  • Lecture 273: Bonus Lecture - Accessing the Labs
  • Section 18: K8s cron jobs
    • p
- architecture: [ logical, physical ]
- community
- core object
- extension
- fundamentals
- networking
- operation
- security
- tool
- usertype
- workload
---
- scale --E

  • Course Sections:

    • 1 Introduction
    • 2 Core Concepts
    • 3 Scheduling
    • 4 Logging + Monitoring
    • 5 Application Lifecycle Management
    • 6 Cluster Maintenance
    • 7 Security
    • 8 Storage
    • 9 Networking: [ pod, cluster, lan, wan ]
    • 10 Design + Install Kubernetes Cluster
    • 11 Install k8s the kubeadm way.
    • 12 End to End Tests on a Kubernetes Cluster
    • 13 Troubleshooting
    • 14 Other Topics
    • 15 Lighning Labs
    • 16 Mock Exams
    • 17 Course Conclusion

  • Addenda:

    • generic cli skills
    • specific cli skills + alpha options
    • tpl: golang + sphinx + jekyll
    • K8s Cron Jobs
    • Github Integrations
    • Skills + Playgrounds
    • Languages
    • Cloud Vendors
    • Dev Ops
      • github actions
      • gitlab ci
      • hashicorp cloud
  • Course Resources
    • Syllabus
    • Video Lectures
    • Lab Playground
    • Hands-On Mock Tests
    • Slack Workspace
Raw
03-github-actions-course.md

Github Actions Course

  • Lecture 2. Introduction to GitHub Workflows & Actions

  • Workflows: Custom automated processes we can set up in our repo to build, test, package, release or deploy any code project on github.

  • Github Event Types

  • These events can be scheduled or triggered.

  • Github Triggering Events Include:

    • Push
    • Pull Request
    • Issue
    • etc

  • Each WORKFLOW runs as a JOB on one or many github servers: workflows --< jobs --< stages --< steps [actions + commands]

  • Runners

    • Any machine with the github actions runner installed on it.
    • A runner us responsible for running our jobs whenever an event happens and displays back the results.
    • It can be hosted by github or you can host your own runner.

  • Github hosted runners

    • Linux, Windows or MacOS virtual envs with commonly used preinstalled software.
    • No customimzation of the hardware configs.

  • Self-Hosted Runners

    • We manage machine ourselves.
    • More control over hardware, os, software than managed runners.

  • Pre-installed software:

    • curl git npm yarn pip
    • languages like python, ruby, nodejs
    • android sdk and xcode

  • Testing in parallel:

    • Desktop OS: ( osx ubuntu win )
    • Server OS: ubuntu freebsd rhel debian
    • Mobile OS: iOS android

  • Trending:

    • Legacy Machines
    • Current Machines
    • Emerging Machines
  • Lecture 3. A Brief Introduction to Writing in YAML Format
    • Word Processors are not ascii text editors...
      • vscode : preferences + extensions: [ prettier + markdown + yaml + yamllint+ yaml2json + json2yaml? ]
      • yq: yaml2json
    • Workflows are written in a yaml format: YAML is a serialized data format.
    • Other ascii data file formats: csv, json, xmls, toml ...
name: Ali
age: 28
address: "eiug:iugd"
active: true
---
key0: 
  key1: value
  key1a: 33
  key1b:
    key2:value
    key2b:value
---
array:
  - item1
  - item2
jsonAray: [item1, item2]
objectsArray:
  - key1: value1
    key1b: value1b
  - key2: value2
    key2b: value2b
---
longText: >
  lqurwygllqurwygllqurwygllqurwygl
  value2bvalue2bvalue2bvalue2b
  value2b

  • Longtext reads well in yaml but the special less than character > keeps this intact as single string.

  • Using special pipe char | will retain string returns.

  • Activate yaml2json to see this in json. Spaces, no tabs.

  • $ yq r -j file.yaml # outputs json

  • Key Value Pairs can be extended whereby the value is itself a dict/map.

  • Sub-maps work too.

  • yaml dict => jsonObject = json but no quotes needed

  • yaml array => jsonArray

  • We want REALLY good text file samples <=> yaml <=> json...

  • gitub workflows leverage yaml

  • Git+github team workflow knowledge are prerequisite.

  • Learn to use github API via gists + gh + terraform.

  • Lecture 4. Let's Create Our First Workflow

  • We set notifications on github actions under settings/notifications tab under target github account.

  • Provision Public Github Repo...

  • SET: GHUB_ACCOUNT_NAME='jeremy-donson'; NEW_PUBLIC_REPO_NAME='github-actions-test'; GH_ACCOUNT="http://github.com/${GHUB_ACCOUNT_NAME}" GH_NEW_EXPECTED_REPO_URL="${GH_ACCOUNT}/${NEW_PUBLIC_REPO_NAME}"

  • GET: ${NEW_GH_REPO_URL}

  • MATCH: ${GH_NEW_EXPECTED_REPO_URL} ${NEW_GH_REPO_URL}

cd ~/repos ; mkdir ${NEW_PUBLIC_REPO_NAME} ; cd $_
git init ; git remote add origin ${NEW_REPO_URL_SSH} ; git remote -v
GITHUB_ACTIONS_DIR_PATH='.github/workflows/'; GITHUB_ACTIONS_FILE='simple-shell-cmds-bash.yml'
mkdir -p ${GITHUB_ACTIONS_DIR_PATH}
echo "${WORKFLOW_TPL}" >> ${GITHUB_ACTIONS_DIR_PATH}${GITHUB_ACTIONS_FILE}
---
name: Simple Shell Commands bash
on: [push]
jobs:
  run-shell-command:
    runs-on: ubuntu-latest
    steps:
    - name: echo a string with error
      run: eccho 'Hello Github'
    - name: multiline script checks node and npm versions
      run: |
        node -v
        npm -v
---
git status
git diff
git add -A ; git commit -m 'First github action: Shell Commands'
git status
time git push --set-upstream origin master # ?? log: logstamp + elapsed + repo_size + ga_start + ga_stop
git status

  • Now we surf to ${NEW_GH_REPO_URL}/actions to watch the action execute and get stuck.

    • Cancel failed job in UI. Examine error.
    • Rerun button and logs filters are available in the web ui.
    • Under ... gui menu in upper right... DownloadLogArchive is a default job artifact.
      • Download and explore job log artifacts.

  • Fix github event code. Git Add-Commit-Push again. Watch ghub action execute and pass.

  • settings/secrets => we will add two new secrets and use them:

    • NAME: ACTIONS_RUNNER_DEBUG, VALUE: true }

    • NAME: ACTIONS_STEP_DEBUG, VALUE: true }

    • This enables github action debugging.

    • Rerun to reveal more verbose output. Check new logArchive artifacts.

    • We want to toggle enabling these github actions debug settings as needed. ??

      • In order to toggle via api: ??

  • Available events that can be used as action triggers

---
name: Bash + Py Shell Commands
on: [push]
jobs:
  run-shells-command:
    runs-on: ubuntu-latest
    steps:
    - name: echo a string with error
      run: eccho 'Hello Github'
    - name: multiline script checks python+node+npm versions
      run: |
        python -v
        node -v
        npm -v
    - name: python platform command
      run: |
        import platform
        print
        (platform.processor())
      shell: python
    - name: bash version command
      run: bash --version
---
git status; git add -A; git commit -m 'Second git workflow: simple-shells-cmds-py.yml'; time git push; git status
# watch job execute and see job ouput
  • Here we offer a bash template for this github actions yaml file...
    • simple-git-hub-action-v1-tpl.sh
GITHUB_ACTION_TPL_FILE_NAME='simple-git-hub-action-v1-tpl.sh'
TPL_STRING="name: ${GHA_NAME}
on: [${ACTION_TRIGGERS}]
jobs:
  ${JOB_NAME_01}:
    runs-on: ${$RUNS_ON_01}
    steps:
    - name: ${STEP_NAME_01}
      run: ${CMD_STR_01}

  ${JOB_NAME_02}:
    runs-on: ${$RUNS_ON_02}
    steps:
    - name: ${STEP_NAME_02}
      run: ${CMD_STR_02}

  ${JOB_NAME_03}:
    runs-on: ${$RUNS_ON_03}
    steps:
    - name: ${STEP_NAME_03_MULTI}
      run: |
      ${03_CMD_STR_01}
      ${03_CMD_STR_02}"

Now we use that v1 bash template to configure the next yaml workflow:

# GHA_NAME ACTION_TRIGGERS JOB_NAME_01 RUNS_ON STEP_NAME_01 CMD_STR_01 [ STEP_NAME_02_MULTI 02_CMD_STR_01 02_CMD_STR_02 ]
# template sections: action => jobs => steps

GITHUB_ACTION_TPL_OUTPUT_FILE_NAME='simple-github-action-from-tpl-v1.yaml'
ACTION_TRIGGERS='[push]';
JOB_NAME_01='runs-windows-commands'; RUNS_ON'windows-latest'
 STEP_NAME_01='Directory List: PowerShell'; CMD_STR_01='get-Location'
 STEP_NAME_02='Directory List: Bash'; CMD_STR_02='pwd'
# STEP_NAME_02_MULTI=''; 02_CMD_STR_01=''; 02_CMD_STR_02=''

echo -en  > ${GITHUB_ACTION_TPL_OUTPUT_FILE_NAME}
cat ${GITHUB_ACTION_TPL_OUTPUT_FILE_NAME}
read -p "Is template output ${GITHUB_ACTION_TPL_OUTPUT_FILE_NAME} ready to push?" Y
  [ "$Y" -eq yes ] || vi ${GITHUB_ACTION_TPL_OUTPUT_FILE_NAME}
git status; git add -A; git commit -m ''; git status; git push

  • Can templating a workflow be automated???

  • Third example: Bash On Windows

    • GITHUB_ACTIONS_FILE='simple-shells-cmds-win.yml'
---
name: Win Bash Shells Commands
on: [push]
jobs:
  run-linux-shell-commands:
    runs-on: ubuntu-latest
    steps:
    - name: python platform command
      run: |
        import platform
        print
        (platform.processor())
      shell: python
    - name Get Directory Bash
      run: pwd
  run-windows-shell-commands:
    runs-on: windows-latest
    needs:["run-linux-shell-commands"] # comment to remove inter-job dependency
    steps:
    - name: Get Directory PowerShell
      run: Get-Location
    - name: Get Directory Bash
      run: pwd
      shell: bash
---
git status; git add -A; git commit -m 'Third github workflow with two jobs on two platforms'; time git push ; git status
# watch job execute and see job ouput. Cancel any stalled jobs and check errors.
  • Jobs run dependent steps sequentially or in parallel.
    • Dependent jobs in same file can be specified using need: ["run-shell-command"] => confirm in github actions web gui
  • Job run on separate machines execute in parallel.

  • Lecture 6. Using a Simple Action

    • So far we have learned about passing in step commands.
    • Actions can also be passed into github workflow job steps.
    • Action template first example.
    • We can reference action master or any action repo release or commit.

  • In this workflow, we use an action: actions.yml

# ACTION_FROM_LOCAL_FILE or ACTION_FROM_REMOTE_REPO
---
name: Actions Workflow
on: [push]
jobs:
  runs-on: ubuntu-latest
  steps:
    - name: Simple JS Action
      id: greet
      uses: actions/hello-world-javascript-action@[master | v1] # See action repo releases to choose a release or commit.
      with: # reference action repo readme
        who-to-greet: Jojo

- name: Log Greeting Time
      run: echo "${{ steps.greet.output.time }}"
---
git status; git add -A; git commit -m 'Github Actions First Demo'; time git push ; git status
  • See output in web job ui log.
  • Lecture 7. The Checkout Action
    • The checkout action is more useful.
name: Action checkout Workflow
on: [push]
jobs:
  runs-on: ubuntu-latest
  steps:
#    - name: Checkout
#      uses: actions/checkout@v1
    - name: Current Dir + List Files In Dir
      run: |
        uname -a
        pwd
        ls -a
        echo 'gh sha: ',$GITHUB_SHA
        echo 'gh repo: ',$GITHUB_REPOSITORY
        echo 'gh workspace: ',$GITHUB_WORKSPACE
        echo "gh token: ${{ github.token }}"
        # git clone git@github:$GITHUB_REPOSITORY --token=${{ github.token }}
        # git checkout $GITHUB_SHA
    - name: Simple JS Action
      id: greet
      uses: actions/hello-world-javascript-action@[master | v1] # See action repo releases to choose a release or commit.
      with: # reference action repo readme
        who-to-greet: Jojo

- name: Log Greeting Time
      run: echo "${{ steps.greet.output.time }}"
---
git status; git add -A; git commit -m 'Github Repo Copy Absent From Worker'; time git push ; git status
  • Check pwd output from virtual runner: /home/runner/work/${GITHUB_REPO_NAME}/${GITHUB_REPO_NAME}/
    • The checkout action is used to copy repo to worker dir above.
    • Uncomment 2 lines above for first step + rename workflow, git commit and push.
  • Lecture 8. Quick Note!
    • Hello, hope you are enjoying the course so far :)
    • I forgot to mention in the previous lecture that the checkout action can also take some inputs as well. You can for instance change the commit that you are checking-out to, override the authentication token with another one, choose the fetch depth and more. You can check the rest of the options here. ??

  • Section 2: Events, Schedules, External Events + Filters

  • Lecture 9. Triggering a Workflow with Github Events & Activity Types

  • actions.yaml

name: Actions Workflow
on: [push, pull_request]
jobs:
  run-github-actions
  runs-on: ubuntu-latest
  steps:
    - name
---
$ git checkout -b develop
$ git status; git add -A; git commit -m ''; time git push --set-upstream origin develop ; git status
=> This push triggers action.
CREATE MERGE BRANCH PULL REQUEST IN WEB UI => Pull req triggers github action.
But closing the PR by default does NOT trigger github action.

  • Events That Trigger Workflows

    • Pull requests have many activity types, only some of which are action triggers

      • opened, synchronize, reopened

    • In cases where we implement action types, we use a map

  • on: [push, pull_request] becomes a map on: push: pull_request: types: [closed, assigned, opened, reopened]

  • $ git status; git add -A; git commit -m ''; git status

    • Reopen the pull request that was closed.
      • See reopen trigger another action workflow.
    • Assign myself to pull request.
      • See assign trigger another action workflow.
    • Close the pull request.
      • See close trigger another action workflow.

  • We can specify pull request branches later.

    • GITHUB_SHA = commit sha
    • GITHUB_REF = branch reference

  • Lecture 10. Setting a Schedule to Trigger Workflows

    • Comment out push directive in yaml file, so this is on pull requests only.
    • Use Crontab Guru to interpret cron schedule config strings
      • 1 * * * *
      • 1,2 * * * *
      • 1-3 * * * *
      • 0/15 * * * *
      • 20/15 * * * *
      • 0 12 * * *
      • 0 12 1 * *
      • 0 12 2 * *
      • 0 12 1 1 *
      • 0 12 1 JAN *
      • 0 1 * 1 *

  • More examples under Crontab Guru

name: Actions Workflow Scheduling
on:
  schedule: # min hr day dom month
    - cron: "0/5 * * * *" # max freq = 5m cadence
...
  • Do NOt let this run ongoing. This is achieved by comenting the schedule section.

  • Lecture Quiz 1: Test Your Cron Expression Knowledge

    • Question 1: What would be the expression for At 22:00 on every day-of-week from Monday through Friday.
    • Question 2: What would be the expression for At 04:00 on every day-of-month from 8 through 14.
    • Question 3: What would be the expression for At 00:05 in August.
    • Question 4: The shortest interval you can run scheduled workflows...
on:
  repository_dispatch:
    types: [build]
...

  • $ curl -s https://api.gitub.com/repos/jeremy-donson/github-actions-test/dispatches # NEW_PUBLIC_REPO_NAME

  • postman:

    • url above + header kvps:
      • Accept : application/vnd.github.everest-preview+json
      • Content-Type : application/json
    • body raw: { "event_type": "buildxxx" }
    • This should fail without github token for auth: settings/developer/personal-access-token => cfg => generate-new-token
      • Note = repo dispatch scopes = repo
    • paste token into basic auth password field => status 204 no content = success
    • git push => sets repo dispatch triggers on type build
    • Now the event build body when well-formed should trigger a workflow
    • body raw: { "event_type": "build" } => success

  • We can post to run dispatches from anywhere.

  • We can also have client payload appended to body:

{
  "event_type": "build",
  "client_payload": {
    "env": "production"
    }
}

  • The above payload is available in the workflow as: github.event.client_payload.env

  • actions-with-payload.yaml

name: Actions Workflow With Payload
on: [push, pull_request]
jobs:
  run-github-actions
  runs-on: ubuntu-latest
  steps:
    - name: payload
      run: echo ${{ github.event.client_payload.env }}
    - name: list files
      run: ls -a1
---
$ git status; git add -A; git commit -m ''; time git push # --set-upstream origin develop ; git status
  • This should trigger NO action.
  • Now when we curl the dispatch url we should trigger the action and query the client payload.
  • The on: key is able to be scoped to certain branches or !branches or files or !files.

  • Lecture 12. Filtering Workflows by Branches, Tags & Paths

    • Scheduling is also controlled by filtering Branches, Tags & Paths.
      • on: push: branches: - master - develop - 'feature/*' # only matches 1 level deep; does not match on /
      • on: push: branches_ignore: - 'temp/**'
      • on: pull_request: branches: - master - develop

  • ** will match all recursively...

  • Branch exclude while include: - '!feature/student'

  • Filter Pattern Cheat Sheet

name: Actions Workflow Branch + Tags + Paths
# This workflow will only be triggered on push of js files on certain branches.
jobs:
  run-github-actions
  runs-on: ubuntu-latest
on:
  push:
    branches:
    - master
    - develop
    - 'feature/*'
    tags:
      - v1.*
    paths:
      - '**.js'
      - '!filename.js'

  • Lecture Quiz 2: Workflows Filtering Quiz!

    • Workflows Filtering Quiz => 7 Questions
    • Now it's time to make sure we correctly understand filtering and patterns.
    • Take a look at this cheat sheet before/while solving this quiz.

  • Question 1: What pattern would we use if we would like to only run our workflow for branches: feature/featA, feature/featB, feature/featC?

  • Question 2: Which of these pattern will match the branch feature/featAC?

  • Question 3: Which of the following patterns not match with branch feature/featAC?

  • Question 4: If we have the following configuration in our workflow file, which of these branches will not trigger a workflow run?

on: 
  push:
    branches:
      - 'feature/feat[A-Z]'
      - '!feature/feat[E-G]'
  • Question 5: If we are filtering our workflow by path using this pattern: '*'. Which of the following paths will it match?
  • Question 6: If we are filtering our workflow by path using this pattern: '/src/'. Which of these files will it match?
  • Question 7: If we are filtering our workflow by path using this pattern: '*.jsx?'. Which of these files will it match?

  • Section 3: Environment Variables, Encryption, Expressions + Context

  • .github/workflows/env.yml

name: ENV Variables
on: push
env:
  WF_ENV: Available to all jobs.
jobs:
  log-env:
    runs-on: ubuntu-latest
    env:
      JOB_ENV: Available only to steps of this job.
    steps:
      - name: Log ENV Variables
        env:
          STEP_ENV: Available only to this job step.
        run: |
          echo "WF_ENV: ${WF_ENV}"
          echo "JOB_ENV: ${JOB_ENV}"
          echo "STEP_ENV: ${STEP_ENV}"
      - name: Log ENV Variables 2
        run: |
          echo "WF_ENV: ${WF_ENV}"
          echo "JOB_ENV: ${JOB_ENV}"
          echo "STEP_ENV: ${STEP_ENV}" # NULL
  log-default-env:
    runs-on: ubuntu-latest
    steps: 
    - name: Default ENV Variables
      run: |
        echo "HOME: ${HOME}"
        echo "GITHUB_WORKFLOW: ${GITHUB_WORKFLOW}"
        echo "GITHUB_ACTION: ${GITHUB_ACTION}"
        echo "GITHUB_ACTIONS: ${GITHUB_ACTIONS}"
        echo "GITHUB_ACTOR: ${GITHUB_ACTOR}"
        echo "GITHUB_REPOSITORY: ${GITHUB_REPOSITORY}"
        echo "GITHUB_EVENT_NAME: ${GITHUB_EVENT_NAME}"
        echo "GITHUB_WORKSPACE: ${GITHUB_WORKSPACE}"
        echo "GITHUB_SHA: ${GITHUB_SHA}"
        echo "WF_ENV: ${WF_ENV}"
        echo "JOB_ENV: ${JOB_ENV}"
        echo "STEP_ENV: ${STEP_ENV}"
---
$ git status; git add -A; git commit -m ''; time git push # --set-upstream origin develop ; git status
  • Lecture 13. Default & Custom Environment Variables
    • Review output of two jobs to reflect on env var scopes.
  • Lecture 14. Encrypting Environment Variables
    • In github repo gui, settings/secrets => add new secret: Name =WF_ENV,Value=hewxfcgvhbjh
...
env:
  WF_ENV: ${{ secrets.WF_ENV }}
...
  • We already have a secret builtin: ${{ secrets.GITHUB_TOKEN }} for push access.
name: Pull request labeler
on:
- pull_request
jobs:
  triage:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/labeler@v2
      with:
        repo-token: ${{ secrets.GITHUB_TOKEN }}
  • Example calling REST API: create-issue-on-commit.yml
name: Create issue on commit
on:
- push
jobs:
  create-issue:
    runs-on: ubuntu-latest
    steps:
    - name: Create issue using REST GITHUB API
        run: |
          curl --request POST \
          --url https://api.github.com/repos/${{ github.repository }}/issues \
          --header 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' \
          --header 'content-type: application/json' \
          --data '{
            "title": "Automated issue for commit: ${{ github.sha }}",
            "body": "This issue was automatically created by the GitHub Action workflow **${{ github.workflow }}**. \n\n The commit hash was: _${{ github.sha }}_."
            }' \
          --fail

$ git status; git add -A; git commit -m ''; time git push # --set-upstream origin develop ; git status

  • Watch workflow execute. Check github.com/[user]/[repo]/issues for new auto-generated issue.

  • We can also use the token to authorize push to repo from the virtual runner.

    • Append step...
steps:
  - name: Push a randomly updated file.
    run: |
      pwd
      ls -a
      git init
      git remote add origin "https://$GITHUB_ACTOR:${{ secrets.GITHUB_TOKEN }}@github.com/$GITUB_REPOSITORY.git"
      # NOT git@github.com:[user]/[repo].git"... we need token due to no ssh key.
      git config --global user.email "jjdonson@gmail.com"
      git config --global user.name "jeremy-donson"
      git fetch
      git checkout master
      git branch --set-upstream-to=origin/master
      git pull
      ls -a
      git status
      echo $RANDOM >> random.txt
      ls -a
      git status; git add -A ; git commit -m 'github action worker uses token for auth to push'; git push; git status
  • If we find a github action to do this, then use it!
  • Lecture 16. Encrypting & Decrypting Files
    • 64kb limit on github secrets
    • for secrets > 64kb... secret.json
cdvsbnjkimaginethissecretisMASSIVE

$ gpg --symmetric --cipher-algo AES256 secret.json # enter passphrase twice => secret.json.gpg
  • Passphrase now needs to be configured as a new secret on github. Then we commit and push the following:
jobs:
  decrypt:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - name: Decrypt file
      run: gpg --quiet --batch --yes --decrypt --passphrase="$PASSPHRASE" --output $HOME/decrypted-secret.json secret.json.gpg
      env:
        PASSPHRASE: ${{ secrets.PASSPHRASE }}
    - name: Print decrypted secret passphrase.
      run: |
        cat $HOME/decrypted-secret.json.gpg
        echo 'NOT SECURE!!'
  • push from desktop to github repo ought to fail because we have to pull first - worker pushed an encrypted file!
  • look at workflow action run output in web ui

  • Lecture 17. Expressions & Contexts

    • Expressions can include booleans, literals, strongs, floats, functions.
    • The secrets context has a lot of available variables and values to pass to yaml keys.
    • Github Docs: Contexts

  • context.yml => note the use of id: in one step

name: Context testing
on: push

jobs:
  dump_contexts_to_log:
    runs-on: ubuntu-latest
    steps:

      - name: Dump GitHub context
        id: github_context_step
        env:
          GITHUB_CONTEXT: ${{ toJSON(github) }}
        run: echo '${{ toJSON(github) }}'

      - name: Dump job context
        env:
          JOB_CONTEXT: ${{ toJSON(job) }}
        run: echo '${{ toJSON(job) }}'

      - name: Dump steps context
        env:
          STEPS_CONTEXT: ${{ toJSON(steps) }}
        run: echo '${{ toJSON(steps) }}'

      - name: Dump runner context
        env:
          RUNNER_CONTEXT: ${{ toJSON(runner) }}
        run: echo '${{ toJSON(runner) }}'

      - name: Dump strategy context
        env:
          STRATEGY_CONTEXT: ${{ toJSON(strategy) }}
        run: echo '${{ toJSON(strategy) }}'

      - name: Dump matrix context
        env:
          MATRIX_CONTEXT: ${{ toJSON(matrix) }}
        run: echo '${{ toJSON(matrix) }}'

      - name: Time of First Step
        run echo 'Time of first step github_context_step: ',${{ steps.github_context_step.outputs.time }}
  • Push + View UI Logs
$ git status; git add -A ; git commit -m 'github action worker uses token for auth to push'; git push; git status

  • Lecture 18. Using Functions in Expressions

    • We used toJson function above.
    • Here we use other functions.

  • context.yml

name: context
on: [push]
jobs:
  job-functions-string-demo
  runs-on: ubuntu-latest
  steps:
  - name: dump-step
    run: |
      echo ${{ contains( 'hello', 'll' ) }}
      echo ${{ startsWith( 'hello', 'he' ) }}
      echo ${{ endsWith( 'hello', 'lo' ) }}
      echo ${{ format( 'Hello {0} {1} {2}', 'World', '!', '!' ) }}
---
$ $ git status; git add -A ; git commit -m 'github action worker uses token for auth to push'; git push; git status
  • Watch job execute.
  • Lecture 19. The If key & Job Status Check Functions
    • Job status functions allow us to monitor and handle job errors.
    • Update/Append the following job to the file context.yml:
name: context
on: [push,pull_request]
jobs:
  job-one-functions:
    runs-on: ubuntu-latest
    steps:
    - ...

  job-two-bash-error:
    runs-on: ubuntu-latest
    if: github.event_name == 'push' # conditional@[job|step] level
    steps:

    - name: bash error step one
      env:
        GITHUB_CONTEXT: ${{ toJson(github) }}
      run: eccho $GITHUB_CONTEXT # SYNTAX ERROR

    - name: Dump GitHub context
      if: failure() # continue even if prior step fails
        env:
          JOB_CONTEXT: ${{ toJSON(job) }}
        run: echo "JOB CONTEXT: $JOB_CONTEXT"
      
  job-three-only-on-push:
    runs-on: ubuntu-latest
    if: github.event_name == 'push' # conditional@[job|step] level
    steps:
    - ...

  job-four-always:
    runs-on: ubuntu-latest
    if: always()
    steps:
    - name: Job 4 Step 4: Runner Context
    env:
      RUNNER_CONTEXT: ${{ toJson(runner) }}
    run: echo "$RUNNER_CONTEXT"
  • Study each of four ifs above and determine how each works.
  • We should state our log output expectations prior to:
    • push
    • viewing git action workflow execution in web ui => failure should not block further workflow steps

  • Section 4: Using Strategy, Matrix + Docker Containers in Jobs

  • Lecture 20. Continue on Error & Timeout Minutes

    • Workflow jobs run in parallel on designated workers.
      • When we want to chain jobs, we use the 'needs: ["run-shell-command"]' key for naming dependency on prior job running successfully.
        • We name job by id. This is how we prevent jobs from running in parallel, which is default.
      • We can now add 'if: failure()' to the second step and see it proceed 1 step thru step 1 error. Steps are serialized.
      • We can now add 'if: failure()' to the second step and see it proceed 1 step thru step 1 error. Steps are serialized.
      • 'continue-on-error: true' is another option for a step to fail without block other steps.
      • jobs: jobname: timeout-minutes: 360

  • Lecture 21. Using the setup-node Action

    • Deployment target matrix for testing across platforms and languages and versions...

  • The setup-node action makes this easy:

  • matrix.yml

name: matrix
on: [push,pull_request]
jobs:
  node-version:
    runs-on: ubuntu-latest
    steps:

    - name: Log Node Version 
      run: node -v

#    - name: Action: Setup Node
    - uses: actions/setup/node@v1
      with:
        node-version: 6

    - name: Log Node Version 
      run: node -v
---
$ git status; git add -A ; git commit -m 'github action setup node v 6'; git push; git status

  • Now we have a choice of strategy options.

  • matrix-strategy.yml

name: matrix-strategy-3x3
on: [push]

jobs:
  node-version:
    strategy:
      matrix:
        os: [macos-latest,ubuntu-latest,windows-latest]
        node_version: [6,8,10]
      max-parallel: 2
#     fail-fast: true

    runs-on: ${{ matrix.os }}
    steps:

    - name: Log Node Version 
      run: node -v

#    - name: Action: Setup Node
    - uses: actions/setup/node@v1
      with:
        node-version: ${{ matrix.node_version }}

    - name: Log Node Version 
      run: node -v
---
$ git status; git add -A ; git commit -m 'github action setup node matrix 3x3'; git push; git status
  • Lecture 22. Creating a Matrix for Running a Job with Different Environments
    • When we want to run the prior workflow multiple times with multiple node versions...
      • We may want to run only a portion of our entire matrix using key exclude:
jobs:
  node-version:
    strategy:
    matrix:
      os: [macos-latest,ubuntu-latest,windows-latest]
      node_version: [6,8,10]
      exclude:
      - os: ubuntu-latest
        node_version: 6
      - os: windows-latest
        node_version: 8
      include:
      - os: ubuntu-latest
        node_version: 8
        is_ubuntu_8: "true". # only for this matrix combo

runs-on: ${{ matrix.os }}
env:
  IS_UBUNTU_8: ${{ is_ubuntu_8 }}
steps:

- name: log node version
  run: |
- uses: actions/setup-node@v1
  with:
    node-version: ${{ matrix.node_version }}

- name: log node version 2
  run: |
    node -v
    echo $IS_UBUNTU_8
...
$ git status; git add -A ; git commit -m 'github action setup node matrix 3x3 + tweaks'; git push; git status

  • Lecture 23. Including & Excluding Matrix Configurations

  • Lecture 24. Using Docker Containers in Jobs

    • https://docs.docker.com/ci-cd/github-actions/

    • We would like to record the trending around size and build times for our docker images.

    • Then we can have our own images hosted under docker hub service.

    • .github/workflows/container.yml

name: Container
on: push

jobs:
  node-docker:
    runs-on: ubuntu-latest
    container: 
      image: node:13.5.0-alpine3.10
    # env port, options
    steps:
    - name: Log Node + OS Version
      run: |
        node -v
        uname -a
        cat /etc/os-release
---
This should fail because no mongodb service is up.
  • Lecture 25. An Overview of a Simple Dockerized NodeJS API
    • app.js
var express = require("express");
var app = express();
var mongoose = require"mongoose"();
var apiController = require("./controllers/apiController");
var port = 3000
app.use("/assets", express.static(__dirname + "/public"));
mongoose.connect("mongodb://localhost:27017/users",
 { useNewUrlParser: true, useUnifiedTopology: true },
  err => {
    console.log(err)
  }
);

apiController(app);
app.listen(port);
---
$ node app.js # fails due to no mongodb service
MongoDB Compass was used locally to create service for local testing.
$ node app.js # succeeds
$ git status; git add -A ; git commit -m 'github action setup node matrix 3x3'; git push; git status

FROM node:10.13-alpine
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm instal
COPY . .
EXPOSE 3000
CMD node app.js
  • docker-compose.yml
version: '2.1'
services:
  app:
    build:
    ports:
    - 3000:3000
  mongo:
    ports:
    - 27017:27017
---
$ docker-compose build
$ docker-compose start
POSTMAN GET http://localhost:3000/api/users
POSTMAN POST http://localhost:3000/api/user '{"username": "user2", "address": "khg" }
POSTMAN GET http://localhost:3000/api/users
  • Lecture 26. Running Multiple Docker Services in our Workflows

    • multiple docker services in github workflows

    • .github/workflows/containers-multi.yml

name: Container Multi Services
on: push

jobs:
  node-docker:
    runs-on: ubuntu-latest
    services: 
      app:
        image: alialaa17/node-api
        ports:
        - 3000:3000
      mongo:
        image: mongo
        ports:
        - 27017:27017
    steps:
    - name: POST a new user 
      run: | # runs in vm
        "curl -X POST http://localhost:3000/api/user -H 'Content-Type: application/json' -d '{\"username\": \"usernew4\",\"address\": \"dwed\"}'"
    - name: GET users 
      run: curl -s GET http://localhost:3000/api/users
---
sed -i 's/localhost\:/mongodb\:/g' app.js > app.js
$ git status; git add -A ; git commit -m 'github action container example'; git push; git status
  • Watch workflow execute in web ui.

  • Lecture 27. Running Docker Containers in Individual Steps

    • Now we achieve what we did with docker compose using github action workflow steps.
    • Note that we can only run containers on ubuntu.

  • Core docker concept: Dockerfile ENTRYPOINT is command that runs when container starts.

    • Literal entrypoint: echo "Hello"
    • Array entrypoint: ENTRYPOINT ['path/script', 'args'] => ['bin/echo', 'Hello']
ENTRYPOINT ['','Hello', 'World']
CMD ['World']
  • .github/workflows/containers-docker-steps.yml
name: Container Using Docker Steps
on: push

jobs:

  docker-steps:
    runs-on: ubuntu-latest
    container:
      image: node:10.18.0-jessie
    steps:
    - name: log node version
      run: node -v
    - name: step with docker
      uses: docker://node:12.14.1-alpine3.10
      with:
        entrypoint: /usr/local/bin/node
        args: -v

  node-docker:
    runs-on: ubuntu-latest
    services: 
      app:
        image: alialaa17/node-api
        ports:
        - 3000:3000
---
$ git status; git add -A ; git commit -m 'ga container steps'; git push; git status

  • Lecture 28. Creating our Own Executable File and Running it in our Steps

    • We can run our own script configured as docker container ENTRYPOINT.

  • script.sh

#!/bin/bash
echo -e "We ran a script with $0 as ENTRYPOINT and $1 + $2 as ARGS"
---
chmod +x script.sh

    - uses: actions/checkout@v1
    - name: step with docker
      uses: docker://node:12.14.1-alpine3.10
      with:
        entrypoint: ./script.sh
        args: two strings
---
$ git status; git add -A ; git commit -m 'ga container steps'; git push; git status

export SLACK_WEBHOOK='https://hooks.slack.com/services/...'
SLACK_MESSAGE="hello" slack-notify
  • In docker container:
export SLACK_WEBHOOK='https://hooks.slack.com/services/...'
docker run -e SLACK_WEBHOOK=$SLACK_WEBHOOK -e SLACK_MESSAGE="hello" -e SLACK_CHANNEL ...

  • We will need to create new slack app at api.slack.com: incoming web hook enabled = we need webhook url as secret

    • secrets.SLACK_WEBHOOK = webhook-url

  • slack-message.yml

name: Send Slack Message Via Docker Container
on: push

jobs:
  docker-steps
    steps:
    - name: run a script
      uses: docker://technosophos/slack-notify
      env:
        SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
        SLACK_MESSAGE: "Hello Slack"

    - name: send a slack message via docker
---
$ git status; git add -A ; git commit -m 'ga container steps'; git push; git status

  • Section 5: Creating a CI/CD Workflow To Automate Testing

  • Lecture 30. Creating a ReactJS Boilerplate Application

APP_NAME='react-ts-app'
time npx create-react-app $APP_NAME --template typescript
cd $APP_NAME
npx create-react-app $APP_NAME --use-npm
npm run start
curl localhost:3000
vi App.js:19 # Edit Text: 'Learn React'
npm run test
vi App.test.js:7 # Edit Text: 'learn react'
# autorun upon save: npm run test
CI=true npm run test
ls -a
CI=true npm run test -- --coverage
ls -a
chrome coverage/index.html
  • Lecture 31. Building & Testing the Application Locally
    • First we read package.json .scripts{start,build,test,eject}
  • To prepare for production, single build folder is produced:
time npm run build
ls -a
ls -a build
  • Lecture 32. Deploying the Application using Surge
    • surge is a static web site hosting service.
time npm run build
npm install --global surge
# must create account and login, then
surge
# append build to the deploy resource path ...  react-app/build
# get publish_url + REUSE publish_url on builds
vi App.js
time npm run build
# get publish_url + REUSE publish_url on builds
curl -s publish_url

  • Lecture 33. Using Prettier to Check for Code Formatting Rules

  • Test drive the prettier playground, where we can make some config choices.

  • Here is standard set of prettier configs in prettier.rc file.

  • VSCode supports prettier configs in ./.prettierrc and ./.prettierignore

$ echo -e "build\ncoverage\nnode_modules\package-lock.json" > .pretterignore
$ npx prettier --check "**/*.js" # compare project js file formatting to prettier config
$ npx prettier --write "**/*.js"
$ npx prettier --check "**/*.js" # run again after change
  • Now we edit package.json under scripts to automate this:
"format:check": "prettier --check \"**/*.{js,json,...}\""
"format:correct": "prettier --write \"**/*.{js,json,...}\""

  • Lecture 34. Let's Discuss Our Workflow Plan

    • Team git and github workflows are never one-size-fits-all.
    • Workflow design + planning will produce artifacts useful for development.
      • Project Repos: [upstream,fork,clone,subtree]
        • Envs: [local,staging,production]
        • Team Branches: [development,master]
        • Local Solo Branch: newFeature
        • Workflow Types
        • Deliverable Types: Features + Tests + Docs

  • We can come up with our own diagramming legend for ci/cd workflows by type:

    • locally cloned fork => {newlyTestedFeature,betterFeatureTests,betterFeatureFixes}
      • commit=>push2fork=>pr=>review=>?approval =>mergeIntoDevelop(branch)=>deploy2staging(env)

  • Example End To End 'Workflow Engine':

    • Two team branches: develop + master, and the workflow follows.
      • These team branches correspond to two environments: staging + production
    • One transitional branch named feature1
    • Branch feature1 is used to generate pull requests, which triggers a workflow to test feature1 code.
    • Merge request (feature1 => develop) requires team approval to be merged into team branch: develop
    • Develop branch code testing follows, which should confirm what we just auto-tested.
    • Deploy from team branch develop to staging env comes with more testing.
    • Now we can create a pull request to merge this same feature into master team branch.
    • Merge request (develop:feature1 => master) requires team approval to be merged into team branch: master
    • This merge again triggers build and deploy workflow, but this time from master team branch to production environment.
    • Yet more testing on master prior to deploy.
    • Environments count: [ local, runners, staging, production ]
    • Workflow counts: [prs:2,deploys:2]

  • Workflow design planning => fork+clone, push2fork, pr2merge2branch, deploy2env

    • Workflow planning includes patterns for workflow file names.

    • 01-----[issue_id].yaml

      1. Workflow File Name: 01-feature1-pull-request-testing-develop.yml
      • Triggered By Push Merge PR => Team Branch: develop
        • Install Dependencies
        • Check Code Formatting
        • Run Automated Tests
        • Upload Code Coverage as Artifact
        • Cache Dependencies
      1. Workflow File Name: 02-feature1-build-develop-deploy-to-staging.yml
      • Triggered By Newly Merged Code in Team Branch => develop
      • Install Dependencies
      • Check Code Formatting
      • Run Automated Tests
      • Upload Code Coverage as Artifact
      • Build Project
      • Upload Build as Artifact
      • Deploy to Staging Server + Confirm Expectations??
      • Cache Dependencies
      1. Workflow File Name: 03-feature1-pull-request-testing-master.yml
      • Triggered By Push Merge PR => Team Branch: master
      1. Workflow File Name: 04-feature1-build-master-deploy-to-production.yml
      • Triggered By Newly Merged Code in Team Branch => develop
      • Updates + Appends:
        • Create a release.
        • Deploy to Production Server
        • Upload Coverage to Codecov

  • We can derive scripts #3 + #4 from workflow templates which reflect scripts #1 + #2.

      1. 01-team-merge-workflows.yml.tpl: from repo:branch to repo:branch
      1. 01-team-deploy-workflows.yml.tpl: from repo:branch:sha to env@URL

  • Note that all github project release info in web ui + api are auto-updated upon new release.

  • Any recurent job failures ought to trigger auto-notification by creating a project issue.

  • Dev Team CI Alerts + Notifications Stream Workflow (chained events + alert actions)

    • Upon new release creation, send slack message.
    • Github Actions Workflow Job Failure Triggers Auto-Creation of Project Issue.
      • Upon new issue creation, slack message is triggered.
  • Lecture 35. Setting Up Our Repository
    • github project file: CODEOWNERS
    • Prerequisites: VSCode Setups + Skills
    • Files in repo MUST have owners: file-pattern @owner => drives single approver pr alerts
cat ./.github/workflows/*.yml
cat ./.github/CODEOWNERS || echo 'see link above in docs'
  • Now we need to create new github remote repo + add as origin so we can commit and push.
    • Create a new github repo.
      • Method 1: Manually via web ui. Delete newly created repo.
      • Method 2: Provision github repos with gh. Delete newly created repo.
      • Method 3: Provision github repos with terraform. Delete newly created repo.
git status; git add -A; git commit -m 'CODEOWNERS' # Who owns ./.github/CODEOWNERS ?
git status; git remote add origin git@github.com:[user]/[repo]; git push -u origin master
git status

  • Now we can begin to implement our workflow plan.

    • Create github project remote branch via web ui: develop
    • Configure this branch with branch protection rules under settings/branches
      • master => PRs Must Be Reviewed Prior To Merge => all options box 1 and box 2 + include admins
      • develop => same configs : git fetch if on remote
      • test with push of local commit to gh project branch without pr+aproval+merge => should fail!!

  • Consider how to fork + clone locally => create + configure branch locally, to then push via cli.

    • Anti-pattern: CLONE_EXTERNAL
    • Forked Repo Chain Pattern: [ external <=> forks <=> clones <=> subtrees ]
      • FORK_EXTERNAL
      • SYNC_EXTERNAL_TO_FORK
      • CLONE_FORK
      • SYNC_FORK_TO_CLONE
      • SYNC_CLONE_TO_FORK
      • SYNC_FORK_TO_EXTERNAL: Pull Request of Action Type = Merge Feature From Branch develop into Branch master
git remote -v # remote relatives?
git diffs # external <=> forks <=> clones
# git checkout -b example_workflow_new_feature_01
# How do we configure master + develop with branch protection rules?  Why?
# What about CODEOWNERS file with newly merged features?
# --set-upstream check
00_NEW_BRANCH_LOCAL_NAME='example-workflow-new-feature-solo-local'
01_NEW_BRANCH_LOCAL_CALL=$( eval "git checkout -b ${NEW_BRANCH_LOCAL_NAME}" )
02_NEW_BRANCH_LOCAL_EXEC=$( ${01_NEW_BRANCH_LOCAL_CALL} )
03_BRANCH_CONFIRM=$( git branch | grep $00_NEW_BRANCH_LOCAL_NAME )
# diffChecks?
# reqsFeature?
# mocks?
# tests?
# fails?
# succeeds?

  • Work remotely with gh for remote cli method to create branch develop + confgure branch protection rules.

  • There are constant tensions to maintain sync between:

    • remote git projects and every team member
    • forks of remote git projects
    • clones (local) of repos (remote)
    • clones (local) of forks (remote)
    • new branches with new code@sha to test + pr
    • horizontal dependencies
    • vertical dependencies
    • requirements
    • testing
    • docs
    • issues
    • releases => patches

  • Lecture 36. Creating the Develop Pull Requests Workflow

    • workflow 1 of 4... triggered by pull requests (iter01)

  • team-ci-example-1-of-4.yml

name: Team CI Example 1 of 4
on:
  pull_request:
    branches: [develop]
jobs:
  build:
    runs-on: ubuntu-latest
  steps:
  - name: Implement Checkout Action
    uses: actions/checkout@v2

  - name: Implement Setup Node Action
    uses: actions/setup-node@v1
    with:
      node-version: "12.x"
  
  - run: npm ci
  - run: npm run format:check
  - run: npm test -- --coverage
    env:
      CI: true
---
$ npm ci --help
$ git branch # confirm example_workflow_new_feature_01 is secure in team branches [develop + master]
# build successful checked
$ git status; git commit -m 'First of four example team workflows that are branch-secured and chained.'; git push
$ git status # local git repos
$ git diffs
$ gh status # remote github repos
$ gh diffs
$ gh pr --help # merge example_workflow_new_feature_01@sha => develop

  • It is helpful to create sample second github user test account so that we can solo-test collaborators.

    • Create new github test user devlead1 with unique email id. Team with write privs: devs
    • Invite new github user to teams + projects. Those privs will be branch-secured.
    • New user accepts invitation via email and logs into github via completely separate browser.
    • CODEOWNERS should reflect at least 1 file owned by this collaborator.
    • Now when that file is locally edited+pushed to fork by devuser1 => pr created => Collaborator prompted??
    • Note that user+system security are not optional, especially with teams.

  • Build has failed, so we must troubleshoot and fix that first.

    • Only then can we review and approve the code for merge.

  • We can also start to develop workflow #2 of 4.

  • Lecture 37. Creating the Develop Merge Pull Request Workflow

    • Very similar to our first workflow, with three intermediary added steps:
      • build project
      • upload build as an artifact
      • deploy to staging server

  • We may create a new file with redundant code

    • Or we might prefer one file for both workflows with conditionals.

  • Note that we need to create two secrets to access surge:

$ SURGE_LOGIN=$( surge whoami ) | gh secret set ...
$ SURGE_TOKEN=$(surge token); echo "${SURGE_TOKEN}" | gh secret set # https://cli.github.com/manual/gh_secret
$ gh secret --help
$ surge list
  • team-ci-example-pr-else-push-develop.yml
name: Team CI Example - PR else Push
on:
  pull_request:
    branches: [develop]
  push:
    branches: [develop]
jobs:
  build:
    runs-on: ubuntu-latest
  steps:
  - name: Implement Checkout Action
    uses: actions/checkout@v2

  - name: Implement Setup Node Action
    uses: actions/setup-node@v1
    with:
      node-version: "12.x"
  
  - run: time npm ci
  - run: time npm run format:check
  - run: time npm test -- --coverage
    env:
      CI: true

  - name: Build Project
    if: github.event_name == 'push'
    run: npm run build

  - name: Install surge
    if: github.event_name == 'push'
    run: npm install -g surge

#  - name: Upload Build as an Artifact if: github.event_name == 'push'    run: 

  - name: Create deployment target url with surge
    if: github.event_name == 'push'
    env:
      SURGE_LOGIN: ${{ secrets.SURGE_LOGIN }}
      SURGE_TOKEN: ${{ secrets.SURGE_TOKEN }}
    run: surge --help; surge list
    run: npx surge --project ./build --domain ${SURGE_TARGET_URL}

  - name: List surge deployment urls + logins.
    if: github.event_name == 'push'
    run: surge list
    run: surge 

  - name: Deploy to Staging Server
    if: github.event_name == 'push'
    run: npx surge --project ./build --domain

  • We test the above with:

    • Push to fork:develop => job runs but NO build nor deploy to staging steps
    • Create pull request on dev. Approve merge. => job runs with all steps

  • Kick off another pull request on develop.

    • Close pull request and reopen to retrigger action.

  • Kick off another merge into develop.

  • Lecture 38. Caching NPM Dependencies

    • Next we learn to:
      • uploaded artifacts
      • cached dependencies => uses: actions/cache => README

  • Second step: team-ci-example-pr-else-push-develop.yml

  - name: Cache node_modules
    uses: actions/cache@v1
    with:
      path: ~/.npm/
      key: ${{ runner.os }}-node-cachekey-${{ hashFiles('**/package-lock.json') }} # any change json file in repo or runner
      restore-keys: |
        ${{ runner.os }}-node-
---
$ git status; git commit -m 'pull request and merge workflows'
$ gh login devnoob1 # teamnoob
$ git push
$ gh pr # develop branch triggered workflow #1
# separate terminal...
$ gh login devreviewer1 # teamnode
$ git merge # develop branch triggered workflow #2
  • View logs of first run. Next run has a cache under this new key. 'Cache restored from key: .....'
    • Close pull request and reopen to retrigger action with that cache key.
    • Test file edits that do NOT trigger key cache key refresh.
    • Test file edits that DO trigger key cache key refresh.

$ vi package-lock.json # saved edits prompts new cache key!! $ vi ./.github/CODEOWNERS # saved edits prompts NO new cache key!! But It Should!!!

  • Lecture 39. Correction
    • In the previous lecture I noticed that I have a typo in restore-keys.
      • It should be restore-keys and not restore-key.
  • Lecture 40. Uploading Artifacts in Our Workflows
    • We insert another step after - run: npm test -- --coverage...
      - name: Upload Test Coverage Artifact
        if: github.event_name == 'push'
        uses: actions/upload-artifact@v1
        with:
          name: code-coverage
          path: coverage
  • We insert another step after - run: npm test -- --coverage...
      - name: Upload Build Folder
        if: github.event_name == 'push'
        uses: actions/upload-artifact@v1
        with:
          name: build
          path: build

  • push + pr => Only those jobs.

  • build-deploy => Those jobs too. Check artifacts

  • Third script is like first script, but from branch develop to branch master.

  • Fourth script is like second script, but from branch master deployed to production.

  • Lecture 41. Semantic Versioning & Conventional Commits

    • major.minor.patch
      • major: breaking changes
      • minor: new features, non-breaking
      • patch: bug fixes

  • As we do commits, we consider whether our changes represent major, minor or patch changes.

  • Conventional Commits:

    • [optional scope]:
    • [optional body]
    • [optional footer]

  • Major Change Example:

fix(cart): change cart endpoint
BREAKING CHANGE: changed cart endpoint
closes issue #12

  • Minor Change Example: feat(auth): added Facebook authentication

  • These do NOT affect release versioning:

    • Docs Change Example: docs(auth): added auth docs
    • CI Example: ci: added new workflow
    • Style Example: style: updated doc styles

  • Scenario: how do the following commit messages impact semver for 2.1.3?

    • feat(auth): added fb auth [ 2.1.3 => 2.2.0 ]
    • fix(auth): fixed authentication [ 2.1.3 => 2.1.4 ]
      • Answer: 2.2.0, because new minor version resets patch num index.

  • Important Reading:

  • team-ci-example-pr-else-deploy-develop-else-master.yml

    • releases
    • deploy to production
name: Team CI Example - PR else Push, develop else master
on:
  pull_request:
    branches: [develop, master]
  push:
    branches: [develop,master]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    ...

$ npm install --save-dev semantic-release
# ./release.configs.js
___
module.exports =. {
  branches: "master",
  repositoryUrl: "https://github.com/jeremy-donson/gha-react-app-sample",
  plugins: [
    "@semantic-release/commit-analyzer",
    "@semantic-release/release-notes-generator",
    "@semantic-release/npm", # not relevant here, but std default
    "@semantic-release/github" ]
}
---
$ npx semantic-release # check all plugins loaded
  • Lecture 43. Running semantic-release in Our Workflow
- name: Create a Release
  if: github.event_name == 'push' && github.ref == 'refs/heads/master'
  run: npx semantic-release
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
---
$ git status; git add -A; git commit -m 'Release now semver-managed...' ; git push; git status
  • Now we need to refactor our script to handle two github events on two branches.
  • This is to be done iteratively.

  • Lecture 44. Uploading Release Assets

  • release.config.js

    ["@semantic-release/github", {
      assets: [
      { path: "build.zip, label: "Compressed Build Artifact" },
      { path: "coverage.zip, label: "Compressed Coverage Artifact" },
...
---
$ npx semantic-release # This should fail because no build nor coverage folders exist.
  • New step before create a release: team-ci-example-pr-else-deploy-develop-else-master.yml
- name: Zip Assets
  if: github.event_name == push && github.ref == 'refs/heads/master'
  run: |
    time zip -r build.zip ./build
    time zip -r coverage.zip ./coverage
---
$ git status; git add -A; git commit -m 'fix: xcvghjk' ; git push; git status
  • Semantic release tool should run and succeed, and the new release should be 1.0.1.
  • Look at url ${GITHUB_REPO}/releases

  • Lecture 45. Deploying to Production when Pushing to Master

    • Deploy to staging and production....
    • We need to generate new target production surge url: $SURGE_TARGET_URL_PRODUCTION

  • Update stagig deployment and create production deployment steps.

- name: Deploy Branch: develop To Staging@surge
  if: github.event_name == 'push' && github.ref == 'refs/heads/develop'
  run: npx surge --project ./build --domain $SURGE_TARGET_URL_PRODUCTION
  env:
    SURGE_LOGIN: ${{ secrets.SURGE_LOGIN }}
    SURGE_TOKEN: ${{ secrets.SURGE_TOKEN }}

- name: Deploy Branch: master To Production@surge
  if: github.event_name == 'push' && github.ref == 'refs/heads/master'
  run: npx surge --project ./build --domain $SURGE_TARGET_URL_STAGING
  env:
    SURGE_LOGIN: ${{ secrets.SURGE_LOGIN }}
    SURGE_TOKEN: ${{ secrets.SURGE_TOKEN }}
---
# We can also move those 2 redundant surge env vars to job level.

  • Lecture 46. Uploading Code Coverage Reports to Codecov

    • Code Coverage Upload and Report => authorize account registration with github oauth
      • Copy token so we can upload to account from action runner cli.
      • Create secret: CODECOV_TOKEN

  • This step runs last.

- name: Upload Coverage reports
  if: github.event_name == 'push' && github.ref == 'refs/heads/master'
  run: npx codecov
  env:
    CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
---
$ git status; git add -A; git commit -m 'Code coverage dir: create + upload to codecov.io' ; git push; git status
  • Check web ui workflow report.
  • Check codecov account report.
  • Lecture 48. Validating Our Commit Messages with Commitlint & Commitizen
npm install --save-dev @commitlint/{config-conventional,cli} husky@4
echo "module.exports = {extends: ['@commitlint/config-conventional']}" > commitlint.config.js
# add husky hook to package.json:21
"husky": { "hooks": { "commit-msg": "commitlint -E HUSKY_GIT_PARAMS" }},
# test before commit
git status; git add -A; git commit -m 'Added commitlint+husky + This commit should fail.' ; git push; git status

npm install commitizen -g ; commitizen init cz-conventional-changelog --save-dev --save-exact
grep commitizen package.json
# update husky hooks to package.json:24
"husky": { "hooks": { "commit-msg": "commitlint -E HUSKY_GIT_PARAMS", "prepare-commit-msg": "exec < /dev/tty && git cz --hook || true" }},
git status; git add -A; git commit ; # We should see husky intervene with commit message prompts.
git push

  • Lecture 49. Sending a Slack Message When a New Release is Published

    • Slack alert :
      • Job Failure => Create New Github Issue
      • Issue Created => Send Slack Message
      • Release Created => Send Slack Message

  • We need to create dedicated test channel in slack workspace we own.

  • Test with curl!!

  • release-alerts-slack.yml

name: Notify Slack On Release
on:
  release:
    types: [published]
jobs:
  slack-message-on-published-release:
    runs-on: ubuntu-latest
    steps:
    - name: Slack Message Upon New Release
      run: |
        curl -X POST -H 'Content-type: application/json'
        --data '{"text":"New release ${{ github.event.release.tag_name }} is out
        <${{ github.event.release.html_url }}|check it out now.>"}' ${{ secrets.SLACK_WEBHOOK }}
  • PROBLEM: GITHUB_TOKEN is already in use for codecov and wrong value is set for slack step.
    • This forces us to create a new personal access token from github to use instead.
      • User Account \ Developer Settings \ PersonalAccessTokens => 'semantic release slack alert' => enable repo => copy token
      • Create new github secret: PERSONAL_ACCESS_TOKEN and implement that instead for npx semantic-release...
- name: Create a Release
  if: github.event_name == 'push' && github.ref == 'refs/heads/master'
  run: npx semantic-release
  env:
    GITHUB_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
---
git status; git add -A; git commit -m; git push

  • Did ci run and create release and notify slack?

  • Do we need dedicated service account with botname?

  • Lecture 50. Opening an Automated Issue when the Workflow Fails
    • Failures can be caught in last step of workflow:
- name: Open Issue
  if: failure() && github.event_name == 'pull_request'
  run: |
  curl .... search repo: "Automated issue for commit:"

  • Now we want to break app testing, commit bug fix and push. New PR...

    • This should reveal test failure and prompt new github issue creation.

  • issue-create-new.yml

name: Notify on Release
on:
  issues:
    types: [opened]
jobs:
  slack-message:
    runs-on: ubuntu-latest
    steps:
    - Slack Alert Message: New Github Issue
      run: |
      curl => reuse statement but change slack message content + target url:  github.event.issue.{title,html_url}

  • Once again we use: ${{ secrets.PERSONAL_ACCESS_TOKEN }}

  • We push to master BUT failure drives creation of new issue + slack alert.

  • events event_types event_errors event_error_exceptions

  • In next section, we use these curl command options to configure our own custom github action.

  • Lecture 51. Adding a Status Badge in README.md

    • Github badges can be used as iconic notifiers of workflow run failures: "test-local:passing" with green color, etc

  • Add the following to the top of the file at ./README.md ... see filters in image url

![](https://github.com/[user]/[repo-name]workflows/CI/badge.svg?branch=development&event=push)
  • Verify readme file displays with badge of job status.

  • Section 6: Creating Our Own Github Action

  • Lecture 52. Github Actions Overview

    • We created slack message curl pattern to encapsulate and parameterize in github action of our own.
    • We will create our own github action to simplify and clean up the implementation.
    • Then we implement that same custom private action twice instead.

  • Review actions.yml from earlier... where we reference a pubic action with uses: actions/checkout@v1

  • Search repo: 'actions/'

  • We can also reference a private local action which we create ourselves.

    • This relies upon js scripts in workflows below, else docker containers.
    • Sample js script action: actions/hello-world-javascript-action@v1

  • Below assumes that we already have:

    • fork the simplest js action repo into our personal github account
    • update that forked release to something slightly different: Hello Linus... From Github Action Runner! => SHA1
    • update that forked release again to contain config: default_language = EN => SHA2
    • show that commitizen and release autoversioning are working
    • implement our own via uses: jeremy-donson/simplest-js-action@$SHA1. with....

  • actions-ci.yaml

name: Actions CI
on:
  pull_requests:
    branches: [develop, master22]
  push:
    branches: [develop, master22]
jobs:
  run-github-actions:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v1
    - name: List Files After Checkout Action
      run: |
        pwd
        ls -a

    - name: Simplest Public JS Action
      uses: actions/hello-world-javascript-action@v1
      id: greetpublic
      with:
        who-to-greet: Linus
    - name: Log Greeting Time
      run: echo "${{ steps.greetpublic.outputs.time }}"

    - name: Private Custom Github Action 1 - Judy
      uses: ./path/to/file
      id: greetprivate
      with:
        who-to-greet: Judy
    - name: Log Greeting Time
      run: echo "${{ steps.greetprivate.outputs.time }}"

    - name: Private Custom Github Action 2
      uses: [user]/[repo-name]@$SHA2
      id: greetprivate
      with:
        who-to-greet: Romeo
  • Action version references will be enforced.

  • Lecture 53. Creating a Simple JavaScript Action

    • See file above.... reminder that we will create our own private action:

  • $mkdir -p ./.github/actions/hello/; cd $_

  • actions-ci.yaml

name: Hello World Action
author: Jeremy Donson
description: first private custom github action
inputs:
  who-to-greet:
    description: "Who to greet"
    required: true
    default: Linus
outputs:
  time:
    description: the greeting time
runs:
  using: "node12"
  main: "dist/index.js"

const core = require('@actions/core')
const github = require('@actions/github')

try {
#    throw(new Error("error message 1"));
    const name = core.getInput('who-to-greet');
    console.log(`Hello ${name}`);

    const time = new Date();
    core.setOutput("time", time.toTimeString());

    console.log(JSON.stringify(github, null, '\t'));
} catch(error) {
    core.setFailed(error.message)
}
---
$ npm install @actions/core --save
$ npm install @actions/github --save

  • We have edited branch: master in actions-ci.yml to temp-block workflow from going to master/production.

    • We use the following file to test this new private action:

  • testing-private-action-steps.yml

on: push
jobs:
  testing-private-action-steps:
    runs-on: ubuntu-latest
    steps:
    - name: public action checkout
      uses: actions/checkout@v2

    - name: private action hello
      uses: ./github/actions/hello
      id: hello
      with:
        who-to-greet: "World"

    - name: report prior step event time
      run: |
      echo "Step Time: ${{ steps.hello.outputs.time }}"
     # echo $EXPORTED_VAR_NAME
  • This leaves us with a problem: node modules are not installed on runners....
    • We could copy those installed node modules, or we could install/bundle them on runner.
      • The latter method is preferred. Consider why that is the case.
  • Lecture 54. Bundling our JavaScript Code into 1 File
    • Actions + Code into single file using zeit:
ls -a .github/actions/hello/dist/
npm i -D @zeit/ncc
time npx ncc build .github/actions/hello/index.js -o .github/actions/hello/dist
ls -a .github/actions/hello/dist
cat .github/actions/hello/dist/index.js # all module code now embedded
du .github/actions/hello/dist
git status; git add -A; git commit -m ; git push

  • Skim private test action run output: Run /./.github/actions/hello

  • Uncomment try:throw new in index.js => commit + push, and watch error thrown...

  • Lecture 55. Let's Discover More About the @githubs/core Package

    • Here we will play with other core actions..
    • Lets look at several ways to use actions/core.

  • Recomment + try something else...try:throw new in index.js:7 => commit + push, and watch error thrown...

...
try {
# ...
  core.debug('Core Action Debug Message')
  core.warning('Core Action Warning Message')
  core.error('Core Action Error Message')

  const name = core.getInput("who-to-greet");
  core.setSecret(name);
  console.log(`Hello ${name}`);

  const time = new Date();
  core.setOutput("time", time.toTimeString())

  core.startGroup('Logging github object')
  console.log(JSO.stringify(github, null, "\t"));
  core.endGroup();
  
  core.exportVariable("EXPORTED_VAR_NAME","this value exported") # uncomment last line in testing-private-action-steps.yml
  
} catch (error) {
  core.setFailure(error.message);
}

  • UNlike core.setFailed(error.message), this will NOT stop the step.

  • Enable Step Debug Logging

    • Create new github repo secret: ACTIONS_STEP_DEBUG = true
      • Edit file, **recompile with npx ncc build ... ** locally, then push... => See how step logs are now more verbose.

  • Lecture 56. Creating a JavaScript Action for Opening Github Issues

    • Now we will create an action that is more useful: open github issue...

  • $ mkdir -p ./.github/actions/issue-create/; cd $_ ; touch action.yml

  • Scroll up to file issue-create-new.yml and reuse contents of curl statement parameters:

    • url: https://api.github/co/repos/${{ github.repository }}/issues
    • header: 'authorization: Bearer ${{ secrets.PERSONAL_ACCESS_TOKEN }}'
    • header: 'context-type: application/json'
    • data: title,body,assignees

  • ./.github/actions/issue-create/action.yml

name: Create Github Issue
author: jeremy-donson
description: opens a new github issue
inputs:
  token:
    description: 'Github Token'
    required: true
  title:
    description: 'Issue Title'
    required: true
  body:
    description: 'Issue Body'
  assignees:
    description: 'Issue Assignees'
outputs:
  issue:   # returns new issue id
  description: 'The issue object as a json string'
runs:
  using: 'node12'
  main: 'dist/index.js'

  • octokit/rest.js sends api requests (not curl nor fetch).

  • ./.github/actions/issue-create/index.js

const core = require('@actions/core');
const github = require('@actions/github');

try {
    const token = core.getInput('token')
    const title = core.getInput('title')
    const body = core.getInput('body')
    const assignees = core.getInput('assignees')
    
    const octokit = new github.GitHub(token)
    const response = octokit.issues.create({
      # owner: github.context.repo.owner,
      # repo: github.context.repo.repo,
      ...github.context.repo,
      title,
      body,
      assignees: assignees ? assignees.split(',') : undefined
    })

    core.setOutput('issue', JSON.stringify(response.data));

    } catch (error) {
      core.setFailed(error.message)
}

jobs:
  testing-action:
    runs-on: ubuntu-latest
    steps:    
    - name: Issue Create Action
      uses: ./.github/actions/issue-create
      id: issue-new
      with:
        token: ${{ secrets.GITHUB_TOKEN }}
        title: Issue Title
        body: Issue Body
        assignees: 'student1,reviewer1' # comma-separated-string, or | block with split('\n')

    - name: Show Details Of Newly Created Issue
      run: |
        echo ${{ steps.issue-new.outputs.issue }}
  • Compile and commit and push:
time npx ncc build .github/actions/issue-create/index.js -o .github/actions/issue-create/dist
ls -a .github/actions/issue-create/dist
du .github/actions/issue-create/dist
git add -A; git commit -m ; git push

  • new issue created?

  • workflow runs without errors? FAIL => index.js:12...

    • octokit async call requires keyword 'await' + async function run { ... } + run()
    • recompile and commit and push = rcp
    • now we should see workflow outputs from issue step

  • Review all github actions toolkit options.

  • Lecture 57. Creating a Simple Docker Action

  • action.yaml

name: Hello World Docker Action
author: Jeremy Donson
description: first private custom github action
inputs:
  who-to-greet:
    description: "Who to greet"
    required: true
    default: Linus
outputs:
  time:
    description: "the greeting time"
runs:
  using: "docker"
  image: "Dockerfile"
  entrypoint: ["/entrypoint.sh"]
  args:
    - ${{ inputs.who-to-greet }}
  • testing-private-action-steps.yml TWO NEW STEPS
- name: test private action hello-docker
  uses: ./.github/actions/hello-docker
  id: hello-docker
  with:
    who-to-greet: "world"
- name: test private action hello-docker
  run: |
    echo "Time:
  • entrypoint.sh => duplicating index.js, but with shell script
#!/bin/sh -l

echo "::debug ::Debug Message"
echo "::warning ::Warning Message"
echo "::error ::Error Message"

echo "::add-mask::$1"
echo "Hello $1"
time=$(date)
echo "::set-output name=time::$time"
echo "::group::Some expandable logs"
echo 'some stuff'
echo 'some stuff'
echo 'some stuff'
echo '::endgroup::'

echo 'set-env name=HELLO::hello'
---
$ chmod +x .github/actions/hello-docker/entrypoint.sh
  • Dockerfile
FROM alpine:3.11
COPY entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

  • We need to define inputs in yaml file, and pass those inputs in the args array as docker cmd args.

  • We still have no error thrown like index.js, but commit+push...

    • Then we add the following to index.js:3
if [ true ]
then
  echo 'error'
  exit 1
fi

  • commit+push to see error thrown

  • We still need to create both private actions using docker container: create new issue and slack alert

  • Lecture 59. Creating a PHP Script for Sending a Slack Message

  • Lecture 60. Running our PHP Script using a Docker Action

  • Lecture 61. Populating our Inputs and Context Information Using Environment Variables

  • Lecture 62. Publishing Github Actions into the Marketplace

  • Lecture 63. Bonus Lecture

  • Github Teams, Pages + Templates

  • Github vs Gitlab vs Argo vs Azure Dev Ops vs Hashicorp

  • Change Management with CI/CD.
    • Post-Mortems
    • Real Time Healing
    • Preventive Policies + Exceptions
Raw
04-prometheus-course.md

Prometheus Course Notes

  • Syllabus:

  • Section 1. Introduction

    • Lecture 1. Introduction to Prometheus
    • Lecture 2. Alternate Monitoring Tools
    • Lecture 3. Basic Terminologies in Prometheus

  • Section 2. Architecture of Prometheus

    • Lecture 4. Architecture of Prometheus
    • Lecture 5. How Prometheus Works behind the scenes (Life Cycle)

  • Section 3. Installation and UI Tour

    • Lecture 6. Virtual Machine Installation
    • Lecture 7. Prometheus Installation
    • Lecture 8. First Look of Prometheus UI
    • Lecture 9. Understanding Prometheus Configuration File
    • Lecture 10. Prometheus First Run

  • Section 4. Exporters - Set 1

    • Lecture 11. What are Exporters?
    • Lecture 12. Node Exporter - Monitoring Linux Systems
    • Lecture 13. WMI Exporter - Monitoring Windows Systems
    • Lecture Quiz 1: Quiz

  • Section 5. PromQL - Prometheus Query Language

    • Lecture 14. Data Types in PromQL
    • Lecture 15. Selectors & Matchers
    • Lecture 16. Binary Operators
    • Lecture 17. 'ignoring' and 'on' keywords
    • Lecture 18. Aggregation Operators
    • Lecture 19. Functions - 'rate' & 'irate'
    • Lecture 20. Functions - changes, deriv, predict_linear
    • Lecture 21. Functions continued
    • Lecture Assignment 1: Assignment

  • Section 6. Client Libraries - Adding Instrumentation To Python

    • Lecture 22. What are Client Libraries and Metric Types
    • Lecture 23. Python app Boilerplate Code
    • Lecture 24. Exposing Metrics from Python app using Prometheus Client
    • Lecture 25. Counter Metrics Exposition
    • Lecture 26. Adding Labels to Exposed Metrics
    • Lecture 27. Gauge Metrics Exposition
    • Lecture 28. Summary Metrics Exposition
    • Lecture 29. Histogram Metrics Exposition

  • Section 7. Client Libraries - Instrumenting Go Applications

    • Lecture 30. GO app Boilerplate Code
    • Lecture 31. Counter Metrics Exposition
    • Lecture 32. Gauge Metrics Exposition
    • Lecture 33. Summary Metrics Exposition
    • Lecture 34. Histogram Metrics Exposition

  • Section 8. Quantification of Instrumentation

    • Lecture 35. What to Instrument?
    • Lecture 36. How much to Instrument?

  • Section 9. Recording Rules

    • Lecture 37. What are Recording Rules?
    • Lecture 38. Reload Prometheus Configurations on-the-fly
    • Lecture 39. Writing Recording Rules
    • Lecture 40. Writing Recording Rules continued
    • Lecture 41. Add Multiple Rules
    • Lecture 42. Best Practices for Recording Rules

  • Section 10. Alerting

    • Lecture 43. What is Alerting?
    • Lecture 44. Writing & Firing the first Alert
    • Lecture 45. 'for' clause
    • Lecture 46. Adding Labels to Alerts
    • Lecture 47. Installing Alertmanager
    • Lecture 48. Adding Alert Notifier - Gmail
    • Lecture 49. Sending Alert Notifications - Gmail
    • Lecture 50. Templating the Alerts

  • Section 11. Create Routing Tree for Alrts - Case Study

    • Lecture 51. Case Study - Problem Statement
    • Lecture 52. Understanding the Use Case for Routing Tree
    • Lecture 53. Write Alerting Rules for the Use Case
    • Lecture 54. Coding the Routing Tree - Part 1
    • Lecture 55. Coding the Routing Tree - Part 2
    • Lecture Assignment 2: Assignment
    • Lecture 56. Coding the Routing Tree - Part 3
    • Lecture 57. Run the Routing Tree
    • Lecture 58. Grouping the Alerts
    • Lecture 59. Throttling the Alerts
    • Lecture 60. Inhibiting the Alerts
    • Lecture 61. Silencing the Alerts
    • Lecture 62. 'continue' clause

  • Section 12. PagerDuty + Slack - Alert Notifications

    • Lecture 63. Slack integration with Prometheus
    • Lecture 64. PagerDuty integration with Prometheus

  • Section 13. Blackbox Exporter + Relabelling

    • Lecture 65. What is Blackbox exporter
    • Lecture 66. Download Blackbox exporter
    • Lecture 67. 'http' probe module
    • Lecture 68. 'tcp' & 'icmp' probe module
    • Lecture 69. 'dns' probe module
    • Lecture 70. Scraping targets via Blackbox
    • Lecture 71. Relabelling

  • Section 14. Pushgateway

    • Lecture 72. Introduction to Pushgateway
    • Lecture 73. Getting Started with Pushgateway
    • Lecture 74. Push metrics to Pushgateway
    • Lecture 75. Automate Pushing metrics using Cron job
    • Lecture 76. Python App pushing metrics to Pushgateway
    • Lecture 77. Pushgateway Pitfalls

  • Section 15. Service Discovery

    • Lecture 78. Introduction to Service Discovery
    • Lecture 79. Static Service Discovery
    • Lecture 80. File-based Service Discovery

  • Section 16. Amazon Cloud with Prometheus

    • Lecture 81. Create Prometheus User in AWS
    • Lecture 82. Spinning up AWS EC2 instance
    • Lecture 83. Node Exporter on EC2 instance
    • Lecture Assignment 3: Assignment
    • Lecture 84. Relabeling the private IP to public
    • Lecture 85. 'keep' and 'drop' targets with Relabeling

  • Section 17. AWS Cloudwatch Exporter

    • Lecture 86. AWS Cloudwatch & UI tour
    • Lecture 87. Installation & Configuration
    • Lecture 88. Running the Cloudwatch Exporter

  • Section 18. Exporters - Set 2

    • Lecture 89. 'mysql' Exporter

  • Section 19. Create Custom Exporter

    • Lecture 90. Introduction
    • Lecture 91. Creating Target application
    • Lecture 92. Writing Custom Exporter

  • Section 20. Prometheus HTTP API

    • Lecture 93. Running Prometheus with HTTP API

  • Section 21. Grafana Dashboards

    • Lecture 94. Grafana Installation
    • Lecture 95. Adding Data source to Grafana
    • Lecture 96. Implementing default dashboards of Grafana
    • Lecture 97. Create Custom Dashboard - Part 1
    • Lecture 98. Create Custom Dashboard - Part 2

  • Section 22. Additional Learnings

    • Lecture 99. When to use Prometheus and When NOT to...
    • Lecture 100. Create Gmail App Passwords
    • Lecture 101. ThankYou

  • Section 23. Bonus

    • Lecture 102: Bonus
Raw
07-terraform-associate.md

Terraform Associate Certification

  • Syllabus

  • Section 1: Course Introduction

    • Lecture 1. Course Introduction
    • Lecture 2. Terraform Exam BluePrint

  • Section 2: Understanding Infrastructure As Code

    • Lecture 3. IAC and IAC Benefits
    • Lecture 4. Cloud Agnostic IAC with Terraform
    • Lecture Practice Test 1: Understanding Infrastructure As Code
    • Lecture 5. Procedure Document and Resource Location

  • Section 3: IAC With Terraform

    • Lecture 6. Terraform Workflow
    • Lecture 7. Terraform Initialization
    • Lecture 8. Terraform key Concepts : Plan, apply & Destroy
    • Lecture 9. SetUp Free-Tier AWS Account (Optional)
    • Lecture 10. Create Cloud Machine for Terraform Execution
    • Lecture 11. $100 Discount Gift on Digital Ocean
    • Lecture 12. Terraform Installation & Verification
    • Lecture 13. Install Terraform on MacOS / Windows
    • Lecture Practice Test 2: IaC with Terraform

  • Section 4: Start With Terraform Basics

    • Lecture 14. Terraform Provider & Initialization
    • Lecture 15. AWS SetUp for Terraform
    • Lecture 16. Create Machine Using Terraform
    • Lecture 17. Provide Creds in Separate Centralised File
    • Lecture 18. Provide Creds in Environment Variables
    • Lecture 19. Create Multiple Instances
    • Lecture 20. Terraform State : The Concept

  • Section 5: Terraform Variables Detailed Explanation

    • Lecture 21. Variables in Terraform
    • Lecture 22. Use of Variable in Conf File
    • Lecture 23. Lab : Use of Variable in Conf File
    • Lecture 24. Lab : List and Map Variables

  • Section 6: Terraform Concepts - Building Blocks

    • Lecture 25. Provision Software with Terraform
    • Lecture 26. Lab : Provision Software with Terraform
    • Lecture 27. DataSource in Terraform
    • Lecture 28. Lab : DataSource in Terraform
    • Lecture 29. Lab 2 : DataSource in Terraform
    • Lecture 30. Output Attribute in TF
    • Lecture 31. Lab : Output Attribute in TF
    • Lecture 32. Remote State in Terraform
    • Lecture 33. Lab : Remote State in Terraform

  • Section 7: Terraform For AWS Cloud - I

    • Lecture 34. AWS VPC Introduction
    • Lecture 35. AWS VPC Introduction II
    • Lecture 36. Demo : AWS VPC & Security Group
    • Lecture 37. Lab : Create AWS VPC & NAT Gateway
    • Lecture 38. Launch EC2 Instance using Custom VPC
    • Lecture 39. Lab : Launch EC2 Instance using Custom VPC
    • Lecture 40. We Need You!!!
    • Lecture 41. Elastic Block Store (EBS) in AWS
    • Lecture 42. Demo : Elastic Block Store (EBS) in AWS
    • Lecture 43. Lab : Elastic Block Store (EBS) in AWS
    • Lecture 44. User Data in AWS
    • Lecture 45. Lab: User Data using Script
    • Lecture 46. Lab : User Data using Cloud Init

  • Section 8: Terraform For AWS Cloud - II

    • Lecture 47. AWS RDS Basics
    • Lecture 48. Lab : Create RDS
    • Lecture 49. AWS Access and Identity Management
    • Lecture 50. Lab : IAM Users and Groups
    • Lecture 51. Lab : AWS IAM Roles
    • Lecture 52. EC2 Instance Autoscaling
    • Lecture 53. Lab : EC2 Instance Autoscaling
    • Lecture 54. Your Reviews are Important!
    • Lecture 55. Load Balancing in AWS
    • Lecture 56. Lab : AWS Load Balancing

  • Section 9: Terraform MOdules - Code Reusability

    • Lecture 57. Terraform Module and Application
    • Lecture 58. Lab : Terraform Source From GITHUB
    • Lecture 59. Lab : Local Path Module
    • Lecture 60. Lab : AWS VPC Module Part I
    • Lecture 61. Lab : AWS VPC Module Part II
    • Lecture 62. Lab : AWS VPC Module Part III

  • Section 10: Conditions + Loops In Terraform

    • Lecture 63. Condition Statements in Terraform
    • Lecture 64. Lab : Condition Statements in Terraform
    • Lecture 65. Terraform Built-In Functions
    • Lecture 66. Lab : Terraform Built-In Functions
    • Lecture 67. Loops in TerraFrom HCL
    • Lecture 68. Terraform Project Structure
    • Lecture 69. Lab : Terraform Project Structure

  • Section 11: Packer and Terraform

    • Lecture 70. Packer Introduction and It's Use
    • Lecture 71. Install Packer
    • Lecture 72. Lab : Create Custom AMI Scenario I
    • Lecture 73. Lab : Create Custom AMI Scenario II
    • Lecture 74. Lab : Terraform + Packer

  • Section 12: Job Scenario 1: End to End Web Application Deployment

    • Lecture 75. Application Overview
    • Lecture 76. Create VPC Network
    • Lecture 77. Create RDS Service
    • Lecture 78. Create WebServer in AWS
    • Lecture 79. Deploy Complete Application

  • Section 13: Job Scenario 2: Terraform + Docker + Kubernetes

    • Lecture 80. AWS EKS Introduction
    • Lecture 81. Lab : SetUp EKS Cluster Using AWS
    • Lecture 82. Lab : SetUp EKS Using AWS CLI
    • Lecture 83. Lab : Access EKS Cluster and Deploy Application
    • Lecture 84. Text Direction : SetUp EKS using AWS CLI
    • Lecture 85. EKS Cluster TerraForm Configuration files
    • Lecture 86. Lab : Deploy EKS Cluster using Terraform

  • Section 14: Job Scenario 3: Terraform + AWS ELK

    • Lecture 87. ELK Basics and Application
    • Lecture 88. Lab : Deploy ELK and Execute ELK
    • Lecture 89. Text Direction : Lab - Deploy ELK and Execute ELK
    • Lecture 90. Lab : Install ELK using Terraform

  • Section 15: Terraform GCP: Introduction

    • Lecture 91. Terraform vs Google Deployment Manager
    • Lecture 92. Setup GCP Project For Terraform
    • Lecture 93. Enable Basic APIs on Cloud Project
    • Lecture 94. Setup Remote State File in Cloud

  • Section 16: Terraform Hashicorp certification Guide

    • Lecture 95. Introduction HashiCorp Certification
    • Lecture 96. Understand Infrastructure as Code (IaC) Concepts
    • Lecture 97. Understand Terraform's Purpose
    • Lecture 98. Understand Terraform Basics I
    • Lecture 99. Understand Terraform Basics II
    • Lecture 100. Use the Terraform CLI
    • Lecture 101. Interact with Terraform modules
    • Lecture 102. Navigate Terraform workflow
    • Lecture 103. Implement and Maintain State
    • Lecture 104. Read, Generate, and Modify Configuration
    • Lecture 105. Understand Terraform Cloud and Enterprise Capabilities
    • Lecture 106. Practice Questions For Terraform Associate Certification I
    • Lecture 107. Practice Questions For Terraform Associate Certification II
    • Lecture 108. Practice Questions For Terraform Associate Certification III
    • Lecture 109. Practice Questions For Terraform Associate Certification IV
    • Lecture 110. Practice Questions For Terraform Associate Certification V
    • Lecture 111. Practice Questions For Terraform Associate Certification VI

  • Section 17: Course completion

    • Lecture 112. Congratulations
    • Lecture 113. Bonus Lecture : What is Next?
Raw
08-ansible-mastery.md

Ansible Mastery Course : Ansible Automation with AWS + GCP

Table of Contents

Prerequisites

  • System Admins manage software for fleets of VMs from a control node:

    • control node: this admin-centric node accesses-installs-configures-orchestrates target host fleets
    • target hosts: usually listed in an ansible-specific host inventory file on the control node
      • pre-provisioned hosts include: machine+OS+IP+sshkey+login+sudoer+pkgmgr => host inventory entry

  • Ansible Control Node Requirements

    • Python Version <= 3.10
    • Ansible Version <= 2.9
    • Ansible Docs

  • Three ways to use ansible:

      1. Command line ansible binary and ansible-* binaries: $ ansible --help
      1. Implement System Python3 Module: ansible
      1. Leverage Virtual Environment For Python3 Module: ansible

  • Target Machines

    • Localhost
    • VirtualBox VMs
    • VMWare VMs
    • OpenStack VMs
    • Cloud Vendor VMs

  • Operating System Flavors@Versions

    • Apple OSX
    • Ubuntu Enterprise Linux Server
    • RedHat Enterprise Linux Server
    • MSoft Windows Server
    • MSoft Windows Desktops
    • Ubuntu Desktops
    • RedHat Desktops

  • Github Accounts

Test Matrix

Public Repo Tour

Syllabus

Raw
09-ansible-setups.sh
#!/bin/bash
# global functions + tests
# {SET,GET} {MEM,FILE}
# GET_FILE
# GET_MEM
# SET_FILE
# SET_MEM
# f prepend on custom function names ?
# Ansible Controllers: control node functions + tests + aliases
# Ansible Inventory + Fleet: inventory node functions + tests + aliases
BASE_STRING='/usr/local/bin/ansible' # $ which ansible
BINARY_NAME=$(echo "${BASE_STRING}" | rev | cut -d'/' -f1 | rev )
declare -a ARRAY_OF_APPENDS=( -community -config -connection -console -doc -galaxy -inventory -playbook -pull -test -vault )
# gen_appends_string() { STRING=''; DELIMS=' -'; for n in $(ls -A1 /usr/local/bin/ansible* ); do STRING+="$DELIMS"; STRING+=$(echo $n | rev | cut -d'-' -f1 | rev); done ;}
# string_to_array() { declare -a ARRAY_OF_APPENDS=("$1") ; }
function stamp_date() { DATE_STAMP=$( date +'%Y%m%d' ) ; echo "${DATE_STAMP}" ; }
function stamp_utime() { UTIME_STAMP=$( date +'%s' ) ; echo "${UTIME_STAMP}" ; }
function stamp_utime_ns() { UTIME_STAMP_NANO=$( date +'%s.%N' ) ; echo "${UTIME_STAMP_NANO}" ; }
DATE_STAMP=$(date_stamp)
function mkdir_projects() { PROJECT_NAME=${1}; mkdir "${PROJECT_NAME}-project-{private,public}"
function py3_venv() { python3 -m venv $1 ; }
function mkdir_helps() { DIRNAME=${1}; mkdir "${BINARY_NAME}-help-pages-"${DATE_STAMP}"
function array_count() { ARRAY_COUNT=${#1} ; }
function vartype() {
local var=$( declare -p $1 ); local reg='^declare -n [^=]+=\"([^\"]+)\"$'
while [[ $var =~ $reg ]]; do var=$( declare -p ${BASH_REMATCH[1]} ); done
case "${var#declare -}" in
a*)
echo "ARRAY"
;;
A*)
echo "HASH"
;;
i*)
echo "INT"
;;
x*)
echo "EXPORT"
;;
*)
echo "OTHER"
;;
esac
}
function gen_help_call_code() { # BINARY_NAME ARRAY_OF_APPENDS CODE_GEN_FILE
if [ "$#" -ne "3"]; then echo 'THREE ARGS NEEDED' && exit
BINARY_NAME="${1}"
ARRAY_CHECK=$( vartype(${2}) ) ; if []; then ARRAY_OF_APPENDS="${2}"; fi
CODE_GEN_FILE_PATH="${3}"
ARRAY_OF_APPENDS_ELEMENT_COUNT=${#ARRAY_OF_APPENDS[@]}
CODE_GET_VERSION="${BINARY_NAME} --version"
CODE_GET_HELP="${BINARY_NAME} --help"
# CODE_GET_DEPS=
for ELEMENT in ${ARRAY_OF_APPENDS[@]}
do
${BINARY_NAME}${$ELEMENT} --help >> ${CODE_GEN_FILE_PATH}
done
function ansible_gen_configs() { }
function grep_configs() { }
function configs_update() { }
function gen_bash_profile() { }
function gen_bash_func_test() { }
```
- Iterative Learning Workflows:
```
read -p 'Do we understand the problem?' YN
case YN:
Y) pass ;;
N) hint_problem() ;;
*) ;;
esac
read -p 'Do we understand how to do this?' YN
case YN:
Y) pass ;;
N) hint_method() ;;
*) ;;
esac
read -p 'Do we know how to test the solution?' YN
case YN:
Y) pass ;;
N) hint_test() ;;
*) ;;
esac
read -p 'Do we have the correct solution?' YN
case YN:
Y) pass ;;
N) hint_solution() ;;
*) ;;
esac
read -p 'Did solution pass test?' YN
case YN:
Y) pass ;;
N) solution() ;;
*) ;;
esac
read -p 'Is this a cluster scenario that we want to reuse?' YN
case YN:
Y) pass ;;
N) preserve_scenario() ;;
*) ;;
esac
read -p 'What templating system will we use??' TPLSYS
case TPLSYS:
bash-heredoc) ;;
py3-jinja2) ;;
golang-tpl) ;;
*) ;;
esac
read -p 'Do we derive template from existing template?' YN
case YN:
Y) derive_template() ;;
N) new_template() ;;
*) ;;
esac
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment