im-knots/sre-rampup.md

## sre-rampup.md

      
    Raw
  

              sre-rampup.md
            
          
    SRE Intro Projects

Basics of Web Development


Create a Git repository on Github or elsewhere and clone it. All of your following work must be commited regularly to this repository
Create a basic web application using a modern front-end framework (Hint: npx create-react-app). Familiarize yourself with the basics of front-end web development. Learn modern Typescript, props, imports. Create multiple pages, navigation, routers. Style it. Import material-ui components. Become fluent and at ease navigating a modern web app code.
Create a REST API using modern backend frameworks (Hint: Flask-Python). Connect your API to an SQL database. Write handlers to CRUD store resources of your choice in a database table. Write a test suite for your API with >90% coverage and be able to quickly run your test suite to validate everything is still working at all times. Become fluent and at ease navigating a modern backend app code.
Use your web client to pull your data from your API and display it nicely. Learn about networking between a client browser and an API. Learn and work around CORS restrictions.
Write end-to-end tests for your web application that asserts that your web client is able to produce and consume data from your API and display it in the intended way.
By that point, you should be fluent working in the modern web world. Keep learning, build an authentication system and a login page, store and display more resources, make your application useful. Your imagination is the limit

Basic Kubernetes Usage


Set up minikube or docker-desktop with kubernetes as its back end on your local machine
Use the API service you created in the first lab.
Use a dockerfile to containerize the api service. Build a docker image or OCI for the application in a local image repo (docker-desktop has a built in one)
Deploy the app container to your local k8s as a pod using kubectl.
Portforward the pod to localhost:8080 on your local machine. Send requests to the api and see if it works.
Use kubectl and yaml to define a service object pointing to the pod, send requests to the api and see if it works.
Use kubectl and yaml to define a deployment object for the api app. Send requests to the api and see if it works.
Use kubectl and yaml to scale the number of api pods for the deployment to 3. Send requests to the service object and observe the behavior in the pod logs.
Use kubectl and yaml to attach a storage volume to the app pods.
Install and play k9s. It is an interface for kubectl that gives you better at a glance functionality. Use it to describe pods, the deployments, services, ect.

Introduction to Helm and Converting Kubernetes Manifests to Helm Templates


Install Helm on your local machine if you haven't already. Helm is a package manager for Kubernetes that helps you manage and deploy applications. You can refer to the official Helm documentation for installation instructions.
Review the Kubernetes manifests (YAML files) you created in the previous task for your API deployment, service, and other resources. Understand the structure and content of these manifest files.
Familiarize yourself with Helm concepts such as charts, templates, values, and releases. Helm provides a templating engine that allows you to define reusable configurations and deploy applications consistently across environments.
Convert your Kubernetes manifests into Helm templates. Helm templates use the Go templating language to parameterize and abstract the resource configurations. Start by creating a new Helm chart for your API application.
Organize the Helm chart directory structure. A Helm chart typically consists of directories such as charts, templates, values, and templates/partials. Create these directories within your Helm chart directory.
Move the converted Kubernetes manifests into the templates directory within your Helm chart. Each Kubernetes manifest file should be renamed with the .yaml or .yml extension to indicate that it's a template.
Modify the Helm chart's values.yaml file to define configurable values for your application deployment. This allows for easy customization during deployment. Define values such as image name, port number, environment variables, and other configuration options.
In each Helm template file, replace the specific values with Helm template variables. Use double curly braces ({{ }}) to enclose the template variables. For example, replace the image name with {{ .Values.image.name }} and the port number with {{ .Values.service.port }}.
Use Helm commands such as helm lint and helm template to validate and render the Helm templates. The helm lint command checks for any syntax errors or issues in your Helm chart. The helm template command generates the Kubernetes manifest files from the Helm templates.
Package your Helm chart using the helm package command. This will create a compressed chart archive (.tgz) file. Run helm package <chart-directory> to create the package.
Deploy your application using the Helm chart. Use the helm install command and provide the necessary values or overrides for the deployment. Run helm install <release-name> <chart-archive> to deploy your application using the Helm chart.
Verify the successful deployment of your application by checking the running pods, services, and other resources using kubectl. Use commands such as kubectl get pods, kubectl get services, and kubectl describe <resource> to inspect the deployed resources.
Explore advanced Helm features such as managing multiple releases, rolling updates, and managing dependencies with subcharts. Experiment with Helm commands such as helm upgrade, helm rollback, and helm uninstall to manage the lifecycle of your application.

By following these steps, you'll gain a more specific understanding of Helm and its role in simplifying the deployment and management of Kubernetes applications. You'll learn how to convert Kubernetes manifest files into Helm templates, define configurable values, and deploy applications using Helm charts.
Basic Monitoring with Prometheus


Install helm on your local machine. Helm is a package manager for kubernetes.
Use helm to install the kube-prometheus-stack
https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
Portforward to prometheus and grafana and poke around the interface.
In grafana, view the built in k8s cluster dashboards.
Use basic PromQL to build your own cluster monitoring dashboard.
Send requests to your api service via curl or artillery and view the network usage on your dashboard.

Basic custom metric instrumentation with Prometheus


Modify your api service so that it has multiple handlers. ex. /, /handler1, /handler2.
Import the prometheus library for your langauge (python, go, node, ect)
Python can be found here: https://github.com/prometheus/client_python
Use the prometheus library to create a metric endpoint handler. eg. /metrics
Use Py Prometheus documentation to implement a counter metric for each handler invocation. eg. Every time /handler1 is invoked, increment a metric named handler1_invocations by 1.
Use Py Prometheus documentation to implement a histogram metric for each handler's run time. eg. Every time /handler1 is invoked, track the start time at the top of the handler function, and observe the time delay at the end of the handler function.
Update your docker image with your code changes.
Use kubectl and yaml to define a ServiceMonitor object for your app.
Portforward to prometheus and check "Targets" see if the service monitor for your app was detected and being scraped.
Portforward to grafana and create a new dashboard displaying your custom metrics. Show invocations for each handler and its response time.

Basic Chaos Engineering for API Service


Identify the critical components of your API service, such as pods, services, and worker nodes, that you want to test for resilience.
Start with simple chaos experiments like killing random pods in the API deployment. Use the following kubectl commands to simulate pod failures:

List all pods in the deployment: kubectl get pods -l <deployment-label>
Select a random pod from the list: kubectl get pods -l <deployment-label> -o jsonpath='{.items[*].metadata.name}' | awk '{print $1}' | shuf -n 1
Delete the selected pod: kubectl delete pod <pod-name>


Expand your chaos experiments by introducing network latency or packet loss to simulate real-world network conditions. Use network policies to control network traffic. You can refer to the Kubernetes documentation on Network Policies to create and apply network policy files for introducing network chaos.
Perform chaos experiments by inducing CPU or memory stress on the worker nodes hosting your API service. Use resource quotas to limit resource consumption. You can refer to the Kubernetes documentation on Resource Quotas to create and apply resource quota files for simulating resource exhaustion scenarios.
Monitor the behavior of your API service during chaos experiments. Use Prometheus and Grafana, or other monitoring tools, to gather metrics and observe the impact of chaos on your application's performance and availability.
Analyze the results of chaos experiments and identify areas of improvement. Use the insights gained from chaos engineering to enhance the resilience and fault tolerance of your API service. Implement necessary changes, such as improving error handling, introducing retries, or enhancing scalability.
Repeat chaos experiments periodically to continuously validate the resilience of your API service. Use the lessons learned from each chaos experiment to further enhance the stability and reliability of your application.

By manually running kubectl commands to introduce chaos into your API service, you can gain insights into the resilience and robustness of your application. Remember to start with simple experiments and gradually increase the complexity as you gain confidence in your application's ability to handle various failure scenarios.