mhausenblas

Demo: Building a custom OpenTelemetry collector for system-level monitoring

In this demo we will build a custom OpenTelemetry collector and collect logs and metrics from the operating system.

Prerequisites:

• You need Go in v1.17 or above locally installed.
• socat
• netstat
• jq (optional, for nicer JSON output rendering)

mhausenblas

AZ,Availability Zone
ASG,Auto Scaling Group
EBS,Elastic Block Store
ENI,Elastic Network Interface
EIP,Elastic IP
IGW,Internet Gateway
CM,AWS Certificate Manager
ADS,Application Discovery Service
GWY,Amazon API Gateway
GWYv2,Amazon API Gateway V2

mhausenblas

# Use `tmux` to launch or `tmux attach` if already running

# First install https://github.com/tmux-plugins/tpm and then
# TRIGGER + I for the plugins. Below works for tmux 2.3.

# Change TRIGGER
unbind C-b
set -g prefix F11
bind F11 send-prefix

mhausenblas

Create cluster:

eksctl create cluster -f cluster-config.yaml

Install Prometheus:

helm install prometheus-for-amp prometheus-community/prometheus \

mhausenblas

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
" Common cloud native abbreviations
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

" Amazon Web Services (AWS)
iabbrev EKS Amazon Elastic Kubernetes Service

" Cloud Native Computing Foundation (CNCF)
iabbrev k8s Kubernetes

mhausenblas

Does ConfigMap data propagate to pods where it is used?

Experiment

Let's see. We first create a config map called testcm:

kubectl create configmap testcm --from-literal=somekey=INITIAL_VALUE

mhausenblas

Using the official resource-consumer as a stress tester.

Setup

Using EKS on Fargate cluster in eu-west-1:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-14T04:25:00Z", GoVersion:"go1.12.13", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.8-eks-b8860f", GitCommit:"b8860f6c40640897e52c143f1b9f011a503d6e46", GitTreeState:"clean", BuildDate:"2019-11-25T00:55:38Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

mhausenblas

EKS on Arm

The xarm-install-graviton2.sh script allows you to install and use Amazon EKS on Arm (xARM) with a single command. In essence, it automates the steps described in the docs.

Make sure you have aws, eksctl, kubectl, and jq installed, this will be checked on start-up and the script will fail if these deps are not present. So far tested with bash on macOS.

$ chmod +x xarm-install-graviton2.sh
$ ./xarm-install-graviton2.sh

mhausenblas

Set Up Fluent Bit as a DaemonSet to Send Logs to CloudWatch

Create a new 1.13 or 1.14 EKS cluster called container-insights.

Enable IRSA:

eksctl utils associate-iam-oidc-provider \
               --name container-insights \
               --approve

mhausenblas

#!/bin/bash

################################################################################
# Generate load for the NGINXs services in ECS

nginxurls=$(ecs-cli ps --desired-status RUNNING | grep nginx | awk '{ split($3, url, "-") ; print(url[1]) }')
while true
do
    printf "Hit " 
    for nginxurl in $nginxurls