Lumogon and Kubernetes example

README.md

The following demo assumes a running minikube instance.

First grab access to the Docker socket for minikube. On *nix:

eval $(minikube docker-env)

On Windows:

& minikube docker-env | Invoke-Expression

Now lets demonstrate running lumogon:

docker run --rm  -v /var/run/docker.sock:/var/run/docker.sock puppet/lumogon scan

For the demo you'll want to install a few python packages.

pip install kubernetes docker

We'll also want some seed data.

kubectl apply -f app1.yaml
kubectl apply -f app2.yaml
kubectl apply -f app3.yaml

Then lets run the first of our scripts, which should output to stdout:

collect_data_from_api_and_lumogon.py

Finally pipe that output into the second script which provides the nice UI for the data.

collect_data_from_api_and_lumogon.py | output_package_table.py

On windows you might run into an encoding issue, eg. LookupError: unknown encoding: cp65001. You can avoid this with set PYTHONIOENCODING=UTF-8 or $Env:PYTHONIOENCODING="UTF-8" if you're using PowerShell.

app1.yaml

apiVersion: apps/v1beta2 # for versions before 1.8.0 use apps/v1beta1
 kind: Deployment
 metadata:
   name: nginx-deployment
 spec:
   selector:
     matchLabels:
       app: nginx
   replicas: 2 # tells deployment to run 2 pods matching the template
   template: # create pods using pod definition in this template
     metadata:
       # unlike pod-nginx.yaml, the name is not included in the meta data as a unique name is
       # generated from the deployment name
       labels:
         app: nginx
         team: team-shamu
     spec:
       containers:
       - name: nginx
         image: nginx:1.7.9
         ports:
         - containerPort: 80

app2.yaml

apiVersion: apps/v1beta2 # for versions before 1.8.0 use apps/v1beta1
 kind: Deployment
 metadata:
   name: mysql
 spec:
   selector:
     matchLabels:
       app: mysql
   strategy:
     type: Recreate
   template:
     metadata:
       labels:
         app: mysql
         team: team-humphrey
     spec:
       containers:
       - image: mysql:5.6
         name: mysql
         env:
           # Use secret in real usage
         - name: MYSQL_ROOT_PASSWORD
           value: password
         ports:
         - containerPort: 3306
           name: mysql

app3.yaml

 apiVersion: apps/v1beta2 # for versions before 1.8.0 use apps/v1beta1
 kind: Deployment
 metadata:
   name: hello-deployment
 spec:
   selector:
     matchLabels:
       app: hello
   replicas: 2 # tells deployment to run 2 pods matching the template
   template: # create pods using pod definition in this template
     metadata:
       # unlike pod-nginx.yaml, the name is not included in the meta data as a unique name is
       # generated from the deployment name
       labels:
         app: hello
         team: team-keiko
     spec:
       containers:
       - name: hello
         image: gcr.io/google-samples/hello-app:1.0
         ports:
         - containerPort: 8080

collect_data_from_api_and_lumogon.py

 #!/usr/bin/env python

 from kubernetes import client, config
 import docker
 import json

 config.load_kube_config()

 pods = {}

 # First we will access the Kubernetes API for a list of pods,
 # and then record the labels and annotations for each
 v1 = client.CoreV1Api()
 ret = v1.list_namespaced_pod("default", watch=False)
 for i in ret.items:
     pods[i.metadata.name] = {
             "labels": i.metadata.labels,
             "annotations": i.metadata.annotations,
             }


 # We'll then use Lumogon, via the Docker API, to collect information
 # about all of the running containers
 client = docker.from_env()
 output = client.containers.run("puppet/lumogon",
                                "scan",
                                volumes={'/var/run/docker.sock': {'bind': '/var/run/docker.sock',
 'mode': 'rw'}},
                                remove=True)
 data = json.loads(output)


 # Finally we'll link the two data sources. The Docker label io.kubernetes.pod.name
 # contains the unique pod name used by the Kubernetes API. In this case we're going to
 # add a list of system packages to our metadata from above
 for container_id in data["containers"]:
     container = data["containers"][container_id]
     pod = container["capabilities"]["label"]["payload"]["io.kubernetes.pod.name"]

     if pod in pods:
         for package_type in ["apk", "dpkg", "rpm"]:
             if package_type in container["capabilities"]:
                 pods[pod]["packages"] = container["capabilities"][package_type]["payload"]


 # For this script we'll just print to stdout for either processing
 # with something like jq
 print(json.dumps(pods))

output_package_table.py

 #!/usr/bin/env python
 from terminaltables import AsciiTable
 import json
 import sys

 from collections import Counter

 TEAM_LABEL = "team"

 data = json.load(sys.stdin)


 # We iterate through all the container information and extract out
 # useful structures, including a list of package versions and a list of
 # users indexed by the package version
 packages = {}
 users = {}
 for container in data:
     for package in data[container]["packages"]:
         if package not in packages:
             packages[package] = []
         version = data[container]["packages"][package]
         packages[package].append(version)
         if package not in users:
             users[package] = {}
         if version not in users[package]:
             users[package][version] = []
         users[package][version].append(data[container]["labels"][TEAM_LABEL])


 # Package versions may appear multiple times, so we collapse that down and keep
 # a count of the occurences
 package_counts = {}
 for package_name in packages:
         package_counts[package_name] = Counter(packages[package_name])


 # Finally we build a table based on the data we've structued above
 table_data = [
     ['Package', 'Version', 'Occurences', 'Teams'],
 ]
 for package in sorted(package_counts):
     for version in package_counts[package]:
         unique_users = list(sorted(set(users[package][version])))
         table_data.append([package, version, package_counts[package][version], "\n".
 join(unique_users)])


 table = AsciiTable(table_data)
 print(table.table)