Skip to content

Instantly share code, notes, and snippets.

@protosam

protosam/leaderless.yaml Secret

Last active Jun 1, 2021
Embed
What would you like to do?
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: task-owners.future.ideas.protosam
spec:
group: future.ideas.protosam
versions:
- name: v1beta1
served: true
storage: true
additionalPrinterColumns:
- name: Owner
type: string
description: "Operator that owns this task"
jsonPath: .owningOperator
- name: Age
type: date
jsonPath: .metadata.creationTimestamp
schema:
openAPIV3Schema:
type: object
properties:
owningOperator:
type: string
default: "PENDING"
scope: Namespaced
names:
plural: task-owners
singular: task-owner
kind: TaskOwner
---
apiVersion: v1
data:
run.sh: |-
#!/bin/sh
export PYTHONUNBUFFERED=1
pip3 install -r requirements.txt > /dev/null 2>/dev/null
python3 operator.py
requirements.txt: |-
kubernetes
operator.py: |-
#!/usr/bin/env python3
from kubernetes import client, config, watch
import secrets
import socket
import time
try:
config.load_kube_config()
except:
# load_kube_config throws if there is no config, but does not document what it throws, so I can't rely on any particular type here
config.load_incluster_config()
print("operator.py is running.")
for event in watch.Watch().stream(client.CustomObjectsApi().list_cluster_custom_object, "future.ideas.protosam", "v1beta1", "task-owners", resource_version=""):
if event["type"] == "ADDED" and event['object']["owningOperator"] == "PENDING":
print("Attempting claim")
event['object']['owningOperator'] = socket.getfqdn()
try:
client.CustomObjectsApi().patch_namespaced_custom_object("future.ideas.protosam", "v1beta1", event['object']['metadata']['namespace'], "task-owners", event['object']['metadata']['name'], event['object'])
except client.exceptions.ApiException:
print("Failed to claim.")
continue
event['object'] = client.CustomObjectsApi().get_namespaced_custom_object("future.ideas.protosam", "v1beta1", event['object']['metadata']['namespace'], "task-owners", event['object']['metadata']['name'])
print(event['object']['metadata']['name'], "was is claimed by me.")
kind: ConfigMap
metadata:
name: leaderless-elections
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: leaderless-operator
automountServiceAccountToken: true
---
# minimal permissions required for the service account
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leaderless-operator
rules:
- apiGroups: [""]
resources: ["task-owners"]
verbs: ["list", "watch", "patch"]
---
# binding the above cluster role (permissions) to the above service account
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: leaderless-operator-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: leaderless-operator
subjects:
- kind: ServiceAccount
name: leaderless-operator
namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: leaderless-operator
spec:
replicas: 3
selector:
matchLabels:
name: leaderless-operator
template:
metadata:
labels:
name: leaderless-operator
spec:
serviceAccountName: leaderless-operator
automountServiceAccountToken: true
containers:
- image: python:3
name: leaderless-operator
command: [ 'bash', 'run.sh' ]
workingDir: /usr/src
volumeMounts:
- mountPath: /usr/src
name: source
volumes:
- name: source
configMap:
defaultMode: 420
name: leaderless-elections
@protosam

This comment has been minimized.

Copy link
Owner Author

@protosam protosam commented May 31, 2021

This is an example of abusing Kubernetes CRDs to have an operator claim work without a leader. In testing, I found that the apiserver has logic built in to prevent multiple updates of the same object source.

You can test this out after deploying leaderless.yaml.

Create a new object:

$ cat <<EOF | kubectl apply -f -
apiVersion: future.ideas.protosam/v1beta1
kind: TaskOwner
metadata:
    name: mytaskname-$(openssl rand -hex 12)
EOF

Check the results.

$ kubectl get task-owners
$ kubectl describe task-owners mytaskname-...

You can also check all the logs.

$ kubectl get pods | awk '/^leaderless/{print "echo "$1"; kubectl logs --tail 10 "$1"; echo ---;"}' | sh

You can also just see who owns the tasks.

$ kubectl get task-owners
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment