Kubernetes Third-Party Resource Users

users.md

NOTICE

This list has moved to a GitHub repo for easier tracking: https://github.com/coreos/awesome-kubernetes-extensions

List

Please comment below if you are using Kubernetes Third-Party Resources and I will add you to the list.

Known Users:

• Rook Operator: https://github.com/rook/rook/tree/master/demo/kubernetes
• Elasticsearch Operator: https://github.com/upmc-enterprises/elasticsearch-operator
• Etcd Operator: https://coreos.com/blog/introducing-the-etcd-operator.html
• Prometheus Operator: https://coreos.com/blog/the-prometheus-operator.html
• CoreOS Tectonic: https://coreos.com/tectonic
• Kelsey Hightower's kube-cert-manager: https://github.com/kelseyhightower/kube-cert-manager
• Gravitational Rigging: https://github.com/gravitational/rigging
• PalmStoneGames kube-cert-manager: https://github.com/PalmStoneGames/kube-cert-manager
• linkerd/namerd: https://linkerd.io/config/0.8.6/namerd/index.html#kubernetes
• Flannel Networking: https://github.com/coreos/flannel
• Project Calico Networking: https://github.com/projectcalico/libcalico-go
• Dex OIDC Identity Provider: https://github.com/coreos/dex
• Kubernetes Secrets Manager: https://github.com/upmc-enterprises/kubernetes-secret-manager
• Giant Swarm's Kubception System: http://blog.kubernetes.io/2017/01/how-we-run-kubernetes-in-kubernetes-kubeception.html
• Digital Ocean's Internal CA: re @fatih
• Icinga2 operator - https://github.com/appscode/searchlight
• HAProxy based Ingress controller - https://github.com/appscode/voyager
• Mirantis App Controller - https://github.com/Mirantis/k8s-AppController
• kubeless https://github.com/skippbox/kubeless
• hypernetes network stack: https://github.com/hyperhq/hypernetes/
• SOHO House Postgres Operator (TPR): to create/destroy databases in the cluster or in the cloud.
• SOHO House EIP Operator (TPR): whitelisting IPs and management of those IPs/labels
• SAP OpenStack operator creates various resources in OpenStack (currently, Keystone domains, projects, users, groups and roles; will expand to Swift accounts, Designate zones etc. later): https://github.com/sapcc/kubernetes-operators/tree/master/openstack-operator
• KubeVirt - run virtual machines on Kubernetes: https://github.com/kubevirt/kubevirt
• Archon manages the physical resources that k8s currently doesn't, like Instance, InstanceGroup, Network (defined as TPR)
• Zalando Platform IAM OAuth integration: zalando-incubator/kubernetes-on-aws#154
• Zalando PostgreSQL cluster definition (operator will create a new PostgreSQL/Patroni cluster)
• Loodse https://beta.kubermatic.io
• Atlassian Smith: https://github.com/atlassian/smith
• Kafka Operator: https://github.com/krallistic/kafka-operator
• CrunchyData PostgreSQL operator: https://github.com/CrunchyData/postgres-operator

TODO:

• Dig through commits on GitHub: https://github.com/search?q=ThirdPartyResource&type=Commits&utf8=%E2%9C%93

Resources:

• Kubernetes Docs
• Kubernetes Feature

I am interested in documenting all of the users and getting some more feedback on what is and isn't working.

Both of the kube-cert-managers use TPRs. https://github.com/kelseyhightower/kube-cert-manager and https://github.com/PalmStoneGames/kube-cert-manager

we use TPR to provide transaction like behavior for resource upgrades in rigging: https://github.com/gravitational/rigging

linkerd/namerd may use TPR for policy storage. https://linkerd.io/config/0.8.6/namerd/index.html#kubernetes

We're trying to use them at Atlassian but struggling with the client libraries.

@adieu link your archon here?

@davecheney besides tpr, we also need to push upstream to pay attention and fix client library.

Kubernetes secret manager: https://github.com/upmc-enterprises/kubernetes-secret-manager

@davecheney - my best take on using client library with it after a day of digging around:

https://github.com/gravitational/rigging/blob/master/changeset.go#L941

We're using TPR in DigitalOcean for our internal CA system to issue certificates. The examples in the client were not useful and I had to dive into the code to make sense on how to use the client when doing CRUD methods on a TPR.

We are using TPRs to manage kubernetes clusters with kubernetes.

See http://blog.kubernetes.io/2017/01/how-we-run-kubernetes-in-kubernetes-kubeception.html

At SAP, we're working on an OpenStack operator that creates various types of things in OpenStack (currently, Keystone domains, projects, users, groups and roles; will expand to Swift accounts, Designate zones etc. later): https://github.com/sapcc/kubernetes-operators/tree/master/openstack-operator

There's a difference between TPRs and Operators. At Soho House I've built a few with different degrees of success. Some of the operators use TPRs, some of them they just listen to events and react to them.

• Postgres Operator (TPR): to create/destroy databases in the cluster or in the cloud.
• Namespace Operator: to initialise namespaces (secrets, ingress, etc), and destroy ns once they're not needed anymore.
• EIP Operator (TPR): whitelisting IPs and management of those IPs/labels

Since I was the only one with kubernetes knowledge, I expose the kube events using NATs to allow the other developers to build logic around what's happening in the cluster. For example, we have a couple of jobs that interface with banks and Salesforce and these jobs need to know what has been executed and what's in execution at runtime. I'm using Nats as a way of turning the Kube API into a kind of pub/sub event queue.

I will be talking at KubeCon about our solution and what it worked and what it didn't.

hypernetes is using TPR resource to manage network. https://github.com/hyperhq/hypernetes/

kubeless https://github.com/skippbox/kubeless is using TPR . I would like to see better support for it in the incubating Python client

I'm using TPRs for Kubestack operators.

We, AppsCode, are using TPRs for

• Icinga2 operator - https://github.com/appscode/searchlight
• HAProxy based Ingress controller - https://github.com/appscode/voyager

Helm project is looking into using TPRs to store release data:

• helm/helm#1650 (comment)

AppController uses TPR to store deployment graphs.

https://github.com/Mirantis/k8s-AppController

WIP for Spark on Kubernetes: kubernetes/kubernetes#34377 (comment)

We, Loodse - https://beta.kubermatic.io, use them but having the following issues:
-They don't get deleted with the namespace
-Having a TPR called node.example.org makes kubectl get nodes return my tpr's instead of the actual cluster nodes

Edit: As reference we used: https://github.com/Mirantis/k8s-externalipcontroller/tree/master/pkg/extensions

WIP using TPR's for Amalgam8 microservice routing rules.

We are using TPRs at Zalando Technology for:

• our own Zalando Platform IAM OAuth integration: zalando-incubator/kubernetes-on-aws#154
• PostgreSQL cluster definition (operator will create a new PostgreSQL/Patroni cluster)

We are building Archon which is a set of controllers that runs on k8s and manages the physical resources that k8s currently doesn't, like Instance, InstanceGroup, Network (defined as TPR). So the cluster managed can scale itself.

@hongchaodeng didn't get notification from github. But luckily my colleague found this gist.

@philips is it possible to put the link to archon repo at the end? I think it will make the format better. Thanks a lot.

I use TPR for a Kafka Operator
https://github.com/krallistic/kafka-operator

We're also using TPRs to manage kubernetes clusters with kubernetes, and develop many controllers like KubernetesController, MachineController, MachineSetController, MachineDeploymentController, etc.

Implement Kubernetes as a Serivce using TPR (in Japanease)

CrunchyData PostgreSQL operator: https://github.com/CrunchyData/postgres-operator

Using TPR in https://github.com/coreos/quartermaster

Using TPR to integrate with Stackstorm – https://github.com/pearsontechnology/st2contrib/blob/master/packs/kubernetes/sensors/third_party_resource.py

https://github.com/upmc-enterprises/kong-operator

I use TPR for a Mongo Operator
https://github.com/kirk-enterprise/mongo-operator

Using TPR for RDBMS Operator to support mainstream version Oracle, MySQL, MariaDB
https://docs.google.com/document/d/1zPigFS-VqysjIn569TZgd42whyBlspKcudRaLSrB5Aw/edit#