Kubernetes Third-Party Resource Users

users.md

NOTICE

This list has moved to a GitHub repo for easier tracking: https://github.com/coreos/awesome-kubernetes-extensions

List

Please comment below if you are using Kubernetes Third-Party Resources and I will add you to the list.

Known Users:

• Rook Operator: https://github.com/rook/rook/tree/master/demo/kubernetes
• Elasticsearch Operator: https://github.com/upmc-enterprises/elasticsearch-operator
• Etcd Operator: https://coreos.com/blog/introducing-the-etcd-operator.html
• Prometheus Operator: https://coreos.com/blog/the-prometheus-operator.html
• CoreOS Tectonic: https://coreos.com/tectonic
• Kelsey Hightower's kube-cert-manager: https://github.com/kelseyhightower/kube-cert-manager
• Gravitational Rigging: https://github.com/gravitational/rigging
• PalmStoneGames kube-cert-manager: https://github.com/PalmStoneGames/kube-cert-manager
• linkerd/namerd: https://linkerd.io/config/0.8.6/namerd/index.html#kubernetes
• Flannel Networking: https://github.com/coreos/flannel
• Project Calico Networking: https://github.com/projectcalico/libcalico-go
• Dex OIDC Identity Provider: https://github.com/coreos/dex
• Kubernetes Secrets Manager: https://github.com/upmc-enterprises/kubernetes-secret-manager
• Giant Swarm's Kubception System: http://blog.kubernetes.io/2017/01/how-we-run-kubernetes-in-kubernetes-kubeception.html
• Digital Ocean's Internal CA: re @fatih
• Icinga2 operator - https://github.com/appscode/searchlight
• HAProxy based Ingress controller - https://github.com/appscode/voyager
• Mirantis App Controller - https://github.com/Mirantis/k8s-AppController
• kubeless https://github.com/skippbox/kubeless
• hypernetes network stack: https://github.com/hyperhq/hypernetes/
• SOHO House Postgres Operator (TPR): to create/destroy databases in the cluster or in the cloud.
• SOHO House EIP Operator (TPR): whitelisting IPs and management of those IPs/labels
• SAP OpenStack operator creates various resources in OpenStack (currently, Keystone domains, projects, users, groups and roles; will expand to Swift accounts, Designate zones etc. later): https://github.com/sapcc/kubernetes-operators/tree/master/openstack-operator
• KubeVirt - run virtual machines on Kubernetes: https://github.com/kubevirt/kubevirt
• Archon manages the physical resources that k8s currently doesn't, like Instance, InstanceGroup, Network (defined as TPR)
• Zalando Platform IAM OAuth integration: zalando-incubator/kubernetes-on-aws#154
• Zalando PostgreSQL cluster definition (operator will create a new PostgreSQL/Patroni cluster)
• Loodse https://beta.kubermatic.io
• Atlassian Smith: https://github.com/atlassian/smith
• Kafka Operator: https://github.com/krallistic/kafka-operator
• CrunchyData PostgreSQL operator: https://github.com/CrunchyData/postgres-operator

TODO:

• Dig through commits on GitHub: https://github.com/search?q=ThirdPartyResource&type=Commits&utf8=%E2%9C%93

Resources:

• Kubernetes Docs
• Kubernetes Feature

I am interested in documenting all of the users and getting some more feedback on what is and isn't working.

we use TPR to provide transaction like behavior for resource upgrades in rigging: https://github.com/gravitational/rigging

linkerd/namerd may use TPR for policy storage. https://linkerd.io/config/0.8.6/namerd/index.html#kubernetes

We're trying to use them at Atlassian but struggling with the client libraries.

@adieu link your archon here?

@davecheney besides tpr, we also need to push upstream to pay attention and fix client library.

Kubernetes secret manager: https://github.com/upmc-enterprises/kubernetes-secret-manager

@davecheney - my best take on using client library with it after a day of digging around:

https://github.com/gravitational/rigging/blob/master/changeset.go#L941

We're using TPR in DigitalOcean for our internal CA system to issue certificates. The examples in the client were not useful and I had to dive into the code to make sense on how to use the client when doing CRUD methods on a TPR.

We are using TPRs to manage kubernetes clusters with kubernetes.

See http://blog.kubernetes.io/2017/01/how-we-run-kubernetes-in-kubernetes-kubeception.html

At SAP, we're working on an OpenStack operator that creates various types of things in OpenStack (currently, Keystone domains, projects, users, groups and roles; will expand to Swift accounts, Designate zones etc. later): https://github.com/sapcc/kubernetes-operators/tree/master/openstack-operator

There's a difference between TPRs and Operators. At Soho House I've built a few with different degrees of success. Some of the operators use TPRs, some of them they just listen to events and react to them.

• Postgres Operator (TPR): to create/destroy databases in the cluster or in the cloud.
• Namespace Operator: to initialise namespaces (secrets, ingress, etc), and destroy ns once they're not needed anymore.
• EIP Operator (TPR): whitelisting IPs and management of those IPs/labels

Since I was the only one with kubernetes knowledge, I expose the kube events using NATs to allow the other developers to build logic around what's happening in the cluster. For example, we have a couple of jobs that interface with banks and Salesforce and these jobs need to know what has been executed and what's in execution at runtime. I'm using Nats as a way of turning the Kube API into a kind of pub/sub event queue.

I will be talking at KubeCon about our solution and what it worked and what it didn't.

hypernetes is using TPR resource to manage network. https://github.com/hyperhq/hypernetes/

kubeless https://github.com/skippbox/kubeless is using TPR . I would like to see better support for it in the incubating Python client

I'm using TPRs for Kubestack operators.

We, AppsCode, are using TPRs for

• Icinga2 operator - https://github.com/appscode/searchlight
• HAProxy based Ingress controller - https://github.com/appscode/voyager

Helm project is looking into using TPRs to store release data:

• helm/helm#1650 (comment)

AppController uses TPR to store deployment graphs.

https://github.com/Mirantis/k8s-AppController

WIP for Spark on Kubernetes: kubernetes/kubernetes#34377 (comment)

We, Loodse - https://beta.kubermatic.io, use them but having the following issues:
-They don't get deleted with the namespace
-Having a TPR called node.example.org makes kubectl get nodes return my tpr's instead of the actual cluster nodes

Edit: As reference we used: https://github.com/Mirantis/k8s-externalipcontroller/tree/master/pkg/extensions

WIP using TPR's for Amalgam8 microservice routing rules.

We are using TPRs at Zalando Technology for:

• our own Zalando Platform IAM OAuth integration: zalando-incubator/kubernetes-on-aws#154
• PostgreSQL cluster definition (operator will create a new PostgreSQL/Patroni cluster)

We are building Archon which is a set of controllers that runs on k8s and manages the physical resources that k8s currently doesn't, like Instance, InstanceGroup, Network (defined as TPR). So the cluster managed can scale itself.

@hongchaodeng didn't get notification from github. But luckily my colleague found this gist.

@philips is it possible to put the link to archon repo at the end? I think it will make the format better. Thanks a lot.

I use TPR for a Kafka Operator
https://github.com/krallistic/kafka-operator

We're also using TPRs to manage kubernetes clusters with kubernetes, and develop many controllers like KubernetesController, MachineController, MachineSetController, MachineDeploymentController, etc.

Implement Kubernetes as a Serivce using TPR (in Japanease)

CrunchyData PostgreSQL operator: https://github.com/CrunchyData/postgres-operator

Using TPR in https://github.com/coreos/quartermaster

Using TPR to integrate with Stackstorm – https://github.com/pearsontechnology/st2contrib/blob/master/packs/kubernetes/sensors/third_party_resource.py

https://github.com/upmc-enterprises/kong-operator

I use TPR for a Mongo Operator
https://github.com/kirk-enterprise/mongo-operator

Using TPR for RDBMS Operator to support mainstream version Oracle, MySQL, MariaDB
https://docs.google.com/document/d/1zPigFS-VqysjIn569TZgd42whyBlspKcudRaLSrB5Aw/edit#