Separating "public" from "internal" CRs

public-internal-crs.md

Summary

It would be great to indicate whether or not a custom resource is intended for general usage as opposed to being a resource used internally to a concern.

Generally speaking, this can help encourage increased custom resource usage while at the same time providing a reasonable user experience.

Background

Kubernetes API as messaging between two concerns

Kubernetes API can be looked as a message bus between two parties:

• One party creates an object (custom resource) that describes what they want done, typically in a spec field
• Another party listens for that object, implements it (if possible) and updates the object in the status field

Essentially this is a request/answer between two systems, and generally it works great.

Most API resources can considered to be usable by a human / other concerns

Directly or Indirectly, vanilla resources in Kubernetes (core, batch, etc.) are suitable for use by a human, and thus could be considered part of some public set of resources. They are considered fundamental building blocks to the kubernetes environment.

Generally speaking, its best if just one controller manages any given resource

Aside from splitting by namespace or some other filter, it is easiest if just one controller is "listening" to what we would consider to be the spec a specific type of resource. It avoids contention, etc.

Its also easier if a controller is only concerned with one type of resource

Its also easier for design, development, testing, scaling, and overall lifecycle if a controller is only concerned with one type of resource. In the world of kubernetes, embracing microservices makes sense, and its much easier for the scheduler to balance a load of 30 microservices than 2 or 3 monolithic controllers.

Problem

Consider for a moment that an operator is defined as a suite of controllers. Each controller is concerned with a specific task, and overall communication happens between controllers through kubernetes api custom resources. So if there are five controllers in this operator, there's also five custom resources.

Its not hard to imagine in this situation that the operator has been designed to be interfaced by external conerns (humans, etc.) through one or two custom resources and the remaining custom resources are used to do some concerns that are internal to the overall operator.

Furthermore, as the operator develops over time, some custom resources (and controllers) may be added, removed, or changed. Perhaps more indiscrimatory/rapidly than the intended "public interface" custom resources.

To the user, without reading the documentation for this operator, there is no indication which resources are something that, under normal circumstances, a user should ignore and which ones are what a user should examine. Users may be conditioned to think that every resource is intended for public usage and may cause unintended actions, or at least feel overwhelmed.

Arguably through proper rbac rules, users could be restricted from creating/viewing/modifying such internal custom resources, but we may be able to do better. It would be nice to allow the user to differentiate between "public" custom resources and ones that are internal.

Possible solution

A possible solution could be to add a well known annotation to custom resource definitions. Imagine if:

• Annotation api.kubernetes.io/internal-resource: true was set on a CRD, it could indicate to a user that the resource is intended to be internal to some other concern.
• We could add --interal-resources=true as a command line flag to kubectl commands to include such internal resources (by default exclude them)

This way the resources can still belong to a group (all or otherwise) but not show up unless a user specifically asked for it. Furthermore, as an opt-in type of situation, there should not be any backward-compatibility issues.