Train of thought for the development of hambone

hambone_train_of_thought.md

• We want an API server for CRUDing nearly identical k8s specs, but want to be able to customize anything about any one of them.
• Chose gRPC/Protocol Buffers for the API, primarily for the free code generation, but also for the advanced networking smarts built into gRPC. Swagger docs and a JSON/Rest gateway can be auto-generated from the proto file.
• A major goal for the project is replica safety. - We want to run multiple copies of the API server to eliminate the majority of types of downtime for clients.
• I went through 2 major re-writes on this project.
  • The first version was a quick spike as a Golang package for use in custom apps. It uses jq (and yq) to render yaml templates and apply them via kubectl.
  • I then decided that with the right abstractions an open source app could be developed (hambone).
  • hambone's first version was built as a standalone binary, with plug-able state storage and k8s adapters. It hoped to avoid executing other processes (often miss-referenced as "shelling out",) and did so by consuming the k8s-server API directly.
  • This version I test drove. The official Golang client lib even has a mock server for developing unit tests.
  • I discovered that working with even minor version differences between the k8s client library and the actual cluster can be expensive. - A different version of hambone would need developed against each minor version of k8s. - Hmm.. this is something that kubectl already handles really well...
  • While reading how kubectl does it, I discovered kustomize (not publicized at the time, but has been since).
  • kustomize has a similar goal to mine, - to repeatably customize a "base" set of k8s specs with minimal work, - except their way of doing it is undoubtedly superior to mine. The tool also follows the unix philosophy of single responsibility, and marries perfectly with kubectl.
  • If there's ever a non-smelly use-case for "shelling out", this might be it..
• Scrap it all. Let's make a CRUD API server that executes kustomize and kubectl, and manages state.
• Now you can use the same version of hambone against (almost) any version of k8s, and conversely upgrade hambone without upgrading your cluster.
• You'll notice there are very few tests in this final version. That's because:
  • The app has very little of it's own logic (FTW).
  • There are mocking libraries for unit testing apps that execute other processes, but A) it didn't seem those libraries were built for this scenario, (I have yet to investigate that further), and B) I was already out of time for this project.
  • I want to add more tests, but had to switch to looking for a job. :)
• No more major re-writes, let's make this an alpha version.