for each hardware piece we need the following components:
- OS
- Container runtime
- Network Connection to a control plane (or be configured itself to be a control plane)
- Potentially storage software
- Ceph looks good here
- Mayastor is new and quite promising
- Things to setup:
- Network connectivity. Kubernetes services need a way to access a "real" IP
- MetalLB works well even for off-the-shelve routers
- Storage Provider
- Connect to what you've chosen above
- Ingress software
- Traefik (Proxy) is very proven and works incredibly well
- Mesh software
- Traefik (Mesh) looks good
- Istio is known
- Linkered looks good and is very simple
- Hashicorp consul is enterprise-focused software.
- Metrics
- Prometheus is standard
- a secondary long-term storage is required.
- Grafana Mimir looks good
- Everyone and their mom comes with a custom prometheus instance. Replace them all.
- Make a list of things to plug into this as you setup other things
- Prometheus is standard
- Dashboarding / Grafana Setup
- A lot of things have dashboard presets. Make a list and add.
- Tracing
- Jaeger appears to be standard
- Grafana Tempo looks very good.
- OpenTelemetry compatability seems important
- S3-compatible thing for storage
- Where possible use kubernetes backed storage
- min.io looks good for providing self-hosted, kubernetes based, S3 compatible storage.
- Ceph has ootb S3 storage. Not sure whether I like it.
The above are the "basics" below are some often used things that will come in handy while you build your first cloud-native app.
Some messaging platform will be criticial to exchange messages between your services. Here are the options:
- no backbone. Use distributed gRPC / HTTP / etc. instead.
- I don't like this, it can become hard to manage and bottlenecks are hard to observe.
- You don't have to maintain extra infrastructure, but this feels irrelevant given the above setup...
- message based backbone.
- The standard here is RabbitMQ.
- IF you can do with just pub/sub Redis can be used. I doubt you'll make this work.
- Stream based backbone.
- Kafka is the standard here, but newer alternatives exist (ie Apache pulsar) this choice is very nuanced.
- You need to model your app around this, while simple messaging can be easier to work with.
- Look at RisingWave too here
- SQL & NoSQL options available. This is fairly nuanced and imo there is no clear winner.
- Redis is the pretty clear winner when it comes to basic caching. Memcached exists.