challenges of implementing AWS S3 style DNS

minio.md

• A stateful set implementing S3 compatible service gets a DNS name such as: ${S3 instance}.${namespace}.svc.cluster.local

  • APIs such as list bucket and make bucket are sent to this DNS address

• Buckets in S3 have their own DNS name and going forward it is the preferred way for communicating with a bucket. (vs path style)

• We can create a new service per bucket (backed by the stateful set) and give it a DNS name ${bucket}.${namespace}.svc.cluster.local. It is mapped to the Stateful Set via Cluster IP.

  • To be compatible with S3 style DNS we need: ${bucket}.${S3 instance}.${namespace}.svc.cluster.local
  • Currently a service name cannot have a “.” in their name.

• One option we have implemented is taking over a custom DNS domain and asking CoreDNS to forward the subdomains for that custom domain. This approach has a few downsides to it:

  • Buckets are no longer k8s services and we lose any ecosystem integration that exists around monitoring and analyzing services.
  • We cannot conveniently automate updating the forwarding rule. Operator needs edit permissions to the configmap in kube-system namespace.
  • Depends on vendor specific solution and is not a k8s standard config
  • Ref: https://github.com/minio/disco

• The landscape for service discovery is not very standardized as well.

• We desire a k8s standardized way to achieve the DNS hierarchy that is compliant with how a lot of cloud vendors layout bucket hierarchy in their DNS.