In my storage quests, I finally decided I want to lazily use S3 for ReadWriteMany and to do do some experiments with.
There are a few options, but to save you some time if you just want what I landed on, I like csi-s3.
Well... this works great! The only problem was that it needed security privileges for mounting. That would be terrible if a container with this power got compromised, so I immediately moved on to getting this a layer away from being managed in-pod.
My initial plan was to just use the nfs-subdir-external-provisioner on top of a multi-replica S3 backed deployment of NFS Ganesha.