I've been using CoreOS as an internal Docker PaaS, and have had a few thoughts.
-
While sub-optimal, there is a need to map in network storage to containers.
For example, Jenkins stores its configuration on the file system. As CoreOS containers can and should be portable around the cluster, unless you're using some kind of distributed storage your Jenkins configs will disappear periodically.
-
Almost everything else should be ephemeral. You should replace CoreOS machines with fresh images periodically. Rather than perform maintenance and cleanup of the CoreOS hosts, just replace them. This is not only convenient, but tests failover constantly.
-
Map in ephemeral storage for apps to use as temporary/scratch disk.
-
Corollary: databases that do their own replication can use scratch disk instead of network storage
-
Leverage etcd for service discovery