Strategies for persistent data storage on CoreOS-cluster

README.md

Persistent data storage on CoreOS-cluster

Storing the data on the host machine

Data directories are created in /home/coreos and mounted into the container using volume cli options of docker run. All data the container writes is stored on the host and as long as the host persists safe against container restarts / recreates.

• Pro
  • No effort, just create the directories and mount them into the container
• Contra
  • Container is bound to host (unable to failover)
  • Backup is hard (it requires backup-strategy working on CoreOS)

Having a cluster-fs on CoreOS

Data is stored using previous method but the directory containing the data of the containers is synched using a cluster filesystem and for that reason available on every machine.

• Pro
  • All machines have the same container data
  • Containers can easily failover and have their data available
• Contra
  • Currently only one cluster-fs is supported and it's alpha
  • Storage on every machine is used for data not required on that machine

Data-Container with exposed volumes

There is a redis-data container exposing /data to redis container. Data is stored inside redis-data and as long as redis-data exists the data is safe even when the redis container is recreated.

• Pro
  • It's the "docker way"
• Contra
  • That container is not moveable
  • Container needs to have backup / restore logic
  • Data-Container needs knowledge about the app container (which volumes are to expose, which dirs to backup)

Backup-Container using volumes from app container

The redis container stores the data inside itself on /data which is mounted into the redis-backup container. The redis-backup container manages backup of the data mounted to itself into some datastore. If the redis container is recreated the redis-backup container restores the data and everything is working.

• Pro
  • Backup-Container can somehow find out which volumes are there and backup those
  • Backup-Container needs no knowledge about the app container
• Contra
  • How does the Backup-Container know whether to backup or restore?
  • Data is restored after startup of the app container. App needs to be able to manage this. (Redis for example isn't)

App-container stores persistent data in cloud storage

For example AWS credentials are passed to the container and the container stores all data into S3. No data is held on the host (except maybe a cache inside the container) and no backup- or failover-strategy is required.

• Pro
  • Best solution as application itself is stateless
• Contra
  • Developer of application needs to support this storage way

Storing the data inside app-container without sync / backup

• Pro
  • Really no effort
• Contra
  • Everything else

One EBS volume per container

• Pro
  • Ability to move data fast by reattaching device to different machine
• Contra
  • Bound to AWS
  • Only one machine can access the data concurrently
  • Failover needs to be done by something capable of moving EBS volumes instead of Fleet or similar

This could be another solution in the future: https://aws.amazon.com/efs/

I was thinking of using Ceph and/or GlusterFS-on-ZFS the question for me is wether CoreOS or the Container should own de FS connection. giving CoreOS acces would mean that most docker images should work but a failure here would mean that all containers will lose their FS acces.

I'm working on a writeup for CoreOS + EFS, as @seiffert had earlier mentioned, using a cloud-config mount unit to pick up the right mount point for the storage.

Any thoughts on strategies specifically for when a container wants to r/w to a sqlite database? Currently my approach is everything NFS4 but this is not ideal due to lack of locking. Only solution I can think of is to use CoreOS local FS and keep that synced up remotely for backup/fail-over. But maybe there is some clustered/distributed FS technology that would work better?