Storage backups are an import part of a fault-tolerant application. In Procore's current state, an S3 Bucket deletion would be catastrophic.
The goal is to have a storage backup solution that is robust, autonomous, scalable, and easily deployed in an emergency.
- Cloudfiles backup - This was our previous solution. At some point this solution stopped working, but its possible to bring it back, possibly through a lambda function.
- Pros: This keeps our replicated files out of the AWS ecosystem, which would protect us from an all out S3 failure.
- Cons: We have no easy and quick way to restore from or switch traffic to the backup.
- Cross-Region replication - S3 has a feature called cross-region replication built in. This allows you to replicate all files in a bucket to another bucket in a different region.
- Pros: We can easily switch customers between S3 Buckets with a little configuration change. Its fully automatic and as robust as S3.
- Cons: We need to replicate to a different region than the files are currently stored in. It also doesnt work retroactively.
- Lambda function with S3 trigger - described in this post, we could have a lambda function replicate our files to another s3 bucket.
- Pros: This allows us to replicate files to the same region that they're currently stored in. It also gives us better metrics and visibility into whether the replication is working.
- Cons: Cost is a huge con here, as is development/maintenance. This solution is also not retroactive.
Given the pros and cons laid out here, we believe the best solution is to enable cross-region replication. In order to do this we'll also need to enable versioning on any bucket we want to replicate, which may require developer support but also gives us an extra layer of protection.