DSC disaster recovery strategy

DSC_DR.md

Keep a hot spare EBS volume in sync with production.

On dsc.cdlib.org;

• run stunnel configured in server mode, set to exec rsync
  • specify connection must have specific client certificate
• dump database to flat file

At 5am every day; start machine (attached boot EBS)

1. mount zpool
2. rsync from dsc.cdlib.org to DSC_DR zpool --rsh stunnel (client cert on this machine)
3. zfs snapshot
4. unmount zpool
5. incremental snapshot DSC_DR ebs volumes to s3

create a script that can bootstrap a DSC/XTF server and mysql in Amazon and automatically test it and tear it down once a week.

In the event of a disaster in the UCOP datacenter, use the DR script to set up a new version of the site in Amazon.

stunnel.md

some hints

http://dozzie.jarowit.net/trac/wiki/RsyncSSL

git clone http://dozzie.jarowit.net/code/rsync-ssl.git

zfs_ebs.md

The mirror could be a zpool built from several EBS volumes

• http://zfsonlinux.org (linux kernal)
• http://omnios.omniti.com (solaris derivative AMI)
• http://www.daemonology.net/freebsd-on-ec2/ (BSD AMI)

These could be stacked in different sizes, with the smallest getting switched out for a larger one when the pool needs to grow:

200 G            200
100 G            100
 50 G             50
 25 G             25
 12.5 G      ->  400 G
-------         ------
387.5 G          775 G