Camlistore Encryption on S3

caenons3.md

Camlistore Encryption on S3

Here's what I found out while setting up Camlistore to sync encrypted blobs to Amazon S3. This works for me, but I'm new to Camlistore so this might not be the best way to do it.

Amazon S3 Config

The encryption handler requires two buckets, one for metadata and one for blobs. In this example, I've created two buckets called my-camlistore-meta and my-camlistore-blob.

Server Config

A low level server config file is required. If you're using the high level format, you can convert it by running:

camtool dumpconfig

You can then add the following to the prefixes section to support S3 and encryption:

"/enc-s3/": {
  "handler": "storage-encrypt",
  "handlerArgs": {
    "I_AGREE": "that encryption support hasn't been peer-reviewed, isn't finished, and its format might change.",
    "meta": "/enc-s3-meta/",
    "blobs": "/enc-s3-blob/",
    "metaIndex": { "type": "memory" },
    "key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  }
},
"/enc-s3-meta/": {
  "handler": "storage-s3",
  "handlerArgs": {
    "aws_access_key": "XXXXXXXXXXXXXXXXXXXX",
    "aws_secret_access_key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "bucket": "my-camlistore-meta"
  }
},
"/enc-s3-blob/": {
  "handler": "storage-s3",
  "handlerArgs": {
    "aws_access_key": "XXXXXXXXXXXXXXXXXXXX",
    "aws_secret_access_key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "bucket": "my-camlistore-blob"
  }
},
"/sync-to-enc-s3/": {
  "handler": "sync",
  "handlerArgs": {
    "from": "/bs/",
    "queue": {
      "file": "/path/to/blobs/sync-to-enc-s3-queue.kv",
      "type": "kv"
    },
    "to": "/enc-s3/"
  }
}

Update the key to a 16 byte hex value. This key is the only way to decrypt your data, so keep a copy somewhere safe. To generate 16 random bytes, you can run:

openssl rand -hex 16

Fill in the aws_access_key and aws_secret_access_key values from your Amazon credentials.

And update the file in /sync-to-enc-s3/ to point to the full path to a new kv file in your blobs directory.

Syncing

To sync your existing files to S3:

camtool sync --all

You can verify that it worked by examining the new files added to your S3 buckets. There should be nothing human-readable.

To sync from S3 to your local blob store:

camtool sync --src http://localhost:3179/enc-s3/ --dest http://localhost:3179/bs/
camtool sync --all  # this will index the new blobs