buckets.md

Non-hierarchycal, shared, versioned storage (modeled after Amazon S3)

• Buckets contain objects.
• Object contain data (Vec<u8>).
• Access to a bucket can be shared: bucket can have multiple owners who can then create/modify/delete objects.
• Any modification to an object (create, update, delete) causes version bump of the bucket. That way, whole bucket can be restored to a previous version. This is much simpler to do than versioning directories in a hierarchical storage (POSIX-like)
• Objects themselves can be unversioned or versioned.
• Buckets are stored in versioned structured data. Objects can be any type of data.

Structure

struct Bucket {
    name: String, // Might be redundant. See "Additional ideas".
    object: Vec<Object>,
}

struct Object {
    name: String,
    id: DataIdentifier,
    version: u64,
    size: u64,
    created_at: Tm,
    updated_at: Tm,
    metadata: Vec<u8>,
    key: secretbox::Key, // Might not be needed. See "Open questions".
}

Operations

Create bucket

1. Create the bucket structure
2. Serialize it
3. Encrypt it using symmetric encryption (the key comes from user's account packet, or somewhere else)
4. Store into versioned structured data

Create object

1. Fetch the bucket to put the object in, decrypt and deserialize it
2. Store the object content in the network (as immutable data, versioned/unversioned structured data, etc...)
3. Append the object info to the bucket's objects list (its DataIdentifier, name, version (if versioned) and other metadata)
4. Update the versioned structured data where the bucket is stored, bumping its version

Retrieve object

1. Fetch, decrypt and deserialize the bucket
2. Look up the object in the objects vec in the bucket. Get its data_identifier and version
3. GET the data given data_identifier. If versioned, use version to fetch the correct version
4. (TBD) decrypt using the symmetric key stored in the object metadata

Update object

1. Fetch, decrypt and deserialize the bucket
2. Retrieve the object
3. Modify it
4. POST the object to the network
5. Update the object metadata in the bucket (updating the version too if versioned)
6. Serialize, encrypt and store the bucket to the network (bumping its version)

Delete object

1. Fetch, decrypt, deserialize the bucket
2. Lookup the object to delete. Get its data_identifier.
3. DELETE the object form the network (using the data_identifier)
4. Remove the object metadata from the objects vec in the bucket.
5. Serialize, encrypt and store the bucket to the network (bumping its version)

Delete bucket

1. Fetch, decrypt, deserialize the bucket
2. Delete all objects
3. Delete the structured data with the bucket.

Fetch previous version of buckets

1. Fetch the desired version of the structured data where the bucket is stored, decrypt and deserialize
2. Done 8)

Revert object to previous version

1. Revert the structured data where the bucket is stored

Revert a versioned object to previous version

1. Fetch, decrypt, deserialize the bucket
2. Lookup the object to revert
3. Modify the version field in the object metadata to the desired version number
4. Serialize, encrypt and store the bucket to the network (bumping its version)

Open questions

• Do we also want read-only access to buckets and objects for non-owners?
• Do we need to encrypt objects? (If the answer to the previous question is yes, we do. Otherwise it might not be necessary).

Additional ideas

User can own more than one bucket, so there needs to be a place where the bucket ids are stored. We can use a "bucket index" struct for that:

struct BucketIndex {
    buckets: Vec<BucketEntry>,
}

struct BucketEntry {
    name: String,
    id: XorName,
    version: u64,
    key: secretbox::Key,
}

This struct would be stored in a structured data (unversioned is probably fine) and its id would be added to the account packet.