YapDatabaseExtensions and CloudKit

README.md

YapDatabaseExtensions and CloudKit

This is a follow-up to this question, seeking advice on how to use YapDatabaseExtensions' value types with the CloudKit extension for YapDatabase.

Dan Thorpe offered some excellent suggestions in the discussion linked above. Many thanks to him. This is my first stab at a solution. I hope I didn't butcher his advice too badly.

Intention

If this is useful to anyone trying to add CloudKit sync to an app using YapDatabaseExtensions and ValueCoding that's terrific.

If this is a misguided and/or needlessly complicated approach I hope to get some feedback and suggestions.

Best case scenario: We eventually end up with some reusable CloudKit helpers for people using YapDatabaseExtensions.

Problem

In the recordHandler block for CloudKit we need to:

• Figure out which of the synced properties were changed in our object
• Store the previous value of changed properties in the recordInfo (for later use in a three-way merge in the mergeBlock).
• Store the current value of synced properties into the CKRecord.

In the mergeBlock for CloudKit we need to:

• Get the current value of local properties to see if the one in the incoming CKRecord was changed.
• Update individual local property values if we spot a new one from CloudKit.

In most of our app we only deal in ValueType objects. So we can't use the MyDatabaseObject base class approach from the CloudKitTodo example, where changes are tracked and stored via KVO.

And besides: A forced common base class for models with hidden but valuable internal state and logic smells a lot like NSManagedObject. Yuck!

So what to do?

The goal

The objects we are being handed in the recordBlock and mergeBlock are our CodingType objects. We want to cast them to something which can help us solve the problem above. This could either be a common coder base class, or it could be a protocol without any Self type restrictions or associated types (otherwise we can't cast to it and treat it as a type).

Inspired by Crusty we go for a protocol which would make us happy.

protocol SyncableObject {
    var allSyncKeys: Set<String> { get }
    var changedSyncKeys: Set<String> { get }
    var previousSyncValues: [String:AnyObject] { get }
    func syncValueForKey(syncKey: String) -> AnyObject?
    mutating func setSyncValue(syncValue: AnyObject, forKey: String)
}

If we can make all our coders conform to SyncableObject we're happy.

If we could even make it easy for them to conform to SyncableObject we would have the beginnings of a reusable approach and we'd be even happier.

The rest of this is all an attempt at making it possible and easy to make our CodingType objects conform to SyncableObject.

Puzzle piece 1: Getting access to the previous value

A coder will need access to the previous coder in order to figure out what changed. Let's express this through a protocol. Again with no further type restrictions so we can cast to it.

protocol HasPreviousCoder {
    var previousCoder: AnyObject? { get set }
}

How and when should we set this variable?

Inside the recordHandler block the old coder has already been overwritten and lost. So we have to grab it earlier.

The YapDatabase objectPreSanitizer block is invoked before the save occurs. Yay! But it has no access to the open write transaction. And if we try to read using our own transaction we end up in a deadlock. Sad!

It's very possible there are better ways to go about this. But what I ended up doing was registering a hook extension before registering the CloudKit extension.

That way we have a chance to modify the coder before CloudKit sees it. And we can do it before the old value is overwritten.

class PreviousCoderHook: YapDatabaseHooks {
    override init() {
        super.init()
        self.willModifyRow = { (transaction, collection, key, proxyObject, proxyMetadata, flags) in
            guard var hasPreviousCoder = proxyObject.realObject as? HasPreviousCoder else { return }
            let index = YapDB.Index(collection: collection, key: key)
            hasPreviousCoder.previousCoder = transaction.readAtIndex(index)
        }
    }
}

Then somewhere in your database setup code you register this extension:

// Register this hook BEFORE registering the CloudKit extension.
db.registerExtension(PreviousCoderHook(), withName: "previousCoderHook")

Puzzle piece 2: Understanding the value type

When dealing with general CodingType coders we don't know anything about the ValueType.

It has to help us out and describe which of its properties should be synced and how to get and set them. Time for yet another protocol:

struct SyncablePropertyLens<T> {
    let get: T -> AnyObject
    let set: (T, AnyObject) -> T
}

protocol Syncable {
    // Getters and setters for each synced value. Stored by their sync key.
    static var syncProperties: [String:SyncablePropertyLens<Self>] { get }
}

It may be tedious, but surely we can make our ValueType types conform to Syncable. Then, in the coder, we have what we need to interact with them in a general manner.

Note: The "lens" terminology above borrows from the functional programming world. It is inspired by Chris Eidhof's blog post and Brandon Williams' presentation. But it's not the type-safe lenses we'd normally use in Swift. These are for producing and consuming Objective-C classes going in and out of things like CKRecord or NSCoder.

Puzzle piece 3: Modifying the value of a coder

While mutability is bad, it's also a little wasteful if we have to create a new coder every time we modify a single property of the value it stores.

We declare yet another protocol for this, adding a setter for the value so we can update it in place:

protocol MutableCodingType: CodingType {
    var value: ValueType { get set }
}

Implementing SyncableObject

Imagine we're lucky enough that the following is the case:

• Our value type conforms to Syncable
• Our coder type conforms to HasPreviousCoder and MutableCodingType

In that case we can automatically make the coder conform to SyncableObject:

extension SyncableObject where Self: MutableCodingType, Self: HasPreviousCoder, Self.ValueType: Syncable {
    var allSyncKeys: Set<String> {
        return Set(ValueType.syncProperties.keys)
    }
    
    var changedSyncKeys: Set<String> {
        guard let previousValue = self.previousValue else { return allSyncKeys }
        let changedKeys = ValueType.syncProperties
            .filter { _, lens in !lens.get(self.value).isEqual(lens.get(previousValue)) }
            .map { key, _ in key }
        return Set(changedKeys)
    }
    
    var previousSyncValues: [String:AnyObject] {
        guard let previousValue = self.previousValue else { return [:] }
        return ValueType.syncProperties.reduce([:]) { (var dict, entry) in
            let (key, lens) = entry
            dict[key] = lens.get(previousValue)
            return dict
        }
    }
    
    func syncValueForKey(syncKey: String) -> AnyObject? {
        return ValueType.syncProperties[syncKey]?.get(self.value)
    }
    
    mutating func setSyncValue(syncValue: AnyObject, forKey syncKey: String) {
        guard let lens = ValueType.syncProperties[syncKey] else { return }
        self.value = lens.set(self.value, syncValue)
    }
}

Reusable bonus helpers for NSCoding

Most of the time we probably want similar property encodings in the database and in CKRecord.

If we go through the trouble of making our ValueType conform to Syncable we can use that to provide some helpers which will make it much easier for the coder to implement NSCoding in the common case.

If you also have some model properties that you don't want to sync you can still use these helpers to encode the synced part of the object.

extension Syncable {
    func encodeSyncProperties(coder: NSCoder) {
        for (key, lens) in Self.syncProperties {
            coder.encodeObject(lens.get(self), forKey: key)
        }
    }
    
    static func decodeSyncProperties(decoder: NSCoder, into value: Self) -> Self {
        return Self.syncProperties.reduce(value) { value, entry in
            let (key, lens) = entry
            guard let decodedValue = decoder.decodeObjectForKey(key) else { return value }
            return lens.set(value, decodedValue)
        }
    }
}

Example model object

That's all the reusable code for today. Let's see what a model looks like.

Imagine a todo-app. We could model a simple todo item like this:

struct Todo {
    var uuid: String
    var title: String
    var done: Bool
    
    init(uuid: String = NSUUID().UUIDString, title: String = "", done: Bool = false) {
        self.uuid = uuid
        self.title = title
        self.done = done
    }
}

We need to be able to encode it and put it in the database (the usual ValueCoding stuff):

extension Todo: Persistable {
    static let collection: String = "Todos"
    
    var identifier: Identifier {
        return self.uuid
    }
}

extension Todo: ValueCoding {
    typealias Coder = TodoCoder
}

And we need to make it conform to Syncable if we want to sync it:

extension Todo: Syncable {
    static let syncProperties = [
        "uuid" : uuidSyncLens,
        "title" : titleSyncLens,
        "done" : doneSyncLens
    ]
    
    private static let uuidSyncLens = SyncablePropertyLens<Todo>(get: { $0.uuid }, set: {
        guard let uuid = $1 as? String else { return $0 }
        return Todo(uuid: uuid, title: $0.title, done: $0.done)
    })

    private static let titleSyncLens = SyncablePropertyLens<Todo>(get: { $0.title }, set: {
        guard let title = $1 as? String else { return $0 }
        return Todo(uuid: $0.uuid, title: title, done: $0.done)
    })

    private static let doneSyncLens = SyncablePropertyLens<Todo>(get: { $0.done }, set: {
        guard let done = ($1 as? NSNumber)?.boolValue else { return $0 }
        return Todo(uuid: $0.uuid, title: $0.title, done: done)
    })
}

The lenses smell of boilerplate, but I think it's the best we can do with Swift at the moment.

Maybe you can write something like Lenso by Maciej Konieczny to generate them automatically.

But all the hard work is done. The coder is very simple and leverages our work with the lenses to implement NSCoding:

class TodoCoder: NSObject, NSCoding, MutableCodingType, HasPreviousCoder, SyncableObject {
    var value: Todo
    var previousCoder: AnyObject?
    
    required init(_ v: Todo) {
        self.value = v
    }

    required init(coder aDecoder: NSCoder) {
        self.value = Todo.decodeSyncProperties(aDecoder, into: Todo())
    }
    
    func encodeWithCoder(aCoder: NSCoder) {
        self.value.encodeSyncProperties(aCoder)
    }
}

You could also implement NSCoding field by field and ignore those Syncable-based helpers.

Using SyncableObject in CloudKit blocks

Where did we come from? Oh yes. The CloudKit recordHandler and mergeBlock!

Now you can do a cast like this from the anonymous object you get from the CloudKit extension, and get the same functionality that MyDatabaseObject used to give you in the CloudKitTodo example:

if let syncableObject = object as? SyncableObject {
	print("Changed sync keys:", syncableObject.changedSyncKeys)
	print("Previous sync values:", syncableObject.previousSyncValues)
}

And that's what we set out to accomplish.

Attachment

If you like this you can grab the SyncHelpers.swift file attached to this gist.

SyncHelpers.swift

import ValueCoding
import YapDatabaseExtensions
import YapDatabase.YapDatabaseHooks

// MARK: - Previous Coder

protocol HasPreviousCoder {
    var previousCoder: AnyObject? { get set }
}

extension CodingType where Self: HasPreviousCoder {
    var previousValue: Self.ValueType? {
        guard let previousCoder = self.previousCoder as? Self else { return nil }
        return previousCoder.value
    }
}

class PreviousCoderHook: YapDatabaseHooks {
    override init() {
        super.init()
        self.willModifyRow = { (transaction, collection, key, proxyObject, proxyMetadata, flags) in
            guard var hasPreviousCoder = proxyObject.realObject as? HasPreviousCoder else { return }
            let index = YapDB.Index(collection: collection, key: key)
            hasPreviousCoder.previousCoder = transaction.readAtIndex(index)
        }
    }
}

// MARK: - Mutable value in coder

protocol MutableCodingType: CodingType {
    var value: ValueType { get set }
}

// MARK: - Syncable value type

struct SyncablePropertyLens<T> {
    let get: T -> AnyObject
    let set: (T, AnyObject) -> T
}

protocol Syncable {
    // Getters and setters for each synced value. Stored by their sync key.
    static var syncProperties: [String:SyncablePropertyLens<Self>] { get }
}

// MARK: - Syncable coder object

protocol SyncableObject {
    var allSyncKeys: Set<String> { get }
    var changedSyncKeys: Set<String> { get }
    var previousSyncValues: [String:AnyObject] { get }
    func syncValueForKey(syncKey: String) -> AnyObject?
    mutating func setSyncValue(syncValue: AnyObject, forKey: String)
}

extension SyncableObject where Self: MutableCodingType, Self: HasPreviousCoder, Self.ValueType: Syncable {
    var allSyncKeys: Set<String> {
        return Set(ValueType.syncProperties.keys)
    }
    
    var changedSyncKeys: Set<String> {
        guard let previousValue = self.previousValue else { return allSyncKeys }
        let changedKeys = ValueType.syncProperties
            .filter { _, lens in !lens.get(self.value).isEqual(lens.get(previousValue)) }
            .map { key, _ in key }
        return Set(changedKeys)
    }
    
    var previousSyncValues: [String:AnyObject] {
        guard let previousValue = self.previousValue else { return [:] }
        return ValueType.syncProperties.reduce([:]) { (var dict, entry) in
            let (key, lens) = entry
            dict[key] = lens.get(previousValue)
            return dict
        }
    }
    
    func syncValueForKey(syncKey: String) -> AnyObject? {
        return ValueType.syncProperties[syncKey]?.get(self.value)
    }
    
    mutating func setSyncValue(syncValue: AnyObject, forKey syncKey: String) {
        guard let lens = ValueType.syncProperties[syncKey] else { return }
        self.value = lens.set(self.value, syncValue)
    }
}

// MARK: - NSCoding helpers

extension Syncable {
    func encodeSyncProperties(coder: NSCoder) {
        for (key, lens) in Self.syncProperties {
            coder.encodeObject(lens.get(self), forKey: key)
        }
    }
    
    static func decodeSyncProperties(decoder: NSCoder, into value: Self) -> Self {
        return Self.syncProperties.reduce(value) { value, entry in
            let (key, lens) = entry
            guard let decodedValue = decoder.decodeObjectForKey(key) else { return value }
            return lens.set(value, decodedValue)
        }
    }
}

I think this is a really well expressed solution to the problem!

In terms of viability, is it something which you think is worthwhile? I did something similar to this in a project last year, and I have considered providing helpers and/or general solutions in YapDatabaseExtensions for this. But, not sure about how framework consumers would feel about the boilerplate (I don't, but some people might consider it too much), and whether anyone else wanted to do this sort of stuff...

I think in general though your solution is really nice. I'm not too familiar with the lens stuff, so I'm gonna read up, and compare it to what I've done, and come back with more feedback. Just wanted to say now, really great - we should certainly discuss about putting it into YapDatabaseExtensions!

Thank you for the kind words!

It's a matter of preference for sure, but I think the boilerplate needed to conform to Syncable is worthwhile.

It gives you control over what should be synced, how it should be represented in the cloud and buys you some NSCoding helpers which can eliminate the (twice repeated) boilerplate of init(coder:) and encodeWithCoder:.

And, for me at least, it was very unclear how to bridge the gap from "simple Swift struct" to "scary, per-field diffing, 3-way merge helper with automatic knowledge of past events in the database". Writing the lenses is tedious work, but conceptually simple in comparison. Especially if an example app is provided.

I'm also wondering what else a Syncable type could help with. For starters, it could store the CloudKit collection name, similar to what Persistable does for the local database. Record zones are another thing to worry about.

With a protocol-oriented solution like this we could even dare to dream of a reusable replacement for the CloudKitManager class from the CloudKitTodo example. With a delegate protocol for app-specific error and event handling.

But I'm only starting to venture into that territory now. Replacing MyDatabaseObject while staying Swifty was the first bridge to cross.

If you think this is a worthwhile feature set to add to YapDatabaseExtensions I'm happy to put in some work to get it into a more shippable state. I should probably finish the CloudKit sync in my proof-of-concept app first, though. Just to dogfood this at least once. But let's discuss it further.