ndb.Model - migrating properties like a boss

ndb-properties-migration.md

ndb.Model - migrating properties like a boss

The problem

Consider following model:

class X(ndb.Model):
    prop1 = ndb.StringProperty()
    prop2 = ndb.StringProperty()
    prop3 = ndb.StringProperty()

Some business rules changed, and we don’t want prop2 and prop3 in the model anymore. Instead, prop4 is introduced, which does the same as prop2 and prop3 together:

class X(ndb.Model):
    prop1 = ndb.StringProperty()
    prop4 = ndb.StringProperty()

But! old X entities in Datastore still have prop2 and prop3, and they don’t have prop4; we don’t want to run migration on them because of Reasons™. This is especially valid if X was denormalized somewhere as a LocalStructuredProperty.

Raw properties and their values

Let’s assume that x is an initialized X entity, freshly obtained from Datastore. Although trying to access x.prop will raise AttributeError, you can see that old values are still returned in x.to_dict():

>>> x.to_dict()
{'prop1': 'a', 'prop2': 'b', 'prop3': 'c', 'prop4': None}

Notice that prop4 is returned, albeit not being in the entity itself. Let's go deeper! All entities have some interesting private properties, like _properties and _values:

>>> x._properties
{'prop1': GenericProperty('prop1', repeated=0), 'prop2': GenericProperty('prop2', repeated=0),  'prop3': GenericProperty('prop3', repeated=0), 'prop4': StringProperty('prop4')}

Notice that prop4 is StringProperty, while prop1 and prop2 are GenericProperties.

>>> x._values
{'prop1': _BaseValue('a'), 'prop2': _BaseValue('b'), 'prop3': _BaseValue('b'), 'prop4': None}

The easiest way to access the value, I believe, is to call property's _get_value, which must be called with the object itself as a first parameter:

>>> x._properties.get('prop2')._get_value(x)
'b'
>>> type(x._properties.get('prop2')._get_value(x))
<type 'str'>

Migrating

After you put an object, old properties don't disappear! What is more, they show up in X.__repr__.

>>> x
X(prop1='a', prop2='b', prop3='c', prop4=None)
>>> x._properties
{'prop1': GenericProperty('prop1', repeated=0), 'prop2': GenericProperty('prop2', repeated=0),  'prop3': GenericProperty('prop3', repeated=0), 'prop4': StringProperty('prop4')}
>>> x.put()
>>> x = x.key.get()
>>> x
X(prop1='a', prop2='b', prop3='c', prop4=None)
>>> x._properties
{'prop1': GenericProperty('prop1', repeated=0), 'prop2': GenericProperty('prop2', repeated=0), 'prop3': GenericProperty('prop3', repeated=0), 'prop4': StringProperty('prop4')}

In order to force them to go away, you need to explicitly remove them from object's _properties dict.

>>> x._properties.pop('prop2', None)
GenericProperty('prop2', repeated=0)
>>> x.put()
>>> x = x.key.get()
>>> x
X(prop1='a', prop3='c', prop4=None)
>>> x._properties
{'prop1': GenericProperty('prop1', repeated=0), 'prop3': GenericProperty('prop3', repeated=0), 'prop4': StringProperty('prop4')}

Sample code

Assume that you have both old and new entities of X in your Datastore, and you need to do something with either prop4, or prop2 and prop3 together. You need to detect which object you're dealing with - prop4 would be None.

x = X.get_by_id(123)
if x.prop4:  # new object
    do_something(x.prop4)
else:  # old object
    prop2 = x._properties.get('prop2')._get_value(x)
    prop3 = x._properties.get('prop3')._get_value(x)
    prop4 = prop2 + prop3
    do_something(prop4)

Testing

You can dynamically emulate old objects by modifying model on the fly (remember to replace it back after test finishes!)

def test_x(self):
    # Migrate X back to what it used to be
    X.prop2 = ndb.StringProperty()
    X.prop3 = ndb.StringProperty()
    prop4 = X._properties['prop4']
    del X.prop4
    X._fix_up_properties()
    x = X(prop1='a', prop2='b', prop3='c')
    x.put()
    x = x.key.get()
    assert x.prop1 == 'a'
    assert x.prop2 == 'b'
    assert x.prop3 == 'c'
    assert not hasattr(x, 'prop4')
    # Now switch back!
    x_key = x.key
    del x
    del X.prop2
    del X.prop3
    X.prop4 = prop4
    X._fix_up_properties()
    x = x_key.get()