Simulated ForeignKeys from MongoEngine Document to Django Models

gistfile1.py

# Using the `@property` decorator allows the MongoEngine Document object to have
# properties that reference Models, and be got/set like normal foreign keys.
# This same pattern can work the other way and allow Models interface with
# Documents.

class Foo(mongoengine.Document):
    # The `id` of the property is stored in the Document.
    user_id         = mongoengine.IntField()

    # Setters and getters to interface with the SQL DB and set Users to the
    # user property directly.
    @property
    def user(self):
        # Lazily dereference the Model, and cache the object in the `_data`
        # property to avoid repeated fetching.
        if 'user' not in self._data:
            self._data['user'] = User.objects.get(pk=self.user_id)
        return self._data['user']
    
    @user.setter
    def user(self, value):
        # Cache the value in the `_data` attribute and set the corresponding
        # `id` property.
        if value and hasattr(value, 'pk'):
            self._data['user'] = value
            self.user_id = value.pk
        else:
            self._data['user'] = None
            self.user_id = None
    
    @user.deleter
    def user(self):
        self._data['user'] = None
        self.user_id = None

Is the user attribute query-able e.g. via, Foo.objects(user='bar')?

No, since it only exists within the app layer; it's not part of the actual data in the database. You can query against user_id. It's probably possible to add to the objects manager so it queries for the user, gets their ID, and queries for the document that uses that user_id. That's pretty inelegant, though.

It would be nice to be able to decorate methods with @Property and then query over them. Particularly in the case of dynamic data that could be treated as a field.

That sort of thing is a bit out of the realm of the ORM/ODM, since it's not dealing with the actual stored data. The best option is probably to have a dictionary that maps each object to the computed value of the field, effectively indexing the objects by that dynamic data.

That sounds intriguing. How does mongoengine actually keep track of what's a field and what isn't? Would it be possible to overload that process and inject dynamic values in a given field?

The fields are all properties that are derivatives of BaseField. Custom fields can be defined and made to return things dynamically. But, that dynamic information wouldn't be queryable since it's not stored in Mongo. Mongoengine only constructs the corresponding query in Mongo's query language and handles the interaction with the database; it doesn't execute the queries itself or otherwise manipulate the data.

Right so I think what would need to happen is the property decorator would need to set the attribute as mongoengine is setting the fields, i.e. at the database level. It seems like it should be possible to hook into this process.

The non-field attributes don't exist at the database level, though. In Mongo, the above document looks something like:

{
    "_id": ObjectId("<some id>"),
    "_types": ["Foo"],
    "_cls": "Foo",
    "user_id": 1
}

Mongoengine operates at the application level, mapping that document to a Foo object, and vice versa, using the _types and _cls values to know what object class to use. The user property is merely a convenience in the application level that avoids having to duplicate the User.objects.get(pk=obj.user_id) lookup all over the place.

Sure but what I'm getting at is using a custom descriptor to actually set the fields dynamically. That would be a huge boon to situations where you might want to do something complex and dynamic that should be query-able. It's unfortunate that mongoengine doesn't already provide a facility like this but I imagine that it could be done by hooking into the mechanics of setting a field.

You can. Just make your own derivative of BaseField and define the necessary to_mongo and to_python methods that do the translation, as specific to your data model. This simulated foreign key thing could be done as a custom field, but it was simpler to use the @property decorator. (I've actually since moved away from using MongoEngine entirely — just using PyMongo directly, with DictShield for validation help.)

Yeah I don't blame you: I'm using PyMongo directly for some things (particularly where speed is a concern). Anyway I think it's even simpler than that, you can basically just write a custom property decorator that sets the attribute on the given model in the scope of the fget method. So long as this is executed at some point, because it's using the customized mongoengine descriptor logic, it will actually be able to query by it. One issue with this approach is that it doesn't work until it's been initialized via getattr for example. Anyway thanks for indulging me.