Navigation Menu

Skip to content

Instantly share code, notes, and snippets.

@anentropic
Last active May 28, 2021 10:39
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save anentropic/0f2d700b5abdc21177bb to your computer and use it in GitHub Desktop.
Save anentropic/0f2d700b5abdc21177bb to your computer and use it in GitHub Desktop.
class CustomQuerySetManager(Manager):
"""
A re-usable Manager to access a custom QuerySet
"""
def __getattr__(self, attr, *args):
try:
return getattr(self.__class__, attr, *args)
except AttributeError:
# don't delegate internal methods to queryset
# NOTE: without this, Manager._copy_to_model will end up calling
# __getstate__ on the *queryset* which causes the qs (as `all()`)
# to evaluate itself as if it was being pickled (`len(self)`)
if attr.startswith('__'):
raise
return getattr(self.get_query_set(), attr, *args)
def get_query_set(self):
return getattr(self.model, 'QuerySet', QuerySet)(self.model, using=self._db)
class CustomModel(models.Model):
objects = CustomQuerySetManager()
QuerySet = CustomQuerySet
value1 = models.IntegerField()
value2 = models.IntegerField()
from django.test import TestCase
class CustomQuerySetManagerTestCase(TestCase):
def test_get_queryset_no_queries(self):
with self.assertNumQueries(0):
CustomModel.objects.get_query_set()
def test_filter_unevaluated_no_queries(self):
with self.assertNumQueries(0):
CustomModel.objects.filter(value1=2)
def test_filter_evaluated_1_query(self):
with self.assertNumQueries(1):
qs = CustomModel.objects.filter(value1=2)
list(qs)
def test_only_unevaluated_no_queries(self):
with self.assertNumQueries(0):
CustomModel.objects.only('value1')
def test_only_evaluated_1_query(self):
with self.assertNumQueries(1):
qs = CustomModel.objects.only('value1')
list(qs)
"""
in the original version of CustomQuerySetManager from http://stackoverflow.com/a/2163921/202168
`test_only_evaluated_1_query` fails because 3 queries are executed instead of 1
If we inspect the two extra queries we see they look like you did
`list(CustomModel.objects.all())` ...twice!
If you have millions of rows in your db this is very bad, you try to do:
`CustomModel.objects.only('value1').filter(value2=99)`
and you end up triggering two unbounded whole-table fetches as well as your actual query!!
As mentioned in the comment above, the reason is because of some Django model internals.
When you add an `only()` clause to your queryset, Django derives a new model class from your
original model (like `CustomModel_deferrer_value1`), as a proxy model. This leads to
`Manager._copy_to_model()` method being called to copy your custom manager to the new proxy
class... this uses `copy.copy()` which internally calls the `__getstate__` method of object
in question, as when pickling.
Unfortunately the original version of CustomQuerySetManager was naïvely delegating
the `__getstate__` call through to the *queryset* instance... and, just like when you pickle
a queryset, this causes it to be evaluated.
"""
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment