Skip to content

Instantly share code, notes, and snippets.

@Skyross
Last active September 28, 2017 21:53
Show Gist options
  • Save Skyross/f5715b2720e6e12a1aeb0976a5372303 to your computer and use it in GitHub Desktop.
Save Skyross/f5715b2720e6e12a1aeb0976a5372303 to your computer and use it in GitHub Desktop.
Django fetching by chunks
from django.db.models import Count, Min, Max
from django.db.models.functions import Coalesce
def chunk_fetch(queryset, chunk_size=100):
"""
:type chunk_size: int
:type queryset: django.db.models.QuerySet
"""
aggregation = queryset.aggregate(
min_id=Coalesce(Min('pk'), 0),
max_id=Coalesce(Max('pk'), -1)
)
min_id, max_id = aggregation['min_id'], aggregation['max_id']
while min_id <= max_id:
queryset_slice = list(queryset.filter(
pk__gte=min_id,
pk__lt=min_id + chunk_size
))
for record in queryset_slice:
yield record
min_id += chunk_size
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment