Skip to content

Instantly share code, notes, and snippets.

@warlo
Created August 21, 2018 08:26
Show Gist options
  • Save warlo/5b46cdf01643bead2a97dada92b03b41 to your computer and use it in GitHub Desktop.
Save warlo/5b46cdf01643bead2a97dada92b03b41 to your computer and use it in GitHub Desktop.
Batch querysets without slicing them for .update() etc.
def queryset_in_batches_non_slicing(queryset, chunk_size=1000):
"""
Iterate over a Django queryset that is ordered by primary key.
Differs from `queryset_batch_iterator` by not slicing the queryset and filters naively on
upper and lower bounds using pk and chunk size. This allows queryset operations to be performed
such as `.update()` and `.delete()`
"""
pk = 0
queryset = queryset.order_by('pk')
last_element = queryset.last()
# Empty queryset
if last_element is None:
return
while pk < last_element.pk:
prev_pk = pk
pk = pk + chunk_size
queryset_to_yield = queryset.filter(pk__gt=prev_pk, pk__lte=pk)
yield queryset_to_yield
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment