Created
August 21, 2018 08:26
-
-
Save warlo/5b46cdf01643bead2a97dada92b03b41 to your computer and use it in GitHub Desktop.
Batch querysets without slicing them for .update() etc.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def queryset_in_batches_non_slicing(queryset, chunk_size=1000): | |
""" | |
Iterate over a Django queryset that is ordered by primary key. | |
Differs from `queryset_batch_iterator` by not slicing the queryset and filters naively on | |
upper and lower bounds using pk and chunk size. This allows queryset operations to be performed | |
such as `.update()` and `.delete()` | |
""" | |
pk = 0 | |
queryset = queryset.order_by('pk') | |
last_element = queryset.last() | |
# Empty queryset | |
if last_element is None: | |
return | |
while pk < last_element.pk: | |
prev_pk = pk | |
pk = pk + chunk_size | |
queryset_to_yield = queryset.filter(pk__gt=prev_pk, pk__lte=pk) | |
yield queryset_to_yield |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment