Skip to content

Instantly share code, notes, and snippets.

@shacker
Last active June 5, 2020 20:58
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save shacker/f8cc14952faae3488d23b9f10c0b6e2f to your computer and use it in GitHub Desktop.
Save shacker/f8cc14952faae3488d23b9f10c0b6e2f to your computer and use it in GitHub Desktop.
# General pattern for a data "chunking" process to prevent a server from hitting memory limits
# when calling .update() or .delete() on large amounts of data. In this example, we enter an
# infinite loop, then keep deleting 1000 records at a time until the records are exhausted,
# then exit the loop. You can't call .update() or .delete() after taking a slice, hence the need
# for two queries rather than one.
t = Territory.objects.get(name="some-territory")
while True:
loc_ids = Location.objects.filter(territory=t)[:1000].values_list("id", flat=True)
locs = Location.objects.filter(id__in=loc_ids)
if locs:
print(f"Deleting 1000 locations starting with {loc_ids[0]}...")
locs.delete()
else:
print("Done")
break
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment