Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
Add IDs and hashes to a KMinHash key backed by a Redis sorted set using pipeline, truncating the set after every batch addition
def update_min_hashes_batch(self, ids_batch):
pipeline = self.redis_client.pipeline()
for element_id in ids_batch:
min_hash = self.__element_hash(element_id)
pipeline.zadd(self.key, min_hash, element_id)
pipeline.zremrangebyrank(self.key, self.k, -1)
pipeline.execute()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment