Skip to content

Instantly share code, notes, and snippets.

@yhemanth

yhemanth/redis-kminhash-time-optimised-add.py Secret

Last active October 29, 2015 16:54
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save yhemanth/5a7c0856ef25dd3ff37d to your computer and use it in GitHub Desktop.
Save yhemanth/5a7c0856ef25dd3ff37d to your computer and use it in GitHub Desktop.
Download ZIP
Add IDs and hashes to a KMinHash key backed by a Redis sorted set using pipeline, truncating the set after every batch addition
Raw
redis-kminhash-time-optimised-add.py
def update_min_hashes_batch(self, ids_batch):
pipeline = self.redis_client.pipeline()
for element_id in ids_batch:
min_hash = self.__element_hash(element_id)
pipeline.zadd(self.key, min_hash, element_id)
pipeline.zremrangebyrank(self.key, self.k, -1)
pipeline.execute()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment