Skip to content

Instantly share code, notes, and snippets.

@dingmaotu
Created August 9, 2019 14:51
Show Gist options
  • Save dingmaotu/b465509f5c5d54dceacf5a2eb985c739 to your computer and use it in GitHub Desktop.
Save dingmaotu/b465509f5c5d54dceacf5a2eb985c739 to your computer and use it in GitHub Desktop.
fast way to remove large number of redis keys by pattern
# to remove all keys matching a pattern in redis
# we could use the recommended way: redis-cli --scan --pattern 'abc:*' | xargs redis-cli del
# but this can be very slow if you have lots of data (like 8G redis cluster)
# we can use the following script to remove keys (considerably faster)
import time
import logging
from rediscluster import StrictRedisCluster
logger = logging.getLogger(__name__)
client = StrictRedisCluster(startup_nodes=hosts, password=password,
skip_full_coverage_check=True)
pattern = "abc:*"
start_time = time.time()
item_count = 0
batch_size = 100000
keys = []
logger.info("Start scanning keys...")
for k in client.scan_iter(pattern, count=batch_size):
keys.append(k)
if len(keys) >= batch_size:
item_count += len(keys)
logger.info("batch delete to {} ...".format(item_count))
client.delete(*keys)
keys = []
if len(keys) > 0:
item_count += len(keys)
logger.info("batch delete to {}".format(item_count))
client.delete(*keys)
end_time = time.time()
logger.info("deleted {0} keys in {1:0.3f} ms.".format(item_count, (end_time - start_time) / 1000.0))
@andsens
Copy link

andsens commented May 28, 2021

Or you could just change xargs to xargs -n500 -P10, which deletes 500 keys per invocation and runs 10 clients in parallel.

@jdhao
Copy link

jdhao commented Jul 20, 2021

Thanks for this gitst. This is much faster than not using count=batch_size. Saves me a lot of the time.

@DaveLanday
Copy link

@andsens I don't think this works with Cluster Mode enabled right? I was getting CROSSSLOT errors

@andsens
Copy link

andsens commented Jun 8, 2022

@DaveLanday hm, no that would fail. A multikey (500 keys in this case) operation, if I understand cluster mode correctly (never used it), has to operate on the same hashing slot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment