public
Last active

Testing storage of millions of keys in Redis

  • Download Gist
gistfile1.py
Python
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56
#! /usr/bin/env python
 
import redis
import random
import pylibmc
import sys
 
r = redis.Redis(host = 'localhost', port = 6389)
mc = pylibmc.Client(['localhost:11222'])
 
REDIS_SETGET = False
REDIS_HSET = False
MC = False
 
NUM_ENTRIES = 1000000
MAX_VAL = 12000000
 
if len(sys.argv) != 2 or sys.argv[1] not in ('redis-normal', 'redis-hashes', 'memcached'):
print 'Specify a test: redis-normal, redis-hashes, memcached'
print 'NOTE: clear out memcached (restart) or Redis (FLUSHALL) before running'
sys.exit(2)
if sys.argv[1] == 'redis-normal':
REDIS_SETGET = True
elif sys.argv[1] == 'redis-hashes':
REDIS_HSET = True
elif sys.argv[1] == 'memcached':
MC = True
 
p = r.pipeline()
for i in range(0, NUM_ENTRIES):
value = random.randint(0, MAX_VAL)
if MC:
mc.set(str(i), value)
elif REDIS_SETGET:
r.set(str(i), value)
elif REDIS_HSET:
bucket = int(i / 500)
p.hset(bucket, i, value)
 
if i % (NUM_ENTRIES/10) == 0:
if REDIS_SETGET or REDIS_HSET:
p.execute()
p = r.pipeline()
print i
 
# one final clear out
if REDIS_SETGET or REDIS_HSET:
p.execute()
 
# get size
if MC:
size = int(mc.get_stats()[0][1]['bytes'])
elif (REDIS_SETGET or REDIS_HSET):
size = int(r.info()['used_memory'])
 
print '%s bytes, %s MB' % (size, size / 1024 / 1024)

I think you can save at least another 20% just by using the remainder instead of i as the key in the hash

Took 49 MB with Memcached and only 15 MB with redis hashes on my debian box with 384 MB of RAM.

I hacked together a quick Groovy script and was able to save another 20% space by doing a custom "map reduce" on the numerical keys you are using. It went from 18mb -> 14mb for me. You can find the Groovy Gist here https://gist.github.com/1337648

Please sign in to comment on this gist.

Something went wrong with that request. Please try again.