-
-
Save mikeyk/1329319 to your computer and use it in GitHub Desktop.
#! /usr/bin/env python | |
import redis | |
import random | |
import pylibmc | |
import sys | |
r = redis.Redis(host = 'localhost', port = 6389) | |
mc = pylibmc.Client(['localhost:11222']) | |
REDIS_SETGET = False | |
REDIS_HSET = False | |
MC = False | |
NUM_ENTRIES = 1000000 | |
MAX_VAL = 12000000 | |
if len(sys.argv) != 2 or sys.argv[1] not in ('redis-normal', 'redis-hashes', 'memcached'): | |
print 'Specify a test: redis-normal, redis-hashes, memcached' | |
print 'NOTE: clear out memcached (restart) or Redis (FLUSHALL) before running' | |
sys.exit(2) | |
if sys.argv[1] == 'redis-normal': | |
REDIS_SETGET = True | |
elif sys.argv[1] == 'redis-hashes': | |
REDIS_HSET = True | |
elif sys.argv[1] == 'memcached': | |
MC = True | |
p = r.pipeline() | |
for i in range(0, NUM_ENTRIES): | |
value = random.randint(0, MAX_VAL) | |
if MC: | |
mc.set(str(i), value) | |
elif REDIS_SETGET: | |
r.set(str(i), value) | |
elif REDIS_HSET: | |
bucket = int(i / 500) | |
p.hset(bucket, i, value) | |
if i % (NUM_ENTRIES/10) == 0: | |
if REDIS_SETGET or REDIS_HSET: | |
p.execute() | |
p = r.pipeline() | |
print i | |
# one final clear out | |
if REDIS_SETGET or REDIS_HSET: | |
p.execute() | |
# get size | |
if MC: | |
size = int(mc.get_stats()[0][1]['bytes']) | |
elif (REDIS_SETGET or REDIS_HSET): | |
size = int(r.info()['used_memory']) | |
print '%s bytes, %s MB' % (size, size / 1024 / 1024) |
Took 49 MB with Memcached and only 15 MB with redis hashes on my debian box with 384 MB of RAM.
I hacked together a quick Groovy script and was able to save another 20% space by doing a custom "map reduce" on the numerical keys you are using. It went from 18mb -> 14mb for me. You can find the Groovy Gist here https://gist.github.com/1337648
Oh, range() is considered harmful. http://rhodesmill.org/brandon/2012/counting-without-counting/
i have used the above code on window 7 ultimate 32 bit system. i tested the code with redis-normal and redis-hashes keys. when i used redis normal it took 54 MB while redis-hashes took 69 MB. i store hash key like HSET media:1 1 7742896.https://gist.github.com/aman2626/d9170dea4c2716a2c7a07c3db9e67285
python version:3.3
redis-py version:redis-2.10.5-py3.3
i think 69 MB is the total amount of space taken by redis (54 MB with normal redis + 15 MB with hash key). am i correct?
I think you can save at least another 20% just by using the remainder instead of
i
as the key in the hash