Create a gist now

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Testing storage of millions of keys in Redis
#! /usr/bin/env python
import redis
import random
import pylibmc
import sys
r = redis.Redis(host = 'localhost', port = 6389)
mc = pylibmc.Client(['localhost:11222'])
REDIS_SETGET = False
REDIS_HSET = False
MC = False
NUM_ENTRIES = 1000000
MAX_VAL = 12000000
if len(sys.argv) != 2 or sys.argv[1] not in ('redis-normal', 'redis-hashes', 'memcached'):
print 'Specify a test: redis-normal, redis-hashes, memcached'
print 'NOTE: clear out memcached (restart) or Redis (FLUSHALL) before running'
sys.exit(2)
if sys.argv[1] == 'redis-normal':
REDIS_SETGET = True
elif sys.argv[1] == 'redis-hashes':
REDIS_HSET = True
elif sys.argv[1] == 'memcached':
MC = True
p = r.pipeline()
for i in range(0, NUM_ENTRIES):
value = random.randint(0, MAX_VAL)
if MC:
mc.set(str(i), value)
elif REDIS_SETGET:
r.set(str(i), value)
elif REDIS_HSET:
bucket = int(i / 500)
p.hset(bucket, i, value)
if i % (NUM_ENTRIES/10) == 0:
if REDIS_SETGET or REDIS_HSET:
p.execute()
p = r.pipeline()
print i
# one final clear out
if REDIS_SETGET or REDIS_HSET:
p.execute()
# get size
if MC:
size = int(mc.get_stats()[0][1]['bytes'])
elif (REDIS_SETGET or REDIS_HSET):
size = int(r.info()['used_memory'])
print '%s bytes, %s MB' % (size, size / 1024 / 1024)
@tanoc

This comment has been minimized.

Show comment
Hide comment
@tanoc

tanoc Nov 1, 2011

I think you can save at least another 20% just by using the remainder instead of i as the key in the hash

tanoc commented Nov 1, 2011

I think you can save at least another 20% just by using the remainder instead of i as the key in the hash

@lushchick

This comment has been minimized.

Show comment
Hide comment
@lushchick

lushchick Nov 1, 2011

Took 49 MB with Memcached and only 15 MB with redis hashes on my debian box with 384 MB of RAM.

Took 49 MB with Memcached and only 15 MB with redis hashes on my debian box with 384 MB of RAM.

@ctoestreich

This comment has been minimized.

Show comment
Hide comment
@ctoestreich

ctoestreich Nov 3, 2011

I hacked together a quick Groovy script and was able to save another 20% space by doing a custom "map reduce" on the numerical keys you are using. It went from 18mb -> 14mb for me. You can find the Groovy Gist here https://gist.github.com/1337648

I hacked together a quick Groovy script and was able to save another 20% space by doing a custom "map reduce" on the numerical keys you are using. It went from 18mb -> 14mb for me. You can find the Groovy Gist here https://gist.github.com/1337648

@pcdinh

This comment has been minimized.

Show comment
Hide comment
@aman2626

This comment has been minimized.

Show comment
Hide comment
@aman2626

aman2626 May 18, 2016

i have used the above code on window 7 ultimate 32 bit system. i tested the code with redis-normal and redis-hashes keys. when i used redis normal it took 54 MB while redis-hashes took 69 MB. i store hash key like HSET media:1 1 7742896.https://gist.github.com/aman2626/d9170dea4c2716a2c7a07c3db9e67285

python version:3.3
redis-py version:redis-2.10.5-py3.3

i have used the above code on window 7 ultimate 32 bit system. i tested the code with redis-normal and redis-hashes keys. when i used redis normal it took 54 MB while redis-hashes took 69 MB. i store hash key like HSET media:1 1 7742896.https://gist.github.com/aman2626/d9170dea4c2716a2c7a07c3db9e67285

python version:3.3
redis-py version:redis-2.10.5-py3.3

@aman2626

This comment has been minimized.

Show comment
Hide comment
@aman2626

aman2626 May 23, 2016

i think 69 MB is the total amount of space taken by redis (54 MB with normal redis + 15 MB with hash key). am i correct?

i think 69 MB is the total amount of space taken by redis (54 MB with normal redis + 15 MB with hash key). am i correct?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment