crazyguitar/redis.md

## redis.md

      
    Raw
  

              redis.md
            
          
    Redis Note

Features


An in-memory database (often shortened to IMDB) which stores data using the main memory of a machine as the primary storage facility, rather than storing the data on a disk as in a typical disk-optimized database.
Redis can store a mapping of keys to values (Key-Value Store) and can even achieve similar performance levels as memcached.
Redis supports the writing of its data to disk
Redis has two different forms of writing in-memory data to disk in a compact format:

point-in-time dump (snapshot): when certain conditions are met (a number of writes in a given period) or when one of the two dump-to-disk commands is called
append-only file: writes every command that alters data in Redis to disk as it happens


Transaction: MULTI, EXEC, DISCARD and WATCH are the foundation of transactions in Redis. They allow the execution of a group of commands in a single step. Guarantee that:

All the commands in a transaction are serialized and executed sequentially
Either all of the commands or none are processed, so a Redis transaction is also atomic


Some basic redis operations

Set, Get, Del

import base64
import uuid
import os

import redis

r = redis.StrictRedis(host='localhost', port=6379)

# Set
for i in range(10000):
    u = str(uuid.uuid4())
    v = base64.b64encode(os.urandom(64))
    r.set(u, v)
    
# Get
for k in r.keys():
    print(k, r.get(k))
    
# Delete
for k in r.keys():
    r.delete(k)
Set, Get List

>>> def fib(n:int):
...     a, b = 0, 1
...     for _ in range(n):
...         yield a
...         b, a = a + b, b
... 
>>> r.rpush("fib", *(f for f in fib(10)))
10
>>> r.lrange("fib", 0, -1)
['0', '1', '1', '2', '3', '5', '8', '13', '21', '34']
>>> r.delete("fib")
>>> r.lpush("fib", *(f for f in fib(10)))
10
>>> r.lrange("fib", 0, -1)
['34', '21', '13', '8', '5', '3', '2', '1', '1', '0']
Set, Get hash

# Set a hash with only one key

>>> rc.hset("redis_key", "key", "value")
1
>>> rc.hget("redis_key", "key")
'value'

# Set a hash with multiple keys

>>> data = {"name": "redis", "age": 10}
>>> r.hmset("data", data)
True
>>> r.hgetall("data")
{'name': 'redis', 'age': '10'}
Pipeline

Using pipeline to improve the performance
import base64
import uuid
import os

import redis

r = redis.StrictRedis(host='localhost', port=6379)

def profile(f):
    @wraps(f)
    def wrapper(*a, **k):
        s = time.time()
        r = f(*a, **k)
        e = time.time()
        print(f'{f.__name__} cost: {e - s} sec')
        return r
    return wrapper

@profile
def setkey(n, r):
    for i in range(n):
        u = str(uuid.uuid4())
        v = base64.b64encode(os.urandom(64))
        r.set(u, v)

@profile
def pipeline_setkey(n, r):
    with r.pipeline() as p:
        for i in range(n):
            u = str(uuid.uuid4())
            v = base64.b64encode(os.urandom(64))
            p.set(u, v)
        p.execute()

@profile
def getall(r):
    for k in r.keys():
        _ = r.get(k)

@profile
def pipeline_getall(r):
    with r.pipeline() as p:
        for k in r.keys():
            p.get(k)

        _ = p.execute()

setkey(10000, rc)
pipeline_setkey(10000, rc)

getall(rc)
pipeline_getall(rc)
output:
setkey cost: 5.000656604766846 sec
pipeline_setkey cost: 0.5845057964324951 sec
getall cost: 7.360884189605713 sec
pipeline_getall cost: 0.9100451469421387 sec

Publish / Subscribe

import base64
import time
import uuid
import os

import redis

from threading import Thread

rc = redis.StrictRedis(host='localhost', port=6379)

def cleankeys(r):
    with r.pipeline() as p:
        for k in r.keys():
            p.delete(k)

        p.execute()


def setkeys(n, r):
    with r.pipeline() as p:
        for i in range(n):
            u = str(uuid.uuid4())
            v = base64.b64encode(os.urandom(64))
            p.set(u, v)

        p.execute()


def publisher(n, r):
    for i in range(n):
        time.sleep(1)
        r.publish('channel', i)
        

cleankeys(rc)
setkeys(10000, rc)

p = rc.pubsub()
p.subscribe(['channel'])

t = Thread(target=publisher, args=(10, rc))
t.start()
count = 0

for i in p.listen():
    print(i)
    count += 1
    if count == 5:
        p.unsubscribe()
output:
{'type': 'subscribe', 'pattern': None, 'channel': 'channel', 'data': 1}
{'type': 'message', 'pattern': None, 'channel': 'channel', 'data': '0'}
{'type': 'message', 'pattern': None, 'channel': 'channel', 'data': '1'}
{'type': 'message', 'pattern': None, 'channel': 'channel', 'data': '2'}
{'type': 'message', 'pattern': None, 'channel': 'channel', 'data': '3'}
{'type': 'unsubscribe', 'pattern': None, 'channel': 'channel', 'data': 0}

Create a redis cluster

In this section, run the redis inside of the docker.
Step 1: Create a docker network for redis

$ docker network create redis
$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
ab6c0c2caa47        bridge              bridge              local
939d0128b956        host                host                local
c366a47b1338        none                null                local
d61c7773bf23        redis               bridge              local
Step 2: create a cluster-config.conf
port 6379
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

Step 3: run the redis containers

$ for i in `seq 1 6`; do docker run -d -v $PWD/cluster-config.conf:/usr/local/etc/redis/redis.conf \
--name "redis-$i" --net redis redis redis-server /usr/local/etc/redis/redis.conf; done
then check the cluster's information
$ docker exec redis-1 redis-cli cluster nodes
17096e2421db0d3e10c09e051e11b7e6a7998a93 172.18.0.3:6379 master - 0 1519778388949 2 connected 5461-10922
f79ee7069ac0d68b0bff7e98cf5a7f417bf01a3a 172.18.0.2:6379 myself,master - 0 0 1 connected 0-5460
ac9f1b0c84a023319a47d7069209b59d52b11ce2 172.18.0.7:6379 slave 17096e2421db0d3e10c09e051e11b7e6a7998a93 0 1519778389470 6 connected
b3e4908d7e7ad2a64bc127ce36a65ddc5485bc1f 172.18.0.6:6379 slave f79ee7069ac0d68b0bff7e98cf5a7f417bf01a3a 0 1519778390495 5 connected
750b459e1414c4782f01ae25385e71f6e66c8d8c 172.18.0.5:6379 slave 08e24b570db61846e9165578d598edd339df9273 0 1519778388433 4 connected
08e24b570db61846e9165578d598edd339df9273 172.18.0.4:6379 master - 0 1519778389980 3 connected 10923-16383
Step 4: get redis containers' ip address

$ for c in `seq 1 6`; do docker inspect -f '{{(index .NetworkSettings.Networks "redis").IPAddress}}' redis-$c; done
172.18.0.2
172.18.0.3
172.18.0.4
172.18.0.5
172.18.0.6
172.18.0.7
Step 5: create the redis cluster

docker run -i --rm --net redis ruby sh -c '\
 gem install redis \
 && wget http://download.redis.io/redis-stable/src/redis-trib.rb \
 && ruby redis-trib.rb create --replicas 1 \
 172.18.0.2:6379 \
 172.18.0.3:6379 \
 172.18.0.4:6379 \
 172.18.0.5:6379 \
 172.18.0.6:6379 \
 172.18.0.7:6379'
Step 6: test your cluster

$ docker run -it --rm --net redis python:3.7-rc /bin/bash
root@317b24bcdba9:/# pip install redis-py-cluster
root@317b24bcdba9:/# python3
Python 3.7.0b1 (default, Feb 15 2018, 20:57:01) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from rediscluster import StrictRedisCluster
>>> n1 = {"host": "redis-1", "port": "6379"}
>>> n2 = {"host": "redis-2", "port": "6379"}
>>> n3 = {"host": "redis-3", "port": "6379"}
>>> nodes = [n1, n2, n3]
>>> rc = StrictRedisCluster(startup_nodes=nodes, decode_responses=True)
>>> rc.set("foo", "bar")
True
>>> print(rc.get("foo"))
bar
Add a new redis node

First, run the redis server in cluster mode.
$ docker run -d -v $PWD/cluster-config.conf:/usr/local/etc/redis/redis.conf \
--name redis-7 --net redis redis redis-server /usr/local/etc/redis/redis.conf
then add node to cluster
docker run -i --rm --net redis ruby sh -c '\                  
 gem install redis \
 && wget http://download.redis.io/redis-stable/src/redis-trib.rb \
 && ruby redis-trib.rb add-node 172.18.0.8:6379 172.18.0.2:6379'
check the cluster info
docker exec redis-1 redis-cli cluster nodes
17096e2421db0d3e10c09e051e11b7e6a7998a93 172.18.0.3:6379 master - 0 1519779643650 2 connected 5461-10922
f79ee7069ac0d68b0bff7e98cf5a7f417bf01a3a 172.18.0.2:6379 myself,master - 0 0 1 connected 0-5460
ac9f1b0c84a023319a47d7069209b59d52b11ce2 172.18.0.7:6379 slave 17096e2421db0d3e10c09e051e11b7e6a7998a93 0 1519779644160 6 connected
b3e4908d7e7ad2a64bc127ce36a65ddc5485bc1f 172.18.0.6:6379 slave f79ee7069ac0d68b0bff7e98cf5a7f417bf01a3a 0 1519779643650 5 connected
889bce5cfd30205a7a9f8bc8a1c73f3060e41233 172.18.0.8:6379 master - 0 1519779643133 0 connected
750b459e1414c4782f01ae25385e71f6e66c8d8c 172.18.0.5:6379 slave 08e24b570db61846e9165578d598edd339df9273 0 1519779644160 4 connected
08e24b570db61846e9165578d598edd339df9273 172.18.0.4:6379 master - 0 1519779644672 3 connected 10923-16383

Delete a node

To remove a node, run the following commands:
# check cluster info
$ docker exec redis-1 redis-cli cluster nodes

# delete the node
$ node_id="ac9f1b0c84a023319a47d7069209b59d52b11ce2"
$ master="172.18.0.2:6379"
$ docker run -i --rm --net redis ruby sh -c "\
 gem install redis \
 && wget http://download.redis.io/redis-stable/src/redis-trib.rb \
 && ruby redis-trib.rb del-node $master $node_id"
Reference:


Redis cluster tutorial
Creating Redis Cluster using Docker
redis-py-cluster