benchmark-redis-store-large-value

benchmark-redis-store-large-value

Since redis uses single thread to process client requests one by one, storing large value for a key blocks others.
I tested how long this takes in different size value.

1.Test environment
  4core 2gb
  ubuntu server 14.04
  redis-version: 3.2.8
  
2.Procedure
  Disable persistence.
  Prepare buffers to be sent
  Send request. Measure elapsed time and memory usage.
  
3.Script

def benchmark_large_value():
    # Prepare values
    lengths = [1, 4, 32, 64, 128, 256, 512]  # M
    values = []
    with open("/dev/urandom", "rb") as f:
        for length in lengths:
            value = f.read(length * 1024 * 1024)
            values.append(value)

    r = redis.StrictRedis()
    # Send request and measure elapsed time
    for value, length in zip(values, lengths):
        key = "key-%d" % length
        start = time.time()
        r.set(key, value)
        end = time.time()
        print 'length: %3dM, spend: %.4fs, memory %.1f%%, swap %.1f%%' % \
              (length, end-start, psutil.virtual_memory().percent, psutil.swap_memory().percent)
              
4.Result

length:   1M, spend: 0.0084s, memory 59.0%, swap 0.0%
length:   4M, spend: 0.0117s, memory 59.6%, swap 0.0%
length:  32M, spend: 0.0626s, memory 62.6%, swap 0.0%
length:  64M, spend: 0.1335s, memory 67.6%, swap 0.0%
length: 128M, spend: 0.2659s, memory 77.3%, swap 0.0%
length: 256M, spend: 0.5842s, memory 96.6%, swap 0.0%
length: 512M, spend: 5.3249s, memory 74.0%, swap 8.8%

5.Conclusion

    This test executes on a 2GB memory machine. After preparing buffers in script, memory usage has grown up to 59%.
    When setting a 512M value, machine runs out of memory and instead uses swap partition. The corresponding elapsed
    time, 5s, is nearly unacceptable for server.
        
    Before 512M, elapsed time increases linearly with the size of value. A 64M value will block others about 100ms on
    this machine.