Benchmark file read from disk, CSV parsing, write and read operations to Redis or Memcached

README.md

Test scenario

Benchmark file read from disk, CSV parsing, write and read operations to Redis or Memcached

Environment info:

$ python --version
Python 2.7.10
$ pip freeze |grep -E 'redis|memcached'
python-memcached==1.59
redis==2.10.6

Test data:

$ ll big.csv
-rw-r--r--  1 ivan.styazhkin  staff   500M Aug 19 12:33 big.csv
$ head -n 1 big.csv
Brent,60,20170813,230000,52.0200000,52.0500000,51.9700000,52.0500000,203

Redis

$ docker run --name test-redis -p 6379:6379 -d redis
$ python test_read.py redis big.csv
file_read 0.402418136597
csv_read 12.0058579445
test_write 4.8344669342
test_read 10.0016810894

Memcached

$ docker run --name test-memcached -p 11211:11211 -d memcached
$ python test_read.py memcached big.csv
file_read 0.386235952377
csv_read 12.3348860741
test_write 0.480508089066
test_read 5.19752502441e-05

test_read.py

import csv
import sys
import time

import redis
import memcache


def timer(fn):
    def wrapper(*args, **kwargs):
        start = time.time()
        result = fn(*args, **kwargs)
        print(fn.__name__, time.time() - start)
        return result

    return wrapper


@timer
def file_read(filename):
    with open(filename) as f:
        return f.read()


@timer
def csv_read(filename):
    with open(filename) as f:
        reader = csv.reader(f)
        for row in reader:
            pass  # Do nothing. Just iterate over csv file


@timer
def test_write(client, key, data):
    return client.set(key, data)


@timer
def test_read(client, key):
    return client.get(key)


if __name__ == '__main__':
    if len(sys.argv) < 3:
        exit(1)

    filename = sys.argv[2]
    text = file_read(filename)

    csv_rows = csv_read(filename)

    name = sys.argv[1]
    if name == 'redis':
        client = redis.Redis()
        test_write(client, filename, text)
        test_read(client, filename)

    elif name == 'memcached':
        client = memcache.Client('localhost')
        test_write(client, filename, text)
        test_read(client, filename)