Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save norbertparti/0cb4659f1fa08e21618bfa1e777fdfa2 to your computer and use it in GitHub Desktop.
Save norbertparti/0cb4659f1fa08e21618bfa1e777fdfa2 to your computer and use it in GitHub Desktop.
Multiprocessing parallel push to redis queue python
import time
from random import random
import json
from multiprocessing import Pool
import redis
r = redis.StrictRedis(host='localhost', port=6379, db=0)
queue_key = 'queue'
parallel = 30
seq = 2500
def push(val):
for i in range(seq):
_json = json.dumps({'key': random(), 'val': i * val})
r.rpush('queue', _json) # loop here for multiple request sequentially
pool = Pool(10)
start = time.time()
args = (i for i in range(parallel))
pool.map(push, args)
print(f'{parallel * seq} request time {time.time() - start}')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment