Skip to content

Instantly share code, notes, and snippets.

@hanneshapke hanneshapke/redis.py
Created May 24, 2018

Embed
What would you like to do?
Load word vectors into a redis db
import bz2
import pickle
from django.conf import settings
from djang_redis import get_redis_connection
from tqdm import tqdm
from .constants import GOOGLE_WORD2VEC_MODEL_NAME
def load_word2vec_into_redis(rs, wvmodel, key=GOOGLE_WORD2VEC_MODEL_NAME):
""" This function loops over all available words in the loaded word2vec model and loads
them into the redis instance via the rs object.
:param rs: redis connection object from django_redis
:param wvmodel: word vector model loaded into the memory of this machine.
Once the loading is completed, the memory will be available again.
:param key: suffix for the redis keys
"""
print("Update Word2Vec model in redis ...")
for word in tqdm(list(wvmodel.vocab.keys())):
rs.set(key + word, bz2.compress(pickle.dumps(wvmodel[word])))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.