Skip to content

Instantly share code, notes, and snippets.

@Hkazanci93
Hkazanci93 / gist:19ad4175b0c6f3b65a177d621c3a8ce9
Created October 19, 2021 10:20
Code to unzip and parse metadata.zip
redis_client.hset(f"article_id:{article_id}",mapping={'title': each_line['title']})
redis_client.hset(f"article_id:{article_id}",mapping={'publication_date':each_line['publish_time']})
@Hkazanci93
Hkazanci93 / gist:f19cf6067be140fe2fe5fed599e66e09
Created October 19, 2021 10:23
Pre-processing task that uses RedisGears
gb = GB('KeysReader')
gb.filter(filter_language)
gb.flatmap(parse_paragraphs)
gb.map(spellcheck_sentences)
gb.foreach(save_sentences)
gb.count()
gb.register('paragraphs:*',keyTypes=['string','hash'], mode="async_local")
@Hkazanci93
Hkazanci93 / gist:ee134f4da01d36ad3852b489b192b6a9
Created October 19, 2021 11:35
Parsing JSON into string
rediscluster_client.set(f”paragraphs:{article_id}“,” “.join(article_body))
bg = GearsBuilder('KeysReader')
bg.foreach(process_item)
bg.count()
bg.register('sentence:*', mode="async_local",onRegistered=OnRegisteredAutomata)
@Hkazanci93
Hkazanci93 / gist:9949d8692210d732004b2768ffb4d0de
Created October 19, 2021 11:37
Turning sentences into edges
bg = GearsBuilder('KeysReader')
bg.foreach(process_item)
bg.count()
bg.register('sentence:*', mode="async_local",onRegistered=OnRegisteredAutomata)
@Hkazanci93
Hkazanci93 / gist:fe073075e38d73d91476f22018ba5f15
Created October 19, 2021 11:38
Creating a stream on each shard
'XADD', 'edges_matched_{%s}' % shard_id, '*','source',f'{source_entity_id}','destination',f'{destination_entity_id}','source_name',source_canonical_name,'destination_name',destination_canonical_name,'rank',1,'year',year)
@Hkazanci93
Hkazanci93 / gist:5ab2b0c31c261b6d9c667220d41a4b2c
Created October 19, 2021 11:39
Increasing sentence score
zincrby(f'edges_scored:{source_entity_id}:{destination_entity_id}',1, sentence_key)
@Hkazanci93
Hkazanci93 / gist:f13db4b8f65f193707410392a109164b
Created October 19, 2021 11:40
Populating RedisGraph from RedisGears
"GRAPH.QUERY", "cord19medical","""MERGE (source: entity { id: '%s', label :'entity', name: '%s'})
ON CREATE SET source.rank=1
ON MATCH SET source.rank=(source.rank+1)
MERGE (destination: entity { id: '%s', label: 'entity', name: '%s' })
ON CREATE SET destination.rank=1
ON MATCH SET destination.rank=(destination.rank+1)
MERGE (source)-[r:related]->(destination)
ON CREATE SET r.rank=1, r.year=%s
ON MATCH SET r.rank=(r.rank+1)
ON CREATE SET r.rank=1
@Hkazanci93
Hkazanci93 / gist:9f0d76f77d7f4dc323a00b44d3cd64c1
Created October 19, 2021 11:43
Querying RedisGraph data during API calls
"WITH $ids as ids MATCH (e:entity)-[r]->(t:entity) where (e.id in ids) and (r.year in $years) RETURN DISTINCT e.id, t.id, max(r.rank), r.year ORDER BY r.rank DESC LIMIT $limits"""
"""WITH $ids as ids MATCH (e:entity) where (e.id in ids) RETURN DISTINCT e.id,e.name,max(e.rank)"""