Skip to content

Instantly share code, notes, and snippets.

View AlexMikhalev's full-sized avatar

Dr Alexander Mikhalev AlexMikhalev

50 followers · 249 following
View GitHub Profile
@AlexMikhalev
AlexMikhalev / redis.conf.j2
Last active May 13, 2020 10:29
Template for docker based ansible deployment
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
dir /data
bind {{ ansible_ztnfallfvg.ipv4.address }}
loadmodule /usr/lib/redis/modules/redisearch.so
loadmodule /usr/lib/redis/modules/redisgraph.so
loadmodule /usr/lib/redis/modules/redisbloom.so
loadmodule /var/opt/redislabs/lib/modules/redisgears.so
@AlexMikhalev
AlexMikhalev / lang_detect_gears.py
Last active May 16, 2020 07:35
Simple language detection script for RedisGears
from langdetect import detect
def detect_language(x):
#detect language of the article
if x['value']!='':
lang=detect(x['value'])
else:
lang="empty"
execute('SET', 'lang_article:' + x['key'], lang)
@AlexMikhalev
AlexMikhalev / requirements_gears_spacy.txt
Created May 19, 2020 07:27
Spacy and dependencies
langdetect==1.0.8
spacy
https://github.com/explosion/spacy-models/releases/download/en_core_web_md-2.2.5/en_core_web_md-2.2.5.tar.gz
@AlexMikhalev
AlexMikhalev / spacy_sentences_geared.py
Created May 19, 2020 08:06
Processing paragraphs to sentence using spacy on RedisGear
import spacy
nlp=spacy.load('en_core_web_md', disable=['ner','tagger'])
nlp.max_length=2000000
def remove_prefix(text, prefix):
return text[text.startswith(prefix) and len(prefix):]
def parse_paragraphs(x):
key_prefix="paragraphs:"
#make sure we only process english article
@AlexMikhalev
AlexMikhalev / dump.txt
Created May 22, 2020 14:01
RedisGearsSegfault
161783:S 22 May 2020 15:59:26.305 # === ASSERTION FAILED ===
161783:S 22 May 2020 15:59:26.305 # ==> src/cluster.c:215 'reply->type == REDIS_REPLY_STATUS' is not true
161783:S 22 May 2020 15:59:26.305 # (forcing SIGSEGV to print the bug report.)
161783:S 22 May 2020 15:59:26.305 # Redis 6.0.1 crashed by signal: 11
161783:S 22 May 2020 15:59:26.305 # Crashed running the instruction at: 0x55dbdad6b99f
161783:S 22 May 2020 15:59:26.305 # Accessing address: 0xffffffffffffffff
161783:S 22 May 2020 15:59:26.305 # Failed assertion: reply->type == REDIS_REPLY_STATUS (src/cluster.c:215)
------ STACK TRACE ------
EIP:
@AlexMikhalev
AlexMikhalev / sphinxpdf-latex
Created June 5, 2020 10:03
Generate pdf from docs
latexpdf:
@$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(PDFBUILDDIR)/latex
@echo "Running LaTeX files through pdflatex..."
make -C $(PDFBUILDDIR)/latex all-pdf
cp $(PDFBUILDDIR)/latex/*.pdf $(PDF)
@echo "pdflatex finished; see $(PDF)"
@AlexMikhalev
AlexMikhalev / couchimport-jq-json-zip.sh
Created June 14, 2020 18:26 — forked from assafmo/couchimport-jq-json-zip.sh
Import an archive full of JSONs into CouchDB
#!/bin/bash
#json per line (same result, but for example can use jq to set _id)
7z x -so $1 | jq -c . | couchimport --db mydb --type jsonl
@AlexMikhalev
AlexMikhalev / QA SearchAPIexample.json
Last active July 13, 2020 16:29
curl -H "Content-Type: application/json" -X POST -d '{"search":"Effectiveness of community contact reduction"}' http://10.144.17.211:8181/qasearch | jq .results
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2979 100 2922 100 57 200 3 0:00:19 0:00:14 0:00:05 667
[
{
"answer": "effectiveness of community contact reduction [SEP]",
"sentence": "In patients with suspected COVID-19 infection, it is recommended that PPE be donned by all clinicians and health care providers.",
"sentencekey": "sentences:0014b643b47d9b881e401ad5c4995622e0a39433:31:{1x3}",
"title": "Heart Rhythm College of Cardiology; and the Electrocardiography and Arrhythmias Committee of the Council on Clinical Cardiology"
@AlexMikhalev
AlexMikhalev / Dockerfile
Created September 10, 2020 15:51
Docker to build tauri app
FROM ubuntu:19.10
ENV DEBIAN_FRONTEND=noninteractive
ENV TZ=Europe/Berlin
# Dev tools
RUN apt-get update && apt-get install -y libwebkit2gtk-4.0-dev build-essential curl libssl-dev gnupg
# Node
RUN curl -sL https://deb.nodesource.com/setup_12.x | bash -
@AlexMikhalev
AlexMikhalev / bash.sh
Created September 21, 2020 21:27
Install Cuda and pytorch from pip
```
pip install --no-cache-dir torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html
```