Skip to content

Instantly share code, notes, and snippets.

View AlexMikhalev's full-sized avatar

Dr Alexander Mikhalev AlexMikhalev

View GitHub Profile
@AlexMikhalev
AlexMikhalev / rclusterlogs.txt
Created May 10, 2021 11:33
docker logs -f rgcluster
docker logs -f rgcluster
Starting 30001
Starting 30002
Starting 30003
Starting 30004
Starting 30005
Starting 30006
Starting 30007
Starting 30008
Starting 30009
curl -H "Content-Type: application/json" -X POST -d '{"search":"How does temperature and humidity affect the transmission of 2019-nCoV?"}' https://api.thepattern.digital/qasearch | jq .results
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 31540 100 31456 100 84 1770 4 0:00:21 0:00:17 0:00:04 6940
[
{
"answer": "community transmission",
"sentence": "At the time of this review April 6 2020 the disease termed covID 19 corona virus disease 2019 had become pandemic and spread to more than 203 countries and territories including community transmission in countries like the United States Germany France sPain Japan Singapore South Korea iRan and Italy and a large scale outbreak with more than 600 cases on the cruise ship Diamond Princess",
"sentencekey": "sentence:0020ab317ef923b740c1d3db52083a4e48495f8a:{5M5}:8",
"title": "SARS-CoV-2 Vaccines: Status Report"
10:M 30 Apr 2021 21:27:20.002 * <rg> GEARS: QA cached called {'event': 'keymiss', 'key': 'cache_{06S}_{PMC261870.xml:{06S}:26}_Laser Correction', 'type': 'empty', 'value': None}
=== REDIS BUG REPORT START: Cut & paste starting from here ===
10:M 30 Apr 2021 21:27:20.002 # === ASSERTION FAILED ===
10:M 30 Apr 2021 21:27:20.002 # ==> module.c:4085 '(c->flags & CLIENT_BLOCKED) == 0' is not true
------ STACK TRACE ------
Backtrace:
@AlexMikhalev
AlexMikhalev / mermaid.md
Created April 22, 2021 09:31
Mermaid parse example
graph TD
    A[Christmas] -->|Get money| B(Go shopping)
    B --> C{Let me think}
    C -->|One| D[Laptop]
    C -->|Two| E[iPhone]
    C -->|Three| F[fa:fa-car Car]
@AlexMikhalev
AlexMikhalev / qa_search.py
Created April 19, 2021 19:55
BERT QA on RedisAI inside RedisGears
### This gears will pre-compute (encode) all sentences using BERT tokenizer for QA
tokenizer = None
def loadTokeniser():
global tokenizer
from transformers import BertTokenizerFast
tokenizer = BertTokenizerFast.from_pretrained("bert-large-uncased-whole-word-masking-finetuned-squad")
return tokenizer
@AlexMikhalev
AlexMikhalev / qa_search.py
Created April 17, 2021 22:58
QA bert interim working
tokenizer = None
import numpy as np
import torch
import os
config_switch=os.getenv('DOCKER', 'local')
if config_switch=='local':
startup_nodes = [{"host": "127.0.0.1", "port": "30001"}, {"host": "127.0.0.1", "port":"30002"}, {"host":"127.0.0.1", "port":"30003"}]
else:
@AlexMikhalev
AlexMikhalev / qa_search.py
Last active April 17, 2021 20:06
QA BERT pre-cashed
tokenizer = None
import numpy as np
import torch
import os
config_switch=os.getenv('DOCKER', 'local')
if config_switch=='local':
startup_nodes = [{"host": "127.0.0.1", "port": "30001"}, {"host": "127.0.0.1", "port":"30002"}, {"host":"127.0.0.1", "port":"30003"}]
else:
@AlexMikhalev
AlexMikhalev / memory.py
Created April 8, 2021 13:40
Check memory usage in python
#standard module
# peak memory usage (kilobytes on Linux, bytes on OS X)
import resource
resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
# pip install psutil
import os, psutil; print(psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2)

Problem

When I use docker to work with the shared workspace with host under Ubuntu, I find that files created by docker user is owned by root. This is not the same with macOS.

Maybe this is becuase docker is run by root user and the default user mapping mechanism is to map container-root to host-user or host-root. So can I map the container-root or container-any-user to host-current-user?

Fortunately the latest docker supports the re-map the container user to any host user via Linux namespace. Refer to this.

Linux namespace

```
redis-cli -c -p 30001 -h 127.0.0.1
127.0.0.1:30001> keys *
1) "processed_docs_stage1_para"
2) "sentence:PMC293432.xml:{06S}"
3) "sentence:PMC270701.xml:{06S}"
4) "edges_matched_{06S}"
5) "sentence:PMC222961.xml:{06S}"
6) "processed_docs_stage3{06S}"
7) "processed_docs_stage2_para{06S}"