Skip to content

Instantly share code, notes, and snippets.

View tamuhey's full-sized avatar
🏠
Working from home

Yohei Tamura tamuhey

🏠
Working from home
View GitHub Profile
class UnionFind:
def __init__(self, objects):
self.objects = list(objects)
self.weights = {i: 1 for i in range(len(self.objects))}
self.parents = list(range(len(self.objects)))
self.obj2num = {k: i for i, k in enumerate(self.objects)}
def add(self, obj):
self.objects.append(obj)
n = len(self.objects)-1
@tamuhey
tamuhey / spacy_multiprocess_pipe.ipynb
Last active October 7, 2019 15:07
multiprocessing for spacy nlp.pipe
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
[tool.poetry]
name = "foo"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
[tool.poetry.dependencies]
python = "^3.7"
[tool.poetry.dev-dependencies]
@tamuhey
tamuhey / notify-slack.sh
Last active October 10, 2019 10:36
notify-slack
#!/bin/bash
SLACK_NOTIFY_ICON=":ghost:"
if [ ! -f ~/.notify-slack-cfg ]; then
echo "ERROR: A Webhook URL is required. Create yours here: https://my.slack.com/services/new/incoming-webhook/"
echo "Once you have your KEY, please create a file at ${HOME}/.notify-slack-cfg containng only the KEY. Eg: T02TKKSAX/B246MJ6HX/WXt2BWPfNhSKxdoFNFblczW9"
return
fi
SLACK_MESSAGE="$1"
curl -H "Authorization: token $GITHUB_TOKEN" \
-H "Content-Type: $(file -b --mime-type $FILE)" \
--data-binary @$FILE \
"https://uploads.github.com/repos/$REPO/releases/$RELEASE/assets?name=$(basename $FILE)"
@tamuhey
tamuhey / upload_github_release_asset.py
Created October 11, 2019 05:44
upload assets to github release
from pathlib import Path
import os
import re
import subprocess
import fire
import requests
ENDPOINT = "https://api.github.com"
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@tamuhey
tamuhey / ja_upper.py
Created November 8, 2019 16:10
拗音を促音に変換する
import string
LOWER = "ぁぃぅぇぉっゃゅょゎァィゥェォッャュョヮ"
UPPER = "あいうえおつやゆよわアイウエオツヤユヨワ"
MAP = str.maketrans(dict(zip(LOWER, UPPER)))
def ja_upper(text: str) -> str:
return text.translate(MAP)
@tamuhey
tamuhey / ldcc2spacygold.py
Created November 19, 2019 17:57
convert livedoor news corpus to spacy gold jsonl
from pathlib import Path
import itertools as it
import copy
import srsly
from tqdm.notebook import tqdm
labels = [
"movie-enter",
"it-life-hack",
@tamuhey
tamuhey / torch_beemsearch.py
Last active January 21, 2020 10:44
simple and efficient beamsearch function for pytorch
import torch
def beamsearch(probs: torch.Tensor, k: int) -> torch.Tensor:
"""Beam search for sequential probabilities.
Args:
data: tensor of shape (length, d). requires d > 0. Assumed all items in `probs` in range [0, 1].
k: beam width
Returns: (k, length) tensor
"""