tamuhey

default_language_version:
    python: python3.7

repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v2.4.0
    hooks:
    - id: check-added-large-files
      args: ['--maxkb=1000']
    - id: check-merge-conflict

tamuhey

Camphr: spaCy plugin for Transformers, Udify, Elmo, etc.

Hi, I'm Yohei Tamura, a software engineer at PKSHA Technology. I recently published a spaCy plugin called Camphr, which helps in seamless integration for a wide variety of techniques from state-of-the-art to conventional ones. You can use Transformers, Udify, ELmo, etc. on spaCy.

This post introduces how to use Camphr in a nutshell.

Why I chose spaCy

spaCy is an awesome NLP framework and in my opinion has following advantages:

tamuhey

"""Script to convert bccwj NER dataset to jsonl

Usage:

$ python bccwj2jsonl xml/ output/

# convert to irex

$ pythonn bccwj2jsonl xml/ output/ irex
"""

tamuhey

MITではなくApache2にする.

• 商標に関する条項があり，今回作成したロゴとソフト名(Camphr)の権利を排他的に保持できるから．
• 特許に関する条項がきちんと書いてあり，CLAを用意する必要がないから (CLAはcontributorへのハードルを上げる & 個人情報管理の手間が増える)
  • https://opensource.guide/legal/#does-my-project-need-an-additional-contributor-agreement
  • https://opensource.stackexchange.com/questions/5585/benefits-of-a-cla-when-using-apache-2-0-license

参考

tamuhey

export MECAB_URL="https://drive.google.com/uc?export=download&id=0B4y35FiV1wh7cENtOXlicTFaRUE" && \
export IPADIC_URL="https://drive.google.com/uc?export=download&id=0B4y35FiV1wh7MWVlSDBCSXZMTXM" && \
cd /tmp && \
    wget --no-check-certificate ${MECAB_URL} -O mecab.tar.gz && \
    tar xzvf mecab.tar.gz && cd mecab-0.996 && ./configure && make && make check && make install && \
    rm -rf /tmp/* && \
cd /tmp && \
    wget --no-check-certificate ${IPADIC_URL} -O ipadic.tar.gz && \
    tar xzvf ipadic.tar.gz && cd mecab-ipadic-2.7.0-20070801 && ./configure --with-charset=utf8 && ldconfig && make && make install && \
    rm -rf /tmp/*

tamuhey

[tool.poetry]
name = "foo"
version = "0.1.0"
description = ""

[tool.poetry.dependencies]
python = "^3.7"
bar = {path = "bar"}

[tool.poetry.dev-dependencies]

tamuhey

pip install gdown --user
gdown https://drive.google.com/uc?id=0B4y35FiV1wh7cENtOXlicTFaRUE
gdown https://drive.google.com/uc?id=0B4y35FiV1wh7MWVlSDBCSXZMTXM
tar xzvf mecab-0.996.tar.gz
cd mecab-0.996
./configure 
make
make check
sudo make install

tamuhey

2つのtokenizationのアラインメントを求めたい．

https://github.com/explosion/spacy-transformers/issues/87

stack overflowで質問した．

https://stackoverflow.com/questions/59497734/how-can-i-get-an-alignment-for-two-different-tokenizations

diffと似ていそう

tamuhey

import torch

def beamsearch(probs: torch.Tensor, k: int) -> torch.Tensor:
    """Beam search for sequential probabilities.

    Args:
        data: tensor of shape (length, d). requires d > 0. Assumed all items in `probs` in range [0, 1].
        k: beam width
    Returns: (k, length) tensor
    """

tamuhey

from pathlib import Path
import itertools as it
import copy
import srsly
from tqdm.notebook import tqdm


labels = [
    "movie-enter",
    "it-life-hack",