tamuhey

How to calculate the alignment between BERT and spaCy tokens effectively and robustly

site: https://tamuhey.github.io/tokenizations/

Natural Language Processing (NLP) has made great progress in recent years because of neural networks, which allows us to solve various tasks with end-to-end architecture. However, many NLP systems still require language-specific pre- and post-processing, especially in tokenizations. In this article, I describe an algorithm that simplifies calculating correspondence between tokens (e.g. BERT vs. spaCy), one such process. And I introduce Python and Rust libraries that implement this algorithm. Here are the library and the demo site links:

• repo: https://github.com/tamuhey/tokenizations

abhishekkrthakur

import os
import requests
import json

SLACK_WEBHOOK= os.environ.get("SLACK_WEBHOOK")


def send_message(messages, channel="abhishek", username="beast"):
    """
    :param messages: list of texts

jaradc

import numpy as np
from scipy.stats import entropy
from math import log, e
import pandas as pd

import timeit

def entropy1(labels, base=None):
  value,counts = np.unique(labels, return_counts=True)
  return entropy(counts, base=base)

nigeljyng

from keras import backend as K
from keras.engine import InputSpec
from keras.engine.topology import Layer
import numpy as np


class TemporalMaxPooling(Layer):
    """
    This pooling layer accepts the temporal sequence output by a recurrent layer
    and performs temporal pooling, looking at only the non-masked portion of the sequence.

empet

staticfloat

dmrd

czlee

cbcunc

yoon