Skip to content

Instantly share code, notes, and snippets.

View kprybol's full-sized avatar

Kevin Prybol kprybol

  • Raleigh, NC
View GitHub Profile
@tamuhey
tamuhey / tokenizations_post.md
Last active December 3, 2023 19:53
How to calculate the alignment between BERT and spaCy tokens effectively and robustly
View tokenizations_post.md

How to calculate the alignment between BERT and spaCy tokens effectively and robustly

image

site: https://tamuhey.github.io/tokenizations/

Natural Language Processing (NLP) has made great progress in recent years because of neural networks, which allows us to solve various tasks with end-to-end architecture. However, many NLP systems still require language-specific pre- and post-processing, especially in tokenizations. In this article, I describe an algorithm that simplifies calculating correspondence between tokens (e.g. BERT vs. spaCy), one such process. And I introduce Python and Rust libraries that implement this algorithm. Here are the library and the demo site links:

@abhishekkrthakur
abhishekkrthakur / slack_notifier.py
Created December 6, 2019 07:53
Slack notification from python
View slack_notifier.py
import os
import requests
import json
SLACK_WEBHOOK= os.environ.get("SLACK_WEBHOOK")
def send_message(messages, channel="abhishek", username="beast"):
"""
:param messages: list of texts
@jaradc
jaradc / entropy_calculation_in_python.py
Last active February 19, 2024 20:43
Four different ways to calculate entropy in Python
View entropy_calculation_in_python.py
import numpy as np
from scipy.stats import entropy
from math import log, e
import pandas as pd
import timeit
def entropy1(labels, base=None):
value,counts = np.unique(labels, return_counts=True)
return entropy(counts, base=base)
@nigeljyng
nigeljyng / TemporalMaxPooling.py
Last active February 21, 2020 15:56
Temporal max pooling as implemented in https://arxiv.org/abs/1511.04108
View TemporalMaxPooling.py
from keras import backend as K
from keras.engine import InputSpec
from keras.engine.topology import Layer
import numpy as np
class TemporalMaxPooling(Layer):
"""
This pooling layer accepts the temporal sequence output by a recurrent layer
and performs temporal pooling, looking at only the non-masked portion of the sequence.
@empet
empet / Plotly-Julia-Set.ipynb
Last active July 4, 2016 14:04
Escape time algorithm to get Plotly plot of a Julia set
View Plotly-Julia-Set.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View TensorFlow.jl - Real DFT.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@dmrd
dmrd / IntroToJulia.ipynb
Created April 29, 2016 01:14
Notebook walking through a few Julia examples
View IntroToJulia.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View powerpairing.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View Numpy4AnalysticsForward.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View julia_menu_problem.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.