Instantly share code, notes, and snippets.

# Jaemin Choj-min

• Sort options
Last active Feb 20, 2020
Sample the next token from a probability distribution using top-k and/or nucleus (top-p) sampling
View top-k-top-p.py
 def top_k_top_p_filtering(logits, top_k=0, top_p=0.0, filter_value=-float('Inf')): """ Filter a distribution of logits using top-k and/or nucleus (top-p) filtering Args: logits: logits distribution shape (vocabulary size) top_k >0: keep only top k tokens with highest probability (top-k filtering). top_p >0.0: keep the top tokens with cumulative probability >= top_p (nucleus filtering). Nucleus filtering is described in Holtzman et al. (http://arxiv.org/abs/1904.09751) """ assert logits.dim() == 1 # batch size 1 for now - could be updated for more but the code would be less clear top_k = min(top_k, logits.size(-1)) # Safety check
Last active Dec 18, 2019
Compact Bilinear Pooling in PyTorch using the new FFT support
View compact_bilinear_pooling.py
 import torch class CompactBilinearPooling(torch.nn.Module): def __init__(self, input_dim1, input_dim2, output_dim, sum_pool = True): super(CompactBilinearPooling, self).__init__() self.output_dim = output_dim self.sum_pool = sum_pool generate_sketch_matrix = lambda rand_h, rand_s, input_dim, output_dim: torch.sparse.FloatTensor(torch.stack([torch.arange(input_dim, out = torch.LongTensor()), rand_h.long()]), rand_s.float(), [input_dim, output_dim]).to_dense() self.sketch1 = torch.nn.Parameter(generate_sketch_matrix(torch.randint(output_dim, size = (input_dim1,)), 2 * torch.randint(2, size = (input_dim1,)) - 1, input_dim1, output_dim), requires_grad = False) self.sketch2 = torch.nn.Parameter(generate_sketch_matrix(torch.randint(output_dim, size = (input_dim2,)), 2 * torch.randint(2, size = (input_dim2,)) - 1, input_dim2, output_dim), requires_grad = False)
Last active Oct 29, 2019
Implementing Dropout as a Bayesian Approximation in TensorFlow
View dropout_bayesian_approximation_tensorflow.py
 import numpy as np import tensorflow as tf import matplotlib.pyplot as plt from tensorflow.contrib.distributions import Bernoulli class VariationalDense: """Variational Dense Layer Class""" def __init__(self, n_in, n_out, model_prob, model_lam): self.model_prob = model_prob
Last active Jan 10, 2020
View internals.md

### A Tour of PyTorch Internals (Part I)

The fundamental unit in PyTorch is the Tensor. This post will serve as an overview for how we implement Tensors in PyTorch, such that the user can interact with it from the Python shell. In particular, we want to answer four main questions:

1. How does PyTorch extend the Python interpreter to define a Tensor type that can be manipulated from Python code?
2. How does PyTorch wrap the C libraries that actually define the Tensor's properties and methods?
3. How does PyTorch cwrap work to generate code for Tensor methods?
4. How does PyTorch's build system take all of these components to compile and generate a workable application?

#### Extending the Python Interpreter

PyTorch defines a new package `torch`. In this post we will consider the `._C` module. This module is known as an "extension module" - a Python module written in C. Such modules allow us to define new built-in object types (e.g. the `Tensor`) and to call C/C++ functions.

Created Apr 16, 2017
Recurrent Weighted Average RNN in pytorch
View RWA.py
 # An implementation of "Machine Learning on Sequential Data Using a Recurrent Weighted Average" using pytorch # https://arxiv.org/pdf/1703.01253.pdf # # # This is a RNN (recurrent neural network) type that uses a weighted average of values seen in the past, rather # than a separate running state. # # Check the test code at the bottom for an example of usage, where you can compare it's performance # against LSTM and GRU, at a classification task from the paper. It handily beats both the LSTM and # GRU :)
Last active Oct 9, 2018
Pytorch TreeRNN
View treernn.py
 """ TreeLSTM[1] implementation in Pytorch Based on dynet benchmarks : https://github.com/neulab/dynet-benchmark/blob/master/dynet-py/treenn.py https://github.com/neulab/dynet-benchmark/blob/master/chainer/treenn.py Other References: https://github.com/pytorch/examples/tree/master/word_language_model https://github.com/pfnet/chainer/blob/29c67fe1f2140fa8637201505b4c5e8556fad809/chainer/functions/activation/slstm.py https://github.com/stanfordnlp/treelstm
Last active Jan 2, 2020
TensorBoard: TF Dev Summit Tutorial
View labels_1024.tsv
We can make this file beautiful and searchable if this error is corrected: No tabs found in this TSV file in line 0.
 7 2 1 0 4 1 4 9 5 9
Created Feb 2, 2017
Batch normalized LSTM Cell for Tensorflow
View bnlstm.py
 """adapted from https://github.com/OlavHN/bnlstm to store separate population statistics per state""" import tensorflow as tf, numpy as np RNNCell = tf.nn.rnn_cell.RNNCell class BNLSTMCell(RNNCell): '''Batch normalized LSTM as described in arxiv.org/abs/1603.09025''' def __init__(self, num_units, is_training_tensor, max_bn_steps, initial_scale=0.1, activation=tf.tanh, decay=0.95): """ * max bn steps is the maximum number of steps for which to store separate population stats """
Last active Jan 23, 2020
how to convert/port gensim word2vec to tensorflow projector board.
View gensim2projector_tf.py
 # required tensorflow 0.12 # required gensim 0.13.3+ for new api model.wv.index2word or just use model.index2word from gensim.models import Word2Vec import tensorflow as tf from tensorflow.contrib.tensorboard.plugins import projector # loading your gensim model = Word2Vec.load("YOUR-MODEL")
Last active Feb 9, 2020