Skip to content

Instantly share code, notes, and snippets.

Vadim Kantorov vadimkantorov

Block or report user

Report or block vadimkantorov

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@vadimkantorov
vadimkantorov / git_rebase_pull_request.sh
Created Oct 16, 2019
How to rebase a PyTorch pull request (PR)
View git_rebase_pull_request.sh
git remote add upstream https://github.com/pytorch/pytorch.git # do once
git fetch upstream master
# git rebase -i HEAD~2 # optional pick + squash
git rebase upstream/master
git push -f
@vadimkantorov
vadimkantorov / find_domain_words.py
Last active Sep 11, 2019
Compare two ARPA language models with KenLM
View find_domain_words.py
# Usage: python3 find_domain_words --ours chats.arpa --theirs ru_wiyalen_no_punkt.arpa.binary > domain_words.txt
import argparse
import kenlm
parser = argparse.ArgumentParser()
parser.add_argument('--ours', required = True)
parser.add_argument('--theirs', required = True)
args = parser.parse_args()
@vadimkantorov
vadimkantorov / perlin.py
Last active Sep 5, 2019
Perlin noise in PyTorch
View perlin.py
# ported from https://github.com/pvigier/perlin-numpy/blob/master/perlin2d.py
import torch
import math
def rand_perlin_2d(shape, res, fade = lambda t: 6*t**5 - 15*t**4 + 10*t**3):
delta = (res[0] / shape[0], res[1] / shape[1])
d = (shape[0] // res[0], shape[1] // res[1])
grid = torch.stack(torch.meshgrid(torch.arange(0, res[0], delta[0]), torch.arange(0, res[1], delta[1])), dim = -1) % 1
@vadimkantorov
vadimkantorov / invconv1x1.py
Last active Aug 20, 2019
Invertible 1x1 convolution in pure PyTorch (extracted from Glow packages)
View invconv1x1.py
# Original code from OpenAI Glow: https://github.com/openai/glow/blob/master/model.py
# This impl is inspired by this PyTorch reference: https://github.com/rosinality/glow-pytorch/blob/master/model.py
# This impl does not include inverse() and log_abs_det_jacobian() computation.
import torch
class InvConvNd(torch.nn.Module):
def __init__(self, in_channels, gain = 1e-3):
super().__init__()
@vadimkantorov
vadimkantorov / larc.py
Last active Aug 13, 2019
LARC gradient clipping in PyTorch
View larc.py
# ported from https://github.com/NVIDIA/OpenSeq2Seq/blob/master/open_seq2seq/optimizers/optimizers.py
# paper: https://arxiv.org/abs/1708.03888
# more advanced PyTorch variant: https://github.com/NVIDIA/apex/blob/master/apex/parallel/LARC.py
# Usage: larc_(optimizer.param_groups, larc_eta = 1e-3)
import torch
def larc_(param_groups, larc_eta = 1e-3, larc_mode = 'clip', min_update = 1e-7, eps = 1e-7):
for group in param_groups:
@vadimkantorov
vadimkantorov / novograd.py
Last active Aug 25, 2019
NovoGrad optimizer in PyTorch
View novograd.py
# ported from https://github.com/NVIDIA/OpenSeq2Seq/blob/master/open_seq2seq/optimizers/novograd.py
# paper: https://arxiv.org/abs/1905.11286
# a recent NVidia's implementation in PyTorch: https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/SpeechRecognition/Jasper/optimizers.py
import torch
class NovoGrad(torch.optim.Optimizer):
def __init__(self, params, lr=1.0, betas = (0.95, 0.98), eps=1e-8, weight_decay=0.0, dampening=False):
defaults = dict(lr=lr, betas=betas, eps=eps, weight_decay=weight_decay, dampening=dampening)
super(NovoGrad, self).__init__(params, defaults)
@vadimkantorov
vadimkantorov / polynomial_decay_lr.py
Last active Aug 11, 2019
Polynomial decay learning rate scheduler for PyTorch (ported from OpenSeq2Seq and TensorFlow)
View polynomial_decay_lr.py
import torch
# ported from https://github.com/NVIDIA/OpenSeq2Seq/blob/master/open_seq2seq/optimizers/lr_policies.py and https://www.tensorflow.org/api_docs/python/tf/train/polynomial_decay
class PolynomialDecayLR(torch.optim.lr_scheduler._LRScheduler):
def __init__(self, optimizer, decay_steps, power = 1.0, begin_decay_at = 0, end_lr = 0.0, warmup_steps = 0, last_epoch = -1):
self.decay_steps = decay_steps
self.power = power
self.begin_decay_at = begin_decay_at
self.end_lr = end_lr
self.warmup_steps = warmup_steps
@vadimkantorov
vadimkantorov / split_audio_by_silence.py
Last active Jul 8, 2019
Python script using webrtcvad for splitting an audio file into voice segments
View split_audio_by_silence.py
# Usage: python3 split_audio_by_silence.py -i input_audio.m4a -o segments
# will save segments in mp3 format into the segments directory
# based on https://github.com/mozilla/DeepSpeech/tree/master/examples/vad_transcriber
# Dependencies: webrtcvad
import os
import argparse
import collections
import subprocess
import webrtcvad
@vadimkantorov
vadimkantorov / reludropoutinplace.py
Last active Oct 15, 2019
PyTorch module for inplace fused ReLU and Dropout
View reludropoutinplace.py
import torch
class ReLUDropout(torch.nn.Dropout):
def forward(self, input):
return relu_dropout(input, p = self.p, training = self.training, inplace = self.inplace)
def relu_dropout(x, p = 0, inplace = False, training = False):
if not training or p == 0:
return x.clamp_(min = 0) if inplace else x.clamp(min = 0)
@vadimkantorov
vadimkantorov / duration.sh
Last active Jun 17, 2019
Bash alias for audio/video file duration
View duration.sh
# https://superuser.com/questions/650291/how-to-get-video-duration-in-seconds
# Usage: duration myfile.wav
alias duration="ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1"
# mkdir -p sample; find segments.* -name '*.wav' -type f | while read filename; do echo "$(duration $filename) $filename"; done | grep "^5\." | while read duration_ filename_; do cp $filename_ sample; done
You can’t perform that action at this time.