Skip to content

Instantly share code, notes, and snippets.

View ben0it8's full-sized avatar

ben_oght_ah_eight ben0it8

  • Aignostics
  • Berlin
View GitHub Profile
@ben0it8
ben0it8 / bert_textprocessor.py
Last active July 18, 2019 14:04
Create bert textprocessor
import torch
from torch.utils.data import TensorDataset, random_split, DataLoader
import numpy as np
import warnings
from tqdm import tqdm_notebook as tqdm
from typing import Tuple
NUM_MAX_POSITIONS = 256
BATCH_SIZE = 32
@ben0it8
ben0it8 / read_clean_imdb_data.py
Last active July 18, 2019 13:56
read and clean imdb data
import pandas as pd
import re
# text and label column names
TEXT_COL = "text"
LABEL_COL = "label"
def clean_html(text: str):
"remove html tags and whitespaces"
cleanr = re.compile('<.*?>')
@ben0it8
ben0it8 / download_imdb.py
Last active July 18, 2019 13:55
read imdb
import os
import requests
import tarfile
from tqdm import tqdm
# path to data
DATA_DIR = os.path.abspath('./data')
# path to IMDB
IMDB_DIR = os.path.join(DATA_DIR, "imdb5k")
@ben0it8
ben0it8 / cfg-init
Last active March 3, 2019 22:01
initialize dotfiles
sh -c "$(curl -fsSL https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"
echo "ZSH=$HOME/.oh-my-zsh" >> ~/.zshrc
echo "source $ZSH/oh-my-zsh.sh" >> ~/.zshrc
echo "ZSH_THEME='robbyrussel'" >> ~/.zshrc
echo "plugins = (git python osx web-search vi-mode dotenv" >> ~/.zshrc
echo "alias config='/usr/bin/git --git-dir=$HOME/.cfg/ --work-tree=$HOME'" >> ~/.zshrc
source ~/.zshrc
echo ".cfg" >> .gitignore
git clone --bare https://github.com/ben0it8/dotfiles.git .cfg/
config checkout