Skip to content

Instantly share code, notes, and snippets.

View iwiwi's full-sized avatar
🏠
Working from home

Takuya Akiba iwiwi

🏠
Working from home
View GitHub Profile
View 2023-09-19_chinchilla-scaling.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View 2023-09-20_tokenizers.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View 2023-09-22 GPT-4 authors.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
View 2023-09-21_tokenizers.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@iwiwi
iwiwi / crawler.rb
Last active September 20, 2023 10:58
Lightweight parallel web graph crawler
View crawler.rb
#!/usr/bin/env ruby
#
# crawler.rb --- Lightweight parallel web graph crawler
#
# Usage:
# ./crawler.rb START_URL TARGET_REGEXP
#
# Output:
# stdout --- edge list (tab separated URLs)
@iwiwi
iwiwi / WebGraphDecoder.java
Last active September 17, 2021 09:32
Download and decode WebGraph format graphs
View WebGraphDecoder.java
import it.unimi.dsi.fastutil.ints.IntArrayFIFOQueue;
import it.unimi.dsi.fastutil.ints.IntArrays;
import it.unimi.dsi.logging.ProgressLogger;
import it.unimi.dsi.webgraph.GraphClassParser;
import it.unimi.dsi.webgraph.ImmutableGraph;
import it.unimi.dsi.webgraph.LazyIntIterator;
import java.io.*;
import java.util.*;
View quantile_transformer.rs
use crate::*;
mod normal_distribution {
const S2PI: f64 = 2.50662827463100050242E0;
// https://github.com/scipy/scipy/blob/v1.5.4/scipy/special/cephes/ndtri.c
const P0: [f64; 5] = [
-5.99633501014107895267E1,
9.80010754185999661536E1,
-5.66762857469070293439E1,
View install_opencv.sh
#!/bin/bash
set -e
TARGET_DIR="$(dirname $(readlink -m $0))/.local"
WORKING_DIR="/tmp/"
LIBJPEG_VERSION="1.5.3"
OPENCV_VERSION="3.4.1"
eval "$(pyenv init -)"
PYENV_VERSION=$(pyenv version | cut -f 1 -d ' ')
View chance-level-ap.md
import numpy as np
from sklearn.metrics import average_precision_score

def chance_level_ap(n_all, n_positive, trials=1000):
    return np.mean([
        average_precision_score([1] * n_positive + [0] * (n_all - n_positive), np.random.permutation(n_all))
 for _ in range(trials)])
View chance-level-ap.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.