Skip to content

Instantly share code, notes, and snippets.

View Ryu1845's full-sized avatar
🎯
Focusing

Sofian Mejjoute Ryu1845

🎯
Focusing
View GitHub Profile
@Ryu1845
Ryu1845 / fsq.py
Created September 28, 2023 08:01
FSQ Implementation from the paper
def round_ste(z):
"""Round with straight through gradients."""
zhat = jnp.round(z)
return z + jax.lax.stop_gradient(zhat - z)
class FSQ:
@Ryu1845
Ryu1845 / zero_init.py
Last active September 23, 2023 12:03
ZerO Initialization copied from the original repo (https://github.com/jiaweizzhao/ZerO-initialization/)
import math
import torch
def hadamard(n: int, dtype=torch.int8):
"""This function is a port of the one in scipy.linalg"""
if n < 1:
lg2 = 0
else:
lg2 = int(math.log(n, 2))
@Ryu1845
Ryu1845 / came.py
Last active July 17, 2023 23:22
CAME: Confidence-guided Adaptive Memory Efficient Optimization from the official repo (https://github.com/huawei-noah/Pretrained-Language-Model/blob/master/CAME/came.py)
import math
import torch
import torch.optim
class CAME(torch.optim.Optimizer):
"""Implements CAME algorithm.
This implementation is based on:
`CAME: Confidence-guided Adaptive Memory Efficient Optimization`
@Ryu1845
Ryu1845 / bst_layer.py
Created June 19, 2023 20:29
JAX implementation of Block-State Transfomer (copied from https://arxiv.org/abs/2306.09539)
"""Block-State Transformer Layer."""
# Block Transformers are non-recurrent and parallelizable.
block_transformer = jax.vmap(BRecT.nonrecurrent_cell)
def BST(u):
"""Block-State Transformer Layer."""
global MF # True if Multi-Filter, False otherwise (SH/MH)
# split inputs into windows (l/w, w, d)
@Ryu1845
Ryu1845 / lru.py
Last active December 13, 2023 13:17
Linear Recurrent Unit (LRU) from the paper ["Resurrecting Recurrent Neural Networks for Long Sequences"](https://arxiv.org/abs/2303.06349)
"""
Simplified Implementation of the Linear Recurrent Unit
------------------------------------------------------
We present here a simplified JAX implementation of the Linear Recurrent Unit (LRU).
The state of the LRU is driven by the input $(u_k)_{k=1}^L$ of sequence length $L$
according to the following formula (and efficiently parallelized using an associative scan):
$x_{k} = \Lambda x_{k-1} +\exp(\gamma^{\log})\odot (B u_{k})$,
and the output is computed at each timestamp $k$ as follows: $y_k = C x_k + D u_k$.
In our code, $B,C$ follow Glorot initialization, with $B$ scaled additionally by a factor 2
to account for halving the state variance by taking the real part of the output projection.
@Ryu1845
Ryu1845 / split.py
Last active March 18, 2023 19:45 — forked from ptrblck/numpy split vs PyTorch split
numpy split vs PyTorch split
import torch
import numpy as np
# numpy
a = np.random.rand(10, 20)
tmp0 = np.split(a, indices_or_sections=5, axis=0) # split into 5 sections
for t in tmp0:
print(t.shape)
# (2, 20)
# (2, 20)
@Ryu1845
Ryu1845 / template.ass
Created January 8, 2023 14:08
vtuber clip template subtitle file (modified from https://github.com/idolactivities/hololive-translations/)
[Script Info]
; Script generated by Aegisub 9010-makepkg-6f546951b
; http://www.aegisub.org/
ScriptType: v4.00+
WrapStyle: 0
ScaledBorderAndShadow: yes
YCbCr Matrix: TV.709
PlayResX: 1280
PlayResY: 720
<!DOCTYPE html>
<html>
<head>
<title>SNScrape Viewer</title>
<script src="https://unpkg.com/mustache@latest"></script>
<script src="https://cdn.tailwindcss.com"></script>
<script>
function checkVisible(elm, threshold, mode) {
threshold = threshold || 0;
gen-username() {
vowel() {
head /dev/urandom | tr -dc 'aeiueoy' | cut -c1
}
consonant() {
head /dev/urandom | tr -dc 'qwrtpsdfghjklzxcvbnm' | cut -c1
}
digit() {
head /dev/urandom | tr -dc '[:digit:]' | cut -c1
}
@Ryu1845
Ryu1845 / Made in Abyss - 01.json
Last active July 4, 2022 20:26
made in abyss cytube
{
"title": "Made in Abyss - 01",
"duration": 10,
"live": false,
"sources": [
{
"url": "https://cloud1.userscloud.com/d/giluki2xtn2fvxijjto4veyddzoaqowmz6atpvgsxyzkfbknhkvlwc2656llywni2ft46kwd/video.mp4",
"contentType": "video/mp4",
"quality": 1080,
"bitrate": 9000