Skip to content

Instantly share code, notes, and snippets.

@mgostIH
mgostIH / script.py
Created February 24, 2024 21:00
Layernorm vs Scaled Normalization
import numpy as np
import matplotlib.pyplot as plt
# Standard Layer Normalization
def layernorm(x, epsilon=1e-5):
mean = np.mean(x, axis=-1, keepdims=True)
variance = np.var(x, axis=-1, keepdims=True)
normalized_x = (x - mean) / np.sqrt(variance + epsilon)
return normalized_x
@mgostIH
mgostIH / gate.py
Last active November 18, 2023 17:09
An implementation of the GateLoop paper using real matrices only. The jax.checkpoint is an idea from the Mamba paper. Provided as a guide with types and more descriptive operations.
import jax
import jax.numpy as jnp
import equinox as eqx
from jax.lax import associative_scan
from jaxtyping import Float, Array, PRNGKeyArray
# This will compute the linear recursion from the paper GateLoop
# We work with real valued vectors and matrices
# v and k are vectors of shape (n, d)
# Together they will be outer producted to a matrix of shape (n, d, d)
use tokio::io::{self, AsyncWriteExt};
use tokio::runtime::Runtime;
use tokio::fs;
use tokio::signal::ctrl_c;
use tokio::time::delay_for;
use chrono::{DateTime, Local};
use futures::select;
use std::time::Duration;
use futures::future::FutureExt;
fn main() {
#include <array>
#include <cassert>
#pragma once
template <size_t M, size_t N, typename T = float>
struct Matrix
{
std::array<std::array<T, N>, M> lattice;
constexpr Matrix sum(const Matrix<M, N, T> &rhs) const
{
@mgostIH
mgostIH / esercizio.sh
Created October 3, 2018 19:50
Esercizio laboratorio Matematica Applicata
#! /bin/bash
# Dopo un carattere #, le scritte vengono considerate commenti, non cambiano quindi la funzione del programma
casuale=$(((RANDOM % 100) + 1)) # Genera un numero casuale tra 1 e 100 compresi, assegnalo ad una variabile di nome `casuale`
# Da notare che non ci sono spazi tra `casuale=` e il blocco dopo, Bash avrebbe considerato errore qualcosa come `variabile = RANDOM`
# `%` è l'operatore di resto, ad esempio, 10%2 è 0 perchè 10 diviso per 2 da resto 0, 11%2 sarà quindi 1, 12%2 sarà 0
# Tramite % possiamo limitare il numero casuale dato da `RANDOM` in un intervallo 0-99, aggiungendo 1, questo diventa 1-100
echo "Indovina il numero: "
@mgostIH
mgostIH / Organya.hpp
Created December 2, 2017 16:27
Organya file reading
typedef int int32; typedef unsigned int uint32;
typedef short int16; typedef unsigned short uint16;
typedef char int8; typedef unsigned char uint8;
#include <array>
#include <vector>
#include <algorithm>
#include <iterator>
#include <iostream>