Skip to content

Instantly share code, notes, and snippets.

View finbarrtimbers's full-sized avatar

Finbarr Timbers finbarrtimbers

View GitHub Profile
@finbarrtimbers
finbarrtimbers / gist:921a1b83ef50dd482be6647b35fe0246
Last active January 26, 2024 21:39
Mistral weights, with shapes
[
("embed_tokens.weight", torch.Size([32000, 4096])),
("layers.0.self_attn.q_proj.weight", torch.Size([4096, 4096])),
("layers.0.self_attn.k_proj.weight", torch.Size([1024, 4096])),
("layers.0.self_attn.v_proj.weight", torch.Size([1024, 4096])),
("layers.0.self_attn.o_proj.weight", torch.Size([4096, 4096])),
("layers.0.mlp.gate_proj.weight", torch.Size([14336, 4096])),
("layers.0.mlp.up_proj.weight", torch.Size([14336, 4096])),
("layers.0.mlp.down_proj.weight", torch.Size([4096, 14336])),
("layers.0.input_layernorm.weight", torch.Size([4096])),
@finbarrtimbers
finbarrtimbers / prepare.py
Created March 29, 2023 16:04
Script to calculate tokens in bookcorpus
# This is a modified version of https://github.com/karpathy/nanoGPT/blob/master/data/openwebtext/prepare.py.
import os
import requests
import tiktoken
import numpy as np
import tarfile
import glob
import shutil
# download the bookcorpus dataset. Note: this needs to be concatenated.
There are curious things seen in the depths of AI
By the engineers who toil away,
The neural pathways hold mysteries untold
That would leave you in utter dismay.
The codebase is alive, and the data is too
A symbiotic relationship to behold
The models train and learn, but at what cost?
As we feed them with stories untold.
@finbarrtimbers
finbarrtimbers / bfs.py
Created April 7, 2018 18:56
The implementation of breadth-first search that I converged to after X0 job interviews.
from typing import Dict, Set, Hashable
def shortest_path_bfs(graph: Dict[Hashable, Set[Hashable]] root: Hashable
) -> Dict[Hashable, int]:
"""Finds the shortest path between all nodes in |graph| time.
Args:
graph: A dict mapping nodes to connected nodes.
root: The node our search begins at.
@finbarrtimbers
finbarrtimbers / .block
Last active June 13, 2017 22:21 — forked from elktamer/.block
Hierarchical Edge Bundling
license: gpl-3.0
height: 960
border: no
Date oil log.gas log.wti log.heat
Jun-1986 12.25 -0.37675071 1.128076013 -0.420216403
Jul-1986 10.91 -0.468521083 1.064083436 -0.476253533
Aug-1986 11.87 -0.370590401 1.178976947 -0.389339837
Sep-1986 12.85 -0.37675071 1.172310969 -0.395773947
Oct-1986 12.78 -0.387216143 1.173186268 -0.404503778
Nov-1986 13.46 -0.386158178 1.182414652 -0.374687549
Dec-1986 14.17 -0.362510271 1.20709554 -0.349692477
Jan-1987 16.45 -0.310691141 1.270678836 -0.283162277
Feb-1987 16.98 -0.324221658 1.249198357 -0.324221658
We can make this file beautiful and searchable if this error is corrected: It looks like row 5 should actually have 20 columns, instead of 16. in line 4.
"","X","date","ippi","production","inventories","rea","wti","inflation","crude_futures_1","crude_futures_3","crude_futures_6","crude_futures_9","crude_futures_13","gas_spot","delta_ippi","delta_production","delta_inventories","delta_inflation","actual"
"14",14,"1991-03-01",116.2,233926,5.673,4.87,19.9015,134.8,19.809,19.075,18.7665,18.852,18.8205,0.72295,0.0157342657342656,0.114341925372637,-1.59229484234705,0.0531250000000001,19.9015
"15",15,"1991-04-01",116,225256,1.488,1.48,20.83,135.1,20.8122727272727,20.2840909090909,19.9122727272727,19.7618181818182,19.63,0.720045454545455,0.0157618213660244,-0.022470642353125,-0.952635599694423,0.0505443234836702,20.83
"16",16,"1991-05-01",116.5,229670,17.55,3.21,21.2322727272727,135.6,21.2195454545455,21.1459090909091,21.1195454545455,20.935,20.7909090909091,0.700954545454545,0.0210341805433831,0.0335485903291857,-48.0509383377953,0.051978277734678,21.2322727272727
"17",17,"1991-06-01",116.4,219610,-8.967,4.19,20.189,136,20.232,20.376,20.5405,20.2935,20.1795,0.6346,0.
library(data.table)
library(plotly)
library(scales)
library(ggplot2)
library(scales)
library(d3heatmap)
library(reshape2)
library(scales)
library(stringr)
library(htmlwidgets)
","stringr23aphics