veekaybee

Anti-hype LLM reading list

Goals: Add links that are reasonable and good explanations of how stuff works. No hype and no vendor content if possible. Practical first-hand accounts of models in prod eagerly sought.

Foundational Concepts

Pre-Transformer Models

mattiasarro

# Taken from https://johanwind.github.io/2023/03/23/rwkv_details.html. 
# I've added additional comments restructured it a tiny bit, which makes it clearer for me.

import numpy as np
from torch import load as torch_load  # Only for loading the model weights
from tokenizers import Tokenizer

exp = np.exp
layer_norm = lambda x, w, b : (x - np.mean(x)) / np.std(x) * w + b
sigmoid = lambda x : 1/(1 + exp(-x))

turbo

Merged from https://github.com/joelparkerhenderson/git_commit_message and https://chris.beams.io/posts/git-commit/

General Rules

• Commit messages must have a subject line and may have body copy. These must be separated by a blank line.
• The subject line must not exceed 50 characters
• The subject line should be capitalized and must not end in a period
• The subject line must be written in imperative mood (Fix, not Fixed / Fixes etc.)
• The body copy must be wrapped at 72 columns
• The body copy must only contain explanations as to what and why, never how. The latter belongs in documentation and implementation.