Skip to content

Instantly share code, notes, and snippets.

View Laeeth's full-sized avatar

Laeeth Isharc Laeeth

View GitHub Profile
@jart
jart / rename-pictures.sh
Created December 12, 2023 15:24
Shell script for renaming all images in a folder
#!/bin/sh
# rename-pictures.sh
# Author: Justine Tunney <jtunney@gmail.com>
# License: Apache 2.0
#
# This shell script can be used to ensure all the images in a folder
# have good descriptive filenames that are written in English. It's
# based on the Mistral 7b and LLaVA v1.5 models.
#
# For example, the following command:
@saharNooby
saharNooby / simple_bpe_tokenizer.py
Last active May 5, 2023 18:57
Probably the dumbest, no-dependencies, pure Python implementation of 20B_tokenizer.json (a BPE tokenizer for GPT-NeoX model)
import regex
import json
import unicodedata
from typing import Tuple, Callable, Union
# Parses the tokenizer config and returns encode and decode functions.
def load_tokenizer(config_path: str) -> Tuple[Callable[[str], list[int]], Callable[[list[int]], str]]:
# Maps any byte 0..255 to a printable Unicode character.
byte_to_unicode: dict[int, str] = {
33: "!",
@jiamingkong
jiamingkong / langchain.ipynb
Created May 1, 2023 04:07
Using RWKV in langchain
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Type your search: What is Germany?
(Document(page_content="Germany has been described as a [[great power]] with [[Economy of Germany|a strong economy]]; it has the [[List of sovereign states in Europe by GDP (nominal)|largest economy in Europe]], the world's [[List of countries by GDP (nominal)|fourth-largest economy by nominal GDP]] and the [[List of countries by GDP (PPP)|fifth-largest by PPP]]. As a global power in industrial, [[Science and technology in Germany|scientific and technological]] sectors, it is both the world's [[List of countries by exports|third-largest exporter]] and [[List of countries by imports|importer]]. As a [[developed country]] it [[Social security in Germany|offers social security]], [[Healthcare in Germany|a universal health care system]] and [[Higher education in Germany|a tuition-free university education]]. Germany is a member of the [[United Nations]], the European Union, [[NATO]], the [[Council of Europe]], the [[G7]], the [[G20]] and the [[OECD]]. It has the [[List of World
@Laeeth
Laeeth / long_gpt.py
Created April 13, 2023 04:17 — forked from NaxAlpha/long_gpt.py
Training script for LongGPT; Fine-tunes GPT-2 (335M) on The Pile Dataset with a context size of 8k tokens. (requires > 16GB RAM)
import time
from contextlib import suppress
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import torch.backends.cuda as cuda
from torch.utils.data import DataLoader, IterableDataset
@saharNooby
saharNooby / RWKV_cache.py
Last active May 3, 2023 20:58
State cache for RWKV language model
# USAGE EXAMPLE
cache = RWKV_Cache()
init_out, init_state = cache.preprocess_prompt(model, prompt_tokens)
for GENERATION in range(NUM_GENERATIONS):
out, state = init_out.clone(), init_state.clone()
cache_key = [*prompt_tokens]
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@NaxAlpha
NaxAlpha / long_gpt.py
Last active October 15, 2023 11:21
Training script for LongGPT; Fine-tunes GPT-2 (335M) on The Pile Dataset with a context size of 8k tokens. (requires > 16GB RAM)
import time
from contextlib import suppress
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import torch.backends.cuda as cuda
from torch.utils.data import DataLoader, IterableDataset
@veekaybee
veekaybee / chatgpt.md
Last active March 15, 2024 18:28
Everything I understand about chatgpt

ChatGPT Resources

Context

ChatGPT appeared like an explosion on all my social media timelines in early December 2022. While I keep up with machine learning as an industry, I wasn't focused so much on this particular corner, and all the screenshots seemed like they came out of nowhere. What was this model? How did the chat prompting work? What was the context of OpenAI doing this work and collecting my prompts for training data?

I decided to do a quick investigation. Here's all the information I've found so far. I'm aggregating and synthesizing it as I go, so it's currently changing pretty frequently.

Model Architecture