veekaybee

ChatGPT Resources

Context

ChatGPT appeared like an explosion on all my social media timelines in early December 2022. While I keep up with machine learning as an industry, I wasn't focused so much on this particular corner, and all the screenshots seemed like they came out of nowhere. What was this model? How did the chat prompting work? What was the context of OpenAI doing this work and collecting my prompts for training data?

I decided to do a quick investigation. Here's all the information I've found so far. I'm aggregating and synthesizing it as I go, so it's currently changing pretty frequently.

Model Architecture

jldodds

Require Import Coq.Strings.String.
Require Import Coq.Strings.Ascii.
Require Import Coq.Lists.List.
Import ListNotations.

Open Scope string_scope.

(* Make it print lists one item per line*)
Notation "[ x ; y ; .. ; z ]" :=  (cons x (cons y .. (cons z nil) ..))
  (format "[ '[' x ;  '//' y ;  '//' .. ;  '//' z ']' ]") : list_scope.

mattjj

import torch
import torch.utils.dlpack
import jax
import jax.dlpack

# A generic mechanism for turning a JAX function into a PyTorch function.

def j2t(x_jax):
  x_torch = torch.utils.dlpack.from_dlpack(jax.dlpack.to_dlpack(x_jax))
  return x_torch

lumaku

CTC segmentation for Speechbrain

CTC segmentation can be used to align utterances in long audio files. This gist introduces how to use CTC segmentation for Speechbrain.

A short example

from speechbrain.pretrained import EncoderDecoderASR
from speechbrain.alignment.ctc_segmentation import CTCSegmentation

daanzu

# !/bin/bash

# Adapted from egs/aishell2/s5/local/nnet3/tuning/finetune_tdnn_1a.sh commit 42a673a5e7f201736dfbf2116e8eaa94745e5a5f

# This script uses weight transfer as a transfer learning method to transfer
# already trained neural net model to a finetune data set.

# Usage: /home/daanzu/kaldi_dirs/local/run_finetune_tdnn_1a_daanzu.sh --src-dir export/tdnn_f.1ep --num-epochs 5 --stage 1 --train-stage -10

# Required Inputs:

mcarilli

# This isn't supposed to run as a bash script, i named it with ".sh" for syntax highlighting.

# https://developer.nvidia.com/nsight-systems
# https://docs.nvidia.com/nsight-systems/profiling/index.html

# My preferred nsys (command line executable used to create profiles) commands
#
# In your script, write
# torch.cuda.nvtx.range_push("region name")
# ...

katef

#!/usr/bin/awk -f

# This program is a copy of guff, a plot device. https://github.com/silentbicycle/guff
# My copy here is written in awk instead of C, has no compelling benefit.
# Public domain. @thingskatedid

# Run as awk -v x=xyz ... or env variables for stuff?
# Assumptions: the data is evenly spaced along the x-axis

# TODO: moving average

mblondel

# Authors: Mathieu Blondel, Vlad Niculae
# License: BSD 3 clause

import numpy as np


def _gen_pairs(gen, max_iter, max_inner, random_state, verbose):
    rng = np.random.RandomState(random_state)

    # if tuple, interpret as randn

treuille

import streamlit as st
import streamlit.ReportThread as ReportThread
from streamlit.server.Server import Server
import time
import functools
import random
import string


# Copied from tvst's great gist:

tvst

import streamlit.ReportThread as ReportThread
from streamlit.server.Server import Server


def get_session_id():
    # Hack to get the session object from Streamlit.

    ctx = ReportThread.get_report_ctx()

    this_session = None