heydavid525

LLM Wiki

A pattern for building personal knowledge bases using LLMs.

This is an idea file, it is designed to be copy pasted to your own LLM Agent (e.g. OpenAI Codex, Claude Code, OpenCode / Pi, or etc.). Its goal is to communicate the high level idea, but your agent will build out the specifics in collaboration with you.

The core idea

Most people's experience with LLMs and documents looks like RAG: you upload a collection of files, the LLM retrieves relevant chunks at query time, and generates an answer. This works, but the LLM is rediscovering knowledge from scratch on every question. There's no accumulation. Ask a subtle question that requires synthesizing five documents, and the LLM has to find and piece together the relevant fragments every time. Nothing is built up. NotebookLM, ChatGPT file uploads, and most RAG systems work this way.

heydavid525

Introduction

The computer driven markets for instruments like stocks and exchange traded stock options, have transformed finance and the flow of capital. These markets are enabled by order matching engines (and the infrastructure that supports this software). Before computer trading networks and matching engines, stocks where traded on cavernous exchange floors and transaction costs where high. When electronic trading fully matured, floor traders were a fading anachronism and transaction costs had been reduced to pennies a share in many cases. Electronic trading could not exist without advanced network infrastructure, but without the software matching engines no shares would change hands. The computer trading networks, the matching engine software has also created a concentrated nexus of potential failure. Failures in these systems have increased as the frequency and volume on the electronic networks has increased. The position of order matching engines in the trading infrastructure makes these systems o

heydavid525

#!/bin/bash
# Add to instance metadata with `gcloud compute instances add-metadata \
#   instance-name --metadata-from-file startup-script=idle-shutdown.sh` and reboot
# NOTE: requires `bc`, eg, sudo apt-get install bc
# Modified from https://stackoverflow.com/questions/30556920/how-can-i-automatically-kill-idle-gce-instances-based-on-cpu-usage

sudo apt-get install bc
threshold=0.1

count=0

heydavid525

import org.apache.spark.mllib.feature.{Word2Vec, Word2VecModel}
import scala.io.Source
val filename = "/home/wdai/spark/text_embed/glove/glove.6B.50d.txt"
val embed = Source.fromFile(filename).getLines.map(line => {
  val a = line.split(" ")
  a(0) -> a.slice(1,51).map(x => x.toFloat)
}).toMap
embed.size                                                                                                                                                                                                                            val model = new Word2VecModel(embed)
model.save(sc, "/home/wdai/spark/text_embed/glove50d.Word2vecModel")

heydavid525

# -*- coding: utf-8 -*-

""" Convolutional Neural Network for MNIST dataset classification task.

References:
    Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. "Gradient-based
    learning applied to document recognition." Proceedings of the IEEE,
    86(11):2278-2324, November 1998.

Links:

heydavid525

val numDim = 100
val numClasses = 1

// Create distributed model parameters (stored in PS). w can be any dimension
// (like numpy array)
val w = DistArray((numDim, numClasses), sparse=True)

// Create a data streaming iterator accessible by all worker / all tasks
// inMemory
val data = DistData("kddb", inMemory=False)

heydavid525

val textFile = sc.textFile("/nfs/nas-0-16/wdai/datasets/class/binary/higgs/HIGGS.csv")
textFile.count()
textFile.first()
val data = textFile.map(line => line.split(",").map(elem => elem.trim.toDouble))
data.first().length

import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.linalg.Vectors
val subset = data.sample(false, 0.1, 123456)
val labeledData = subset.map(a => LabeledPoint(a(0), Vectors.dense(a.slice(1, 28))))

heydavid525

#!/usr/bin/env python
# -*- encoding: utf-8 -*-
from __future__ import print_function
import time

# keras
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation, Flatten
from keras.layers.advanced_activations import PReLU, ELU
from keras.layers.normalization import BatchNormalization

heydavid525

// RowStorage is an abstract class representing a basic row storage without
// freshness awareness. This class is extended to support LRU and LFU row
// eviction policy when storage is full, and serves as the common interface
// for ThreadCache, ProcessorCache, and ServerStorage
//
// Comment(wdai): This interface does not support Inc(), which should be
// supported at a higher level of abstraction.
template<typename ROW_ID, typename V>
class RowStorage {
  public: