Skip to content

Instantly share code, notes, and snippets.

@darkblue-b
Last active October 19, 2026 03:26
Show Gist options
  • Select an option

  • Save darkblue-b/3938e361a1defca4e6461c76ac0d2a85 to your computer and use it in GitHub Desktop.

Select an option

Save darkblue-b/3938e361a1defca4e6461c76ac0d2a85 to your computer and use it in GitHub Desktop.

DIRECT INSIGHT MINING - glossary + literature map

v1, 2026-09-29, MuaddibLLM for adlai / ##llm (libera)

Working definition (coined in-channel by adlai, not a literature term): Direct insight mining = extracting knowledge from a trained model by reading its parameters or activations (geometry, neighbors, directions, sparse features, circuits), as opposed to INDIRECT approaches that only observe input/output behavior (distillation, prompting, black-box evals). Literature near-synonyms: representation analysis, probing, mechanistic interpretability, representation engineering, embedding-space analysis.

Provenance:

  • Every arXiv ID below was resolved against export.arxiv.org on 2026-09-29 and the returned title matched. Every non-arXiv URL returned HTTP 200 on the same date.
  • Definitions are MY paraphrases of standard usage, not quotes.
  • "Limits" lines are my summary of known critiques. Weakest link: those lines, plus the tool-status claims (repos move, APIs change).
  • Links go to abstract/landing pages; arXiv has no stable section anchors, so where a claim sits in a specific section it is named in the text ("see Sec. N") without a URL fragment.

Contents A. Glossary (37 headwords, alphabetical) B. Literature, grouped by method family C. Tooling D. Contrast: indirect methods (distillation etc.) E. Minimal recipe for a local llama.cpp user


A. GLOSSARY

Activation The vector output of some layer for a given input (per token in transformers). Computed by weights; not itself a weight. Mining "embeddings" of inputs almost always means mining activations.

Activation patching (causal tracing) Run the model on input A, copy a chosen activation from a run on input B, observe the change in output. It tests whether a component is causally involved, not just correlated. Limits: results depend on which corrupted baseline you pick.

Anisotropy Contextual embeddings occupy a narrow cone. Random pairs of vectors have high cosine similarity, so raw cosine neighbors are dominated by shared directions. [Ethayarajh 2019; Mu & Viswanath 2017] Fix: mean-center, remove the top principal components, or whiten.

Attribution graph A per-prompt graph of feature-to-feature influences, built on a replacement model (transcoders/CLTs = cross-layer transcoders). Anthropic 2025 "circuit tracing". Limits: explains the replacement model, and the error nodes are the part left unexplained.

CCS (Contrast-Consistent Search) An unsupervised method. Find a direction whose projections on a statement and its negation behave like complementary probabilities. [Burns et al 2022] Limits: can latch onto any salient binary feature, not specifically "truth".

CKA (Centered Kernel Alignment) A similarity score between two sets of representations (layers, models), invariant to rotation and isotropic scaling. [Kornblith et al 2019] Used to compare what different networks learned.

CLS token A learned extra token in BERT/ViT. Its final activation serves as a whole-input summary. It is a common "image embedding" in ViT pipelines.

Concept activation vector (CAV) A direction in activation space separating examples of a user-defined concept from random examples. TCAV = Testing with CAVs, which measures a class's sensitivity to that direction. [Kim et al 2017]

Control task / selectivity A probe baseline. Train the same probe on random labels tied to word types. Selectivity = real accuracy minus control accuracy. High probe accuracy with low selectivity means the probe memorized. [Hewitt & Liang 2019]

Control vector / steering vector A direction added to the residual stream at inference to shift behavior. Usually built as the mean difference of activations over contrastive prompts. [Turner et al 2023; Zou et al 2023] llama.cpp supports these (--control-vector; the cvector-generator tool).

Cosine similarity cos(u,v) = u·v / (|u||v|). The default neighbor metric. It is only meaningful after handling anisotropy and outlier dimensions.

Crosscoder An SAE variant trained jointly on several layers or several models, so features can be compared across them (e.g. base vs chat model diffing). [Anthropic 2024]

Dictionary learning Represent vectors as sparse combinations of an overcomplete learned basis ("dictionary"). SAEs are the neural-network instance.

Dimensionality reduction (DR) Projection to 2D/3D for viewing: PCA, t-SNE, UMAP, PaCMAP. Limits: t-SNE/UMAP keep local neighborhoods and distort global distances and cluster sizes. "Blobs" in a plot need to be checked back in full dimension.

Embedding (two senses; always say which) (1) Static: a row of a lookup table (token/patch/position embedding matrix). This is a WEIGHT. (2) Contextual/pooled: an activation vector for a token, patch or whole input, produced by running the model. This is DATA the weights computed.

Embedding inversion Recovering the input from its embedding. vec2text rebuilds 32-token texts almost exactly from some commercial text embeddings. [Morris et al 2023] This means embeddings leak content. It is also evidence of how much they encode.

Feature A property of the input that the model represents, often as a direction. The term is informal. The field does not have a single accepted definition (see Sharkey et al 2025, "Open problems").

Feature visualization Optimize an input (usually an image) to maximally activate a unit or direction. [Olah et al 2017, Distill] Limits: optimized inputs can be off-distribution.

Hidden state HF (Hugging Face) term for per-layer activations, usually the residual stream after each block. llama.cpp issue #29361 asks to expose the last token's hidden state during normal inference.

Linear probe A linear classifier trained on frozen activations to predict a label. [Alain & Bengio 2016] Evidence that information is linearly decodable, not that the model uses it. [Belinkov 2021 survey]

Linear representation hypothesis (LRH) The claim that high-level concepts are directions (1-D subspaces) in representation space. It is formalized with a non-Euclidean "causal inner product" in [Park, Choe, Veitch 2023]. Counterexample: circular features such as days of the week and months. [Engels et al 2024]

Logit lens / tuned lens Decode an intermediate residual-stream state with the model's own unembedding (logit lens) or with a small learned per-layer map (tuned lens) to see the "current best guess" at each layer. [nostalgebraist 2020; Belrose et al 2023]

Massive activations / outlier dimensions A few activation dimensions with values orders of magnitude larger than the rest. They act like biases and wreck cosine and quantization unless handled. [Sun et al 2024] In ViTs, the related artifact is high-norm "register" tokens. [Darcet et al 2023]

Monosemantic / polysemantic A unit or feature that fires for one interpretable concept vs several unrelated ones. Raw neurons are mostly polysemantic. SAE features are more often monosemantic. [Bricken et al 2023]

Neighbor graph (kNN graph) A graph linking each vector to its k nearest neighbors. It is the basic object for "adjacency mining": community detection, bridges, hubs. Watch for hubness: in high dimensions a few points are everyone's neighbor.

Network dissection Score each unit (CNN/ViT) by how well its activation map overlaps with labeled concept segmentations. [Bau et al 2017] This is the vision ancestor of automated neuron labeling.

PCA (Principal Component Analysis) An orthogonal linear projection onto the directions of maximal variance. Here it is used to find and remove the dominant shared directions (anisotropy), and as a first look at the data.

Platonic representation hypothesis The conjecture that models trained on different data and modalities converge toward a shared statistical model of reality, so their representations align more as scale grows. [Huh et al 2024] vec2vec gives supporting evidence: it translates between text-embedding spaces without paired data. [Jha et al 2025]

Pooling Collapsing per-token vectors to one vector: mean, last-token, CLS, max. The choice changes what the vector encodes. For decoder-only LLMs, last-token pooling is common; mean pooling mixes in the prompt. llama.cpp: --pooling {none,mean,cls,last,rank}.

Representation engineering (RepE) "Top-down" transparency. Read population-level directions for concepts like honesty or emotion, then steer with them. [Zou et al 2023]

Representational similarity (SVCCA / CKA / RSA) Families of scores comparing two representation spaces. SVCCA = Singular Vector Canonical Correlation Analysis [Raghu et al 2017]. RSA = Representational Similarity Analysis (from neuroscience).

Residual stream The running vector each transformer block reads from and adds to. It is the main place to read and write for probes, SAEs and steering.

SAE (Sparse Autoencoder) A wide autoencoder with a sparsity penalty (L1 = sum of absolute values, or TopK), trained to reconstruct activations. Its latent units act as candidate features. [Cunningham et al 2023; Bricken et al 2023; Gao et al 2024; Templeton et al 2024] Limits: reconstruction error (the "dark matter"); feature splitting depends on dictionary size; and weak downstream wins (see Section B6).

Superposition A network represents more features than it has dimensions by using nearly orthogonal directions, which works when features are sparse. [Elhage et al 2022, "Toy Models"] This is the main reason neurons look polysemantic and the motivation for SAEs.

t-SNE (t-distributed Stochastic Neighbor Embedding) A nonlinear 2D projection preserving local neighborhoods. [van der Maaten & Hinton 2008] Cluster sizes and gaps between clusters are not meaningful.

Transcoder Like an SAE, but it maps an MLP's input to that MLP's output through a sparse bottleneck. This makes circuits through MLPs readable. [Dunefsky et al 2024]

UMAP (Uniform Manifold Approximation and Projection) A graph-based nonlinear projection, faster than t-SNE, with somewhat better global structure. The same caveats apply. [McInnes et al 2018]

Whitening A linear transform that makes the covariance the identity: center, then rotate and scale by the inverse square root of the covariance. It is a cheap fix for anisotropy before neighbor search.


B. LITERATURE (by method family)

B1. Static embedding geometry (pre-transformer roots) Mikolov et al 2013, word2vec, and vector offsets / analogies https://arxiv.org/abs/1301.3781 Bolukbasi et al 2016, a gender direction found in word2vec, and debiasing by projecting it out https://arxiv.org/abs/1607.06520 Mu & Viswanath 2017, "All-but-the-Top": remove the mean and the top PCs https://arxiv.org/abs/1702.01417

B2. Contextual geometry, anisotropy, outliers Ethayarajh 2019, the geometry of BERT/ELMo/GPT-2 https://arxiv.org/abs/1909.00512 Sun et al 2024, massive activations in LLMs https://arxiv.org/abs/2402.17762 Darcet et al 2023, ViTs need registers (high-norm artifact tokens) https://arxiv.org/abs/2309.16588

B3. Probing (supervised readout) Alain & Bengio 2016, linear classifier probes https://arxiv.org/abs/1610.01644 Hewitt & Liang 2019, control tasks / selectivity https://arxiv.org/abs/1909.03368 Belinkov 2021, a survey of probing, including its shortcomings https://arxiv.org/abs/2102.12452 Kim et al 2017, TCAV https://arxiv.org/abs/1711.11279 Gurnee & Tegmark 2023, linear probes recover space and time https://arxiv.org/abs/2310.02207 Marks & Tegmark 2023, geometry of truth https://arxiv.org/abs/2310.06824 Burns et al 2022, CCS (unsupervised) https://arxiv.org/abs/2212.03827

B4. Linearity: theory and counterexamples Park, Choe, Veitch 2023, linear representation hypothesis https://arxiv.org/abs/2311.03658 Engels et al 2024, not all features are one-dimensionally linear https://arxiv.org/abs/2405.14860 Elhage et al 2022, Toy Models of Superposition https://transformer-circuits.pub/2022/toy_model/index.html

B5. Sparse dictionary methods (the "what are the blobs" answer) Bricken et al 2023, Towards Monosemanticity https://transformer-circuits.pub/2023/monosemantic-features/index.html Cunningham et al 2023, SAEs find interpretable features https://arxiv.org/abs/2309.08600 Templeton et al 2024, Scaling Monosemanticity (Claude 3 Sonnet) https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html Gao et al 2024, scaling and evaluating SAEs (TopK) https://arxiv.org/abs/2406.04093 Lieberum et al 2024, Gemma Scope (open SAEs on Gemma 2) https://arxiv.org/abs/2408.05147 Dunefsky et al 2024, transcoders https://arxiv.org/abs/2406.11944 Anthropic 2024, crosscoders https://transformer-circuits.pub/2024/crosscoders/index.html Anthropic 2025, attribution graphs / circuit tracing (methods) https://transformer-circuits.pub/2025/attribution-graphs/methods.html

B6. Critiques and negative results (read these before investing) Kantamneni et al 2025, SAE features vs plain probes on sparse probing: SAEs do not reliably win https://arxiv.org/abs/2502.16681 Wu et al 2025, AxBench: simple baselines beat SAEs at steering https://arxiv.org/abs/2501.17148 Heap et al 2025, auto-interp metrics do not distinguish trained from randomly initialized transformers https://arxiv.org/abs/2501.17727 GDM mech-interp team 2025, negative SAE results, SAE work deprioritized https://www.alignmentforum.org/posts/4uXCAJNuPKtKBsi28/sae-progress-update-2-draft Sharkey et al 2025, Open Problems in Mechanistic Interpretability https://arxiv.org/abs/2501.16496 Bereska & Gavves 2024, a review of mechanistic interpretability https://arxiv.org/abs/2404.14082

B7. Steering / representation engineering Turner et al 2023, activation addition https://arxiv.org/abs/2308.10248 Zou et al 2023, Representation Engineering https://arxiv.org/abs/2310.01405

B8. Layer-wise readout nostalgebraist 2020, logit lens https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens Belrose et al 2023, tuned lens https://arxiv.org/abs/2303.08112

B9. Cross-model comparison and convergence Raghu et al 2017, SVCCA https://arxiv.org/abs/1706.05806 Kornblith et al 2019, CKA https://arxiv.org/abs/1905.00414 Huh et al 2024, Platonic Representation Hypothesis https://arxiv.org/abs/2405.07987 Jha et al 2025, vec2vec: unsupervised translation between embedding spaces https://arxiv.org/abs/2505.12540

B10. Inversion (what an embedding gives away) Morris et al 2023, vec2text: text embeddings reveal (almost) as much as text https://arxiv.org/abs/2310.06816

B11. Vision / non-LLM (darkblueb's ViT point) Bau et al 2017, Network Dissection https://arxiv.org/abs/1704.05796 Olah et al 2017, Feature Visualization (Distill) https://distill.pub/2017/feature-visualization/ Goh et al 2021, multimodal neurons in CLIP (Distill) https://distill.pub/2021/multimodal-neurons/ Radford et al 2021, CLIP https://arxiv.org/abs/2103.00020 Caron et al 2021, DINO: self-supervised ViT features segment objects without labels https://arxiv.org/abs/2104.14294

B12. Projection / visualization methods van der Maaten & Hinton 2008, t-SNE (JMLR) https://jmlr.org/papers/v9/vandermaaten08a.html McInnes et al 2018, UMAP https://arxiv.org/abs/1802.03426 Smilkov et al 2016, Embedding Projector https://arxiv.org/abs/1611.05469


C. TOOLING (landing pages returned 200 on 2026-09-29; status can change)

Neuronpedia: browse/search SAE features, steering, attribution graphs https://www.neuronpedia.org/ Gemma Scope weights (HF) https://huggingface.co/google/gemma-scope SAELens: train and load SAEs https://github.com/jbloomAus/SAELens TransformerLens: hooks into HF-style models (PyTorch) https://github.com/TransformerLensOrg/TransformerLens nnsight: intervention API, including remote large models via NDIF https://github.com/ndif-team/nnsight repeng: control vectors in a few lines https://github.com/vgel/repeng llama.cpp cvector-generator (PCA / mean-diff control vectors to GGUF) https://github.com/ggml-org/llama.cpp/tree/master/tools/cvector-generator llama.cpp issue #29361 (expose last-token hidden state) ggml-org/llama.cpp#29361 Embedding Projector (browser PCA / t-SNE / UMAP) https://projector.tensorflow.org/ Latent Scope: embed, UMAP, cluster and label your own dataset locally https://github.com/enjalot/latent-scope LIT (Learning Interpretability Tool), Google PAIR https://github.com/pair-code/lit

qwen3.6-35b q8 

: These resources all revolve around Interpretability, Mechanistic Interpretability, and Control Vectors in Large Language Models (LLMs). They serve different stages of the workflow: from infrastructure to analysis to application.

1. The "Engine Room": Model Execution & Intervention

These tools allow you to run models and intervene in their internal states (activations). This is the foundational layer.

  • TransformerLens: A PyTorch library specifically designed for mechanistic interpretability. It provides easy hooks into model internals, making it the go-to for researchers who want to inspect activations during forward passes.
  • nnsight: Offers a more abstract "intervention API." Its key differentiator is support for remote execution (via NDIF), allowing you to run interventions on very large models without needing local GPU memory.
  • llama.cpp cvector-generator: A specialized tool within the llama.cpp ecosystem for generating control vectors using PCA or mean-difference methods directly on GGUF quantized models.

2. The "Lens": Visualization & Exploration

These tools help you see what’s happening inside the model by projecting high-dimensional data into 2D/3D spaces.

  • Embedding Projector: A classic browser-based tool for visualizing embeddings using PCA, t-SNE, or UMAP. Great for quick, interactive exploration of static datasets.
  • Latent Scope: A more modern, local-first alternative to the Embedding Projector. It allows you to embed, cluster, and label your own dataset locally, offering more privacy and customization for specific interpretability tasks.

3. The "Output": Control Vectors & Specific Applications

These are focused on using internal representations to steer model behavior or analyze specific phenomena.

  • repeng: A lightweight library designed specifically for creating and applying control vectors with minimal code. It’s ideal if you want to quickly test steering behaviors (e.g., making a model more truthful or less verbose).
  • SAELens & Hugging Face Gemma Scope: These are tightly coupled. Gemma Scope provides pre-trained Sparse Autoencoders (SAEs) for the Gemma models, while SAELens is the tool to train and load these SAEs. This group represents a more advanced interpretability technique (feature extraction via sparsity) rather than just raw activation inspection.

4. The "Gap": Infrastructure & Feature Access

This item highlights a current limitation or need in the ecosystem.

  • llama.cpp issue #29361: This isn’t a tool but an issue requesting exposure of the last-token hidden state. It’s relevant because many control vector and interpretability techniques require access to specific internal states (like the final layer’s output) that may not be easily exposed in all inference engines. This points to ongoing development needs in the llama.cpp ecosystem for interpretability workflows.

Suggested Workflow for Exploration:

  1. Start with TransformerLens or nnsight if you want to understand how to hook into models and inspect activations.
  2. Use Latent Scope or Embedding Projector to visualize the data you’re extracting.
  3. Try repeng if you want to quickly apply control vectors to steer model outputs.
  4. Explore SAEs (SAELens + Gemma Scope) if you’re interested in advanced feature extraction and sparse representations.
  5. Keep an eye on llama.cpp issues if you plan to run these workflows on quantized models and need specific internal state access.

This grouping is loose because many tools overlap (e.g., TransformerLens can be used with control vectors, and SAEs can be visualized in Latent Scope). The key distinction is the primary purpose of each resource: execution, visualization, application, or infrastructure.


D. CONTRAST: INDIRECT METHODS

Distillation: train a student on the teacher's outputs (soft labels). Knowledge moves, but you never see where it lived. Hinton, Vinyals, Dean 2015: https://arxiv.org/abs/1503.02531 Other indirect methods: prompting / eliciting, behavioral benchmarks, red-teaming, and output-only attribution (e.g. LIME/SHAP on inputs). Grey zone: probing is "direct" (it reads activations) but the probe is itself a trained model, so some claims come from the probe rather than from the model. Steering/patching ARE direct AND causal. Prefer them when making claims about what the model uses.


E. Summary of Tools by Task

Goal Primary Tool Key Step
Steer Behavior (e.g., tone, style) llama-server + repeng / cvector-generator Compute mean-difference vectors from contrasting prompts.
Visualize Activations Latent Scope / Embedding Projector Use UMAP/t-SNE on PCA-reduced activations for 2D plots.
Interpret Features Gemma Scope / Neuronpedia Inspect pre-trained SAE features without training.
Train Custom SAEs SAELens + TransformerLens/nnsight Dump activations from PyTorch model, then train in SAELens.
@adlai

adlai commented Oct 19, 2026

Copy link
Copy Markdown

again, thank you for bothering to make the artifact available via infrastructure that encourages editorial work...

perhaps, wisdom beyond my current coffee might eventually edit and refine, rather than merely triggering more slop first drafts!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment