Skip to content

Instantly share code, notes, and snippets.

@NovCog
Last active March 16, 2026 22:28
Show Gist options
  • Select an option

  • Save NovCog/d1b5c6c5d260fc36400d6d98d39ac879 to your computer and use it in GitHub Desktop.

Select an option

Save NovCog/d1b5c6c5d260fc36400d6d98d39ac879 to your computer and use it in GitHub Desktop.
Building an AI Memory System with Vector Embeddings — NovCog AI Practitioner Series by Guerin Green

Building an AI Memory System with Vector Embeddings

By Guerin Green / Novel Cognition AI Strategist | Federal Courthouse AI Presenter


Part of the NovCog AI Practitioner Series


Language models are stateless. Every conversation starts from zero. The model has its training data and whatever context you provide in the current session — nothing else. For practitioners who work across dozens of projects with interconnected methodologies, this statelessness is a fundamental limitation.

The NovCog Brain is an AI memory system built to solve this: a persistent, searchable knowledge base that feeds context to AI assistants via the Model Context Protocol (MCP). Here's how it works.

Architecture Overview

The system has four layers:

1. Capture layer — A mechanism to store thoughts, observations, discoveries, and methodology notes as they occur during work sessions. Each thought is a short text entry (typically 1-3 sentences) tagged with optional metadata.

2. Embedding layer — Each captured thought is converted into a dense vector embedding using an embedding model (e.g., OpenAI's text-embedding-3-small). The embedding captures the semantic meaning of the thought in a 1536-dimensional vector space.

3. Storage layer — Supabase (PostgreSQL with pgvector extension) stores both the raw text and its vector embedding. Supabase provides SQL querying for exact matches and pgvector for similarity search — covering both retrieval modes.

4. Retrieval layer — An MCP server exposes the memory system to AI assistants (Claude, etc.) as a tool. When the AI needs context about a previous project, methodology, or discovery, it queries the memory system and receives semantically relevant thoughts.

Semantic Search vs. Keyword Search

The critical advantage of vector-based memory over traditional notes or documentation:

Keyword search finds exact matches. If you stored a thought about "Googlebot verification via reverse DNS" and later search for "bot authentication method," keyword search returns nothing. The words don't match.

Semantic search finds meaning matches. The embeddings for "Googlebot verification via reverse DNS" and "bot authentication method" are close in vector space because they describe related concepts. A cosine similarity search returns the relevant thought even though zero words overlap.

This is the same principle behind RAG systems, applied to personal knowledge management. The embedding model bridges vocabulary gaps, synonym differences, and conceptual abstractions.

For a deeper explanation of how embeddings work, see the AI Glossary and the RAG Glossary.

Thought Capture Patterns

Effective AI memory isn't a knowledge base — it's a thought stream. The capture patterns that work best:

Discoveries during work. "Blogger posts are crawled within 3 minutes of publish via PubSubHubbub — faster than any other platform tested." This is a specific, contextualized observation that semantic search can surface when relevant.

Methodology decisions. "Per-page footer variation with 3 links per page across 6 pages distributes to 18 domains without looking over-optimized." This captures a specific technique with its rationale.

Cross-project connections. "The entity authority pattern from Project A also applies to Project B — the sameAs chain needs the same Wikidata link." This creates a semantic bridge between projects that the AI can traverse later.

Failure notes. "WordPress.com canonicalization interfered with outbound link crawl priority — switching to Blogger for crawl triggers." Negative results are as valuable as positive ones for avoiding repeated mistakes.

The capture should be atomic — one thought, one observation, one connection per entry. Dense, multi-topic entries get poor embeddings because the vector tries to average over too much semantic space (the same chunking problem that plagues RAG systems).

MCP Integration

The Model Context Protocol allows AI assistants to use external tools during conversation. The NovCog Brain MCP server exposes functions like:

  • capture_thought — Store a new observation with optional tags
  • search_thoughts — Semantic search across all stored thoughts
  • list_thoughts — Browse recent captures chronologically
  • thought_stats — Overview of the knowledge base (count, topics, recency)

When Claude (or another MCP-capable AI) encounters a question that might benefit from historical context, it can call search_thoughts with a natural language query. The server embeds the query, performs a cosine similarity search against stored thought embeddings, and returns the most relevant matches.

The AI then has context it wouldn't otherwise have — specific methodology details, previous project outcomes, discovered patterns — without the user needing to manually provide it.

Why This Matters for Practitioners

The combination of persistent memory, semantic search, and MCP integration means:

  • Methodology evolves cumulatively. Each session builds on all previous sessions, not just the current context window.
  • Cross-project learning is automatic. A discovery in one project surfaces when it's relevant to another.
  • Institutional knowledge is preserved. When working across dozens of properties in a DAN, the memory system tracks what's been done, what worked, and what needs attention.
  • The AI assistant becomes a genuine collaborator with access to shared context, not a stateless tool that starts fresh every time.

Building an AI memory system isn't technically difficult. Supabase, pgvector, and an embedding API are all accessible to anyone with basic development skills. The hard part is the discipline of capturing thoughts consistently and the judgment to capture the right granularity.


Series Navigation


Burstiness & Perplexity Community | Hidden State Drift | Novel Cognition

#hiddenstatedrift #burstinessandperplexity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment