intuition-probe: let an LLM hallucinate the API/UX it expects, then conform your design to the guess. Companion to 'Complicated Thoughts On LLMs' (theocharis.dev)

intuition-probe.md

name

intuition-probe

description

Use when you want to know what shape a developer or user reaches for when they first meet your API, config format, CLI, or UI, so you can make your design that shape. Spawns a blind agent that commits the design it EXPECTS before seeing the real thing, then reports the shape they reached for and conform-first recommendations (make the API whatever the guess is). Triggers on "is this intuitive", "would a developer guess this", "intuition probe", "blind-test this API/config/CLI", "test the DX/UX of", "what would someone expect here", "make the API whatever the LLM guesses".

Intuition Probe

Discover the shape a developer (or an LLM) reaches for when they meet your API, config, CLI, or UI for the first time, then make your design that shape. An agent guesses how to use the artifact blind, before it sees the real thing. The guess is not an error to grade; it is a design proposal. Where the guess diverges from your API, the default move is to conform the API to the guess, because the guess is what feels familiar, and familiar is what gets adopted.

This follows the Jazz principle "make the API whatever the LLM guesses": you only hear from the few users who push through an unfamiliar design, but an LLM shows you what people reach for cheaply and at volume. See references/going-deeper.md for the theory, the conform-first default, and when NOT to conform.

What it measures (and the honest limit)

It surfaces what the model's prior reaches for, the best cheap proxy for what is familiar to a developer. At the default N=1 it surfaces CANDIDATES, it does not CONFIRM: a confident divergent guess is a candidate shape to conform to, but a single draw cannot prove it is the familiar default, and a single match does not prove your API is familiar. Dial N up; only convergence across agents confirms a shape worth conforming to.

Procedure

Step order matters: freeze prompt (3) before reading the artifact (4) before spawning agents (5). Reversing 3 and 4 contaminates the blind test.

1. Gather inputs

Ask the user for: the artifact under test (a file path, package name, CLI command, or URL -- e.g. plugins/foo/SKILL.md or git commit), a rough goal, an optional read-set (a README, a docs URL, or a snippet), and N (default 1). If the artifact is a path/package/command, note it; do not read it yet.

2. Sanitize the goal

Restate the rough goal as a pure outcome. Strip any leaked identifiers: method names, config keys, flags, exact labels. Show the user:

Sanitized goal: "" is that the task? (y / edit)

Wait for confirmation. The read-set is NEVER sanitized (a README is allowed to contain the answer; that's the doc-informed test).

3. Freeze the blind prompt (BEFORE reading the answer key)

Take the template in references/blind-agent-prompt.md and fill its placeholders (sanitized goal, system name, read-set if any) into a frozen working prompt that you hold in your own context. Do NOT edit the shipped template file. Also record N and the mode (cold / doc-informed) next to the frozen prompt so step 8 has them. Freeze the prompt now and do not revise it after step 4, so the real implementation cannot leak back into it. If the read-set is a URL, you (the orchestrator) fetch it now and inline its content into the frozen prompt; the blind agent is forbidden from opening URLs, so it must receive the doc text, never a bare link.

4. Read the answer key

Read the real surface from the repo: source/signatures (API), schema/examples (config), --help/command defs (CLI), component/route source (UI). If you cannot read it, ask the user to paste the real surface. Keep it private; it never goes into the blind prompt.

5. Run the blind agent(s)

Spawn N agents via the Agent tool, each with the frozen prompt text from step 3 as its prompt. Use subagent_type: general-purpose. Collect each agent's decision_points JSON. (For N=1, one call. For N>1, issue the Agent calls in parallel.)

6. Score

For each agent's guess, apply the references/scorer-rubric.md classification against the answer key, either inline (you, the orchestrator) or by spawning a scorer Agent. Produce the findings JSON. Each finding names the familiar anchor the agent reached for and a conform-first recommendation; the adoption signal is read from the agent's recorded confidence (and, at N>1, convergence). Do not invent signal strength from hindsight. If you spawn a separate scorer Agent rather than scoring inline, pass it three things: the agent's decision_points JSON, the answer key from step 4, and references/scorer-rubric.md.

7. (N>1) Cluster + cross-validate

Group findings by decision point across agents. Mark a finding priority only if at least 2 independent agents hit it. Single-agent findings are listed, de-prioritized.

8. Report

Render per references/report-format.md: lead with the shape the prior reached for (the candidate spec) and a conform-first recommendation per divergence. If N is 1, include the candidate banner and never report the design as a confirmed familiar default. Offer to save the report to a scratch/artifacts folder. Never write to the repo under test.

reference-blind-agent-prompt.md

Blind-agent prompt template

The orchestrator fills {{SANITIZED_GOAL}}, {{SYSTEM_NAME}}, and optionally {{READ_SET}}, then spawns this as an Agent. It MUST NOT include the real implementation, file paths, or exact identifier names of the artifact.

---

You are a developer about to use {{SYSTEM_NAME}} for the first time.

Your task: {{SANITIZED_GOAL}}

{{#if READ_SET}}You have this material available; read it, then proceed: {{READ_SET}}{{/if}}

Write the code/config/commands you EXPECT to work, from intuition and prior knowledge alone. This is a first-instinct capture.

RULES:

• Do NOT search for, grep, open, or read the real implementation or docs of {{SYSTEM_NAME}} (beyond any material quoted above). Write what you'd expect.
• Emit your full guess FIRST, before any exploration.
• For EACH distinct decision you make (which call to use, where logic goes, which key/flag, what shape), record your confidence honestly.
• Split decisions at each independent choice: which call to make, what arguments to pass, and what overall shape are separate decision points.
• For each decision, name the familiar anchor: the known API, library, or idiom you are pattern-matching to (e.g. "zod", "fetch()", "EventEmitter .on()"). Write "none" only if you truly reached for nothing familiar.

Output exactly this JSON, nothing else:

{
  "decision_points": [
    {
      "decision": "<short label, e.g. 'how to read current status'>",
      "guess": "<the exact code/config/command you'd write>",
      "familiar_anchor": "<the known API/library/idiom this resembles, or 'none'>",
      "confidence": "<high|medium|low>",
      "reasoning": "<one line: why you expect this>"
    }
  ],
  "wished_existed": ["<anything you reached for and expected to be there>"]
}

---

Why the familiar anchor and confidence matter

The scorer turns these into an adoption signal BEFORE the agent sees reality. A high-confidence guess anchored on a well-known idiom is the strongest evidence that the API should adopt that shape (the guess is the spec). Report the confidence you actually feel; do not defensively mark everything "low". Name the familiar anchor whenever your guess is really "the way X does it", because that is what makes the shape worth conforming to.

reference-going-deeper.md

Going deeper

Theory (the guess is the spec)

The model's prior is a compression of how APIs, configs, and CLIs are conventionally shaped, so what it reaches for is the best cheap proxy for what feels familiar to a developer. Familiar is what gets adopted. You normally only hear from the few users who push through an unfamiliar design; an LLM shows you what people reach for cheaply and at volume (it "fails" thousands of times a day). The move is to stop fighting the guess and make your API whatever the prior reaches for. Guesses are systematic and reproducible (see the slopsquatting research), so a convergent guess is a real signal, not noise. Caveat: capable agents are "too competent" and can reason around an unfamiliar design from first principles, so withhold the surface and capture the first instinct before exploration.

Modes

• Cold (no read-set): what they reach for with no help. The purest read on what is familiar.
• Doc-informed (read-set provided): whether your docs make the right shape familiar. Hand it a README or the docs site (the orchestrator fetches a URL and inlines its content; the agent never opens links itself); the agent reads it then guesses.
• Doc-swap converge loop: when you choose the teach fallback instead of conforming, this checks the teaching worked. README v1 produces a divergent guess, change one sentence, re-run with README v2, get the conforming guess. The implementation is never touched.

When NOT to conform (the fallback)

Conforming is the default, not a law. Keep your API and teach instead when the guessed shape is unsafe, violates a hard constraint, or would break a real invariant. Counter-example: a framework where a reachability check must not perform I/O. Agents may confidently reach for an I/O-in-the-read shape, but adopting it would break the contract, so the move is to keep the API and make the unfamiliar constraint loud in the docs (and use the doc-swap loop to confirm the teaching lands). A real instance of this fallback was a connection worker in one of our systems: agents put the network dial in the wrong place, and the fix was a one-sentence README change, not an API change. Treat the fallback as the exception you justify, not the reflex.

Deferred (not in v1)

• Diagnostic ladder: same task at 3 context tiers (blind / signatures-only / full-docs) to localize whether the gap is naming, shape, or docs.
• Multi-model / persona diversity: junior/senior/other-ecosystem agents.
• Live UI driving (Playwright): drive the real product instead of reading component source for the answer key.
• Auto-applying conform moves: the skill only ever recommends; it never edits the API under test.

reference-scorer-rubric.md

Scorer rubric

The scorer receives: the blind agent's decision_points JSON + the real surface (answer key). For each decision point, map the guess to the ONE bucket that names the adoption move it implies. The guess is a design proposal: be honest about divergences, and do not rationalize a divergence into a match just because you know the real design. When a decision sits between two buckets, pick the one that implies the larger change to the API; that is the more honest signal. Score every decision point: if one is a self-described hedge or a duplicate of another, fold it into the related finding and say so in that finding's recommendation, rather than dropping it silently. Also turn strong items from the guess's wished_existed list into missing-affordance (or hallucinated) findings.

Bucket	Meaning	Conform-first move (fallback: teach)
match	the API is already what they reached for	keep it
naming-mismatch	right shape, they reached for a different name	adopt their name (rename or add an alias)
shape-mismatch	they reached for a different structure or idiom	adopt their shape (reshape toward the familiar idiom). Fallback, only if a hard constraint or safety issue forbids it: keep the API and make the unfamiliar part loud in the docs
missing-affordance	they reached for something that does not exist	build what they reached for
hallucinated	they confidently invented a call/key that does not exist	build it (the strongest conform signal, the literal Jazz move). Decline only if it is wrong or unsafe

Adoption signal: how strong is the mandate to conform

Signal = reach strength x conformance value.

• Reach strength: at N=1, the agent's recorded confidence; at N>1, how many agents reached for the same shape (convergence).
• Conformance value: a guess anchored on a well-known idiom (named in familiar_anchor) is worth more than a one-off.

Read the base signal off the agent's confidence:

Bucket	confidence high	confidence medium	confidence low
match	(no finding)	(no finding)	(no finding)
naming-mismatch	MEDIUM	MEDIUM	LOW
missing-affordance	MEDIUM	LOW	LOW
hallucinated	HIGH	MEDIUM	LOW
shape-mismatch	HIGH	HIGH	MEDIUM

Bumps: raise one level if the guess names a well-known familiar_anchor (the language's standard library, or a widely used package or idiom, e.g. a top-500 npm package), or (at N>1) if at least 2 agents converged on it. shape-mismatch and hallucinated carry the most weight: adopting them changes what the API fundamentally is, which is exactly the high-value conform move.

Scorer output

Output exactly this JSON:

{
  "findings": [
    {
      "decision": "<label from the guess>",
      "expected": "<what the agent guessed>",
      "familiar_anchor": "<the known idiom they reached for, from the guess, or 'none'>",
      "reality": "<what the real surface actually is>",
      "bucket": "<one of: match | naming-mismatch | shape-mismatch | missing-affordance | hallucinated>",
      "signal": "<one of: HIGH | MEDIUM | LOW>",
      "recommendation": "<conform-first action: adopt their name/shape, or build what they reached for. Use the fallback (keep the API, teach via docs) ONLY when conforming is unsafe or blocked by a hard constraint, and say which constraint.>"
    }
  ]
}