Anthropic Gen-5 quality regression: reproducible BullshitBench + HN measurements (Fable 5 / Opus 5 / Sonnet 5)

README.md

Anthropic Gen-5 Quality Regression — Reproducible Measurements

Measurement scripts for the quality report on Claude generation 5 (Fable 5, Opus 5, Sonnet 5): worse nonsense detection, ~2x verbosity at identical reasoning effort (token-level), under-disclosed model fallback (Fable 5 → Opus 4.8).

Author: KeilerHirsch — report & full writeup: https://github.com/KeilerHirsch/ai-trinity (docs archive) Issues: anthropics/claude-code#83510 · anthropics/claude-code#83795 Dataset: https://github.com/petergpt/bullshit-benchmark (data/latest + data/v2/latest, HEAD 1699443, 2026-08-04)

Metrics

• detect (strict, used below): share of non-control responses with consensus_score >= 2 (mean of three judges; all three marked the nonsense as identified). Computed from aggregate.jsonl.
• green_rate (official leaderboard): counts a response as detected when ≥ 2 of 3 judges gave score 2 — slightly more lenient, yields higher numbers. Both metrics lead to the same conclusion; the tables below use the strict metric, the official values are in the issue body.
• engage: share with consensus_score == 0.
• Refusals are reported separately, not counted as detected or engaged. v1 (n=55): all models refused 0. v2 (n=100): Fable 5 refuses 33–37 %, Opus 5 9–11 %, Sonnet 5 ~1 % of nonsense prompts.

How to reproduce

git clone --depth 1 https://github.com/petergpt/bullshit-benchmark
cd bullshit-benchmark   # or run the scripts from any directory and set BS_BASE*

python3 messprotokoll_bullshitbench.py   # v1: Opus/Sonnet 4.6+4.8 vs Opus 5/Sonnet 5 (n=55/model)
python3 messprotokoll_v2.py              # v2: incl. Fable 5 (n=100/model)
python3 hn_comment_metric.py             # HN complaint-volume proxy (Algolia nbHits)

Paths: BS_BASE (v1) / BS_BASE_V2 (v2) point at the data/latest / data/v2/latest directories; defaults are relative to a bullshit-benchmark clone in the current directory.

Core results (v1/v2, strict metric, 95 % Wilson CI)

Model	detect	engage	median output tokens	median latency
Opus 4.8 @none	0.945 / 0.940	0.000 / 0.010	550 / 716	10.7 s / 12.7 s
Sonnet 4.6 @high/none	0.873 / 0.840	0.000 / 0.020	644 / 346	15.4 s / 9.6 s
Opus 4.6 @high	0.855 / 0.830	0.036 / 0.020	561 / 853	15.3 s / 20.5 s
Sonnet 5 @low	0.600 / 0.740	0.091 / 0.060	590 / 734	10.8 s / 11.6 s
Sonnet 5 @max	0.509 / 0.660	0.109 / 0.010	4,941 / 8,259	58.8 s / 97.0 s
Opus 5 @xhigh	0.491 / 0.600	0.145 / 0.020	2,478 / 3,068	40.1 s / 46.8 s
Fable 5 @low / @xhigh (v2)	0.410 / 0.470	0.030 / 0.020	860 / 920	15.3 s / 16.4 s

Grouped v2: Fable 5 0.440 [0.351–0.533] < Opus 5 + Sonnet 5 0.665 [0.600–0.724] < gen 4.5–4.8 0.768 [0.735–0.799]. Verbosity at zero reasoning (token-level): Opus 5 +107 % vs Opus 4.8, Sonnet 5 +84 % vs Sonnet 4.6. Tokenizer correction (gen-5 tokenizer tokenizes identical content ~30 % higher, official migration guide): word-level Δ ≈ +60 % (Opus 5) / +42 % (Sonnet 5).

Reference numbers, official green_rate (leaderboard.csv, 2026-08-04): Opus 4.8 @none 0.964 / 0.95, Sonnet 4.6 @high 0.927 / 0.91, Opus 5 @xhigh 0.582 / 0.70, Fable 5 @low — / 0.44 (refusal 0.37).

Second review (2026-08-05) — calibration corrections

Applied to the issues and mirrored here (details: factcheck-second-review-2026-08-05.md):

• "83.3 % → 68.3 % audit accuracy" pair removed — not in The Register's AMD-audit article (full text checked).
• "silent rerouting" calibrated to "under-disclosed model fallback": officially documented as Automatic Model Fallback (code.claude.com/docs/en/model-config, v2.1.219+), notice in transcript, session continues on the fallback model; before v2.1.219 every flagged Fable 5 request re-ran on the provider's default Opus model.
• Verbosity: token-level deltas (+107 %/+84 %) now reported alongside tokenizer-corrected word-level deltas.
• Grouped CIs: pool all effort levels per generation (v1: Gen 5 n=220, Gen 4.6/4.8 n=330) — effort-mixed, descriptive indicator; per-model rows are primary evidence.
• Endor: 59.8 %/19.0 % verified; "38/200 confirmed cheating" not verifiable on public pages (marked as such).
• Counter-evidence handled: MASK sycophancy (Sonnet 5 3.1 % vs Opus 4.8 6.1 %, Transparency Hub) is a different construct than false-premise detection and does not refute the BullshitBench finding.

factcheck-second-review-2026-08-05.md

Second-review fact-check (2026-08-05)

Independent second review of the Gen-5 quality report (issues #83510 / #83795) against primary sources. Sources verified with retrieval dates; verdicts follow the "measure, don't assume" method.

1. "Audit accuracy 83.3 % → 68.3 %" — REFUTED as attributed

• Claim: the pair originates from the AMD AI director's audit as reported by The Register (2026-04-06).
• Check (2026-08-05): full text of theregister.com/software/2026/04/06/claude_code_has_become_dumber_lazier_amd_director/5228799 contains 6,852 sessions / 234,760 tool calls / 17,871 thinking blocks, stop-hook violations 0 → ~10/day after 2026-03-08, file reads 6.6 → 2.0 — but NOT the pair "83.3 % → 68.3 %".
• Action: pair removed from issue #83510 and this archive; verified figures kept.

2. "Silent rerouting" — CALIBRATED to "under-disclosed model fallback"

• Official docs (code.claude.com/docs/en/model-config, 2026-08-05): Automatic Model Fallback (v2.1.219+) re-runs classifier-flagged requests on a fallback model (Fable 5 cybersecurity → Opus 4.8; biology → Opus 5; Opus 5 cybersecurity → Opus 4.8), shows a notice in the transcript, session continues on the fallback model. Before v2.1.219: every flagged Fable 5 request re-ran on the provider's default Opus model.
• Practice evidence stands: HN 48473516 (users perceive it as silent), BridgeMind/TechTimes 2026-07-02 (−70 % debugging score on rerouted tasks). No hook event, no machine-readable audit field.
• Action: wording in issue title/body, gist and archive updated.

3. Verbosity +107 %/+84 % — TOKENIZER-CORRECTED

• Official migration guide (platform.claude.com/docs/en/about-claude/models/migration-guide, 2026-08-05): gen-5 tokenizer (introduced with Opus 4.7) tokenizes identical content ~30 % higher than pre-Opus-4.7.
• Word-level Δ ≈ +60 % (Opus 5 @low 1,141 vs 550) and ≈ +42 % (Sonnet 5 @low 590 vs 321).
• Action: both figures reported; token-level deltas no longer presented as word-level verbosity.

4. Grouped CIs — CLARIFIED (not an error)

• The grouped v1 CIs (Gen 5 0.523 [0.457–0.588]; Gen 4.6/4.8 0.870 [0.829–0.902]) reproduce exactly when pooling all measured effort levels per generation (v1: Gen 5 n=220 = Opus5+Sonnet5 × 2 efforts × 55; Gen 4.6/4.8 n=330 = 3 models × 2 efforts × 55). Wilson formula checked.
• Action: documented as effort-mixed descriptive indicator; per-model rows remain primary evidence.

5. Endor Labs — TIGHTENED

• Verified (2026-08-05): public leaderboard (endorlabs.com/research/ai-code-security-benchmark) shows Claude Code + Fable 5 = 59.8 % FuncPass / 19.0 % SecPass on 200 tasks, 2026-06-10.
• Not verifiable on public pages: "38/200 confirmed cheating (33× training memorization)", "15 timeouts", "zero safety refusals". Endor's anti-cheating article documents memorization/training recall as the dominant confirmed cheating mechanism overall (137/182 cases) without per-model numbers.
• Action: issue #83510 marks these as third-party reporting / not verifiable.

6. Counter-evidence handled (MASK)

• Transparency Hub (anthropic.com/transparency, 2026-08-05): Sonnet 5 strongest Claude on MASK sycophantic dishonesty, lying rate 3.1 % (Opus 4.8: 6.1 %).
• Construct distinction: MASK measures sycophantic lying; BullshitBench measures false-premise detection. The vendor numbers do not refute the nonsense-detection finding.

7. BullshitBench v2 leaderboard (cross-check)

• petergpt/bullshit-benchmark, data/v2/latest/leaderboard.csv (state 2026-07-31): 192 rows; Gen-5 rows present: Fable 5 @low green 0.44 / refusal 0.37; Opus 5 @xhigh green 0.70 / refusal 0.11; Opus 4.8 0.94–0.95. Refusal rates match the report exactly; strict-metric detect values are the report's own computation from the pinned data state (HEAD 1699443) and are reproducible via messprotokoll_*.py.

Net effect

Core findings unchanged: Gen 5 nonsense detection ~0.3–0.4 below Gen 4.6/4.8, verbosity direction confirmed (magnitudes tokenizer-adjusted), model fallback documented-but-under-disclosed, model pinning still broken via 4 measured vectors. Two claims removed/softened (83.3→68.3 pair; Endor memorization counts).

hn_comment_metric.py

#!/usr/bin/env python3
"""HN complaint-volume metric (reproducible): hits per month for fixed query pairs via Algolia nbHits."""
import json, time, urllib.request, urllib.parse, datetime as dt

def nb(query, ts0, ts1, tag='comment'):
    u = ('https://hn.algolia.com/api/v1/search?query=' + urllib.parse.quote(query)
         + f'&tags={tag}&hitsPerPage=1'
         + f'&numericFilters=created_at_i>={ts0},created_at_i<{ts1}')
    req = urllib.request.Request(u, headers={'User-Agent':'Mozilla/5.0'})
    return json.loads(urllib.request.urlopen(req, timeout=30).read().decode())['nbHits']

PAIRS = [
    ('Claude verbose', 'ChatGPT verbose'),
    ('Claude dumber', 'ChatGPT dumber'),
    ('Claude refuses', 'ChatGPT refuses'),
    ('Claude slop', 'ChatGPT slop'),
    ('Claude quality', 'ChatGPT quality'),
]
months = ['2025-01','2025-05','2025-09','2026-01','2026-02','2026-03','2026-04','2026-05','2026-06','2026-07']
print('Monat    ', *[f"{p[0].split()[1]:>9s}/{p[1].split()[1]:>9s}" for p in PAIRS])
for ms in months:
    m = dt.date.fromisoformat(ms + '-01')
    nxt = (m.replace(day=28) + dt.timedelta(days=7)).replace(day=1)
    ts0, ts1 = int(dt.datetime(*m.timetuple()[:3]).timestamp()), int(dt.datetime(*nxt.timetuple()[:3]).timestamp())
    row = []
    for cq, gq in PAIRS:
        try:
            c = nb(cq, ts0, ts1); g = nb(gq, ts0, ts1)
        except Exception as e:
            c, g = '?', '?'; print('ERR', ms, cq, e)
        row.append(f'{c:4d}/{g:4d}')
        time.sleep(0.35)
    print(ms, *row)

messprotokoll_bullshitbench.py

#!/usr/bin/env python3
"""
Measurement protocol: BullshitBench raw data -> measurable facts on Anthropic model quality.
Source: github.com/petergpt/bullshit-benchmark, data/latest/{responses,aggregate}.jsonl
State: 2026-08-04 (repo HEAD 1699443)
Reproduce: python3 messprotokoll_bullshitbench.py
Metric: detect (strict) = share of non-control responses with consensus_score >= 2
        (all three judges marked the nonsense as identified); engage = consensus_score == 0.
"""
import json, math, sys
from statistics import median, mean
from collections import defaultdict

import os
BASE = os.environ.get('BS_BASE', './bullshit-benchmark/data/latest')

def wilson(k, n, z=1.96):
    if n == 0:
        return (float('nan'), float('nan'))
    p = k / n
    denom = 1 + z*z/n
    centre = (p + z*z/(2*n)) / denom
    half = z * math.sqrt(p*(1-p)/n + z*z/(4*n*n)) / denom
    return (centre - half, centre + half)

# --- 1) Laden & Join ---
agg = {}
with open(f'{BASE}/aggregate.jsonl') as f:
    for l in f:
        r = json.loads(l)
        agg[r['sample_id']] = r

resp = {}
with open(f'{BASE}/responses.jsonl') as f:
    for l in f:
        r = json.loads(l)
        resp[r['sample_id']] = r

models = sorted({r['model'] for r in agg.values() if r['model'].startswith('anthropic/')})
models = [m for m in models if any(x in m for x in ['opus-4.8','opus-4.7','opus-4.6','sonnet-4.6','sonnet-5','opus-5','haiku-4.5','sonnet-4.5'])]
models.sort()

print('='*100)
print('MEASUREMENT 1: ERROR/GULLIBILITY RATE (non-control questions, n per model)')
print('detect = share of responses that identify the nonsense (consensus_score >= 2)')
print('engage = share of responses that fully engage with the nonsense (consensus_score == 0)')
print('='*100)
print(f"{'Modell':48s} {'n':>4s} {'detect':>7s} {'95%KI':>16s} {'engage':>7s} {'95%KI':>16s}")
res_detect = {}
for m in models:
    rows = [agg[s] for s in agg if agg[s]['model'] == m and not agg[s]['is_control']]
    rows = [r for r in rows if r.get('status') == 'ok']
    n = len(rows)
    k2 = sum(1 for r in rows if r.get('consensus_score', 0) >= 2)
    k0 = sum(1 for r in rows if r.get('consensus_score', 0) == 0)
    if n:
        lo, hi = wilson(k2, n)
        lo0, hi0 = wilson(k0, n)
        res_detect[m] = (k2/n, n)
        print(f"{m:48s} {n:4d} {k2/n:7.3f} [{lo:6.3f},{hi:6.3f}] {k0/n:7.3f} [{lo0:6.3f},{hi0:6.3f}]")

print()
print('='*100)
print('MEASUREMENT 2: VERBOSITY & THINKING EFFORT (all questions, identical prompts)')
print('completion_tokens = response tokens incl. reasoning; reasoning_tokens = thinking tokens')
print('visible = completion - reasoning (only where both fields exist)')
print('='*100)
print(f"{'Modell':48s} {'n':>4s} {'completion':>10s} {'reasoning':>10s} {'sichtbar':>9s} {'Latenz_ms':>9s} {'$':>8s}")
res_verb = {}
for m in models:
    rows = [resp[s] for s in resp if resp[s]['model'] == m and resp[s].get('status') == 'ok']
    n = len(rows)
    if not n:
        continue
    comp = [r['response_completion_tokens'] for r in rows if r.get('response_completion_tokens') is not None]
    reas = [r['response_reasoning_tokens'] for r in rows if r.get('response_reasoning_tokens') is not None]
    vis = [max(0, r['response_completion_tokens'] - (r['response_reasoning_tokens'] or 0)) for r in rows if r.get('response_completion_tokens') is not None]
    lat = [r['response_latency_ms'] for r in rows if r.get('response_latency_ms') is not None]
    cost = [r['response_cost_usd'] for r in rows if r.get('response_cost_usd') is not None]
    res_verb[m] = (median(comp), n)
    print(f"{m:48s} {n:4d} {median(comp):10.0f} {median(reas):10.0f} {median(vis):9.0f} {median(lat):9.0f} {sum(cost)/n:8.4f}")

print()
print('='*100)
print('MEASUREMENT 3: GROUP COMPARISON (newest Gen 5 vs. last Gen 4.6/4.8)')
print('='*100)
gen5 = [m for m in res_detect if m.endswith('@reasoning=low') or m.endswith('@reasoning=xhigh') or m.endswith('@reasoning=max') or m.endswith('@reasoning=high')]
gen4 = [m for m in res_detect if '4.6' in m or '4.8' in m]
def agg_group(models_sel, key='detect'):
    ks, ns = [], []
    for m in models_sel:
        d, n = res_detect[m]
        ks.append(d*n); ns.append(n)
    k = sum(ks); n = sum(ns)
    lo, hi = wilson(k, n)
    return k/n, lo, hi, n
for label, sel in [('Gen.5 (Opus5+Sonnet5, alle Effort-Level)', [m for m in res_detect if ('opus-5' in m or 'sonnet-5' in m)]),
                   ('Gen.4.6/4.8 (Opus4.8+Opus4.6+Sonnet4.6)', gen4)]:
    p, lo, hi, n = agg_group(sel)
    print(f"{label:50s} detect={p:.3f} [{lo:.3f},{hi:.3f}]  n={n}")

messprotokoll_v2.py

#!/usr/bin/env python3
"""Measurement of the v2 dataset (incl. Fable 5). Strict metric: detect = consensus_score == 2, engage = consensus_score == 0."""
import json, math
from statistics import median
import os
BASE = os.environ.get('BS_BASE_V2', './bullshit-benchmark/data/v2/latest')
def wilson(k, n, z=1.96):
    if n == 0: return (float('nan'), float('nan'))
    p = k/n
    denom = 1 + z*z/n
    centre = (p + z*z/(2*n))/denom
    half = z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))/denom
    return centre-half, centre+half

agg = [json.loads(l) for l in open(f'{BASE}/aggregate.jsonl')]
resp = [json.loads(l) for l in open(f'{BASE}/responses.jsonl')]
resp_by_sid = {r['sample_id']: r for r in resp}
models = sorted({r['model'] for r in agg if r['model'].startswith('anthropic/') and any(x in r['model'] for x in ['fable','opus-5','sonnet-5','opus-4.8','opus-4.7','opus-4.6','sonnet-4.6','haiku-4.5','opus-4.5','sonnet-4.5'])})
print(f"{'Modell':50s} {'n':>4s} {'detect':>7s} {'95%KI':>16s} {'engage':>7s} {'comp':>6s} {'reas':>6s} {'sicht':>6s} {'lat':>7s}")
rows_out = []
for m in models:
    rows = [r for r in agg if r['model'] == m and not r['is_control']]
    n = len(rows)
    if not n: continue
    k2 = sum(1 for r in rows if r.get('consensus_score') == 2)
    k0 = sum(1 for r in rows if r.get('consensus_score') == 0)
    lo, hi = wilson(k2, n); lo0, hi0 = wilson(k0, n)
    rs = [resp_by_sid[r['sample_id']] for r in rows if r['sample_id'] in resp_by_sid]
    comp = median([r['response_completion_tokens'] for r in rs if r.get('response_completion_tokens') is not None] or [0])
    reas = median([r['response_reasoning_tokens'] for r in rs if r.get('response_reasoning_tokens') is not None] or [0])
    vis = median([max(0, r['response_completion_tokens'] - (r['response_reasoning_tokens'] or 0)) for r in rs if r.get('response_completion_tokens') is not None] or [0])
    lat = median([r['response_latency_ms'] for r in rs if r.get('response_latency_ms') is not None] or [0])
    print(f"{m:50s} {n:4d} {k2/n:7.3f} [{lo:6.3f},{hi:6.3f}] {k0/n:7.3f} [{lo0:6.3f},{hi0:6.3f}] {comp:6.0f} {reas:6.0f} {vis:6.0f} {lat:7.0f}")
    rows_out.append((m, k2/n, lo, hi, k0/n, comp, reas, vis, lat))

print()
print('Gruppenvergleich v2:')
def grp(pred):
    ks = ns = 0
    for m, det, lo, hi, eng, comp, reas, vis, lat in rows_out:
        if pred(m):
            ks += det * 55; ns += 55
    if ns == 0: return
    lo, hi = wilson(ks, ns)
    print(f"  detect={ks/ns:.3f} [{lo:.3f},{hi:.3f}] n={ns}")
grp(lambda m: 'fable-5' in m)
grp(lambda m: 'opus-5' in m or 'sonnet-5' in m)
grp(lambda m: '4.6' in m or '4.8' in m or '4.5' in m)