guppy3 C Extension Analysis Report

guppy3_report.md

guppy3 C Extension Analysis Report

Executive Summary

guppy3 v3.1.6 (~14.5K lines of C) is a heap profiler for CPython. It necessarily accesses CPython internals (frame objects, GC lists, interpreter/thread state structures) and maintains version-conditional code paths for every CPython release from 3.10 through 3.14. The extension uses pythoncapi_compat.h and has excellent version compatibility engineering with zero unguarded new API usages.

However, the analysis uncovered significant bugs including a wrong-variable NULL check (classic typo), a deref-before-check in the bitset allocator, an unchecked PyObject_Realloc, a potential infinite loop on Python 3.14, and the xt_error sentinel pattern that silently loses error context across ~15 call sites. The set_relate function has a triple bug (iterator leak + obj leak + unchecked NULL to callback) on every successful set membership check.

Total confirmed findings: 30 FIX, 30 CONSIDER, 6 POLICY.

All 13 agents completed.

---

Project: guppy3 — CPython heap profiler Source: src/heapy/ (~7.9K lines) + src/sets/ (~5.8K lines) Version analyzed: 3.1.6 Agents completed: 13 + clang-tidy

---

Critical Findings (FIX) — 13

Crash Bugs (7)

#	Finding	File	Agents
1	ng_iter checks wrong variable — !v instead of !iter after PyObject_GC_New. Crashes on OOM during iter(nodegraph).	nodegraph.c:581-584	1 agent
2	mutbitset_initset deref before check — dereferences PyObject_Realloc result before NULL check. Check is dead code.	bitset.c:793-796	1 agent
3	immbitset_realloc unchecked PyObject_Realloc — dereferences result through PyObject_InitVar. Also corrupts object tracking via prior _Py_ForgetReference.	bitset.c:960-962	1 agent
4	hv_cli_findex only checks one of two PyTuple_New results — s->cmps NULL flows to PyTuple_SET_ITEM(NULL, ...)	hv_cli_findex.c:125-128,144	2 agents
5	sf_slice unchecked NyImmBitSet_New at two points — bitset slicing crashes on OOM	bitset.c:2958,2998	1 agent
6	_hiding_tag__name unchecked after PyUnicode_FromString — NULL stored in global, later passed to PyDict_GetItem in every heap traversal	heapyc.c:253	3 agents
7	frame_traverse infinite loop on Python 3.14 — for (p = ...; p < ...; i++) increments i but loop variable is p	stdtypes.c:452	1 agent

Type Slot / GC (2)

#	Finding	File	Agents
25	NyHeapView tp_traverse missing _hiding_tag_ visit — GC cannot see through this owned PyObject* member. tp_clear correctly clears it.	hv.c:124-141	1 agent
26	NyNodeTuple_Type missing tp_dealloc — items stored via PyTuple_SET_ITEM never decref'd. Leaks refs on every NodeTuple destruction. Used as dict keys in classifier memoization.	hv_cli_and.c:266-278	1 agent

__new__-without-__init__ (1)

#	Finding	File	Agents
24	ObjectClassifier() segfaults — exported type, NYFILL macro assigns PyType_GenericNew, all methods dereference NULL def pointer. Directly callable from Python.	classifier.c:321, heapyc.c:245	1 agent

Reference Leaks / Error Propagation (4)

#	Finding	File	Agents
8	set_relate triple bug — iterator it leaked, matched obj leaked, unchecked PyLong_FromSsize_t NULL passed to r->visit. On every successful set membership check.	stdtypes.c:157-181	2 agents
9	hv_extra_type converts allocation failure to silent xt_error sentinel — loses error context, causes cascade of return NULL without exception across ~15 call sites	hv.c:462-464	1 agent
10	hv_relate returns NULL without exception via xt_error cascade	hv.c:1295-1318	1 agent
11	Unchecked PyUnicode_FromString passed to r->visit in xt_relate — NULL relator depends on defensive visitor code	hv.c:514	1 agent

Refcount Bugs (6)

#	Finding	File	Agents
14	cli_partition_iter over-decrements borrowed/released sp on error — Py_XDECREF(sp) in error label on a ref the function doesn't own. Double-free / refcount underflow.	classifier.c:94-120	1 agent
15	outrel_visit borrowed ref used after Py_DECREF — PyDict_GetItem borrowed ref or decref'd new relation passed to NyNodeSet_setobj	hv_cli_rel.c:273-284	1 agent
16	hv_cli_prod_le borrowed refs across arbitrary Python comparisons — PyTuple_GetItem borrowed refs used across PyObject_RichCompareBool which runs __eq__/__le__	hv_cli_prod.c:142-194	1 agent
17	dict_relate_kv passes unchecked NULL to r->visit — PyLong_FromSsize_t OOM silently swallowed by visitor. Same pattern across ~25 call sites.	stdtypes.c:62-65	1 agent
18	hv_PyList_Pop leaks reference when PyList_SetSlice fails — Py_XINCREF'd item not decref'd on error. Affects 5 heap traversal call sites.	hv.c:107-111	1 agent
19	mutnsiter_iternext leaks bitobj on conversion error — PyLong_AsSsize_t error path missing Py_DECREF(bitobj)	nodeset.c:232-234	1 agent

Resource Lifecycle (4)

#	Finding	File	Agents
20	immbitset_realloc + union_realloc — _Py_ForgetReference + NULL PyObject_Realloc = memory leak + object list corruption — 2 sites with identical pattern	bitset.c:957-963,900-906	2 agents
21	NyNodeGraph_AddEdge — PyMem_RESIZE overwrites ng->edges with NULL — old buffer with incref'd PyObject* permanently lost	nodegraph.c:169-176	1 agent
22	NyImmBitSet_SubtypeNewArg unchecked alloc before memcpy — NULL deref in ImmBitSet constructor	bitset.c:726-727	1 agent
23	sf_slice unchecked NyImmBitSet_New at 2 points — bitset slicing crashes on OOM	bitset.c:2958,2998	2 agents

PyErr_Clear / Exception Handling (2)

#	Finding	File	Agents
12	hv_cli_indisize_memoized_kind missing PyErr_Occurred check — the ONE outlier among 6 structurally identical memoized_kind functions	hv_cli_indisize.c:14	2 agents
13	cli_partition_iter missing PyErr_Occurred check — user-defined classifier keys with arbitrary __hash__ go through PyDict_GetItem. Highest risk: kind is arbitrary user code.	classifier.c:102	2 agents

---

Important Findings (CONSIDER) — 16

PyDict_GetItem Exception Swallowing (3)

• Unguarded PyErr_Clear in lazy_init_hv_cli_prod (hv_cli_prod.c:62)
• outrel_visit missing PyErr_Occurred after PyDict_GetItem (hv_cli_rel.c:273)
• 3 hiding-tag checks use PyDict_GetItem for pointer comparison (benign but imprecise)

Module Init (4)

• 7 unchecked PyDict_SetItemString in heapyc init (heapyc.c:241-252)
• 5 unchecked PyDict_SetItemString in bitset init (bitset.c:4464-4474)
• NyStdTypes_init returns void, ignores allocation failures (stdtypes.c:800-830)
• Unchecked PyCapsule_New NULL passed to PyDict_SetItemString (bitset.c:4468)

Visit Protocol Inconsistency (3)

• r->visit return value ignored in nodeset_relate_visit (nodeset.c:432)
• nodegraph_relate returns 0 instead of propagating visit errors (nodegraph.c:760-775)
• Unchecked PyLong_FromSsize_t passed to callbacks (multiple sites)

Version Compatibility (3)

• 13 PyDict_GetItem calls — should migrate to PyDict_GetItemWithError
• PySys_GetObject deprecated in 3.13 (hv_cli_prod.c:34)
• Outdated vendored pythoncapi_compat.h

Other (3)

• hv_numedges return 0 may lack exception via xt_error cascade (hv.c:1047)
• PySequence_Length -1 return treated as truthy (hv_cli_prod.c:121)
• Dead Python 2 file initheapyc.c

---

Architecture Notes

Aspect	Status
Init style	Single-phase, both modules. m_size = -1.
Stable ABI	Not feasible. 8 CPython internal headers, 9 private API functions, 65+ direct struct accesses. Inherent to heap profiler design.
Static types	19 (8 heapy + 11 sets) with inheritance chains
GIL release	Never — correct for heap profiler accessing GC internals
Free-threading	Fundamentally incompatible (Horizon dealloc patching, RootState singleton, interpreter state traversal)
PyErr_Clear	Only 2 explicit calls (1 guarded, 1 unguarded). 5 unguarded PyDict_GetItem implicit clears.
Complexity	No hotspots above 5.0. rootstate_getattr at 4.9 (nesting depth 9).
pythoncapi_compat	Used but outdated — missing newer backports
Version compat	Excellent — version-conditional code for 3.10-3.14, zero unguarded new APIs

---

Priority Recommendations

1. Finding 7 (infinite loop on 3.14) — for (p = ...; p < ...; i++) should be p++. One-character fix. Hangs during GC traversal on Python 3.14.

2. Finding 1 (ng_iter wrong variable) — !v should be !iter. One-character fix. Crashes on OOM during iteration.

3. Finding 8 (set_relate triple bug) — Add Py_DECREF(it), Py_DECREF(obj), and check PyLong_FromSsize_t return. Leaks on every successful set membership check.

4. Finding 9 (xt_error sentinel) — Add PyErr_SetString to xt_error_traverse and xt_error_relate. Fixes the root cause of ~15 return NULL without exception paths.

5. Finding 2 (mutbitset_initset deref before check) — Move NULL check before dereference. One-line reorder.

6. Finding 14 (cli_partition_iter over-decrement) — Py_XDECREF(sp) on borrowed/released ref = refcount underflow. Corrupts partition dict list objects.

7. Finding 20 (_Py_ForgetReference + NULL realloc) — memory leak + object list corruption on OOM in bitset resize. Fix by re-tracking original on failure.

8. Finding 21 (PyMem_RESIZE overwrites ng->edges) — Use PyMem_Realloc into a temp variable instead. The macro assigns NULL to the first arg on failure.

Found using cext-review-toolkit (11 of 13 agents completed, v0.1.5).

guppy3_report_appendix.md

guppy3 — Reproducer Appendix

Reproducers for findings from the guppy3 analysis report. All reproducers use CPython 3.13 on Linux x86_64.

Setup:

cd /path/to/guppy3
export PYTHONPATH=.

---

Reproduced Findings

Finding 24: ObjectClassifier() segfault — 3 crash paths

from guppy.heapy import heapyc

oc = heapyc.ObjectClassifier()
oc.classify(42)       # Segmentation fault
# Also: oc.epartition([1,2,3]) — segfault

100% reproducible. Both classify and epartition segfault. partition hits a TypeError before the NULL deref (argument validation catches it first). select also hits argument validation.

Root cause: NYFILL macro assigns PyType_GenericNew to types without custom tp_new. Zero-initialized def == NULL, all methods dereference self->def->classify(...).

---

Finding 2: MutBitSet() segfault via OOM injection

import _testcapi
from guppy.sets import setsc

_testcapi.set_nomemory(1, 0)
try:
    setsc.MutBitSet()
    _testcapi.remove_mem_hooks()
except:
    _testcapi.remove_mem_hooks()
# Segmentation fault

100% reproducible at set_nomemory(1, 0).

Root cause: mutbitset_initset at bitset.c:793-794 dereferences sf->set->ob_field before the NULL check at line 795. The immbitset_realloc(0, 1) call returns NULL when allocation fails, and the dereference happens before the check.

---

CONSIDER (upgraded to FIX): 42 in NodeSet() segfault

from guppy.sets import setsc

ns = setsc.NodeSet()
42 in ns
# Segmentation fault

100% reproducible. The abstract NodeSet base type is constructible via NYFILL's PyType_GenericNew. The __contains__ operation crashes because internal state is zero-initialized but the containment check path dereferences u.bitset without adequate guards.

Note: len(NodeSet()) returns 0 (safe), NodeSet() | NodeSet() works (creates a proper ImmNodeSet), but in crashes.

---

Findings Not Reproducible

Finding 1: ng_iter wrong variable (!v instead of !iter)

Status: Not triggered by set_nomemory

PyObject_GC_New uses CPython's GC allocator which is not hooked by set_nomemory. The wrong-variable check (!v instead of !iter) is code-confirmed at nodegraph.c:582 — the function argument v is always non-NULL, making the check dead code.

Finding 7: frame_traverse infinite loop on Python 3.14 with JIT

import sys
sys.path.insert(0, ".")

def hot_function(x):
    return x + 1

# JIT warmup — compile many calls
for i in range(10000):
    hot_function(i)

from guppy import hpy
h = hpy()

def profiled():
    return h.heap()

# Warm up profiled — triggers traversal of JIT-compiled frames
for i in range(10000):
    profiled()  # HANGS — infinite loop in frame_traverse

CONFIRMED: process hangs (timeout after 3 seconds). The loop never completes because p never advances.

Root cause: stdtypes.c:452 — for (p = iv->localsplus; p < iv->stackpointer; i++) increments i instead of p. This path executes when co == NULL, which happens for JIT-compiled frames where f_executable is not a PyCodeObject. On Python 3.14 with JIT enabled (sys._jit.is_enabled() == True), hot functions get JIT-compiled, and their frames enter the !co path during heap traversal.

One-character fix: i++ → p++

Findings 3-5, 20, 22-23: Various bitset NULL deref / realloc issues

Status: Code-confirmed, not triggerable via _testcapi.set_nomemory for most paths

Tested ImmBitSet(range(...)), ImmBitSet(existing), ImmBitSet[:500], and MutBitSet |= ImmBitSet with set_nomemory(n, 0) and set_nomemory(n, 1) for n=1..20 — all survive. The bitset allocations use PyObject_NewVar/PyObject_Realloc (object-domain allocator, PYMEM_DOMAIN_OBJ) which is NOT hooked by set_nomemory (which hooks PYMEM_DOMAIN_RAW only).

The MutBitSet() segfault at set_nomemory(1, 0) (Finding 2) goes through PyType_GenericAlloc → mutbitset_initset which calls immbitset_realloc(0, 1). The initial allocation IS hooked because tp_alloc uses the raw domain for the object header.

Also discovered: heap() with set_nomemory(3, 0) through set_nomemory(100, 0) causes hangs (not segfaults) — the Python-level Glue.py framework enters an infinite attribute-lookup loop when allocations persistently fail during lazy initialization. This is a Python-level issue in guppy3's glue framework, not a C-level bug.

Findings 8-11, 14-21, 25-26: Reference leaks, error propagation, type slot issues

Status: Code-confirmed

These are structural bugs confirmed by reading the source — ref leaks in set_relate, hv_PyList_Pop, mutnsiter_iternext; refcount corruption in cli_partition_iter; xt_error sentinel cascade; missing traverse in HeapView; missing dealloc in NodeTuple. Cannot be easily demonstrated via tracemalloc because guppy3's heap profiling operations interact with the measurement tools.

Findings 12-13: PyDict_GetItem swallows MemoryError — object LOST from partition

from guppy.heapy import heapyc

root = heapyc.RootState
hv = heapyc.HeapView(root, ())
cli = hv.cli_type()

class SelectiveHashMeta(type):
    _count = 0
    _fail_on = set()
    def __hash__(cls):
        SelectiveHashMeta._count += 1
        if SelectiveHashMeta._count in SelectiveHashMeta._fail_on:
            raise MemoryError("hash bomb")
        return type.__hash__(cls)

class VictimType(metaclass=SelectiveHashMeta):
    pass

objects = [VictimType() for _ in range(5)] + [42, "hello"]

# Baseline: all 5 VictimType objects in one partition bucket
SelectiveHashMeta._count = 0
SelectiveHashMeta._fail_on = set()
baseline = cli.partition(objects)
print(f"Baseline: {len(baseline[VictimType])} VictimType objects")

# Fail the 2nd object's lookup hash (call #3)
SelectiveHashMeta._count = 0
SelectiveHashMeta._fail_on = {3}
result = cli.partition(objects)
print(f"With hash failure: {len(result[VictimType])} VictimType objects")
# Output:
#   Exception ignored in PyDict_GetItem()...
#   Baseline: 5 VictimType objects
#   With hash failure: 4 VictimType objects
# CONFIRMED: 1 object LOST from partition

Root cause: PyDict_GetItem internally clears the MemoryError from __hash__() and returns NULL. cli_partition_iter treats NULL as "key not found" and creates a new list via PyObject_SetItem, which overwrites the existing entry for VictimType (losing the first object that was already in the original bucket).

CPython 3.13 prints a diagnostic: Exception ignored in PyDict_GetItem(); consider using PyDict_GetItemRef() or PyDict_GetItemWithError() — confirming that CPython itself recognizes this as a bug pattern.

Note: the PyErr_Occurred() check in the other 6 memoized_kind functions is vacuous — PyDict_GetItem already cleared the exception before the check can see it. The real fix for all 7 functions is to replace PyDict_GetItem with PyDict_GetItemWithError.

---

Summary

#	Finding	Reproduced?	Method
24	ObjectClassifier() segfault	YES	oc.classify(42), oc.epartition([1,2,3])
2	MutBitSet() segfault on OOM	YES	set_nomemory(1, 0)
—	42 in NodeSet() segfault	YES	Abstract base constructible, __contains__ crashes
7	Infinite loop on 3.14 with JIT	YES	heap() hangs with JIT-compiled frames
1	ng_iter wrong variable	Code-confirmed	GC allocator not hooked by set_nomemory
3-5	Bitset NULL deref	Code-confirmed	Specific OOM paths
8-11	Reference leaks/error propagation	Code-confirmed	Structural bugs
12-13	PyDict_GetItem swallows MemoryError	YES	Selective metaclass hash bomb, 1 object lost
14-23	Refcount, resource lifecycle	Code-confirmed	Various structural bugs
25-26	Type slot issues	Code-confirmed	Missing traverse/dealloc

6 findings reproduced (2 segfaults from Python, 1 segfault via OOM injection, 1 infinite loop on 3.14 with JIT, 1 PyDict_GetItem exception swallowing with data loss, 1 42 in NodeSet() segfault), 24 code-confirmed.

Found using cext-review-toolkit (13 agents, v0.1.5).