eregon/node_id_instability.md Secret

## node_id_instability.md

      
    Raw
  

              node_id_instability.md
            
          
    node_id Instability Across Prism Versions

The key point is that node_id is an allocation sequence number
(++parser->node_id), not an identity derived from a node's type or source
location.
Merely reordering node definitions in config.yml does not affect node_id.
Changing the number or order of node allocations does.
Historical examples

An ordinary parser fix changed the number of nodes

The reproduction from
note #13 of Feature #21795
changed from 25 to 26 nodes between Prism versions.
This can be bisected to
Ensure context terminators terminate expressions:
case 1
in 2
  A.print message:
in 3
  A.print message:
end
Before the fix, the second in was represented by a MatchPredicateNode
nested inside the first clause. After the fix, it was represented by a second
InNode.
The commit was an ordinary six-line parser fix and did not change
config.yml. Consequently, a node-definition ABI version would not have
detected this change.
ItParametersNode added an allocation

Internal support for node_id was added on February 8, 2024. Less than two
weeks later,
ItParametersNode was added.
The unchanged source:
-> { it }
gained an extra node. This shifted the LambdaNode and every node allocated
after it.
ShareableConstantNode added a wrapper

A month later,
ShareableConstantNode was added:
# shareable_constant_value: literal
A = 1

def target
end
The constant write gained a ShareableConstantNode wrapper before target,
shifting the DefNode ID.
It is also believable that this node could disappear in the future. For
example, the shareability mode could be stored as flags directly on the
constant-write nodes instead of using a wrapper. The same thing could happen
if this behavior became the default.
ConstantPathNode lost a child node

Shortly afterward,
ConstantPathNode#child was replaced with name and name_loc:
Foo::Bar

def target
end
Previously, Foo::Bar contained a child ConstantReadNode for Bar. After
the change, Bar was stored directly as a name and location on the
ConstantPathNode.
This removed one node allocation, shifting the ID of target down by one.
The commit explicitly described itself as an API-breaking representation
change.
Temporary nodes also consume IDs

The instability is broader than changes to the final AST. Nodes that are
allocated and later discarded still consume IDs.
For example:
A = 1

def target
end
currently has visible nodes with IDs resembling:
StatementsNode       1
# ID 2: temporary ConstantReadNode, absent from the final tree
IntegerNode          3
ConstantWriteNode    4
DefNode              5
ProgramNode          6

Prism first parses A as a read/target and then converts it into a write. A
reasonable optimization could construct the ConstantWriteNode directly,
eliminating the invisible allocation. That would change every later ID
without changing the final AST or the node definitions at all.
This means an ABI version that changes only when config.yml changes would
not be sufficient.
Allocation order can change without changing the tree

IDs can also move when the same nodes are allocated in a different order.
For example, DefNode, LambdaNode, and BlockNode are currently allocated
after their bodies have been parsed. A parser refactor could instead allocate
the outer node when its opening token is encountered and fill in its body
later.
Such a refactor would change the ID of exactly the node returned by
Method#ast or Proc#ast, without adding or removing any nodes.
Similarly, a CallNode is commonly allocated before its arguments. Building
the arguments first and the call afterward would change the CallNode ID
relevant to Thread::Backtrace::Location#ast, even if the final AST were
structurally equivalent.
Nodes that are especially likely to change

ImplicitNode and ImplicitRestNode are particularly plausible candidates.
Prism describes these as synthetic nodes that do not directly correspond to
source syntax:
{ foo: }        # ImplicitNode
foo { |bar,| }  # ImplicitRestNode
foo in [bar,]   # ImplicitRestNode
foo, = bar      # ImplicitRestNode
These could naturally be replaced with flags or nullable fields on nodes such
as AssocNode, ParametersNode, ArrayPatternNode, or MultiTargetNode.
Other plausible areas include:


ShareableConstantNode, which could be folded into constant-write flags.


ItParametersNode and NumberedParametersNode, which could be represented
as modes on another parameter node.


Grouping nodes such as StatementsNode, ArgumentsNode,
BlockParametersNode, and KeywordHashNode.


The many specialized target and write nodes, which could be merged, split,
or replaced with flags.


There are historical precedents for these kinds of changes even before
node_id was introduced: Prism split CaseNode and CaseMatchNode, split
CallNode in target position, and replaced
RequiredDestructuredParameterNode with MultiTargetNode.
Conclusion

Stable node_id values require the exact same parser implementation, parser
options, and source—not merely synchronized node definitions.
Normal bug fixes, temporary-node optimizations, representation changes, and
allocation-order refactors can all change IDs. Some of these changes do not
alter the node ABI or even the shape of the final AST.
Start and end source offsets are materially more robust across Prism versions
because they describe actual positions in the source rather than incidental
parser allocation order.
Source	Numeric child
`1i`	`IntegerNode`
`1.0i`	`FloatNode`
`1ri`	`RationalNode`
`1.0ri`	`RationalNode`
Commit	Release transition	Cause	Example ID change	`Prism.find` impact
`871ed4b46`	1.3.2 → 1.4.0	Added `ParenthesesNode` for `not()`	Following `DefNode(:a)` 3 → 4	The relevant CRubies did not yet expose method node IDs
`07202005c`	1.3.2 → 1.4.0	Rescue-modifier precedence changed	`DefNode(:target)` 8 → 6	Same pre-exposure limitation
`915f6b3ae`	1.4.0 → 1.5.0	Context terminators changed `case`/`in` structure	Later methods shift by one	Proven `Prism.find` failure
`cf3bbf9d2`	1.4.0 → 1.5.0	Removed `ItParametersNode` for `it = it`	`DefNode(:a)` 10 → 9	Proven failures for methods, lambdas, procs, and backtraces
`211677000`	1.6.0 → 1.7.0	Added `InterpolatedStringNode` around heredoc-interrupted `%w` parts	Methods 11/12 → 12/13	Proven `Prism.find` failure
`Fix in handling`	1.8.0 → 1.8.1; 1.9.0 → master	Restored `MatchPredicateNode` inside an `in` body	Outer `InNode` 9 → 12	Only `InNode`, pinned, and local-read IDs move; no exposed failure found
`b54865697`	post-1.9/master	Templated node construction changed allocation order	`ImaginaryNode`/numeric child IDs swap 2 ↔ 3	Internal only; not exposed through CRuby
`ab9412cfc`	post-1.9/master	Rescue associativity changed	`DefNode(:target)` 6 → 8; ID 6 becomes `CallNode(:c)`	Proven failure with Ruby 4.0.6