Skip to content

Instantly share code, notes, and snippets.

@ruv
Last active September 4, 2026 08:35
Show Gist options
  • Select an option

  • Save ruv/07a892e0e7e376b6721ec003aeab08d9 to your computer and use it in GitHub Desktop.

Select an option

Save ruv/07a892e0e7e376b6721ec003aeab08d9 to your computer and use it in GitHub Desktop.
Execution token clarification -- Forth proposal
\ 2026-06-03
\ In the test cases below, we assume the following.
\ 1) Tick `'` returns the execution token of a word.
\ 2) The execution token of a word identifies
\ the execution semantics of the word
\ (either specified by the standard or implementation dependent).
\ 3) Performing the execution semantics of a word
\ in interpretation state leads to performing
\ the interpretation semantics for the word.
\
\ Thus, we test whether the execution semantics, which are identified
\ by the execution token of a word, perform (or implement),
\ in interpretation state, the interpretation semantics for the word.
cr
.( Testing the execution semantics of `s"` ) cr
t{ ' s" execute abc" s" abc" compare -> 0 }t
.( Testing the execution semantics of `s"` that were compiled ) cr
t{ :noname [ ' s" compile, ] ; execute abc" s" abc" compare -> 0 }t
.( Testing the execution semantics of `action-of` ) cr
t{ defer [v] immediate -> }t
t{ : v1 111 ; ' v1 is [v] -> }t
t{ ' action-of execute [v] -> ' v1 }t
.( Testing the execution semantics of `action-of` that were compiled ) cr
t{ :noname [ ' action-of compile, ] ; execute [v] -> ' v1 }t
.( Testing whether the execution semantics of `action-of` do parsing ) cr
t{ :noname action-of [v] ; execute -> ' v1 }t
.( Testing the execution semantics of `is` ) cr
t{ : v2 222 ; -> }t
t{ ' v2 ' is execute [v] -> }t
t{ [v] -> 222 }t
.( Testing the execution semantics of `is` that were compiled ) cr
t{ :noname ['] v1 [ ' is compile, ] ; execute [v] -> }t
t{ [v] -> 111 }t
.( Testing whether the execution semantics of `is` do parsing ) cr
t{ :noname ['] v2 is [v] ; execute -> }t
t{ [v] -> 222 }t
.( Testing the execution semantics of `to` ) cr
t{ 0 value x -> }t
t{ 0 value y -> }t
t{ 2 ' to execute x x -> 2 }t
t{ : xt-of(to) ['] to ; -> }t
t{ 3 xt-of(to) execute x x -> 3 }t
.( Testing whether the execution semantics of `to` do parsing ) cr
t{ :noname 4 xt-of(to) execute y ; execute x y x -> 0 0 4 }t
\ Another test for the execution semantics of `to`
t{ 0 value [some-option] immediate -> }t
t{ : set-some-option ( x -- ) 0<> to [some-option] ; -> }t
t{ : foo ( -- n ) [some-option] [if] 1 [else] 2 [then] ; -> }t
t{ foo -> 2 }t
t{ true set-some-option -> }t
t{ : bar ( -- n ) [some-option] [if] 1 [else] 2 [then] ; -> }t
t{ bar -> 1 }t
.( All test cases are complete ) cr
\ 2026-06-04
\ Test applying `postpone` to `s"`, `is`, `action-of`, `to`.
\ NB: the word `[compile]` is obsolescent, there is no need to test it.
\ At this version, we test the results only in compilation state
0 value is-parsing-detected
: execute-watchfully ( any xt -- any )
>in @ >r
execute
>in @ r> ( u2 u1 )
<> to is-parsing-detected
;
cr
.( Testing the compilation semantics of `s"` ) cr
t{ : _s" postpone s" ; immediate -> }t
t{ :noname _s" abc" ; execute s" abc" compare -> 0 }t
.( Testing the compilation semantics of `is` ) cr
t{ : comp(is) postpone is ; -> }t
t{ : _is ['] comp(is) execute-watchfully ; immediate -> }t
defer v : v1 111 ; : v2 222 ;
' v1 is v
t{ v -> 111 }t
t{ :noname ['] v2 _is v ; execute v -> 222 }t
t{ is-parsing-detected -> true }t
.( Testing the compilation semantics of `action-of` ) cr
t{ : comp(action-of) postpone action-of ; -> }t
t{ : _action-of ['] comp(action-of) execute-watchfully ; immediate -> }t
t{ :noname _action-of v ; execute execute -> 222 }t
t{ is-parsing-detected -> true }t
.( Testing the compilation semantics of `to` ) cr
0 value x
t{ : comp(to) postpone to ; -> }t
t{ : _to ['] comp(to) execute-watchfully ; immediate -> }t
t{ :noname 1 _to x ; execute x -> 1 }t
t{ is-parsing-detected -> true }t
.( All test cases are complete ) cr

Author

Ruv

Change Log

(the latest at the top)

  • 2026-06-04 (to be updated) Narrow down ambiguous conditions. Clarify meaning of terms in the typical usage examples. Improve wording in some places. Remove ambiguous conditions for postpone and [compile]. Add more test cases.
  • 2025-10-03 Correct some normative statements; add ambiguous conditions; make better wording. Add more rationale. Correct some typos.
  • 2025-09-30 Add some rationale; correct some typos and grammar mistakes; minor rewording; add consequences.
  • 2025-09-29 Better wording in some places; corrections; update in ambiguous conditions.
  • 2025-09-28 Huge update; incorporate proposals [249], [212] (partially), [163], [122] (partially); add tests. According to my comment on 2024-09-24, this was planned.
  • 2022-09-19 explicitly allow a short formula, describe what it means, better wording, fix some typos
  • 2022-08-13 Initial version

Preceding history

(the latest at the top)

Problem

By the definition of the term "execution token" in Forth-94 and Forth-2012, it's a value that identifies execution semantics. Can such a value identify other behavior, e.g., some interpretation semantics or compilation semantics? It's unclear at the first glance.

Another problem is that, following unfortunate change to the term "execution token" in the very quickly accepted proposal [157], the standard does not formally state that an execution token identifies anything at all.

Yet another problem is that it is unclear what behavior are identified by the execution token of a word whose execution semantics or interpretation semantics are not specified by the standard.

There are five words for wich the standard defines interpretation semantics but not execution semantics: s", s\", to, is, action-of. Obtaining execution token (and the "tick" operation) for the last three of them is prohibited, while the behavior identified by the execution tokens of the first two is unclear in edge cases (it is neither explicitly defined nor undefined in compilation state).

Solution

Actually, an execution token can identify (and does identify) other semantics too, but only if they are equivalent to the execution semantics that this token also identifies.

Example 1

:noname postpone if ; ( xt )
  1. xt, which is left on the stack, identifies the execution semantics of this unnamed Forth definition.
  2. The execution semantics of this definition are equivalent to the compilation semantics for if (this follows from the standard).
  3. Then, this xt also identifies the compilation semantics for if (identifies in terms of behavior, not object code).

Example 2

: foo postpone if ;
  1. xt of foo identifies the execution semantics of foo.
  2. The execution semantics of foo are equivalent to the compilation semantics for if (this follows from the standard).
  3. Then, this xt also identifies the compilation semantics for if.

Note that the Forth system may provide system-dependent execution semantics for if that are not equivalent to the execution semantics of foo.

Reasoning

Thus, for any execution token there exists at least one Forth definition (named or unnamed) the execution semantics of which are identified by this execution token. So, in any case, an execution token always identifies some execution semantics, but accidentally (or intentionally) these semantics can be equivalent to some interpretation semantics, or some compilation semantics, and then it identifies them too. It's unnecessary that they connected to the same Forth definition. Also, consequently, it's impossible that an execution token identifies some compilation semantics, or some interpretation semantics, but doesn't identify the equivalent execution semantics.

Note that there are cases where the semantics cannot be identified by an execution token in a Forth system, because the implementation of the system does not have or cannot have a Forth definition with equivalent execution semantics.

Examples of semantics that cannot be identified by any execution token:

  • in some systems (in which the phrase postpone literal postpone execute is equivalent to compile,), the formally specified execution semantics of >r;
  • in some systems (where FVM does not have access to the underlying return stack, e.g. WAForth), the initiation semantics of :noname;
  • typically, the run-time semantics of if (which are appended to the current definition by the compilation semantics of if);

Of course, the standard allows such implementations and disallows programs to obtain an execution token of the corresponding semantics, or even does not provide a way to obtain it.

Roadmap

To solve the initial problem, we can:

  • formally and explicitly state the basics described above,
  • specify what particular semantics are identified by the execution token of a word (in which cases they are defined by the standard and in which by the implementation, and to what extent),
  • update the definition of the "execution token" term to say what these tokens identify.

Clearly defining execution tokens for edge cases eliminates ambiguity when obtaining them for the words to, is, and action-of.

Furthermore, the ambiguous conditions on applying postpone and [compile] to these words can be removed. These conditions were inherited from the word to in Forth-94, where they were introduced to provide a room for both "parsing" and "non-parsing" implementations. In reality "non-parsing" implementations were impossible because a child of value is allowed to be immediate.

In Forth-2012, there is also no room for implementation variance of postpone due to name>compile.

A close topic was also discussed in comp.lang.forth in 2024, Subject: "single-xt approach in the standard".

  • Message-ID: <2024Sep23.185202@mips.complang.tuwien.ac.at>
    • Date: 2024-09-23 16:52:02 +00:00
  • Message-ID: <vd0e11$3itr4$1@dont-email.me>
    • Date: 2024-09-25 07:27:28 +00:00

Typical use

  • "The execution semantics identified by xt are equivalent to the interpretation semantics of BAZ"

    • This seems pretty clear.
    • Note that this means that the execution semantics identified by xt are independent of STATE.
  • "xt identifies the interpretation semantics for the word BAZ"

    • This means that the execution token xt identifies the execution semantics which are equivalent to the interpretation semantics for the word BAZ.
    • Also, this means that the execution semantics identified by xt are independent of STATE.
    • At the same time, the execution semantics of the word BAZ may differ from the execution semantics identified by this xt.
  • "xt identifies the compilation semantics for the word FOO"

    • This means that the execution token xt identifies the execution semantics which are equivalent to the compilation semantics for the word FOO.
    • Also, this means that the execution semantics identified by xt are independent of STATE.
    • At the same time, the execution semantics of the word FOO may differ from the execution semantics identified by this xt.
  • "xt of the word BAR"

    • This means that xt identifies the execution semantics of the word BAR.
    • Whether this xt also identifies the compilation semantics, or the interpretation semantics, or both of them, or neither, for the word BAR depends on the word BAR (on how it is defined or specified).
    • Regardless of how BAR is defined, executing xt in interpretation state performs the interpretation semantics for BAR.

Incorrect use

Actually, the standard contains only one place where the "execution token" term is used ambiguously in a normative part — the glossary entry for FIND. The problem is that it says that FIND returns the execution token of the found Forth definition ("its execution token", so it should identify the execution semantics of that definition), but:

  • FIND may return two different execution tokens (one while compiling and another while not compiling) for the same input string (and, formally, the same Forth definition), which may identify different semantics; then, formally, at least one of them does not identify the execution semantics of the found definition (contrary to the semantic description).

In the glossary entry for NAME>INTERPRET, the language is just slightly non normative, since it uses the form "xt represents" instead of the form "xt identifies".

These glossary entries also have some other problems, so they should be corrected anyway; my other proposals on this matter are in progress.

Proposal

Update "execution token" term

In the section 2.1 Definitions of terms, change (as of 2021-09-13):

execution token: A value that can be passed to EXECUTE (6.1.1370)

into

execution token: A value that identifies the execution semantics of a definition.

Rationale

  • This is necessary to correctly describe the "execution token" data type.

Update "execution token" data type description

In the section 3.1.3.5 Execution tokens, make the following changes.

1. Add the following paragraphs to the beginning

For any valid execution token in the system, there is at least one Forth definition (named or unnamed) whose execution semantics are identified by that execution token.

The execution semantics identified by an execution token may be equivalent to the interpretation semantics, compilation semantics, or other semantics for some named Forth definition (or definitions). In such cases, the execution token also identifies those interpretation, compilation, or other semantics.

The system does not need to identify every specified semantics by any execution token.

The execution token of a Forth definition, if available, identifies the execution semantics that are either specified by this standard for that definition or are implementation dependent (if allowed).

If the interpretation semantics for a Forth definition are defined by this standard, the execution token of that definition shall be available.

Rationale

  • We use the clause "valid" in "any valid execution token" because an execution token may become invalid after using words like forget and children of marker.
  • The "if available" clause is used since find (in interpretation state) and name>interpret may return zero for some existing (but not user-defined) words; this effectively means that the execution token is not available for those words, and search-wordlist should also return zero for them.
  • The last paragraph guarantees that any word that is allowed to be Ticked has an execution token.

2. Add the following paragraph at the end

See also: A.3.1.3.5 Execution tokens.

Update "Execution semantics" notion

In the section 3.4.3.1 Execution semantics, make the following changes.

1. Change the first paragraph

The execution semantics of each Forth definition are specified in an "Execution:" section of its glossary entry. When a definition has only one specified behavior, the label is omitted.

into

The execution semantics of a Forth definition are specified in an "Execution:" section of its glossary entry. When a definition has only one specified behavior, the label is omitted.

Rationale

  • For some words, the execution semantics are not specified by the standard.

2. Add the following paragraphs after the first one

If the execution semantics for a Forth definition are specified by this standard and the glossary entry of that definition does not have an "Interpretation:" section, the execution token of that definition identifies the specified execution semantics. Otherwise, the execution token of that definition, if available, identifies the implementation dependent execution semantics.

Rationale

  • We have to rely on the absence of an "Interpretation:" section to refer to ordinary and immediate words until words like >r will be defined using a "Run-Time:" section instead of an "Execution:" section in their glossary entries (commented on 2019-06-21, 2020-08-30).
  • We use the term "implementation dependent" rather than "implementation defined" because this allows implementors to avoid documenting the provided execution semantics of standard words whose execution semantics are not specified by the standard.

The execution semantics identified by the execution token of a Forth definition, when they are performed in interpretation state, shall perform the interpretation semantics of that definition.

Rationale

  • This guarantees that, while interpreting, ' foo execute is equivalent to foo, even when the execution token (xt) of foo identifies implementation dependent execution semantics.

An ambiguous condition exists if the execution semantics of a Forth definition are not specified by this standard and its execution token is executed in compilation state under conditions other than those specified in this standard.

Rationale

  • That behavior in compilation state might depend on the implementation. Namely, if such a word (e.g. s") is implemented using the single-xt approach (all classic Forth systems), this performs the compilation semantics for the word, but if such a word (e.g. s") is implemented using the dual-xt approach, this typically performs the interpretation semantics for the word.
  • An example of conditions under which execution in compilation state of the execution token of such a word (e.g. s") is not ambiguous: if find in compilation state returns ( xt1 1 ) for a word, and, coincidentally, xt1 is the execution token of the word.

An ambiguous condition exists if the interpretation semantics of a Forth definition are undefined by this standard and its execution token is executed under conditions other than those specified in this standard.

Rationale

  • The execution semantics of the definition :noname s" if" forth-worldlist search-wordlist if execute then ; is ambiguous, regardless of whether they are performed while interpreting or compiling, because they execute (at least on some systems) the execution token of the word if, whose interpretation semantics are undefined by this standard.
  • In the same time, the execution semantics of the definition :noname c" if" find 1 = if execute then ; in compilation state is not ambiguous even if find in this case returns the same execution token as ['] if (which identifies the implementation dependent execution semantics of if).

Update ambiguous conditions

In the section 4.1.2 Ambiguous conditions,

replace the phrase:

attempting to obtain the execution token, (e.g., with 6.1.0070 ', 6.1.1550 FIND, etc. of a definition with undefined interpretation semantics;

with the phrase:

attempting to obtain, using the words 6.1.0070 ' or 6.1.2510 ['], the execution token of a definition with undefined interpretation semantics;

Rationale

  • "find" and "etc." are excluded from the list, because find (in interpretation state), search-wordlist, and name>interpret return either the execution token of the word or zero, and there is no ambiguous condition in this regard.

Update glossary entries

In the glossary entries 6.2.2295 TO, 6.2.1725 IS, 6.2.0698 ACTION-OF,

remove the sentence that starts with:

An ambiguous condition exists if any of POSTPONE, [COMPILE], ' or ['] are applied

Editorial note: in the first case the form “are applied” is used, in the second and third cases the form “is applied” is used.

Rationale

  • Historically, there were two approaches to implement to: "parsing" and "non-parsing". Forth-94 formally specified the "parsing" approach. It also declared an ambiguous condition, perhaps, as a concession to implementations that adhered to the "non-parsing" approach. Note that no similar exception was made for the word s", since it always do parsing of the parse area. In Forth-2012, more exceptions for the word to were added without obvious ground, and also for the words is and action-of.
  • It is now specified that the execution tokens of these words identify implementation dependent execution semantics that, when performed in interpretation state, shall perform the specified interpretation semantics for these words.
  • Also, since these words shall parse the parse area for their immediate argument, there is no any reason for an ambiguous condition when applying postpone or [compile] to these words (just as there is no such reason when applying them to s").
  • Note that [compile] is obsolescent.

Consequences

This change specifies all the conditions under wich an xt returned by search-wordlist, ' (Tick), and ['] (Bracket Tick) can (or cannot) be used by a standard program.

This change removes prohibition on Ticking the words to, is, action-of, and specifies that executing of the returned xt in interpretation state shall perform the interpretation semantics for the word. Note that executing can be performed directly by execute, or indirectly by executing of the definition in which this xt is compiled using compile,.

Also, this change removes prohibition on applying postpone and [compile] to the words to, is, action-of.

If the system throws an error on postponing these words, or on ticking these words, or does not provide a correct xt for them, it should be updated to be compliant.

Most Forth systems in which these words are parsing words (as the standard requires) already comply with this change.

According to Anton Ertl's testing of five Forth systems (comment [r885]) in 2022, iForth 5.0.27 and VfxForth 5.11 did not comply with this change with respect to the execution tokens of the word to.

In VfxForth 5.43, the words is and action-of do parsing (as specified in the standard), but the word to does not. Another issue is that the execution tokens of these three words, as well as the word s" (ticking of which is allowed), are not compiled correctly by compile,.

This change does not affect the existing standard programs. However, it allows the use of certain new techniques (previously disallowed) in standard programs.

This change answers to the following requests for clarification:

Testing

See:

@ruv

ruv commented Jun 3, 2026

Copy link
Copy Markdown
Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment