conartist6

// Node objects are immutable
// Also immutable are: properties, attributes, children, terminals, and any arrays
// Immutable trees can be cached as valid with regard to a particular grammar!
const freeze = (node) => Object.freeze(Object.seal(node));

// Helpers to make the following code less verbose
let t = {
  token: (type, str, attributes) => t.node(type, [t.str([str])], {}, attributes),
  node: (type, children, properties, attributes) =>
    freeze({

conartist6

// eat(<| Token 'foo' |> 
{
  "tagName": {
    "language": "Instruction",
    "type": "Call"
  },
  "children": [
    {
      "type": "Reference",
      "value": "verb"

conartist6

<!docype cstml>
<!parsers>
  <CSTML "https://url/to/grammar"/>
  <Regex "https://url/to/re/grammar"/>
</>
<CSTML:TokenTag [Tag]>
  <| Punctuator "<|" balanced="|>" startSpan="Tag" |>
  <Identifier path="tagName">
    <| Identifier "ID" |>
  </>

conartist6

/*
This file contains a formal grammar defined by yielding instructions to a state machine.
The grammar is extensible, because you can always wrap it in a higher-order grammar!
Formally the system is a VM exectuting an https://en.wikipedia.org/wiki/Earley_parser
The VM is not yet sufficiently finished to execute this grammar, but it soon will be.

Usage will be:

```js
import { parse } from 'cst-tokens';

conartist6

(1) MINOTAUR                                                                                     RIPTIDE (2)
(32) FUSION                                                                                    SHATTER! (31)
              MINOTAUR                                                          RIPTIDE
              SWITCHBACK                                                        HYPERSHOCK
(16) SWITCHBACK                                                                                   LUCKY (15)
(17) MALICE                                                                                  HYPERSHOCK (18)
                        MINOTAUR                                    RIPTIDE
                        COBALT                                      TANTRUM
(8) SAWBLAZE                                                                                    END GAME (7)
(25) BLIP                                                                                       TANTRUM (26)

conartist6

Who am I?

I am Conrad Buck (also known as conartist6): a software engineer, open source maintainer, and currently full time self-employed inventor.

I specialize in the creation of tools and platforms, with a particular attention to human factors both in the process of software develpment and in the way software affects the lives of the people who use it. I have an abiding love of all things playful, and see deep symmetry between the artistic, scientific, and engineering processes. I am a private pilot.

My history

Between the 2012 and 2020 I was a Frontend Engineer based in San Francisco, where I worked for MyVR, Spruce Media, Switchfly, VMWare, Facebook, and Plangrid (departed due to pandemic). I have most (-.5 credits) of a Bachelor of Arts in Computer Science degree from Bucknell University. A full accounting of this history is on [LinkedIn](https://www.linkedin.com/i

conartist6

  A range is represented as a [Token, Token] tuple.

  A range can only be iterated backwards, which creates the need to store tokens in an array!

  Each node must have its own tokens array to enable efficient editing.

  A context object often abbreviated `ctx` holds WeakMaps, including `paths` and `previousTokens`.

  Tokens reference their preceding token using the `previousTokens` WeakMap.

conartist6

import { prod, tok, sym, Grammar } from '@/helpers';

new Grammar({
  productions: {
  
    // This version of ImportSpecifier uses helpers to be concise:
    *ImportSpecifier() {
      yield eat(prod`Identifier:imported`);
      yield eatMatch(tok`Keyword:as`, prod`Identifier:local`);
    },

conartist6

const _ = Symbol('private');

class ReadOnlyMap {
  constructor(map) {
    this[_] = map;
  }

  static from(entries) {
    return new ReadOnlyMap(new Map(entries));
  }

conartist6

class Tokenizer {
  constructor(source) {
    this.source = source;
    this.result = null; // result is a token, i.e. a stack of tokens
    this.pathRangesByToken = new WeakMap();
    this.prevTokensByToken = new WeakMap();
  }

  ownTokensFor(path) {
    const { prevTokensByToken, pathRangesByToken } = this;