tokenize-ebnf.scm
defines in it a tokenizer (aka lexer, or scanner) for EBNF. This is compliant to ISO but several other meta-structures such as RegExp (/.../
) are also consumed. Comments starting withg #
are tokenized, consumed and appended to the token stream. Do with them as you wish!
This was an excercise at Scheme --- and so much more, an excercise at hand-rollling lexers in an stateless language with functional paradigm built in.
You can use this to, for example, translate EBNF to LaTEx, HTML, PostScript, ROFF --- or maybe define a parser for it, and convert it to Yacc, PEG, ANTLR, Flex, Re2C, etc.