README.md

A proof-of-concept Python code rewriter, transforming JavaScript-like tagged template strings (tag"foo{1+2}bar") to plain function calls (tag(["foo", "bar"], [1+2])).

Note that the code requires tagged version 0.0.2 or higher.

You can use ./encoder.py file to transform STDIN input to STDOUT:

$ python3 encoder.py < example_input.py

The output looks something like this:

from mylibrary import mytag

mytag (['<div>','</div>'],[(mytag (['<span/>'],[]))])

encoder.py

import io
import ast
import token
import tokenize
import tagged


def rewrite(tag, string):
    strings, exprs = tagged.split(ast.literal_eval(string))

    yield token.NAME, tag
    yield token.OP, "("
    yield token.OP, "["
    for i, s in enumerate(strings):
        if i > 0:
            yield token.OP, ","
        yield token.STRING, repr(s)
    yield token.OP, "]"
    yield token.OP, ","
    yield token.OP, "["
    for i, e in enumerate(exprs):
        if i > 0:
            yield token.OP, ","
        yield token.OP, "("
        yield from coding(io.BytesIO(e.encode("utf-8")))
        yield token.OP, ")"
    yield token.OP, "]"
    yield token.OP, ")"


def coding(stream):
    reader = io.BufferedReader(stream)

    previous = None
    for current in tokenize.tokenize(reader.readline):
        if previous and previous.type == token.NAME and current.type == token.STRING:
            yield from rewrite(previous.string, current.string)
            previous = None
        else:
            if previous:
                yield previous
            previous = current
    if previous:
        yield previous


if __name__ == "__main__":
    import sys
    sys.stdout.buffer.write(tokenize.untokenize(coding(sys.stdin.buffer)))

example_input.py

from mylibrary import mytag

mytag"""<div>{mytag"<span/>"}</div>"""

1. This is a way cleaner implementation than what I did in htm-pyx - didn't know about tokenize (will be using that in the future).
2. I think your points about invalid syntax were good though, so unfortunately I think we'll have to drop the mytag"<div/>" syntax.