Skip to content

Instantly share code, notes, and snippets.

@jviide
Last active October 4, 2019 21:10
Show Gist options
  • Save jviide/8805c46a404fd613b395ca8525fd3f43 to your computer and use it in GitHub Desktop.
Save jviide/8805c46a404fd613b395ca8525fd3f43 to your computer and use it in GitHub Desktop.

A proof-of-concept Python code rewriter, transforming JavaScript-like tagged template strings (tag"foo{1+2}bar") to plain function calls (tag(["foo", "bar"], [1+2])).

Note that the code requires tagged version 0.0.2 or higher.

You can use ./encoder.py file to transform STDIN input to STDOUT:

$ python3 encoder.py < example_input.py

The output looks something like this:

from mylibrary import mytag

mytag (['<div>','</div>'],[(mytag (['<span/>'],[]))])
import io
import ast
import token
import tokenize
import tagged
def rewrite(tag, string):
strings, exprs = tagged.split(ast.literal_eval(string))
yield token.NAME, tag
yield token.OP, "("
yield token.OP, "["
for i, s in enumerate(strings):
if i > 0:
yield token.OP, ","
yield token.STRING, repr(s)
yield token.OP, "]"
yield token.OP, ","
yield token.OP, "["
for i, e in enumerate(exprs):
if i > 0:
yield token.OP, ","
yield token.OP, "("
yield from coding(io.BytesIO(e.encode("utf-8")))
yield token.OP, ")"
yield token.OP, "]"
yield token.OP, ")"
def coding(stream):
reader = io.BufferedReader(stream)
previous = None
for current in tokenize.tokenize(reader.readline):
if previous and previous.type == token.NAME and current.type == token.STRING:
yield from rewrite(previous.string, current.string)
previous = None
else:
if previous:
yield previous
previous = current
if previous:
yield previous
if __name__ == "__main__":
import sys
sys.stdout.buffer.write(tokenize.untokenize(coding(sys.stdin.buffer)))
from mylibrary import mytag
mytag"""<div>{mytag"<span/>"}</div>"""
@rmorshea
Copy link

rmorshea commented Oct 4, 2019

  1. This is a way cleaner implementation than what I did in htm-pyx - didn't know about tokenize (will be using that in the future).
  2. I think your points about invalid syntax were good though, so unfortunately I think we'll have to drop the mytag"<div/>" syntax.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment