Hi Carl,
Congratulations with getting the grant!
I will be looking forward to your upcoming blogs due to two reasons: 1) plain curiosity, 2) I'm the guy behind http://scalamacros.org/, so it'd be very interesting to learn how you solve certain design problems in your macro system.
Cheers, Eugene
Hi Eugene,
How nice to receive an email from someone else who's implemented macros! I hadn't come across http://scalamacros.org/ before, but I'm already perusing it. Looks very interesting indeed. A bit of a different factoring from what's already in S06, but also many points of contact.
Thanks for taking the time to send a note.
Kindly, // Carl
Hey Carl,
Just read your blog post about quasiquotes in Perl. Got some questions:
Why do you need different AST flavors? How do they differ?
Can "quasi" blocks be written outside of macros? E.g. in helper functions? Does hygiene still work then?
What do you think about making "quasi" a macro that expands into an AST?
In the last code snippet, could a quasiquote refer to the $a which is local to the macro? E.g. via quasi { $a }.
Cheers, Eugene
Eugene (>):
Hey Carl,
Just read your blog post about quasiquotes in Perl. Got some questions:
Cool! I had a look through your work on Scala's macros last time you wrote. Quite probably your email today will trigger another exploration.
- Why do you need different AST flavors? How do they differ?
This is the deepest of your four questions, and one I do not answer fully in the post. It's directly connected to the problem that took me months to solve. I will attempt to give a satisfactory answer here. Feel free to follow up with more questions if anything is unclear.
A macro definition must out of necessity come textually before a
corresponding macro invocation. Even so macro, arguments are built
very "early" in the compile-run cycle, whereas quasi ASTs are built
relatively "late". This is because as a macro is invoked, the parser
essentially pauses and allows the runtime to execute the macro
routine, in what is called BEGIN time in Perl. (BEGIN time is any
runtime activity which happens ASAP during parsing.)
So this is what distinguishes quasi ASTs from argument ASTs: the different kinds of environments in which they are constructed. Quasi ASTs have a full runtime context to adopt as theirs, whereas the argument ASTs only have the static compilation context that exists during parsing.
Here's an example to highlight the difference between static lexpads and runtime lexpads.
<masak> r: my $i = 42; BEGIN { say $i }; say $i
<p6eval> rakudo 170c90: OUTPUT«Any()42»
Since BEGIN runs ASAP during parsing, you're seeing Any() which is a
lookup of $i in the static lexpad. Parsing finishes, the program as
such starts running, the assignment happens, and the second say gives
42 back from the runtime lexpad.
Things involving the static lexpads don't happen a lot... except when
you're mucking around running code ASAP, like BEGIN blocks and macros.
Another case where it happens is role composition. At the time a role
is composed into a class, which happens during parsing, the role block
runs, and risks binding things to the static environment, which is the
only one that exists during parsing.
<masak> r: my $x = 42; role R { method foo { $x } }; class C is R {}; say C.new.foo
<p6eval> rakudo 7c0820: OUTPUT«42»
At the time R is composed into C, there's only the static lexpad
containing $x with the value Any. Generally these things just sort
themselves out by ordinary closure cloning by the runtime, but it
doesn't here, because C has forced R to evaluate during parsing.
Rakudo makes sure things like R.foo get the right scope eventually by
inserting "fixups", code that runs whenever the block containing the
declaration of R (the mainline block, in this case) is entered.
Now, the components of the above challenging situation are exactly the ones we find when expanding a macro: BEGIN-time evaluation of code, forced binding of static contexts, and a wish to re-bind later to the actual runtime contexts.
<masak> r: my $x = 42; macro foo($value) { quasi { say {{{$value}}} } }; foo $x
<p6eval> rakudo 7c0820: OUTPUT«42»
{{{$value}}} expands to $x, yes, but it needs to expand to $x with the
context being the mainline. The only such context that exists at the
time of parsing is the static context, in which $x is Any. So we need
the context to be re-bound to the mainline runtime context as soon as
it exists, because that's where the 42 is.
So that's why quasi ASTs and argument ASTs are different and why where they're coming from matters quite a bit. The latter have to worry about static contexts.
(I'm still a bit shocked at the simplicity of the eventual solution: we simply dump a fully-formed closure into the argument AST, along with an instruction to call it. The closure gets automatically cloned by regular mechanisms, and so the context ends up being the right one without any magic.)
- Can "quasi" blocks be written outside of macros? E.g. in helper functions? Does hygiene still work then?
Yes and yes.
<masak> r: sub foo { my $greeting = "OH HAI"; quasi { say $greeting } }; BEGIN my $gast = foo; macro bar { $gast }; bar()
<p6eval> rakudo 7c0820: OUTPUT«OH HAI»
(Code is unnecessarily convoluted because doing it in the obvious, simpler way triggered a bug I wasn't aware of until now.)
- What do you think about making "quasi" a macro that expands into an AST?
The question makes little sense to me; we're probably coming from different directions here.
Perhaps you are assuming more metacircularity than we currently have in Rakudo. The compiler provides macros in userland, but does not itself have them. Hence, quasis can not easily be implemented using macros.
- In the last code snippet, could a quasiquote refer to the $a which is local to the macro? E.g. via
quasi { $a }.
Yes, and that's definitely a feature. In that sense, the quasi is just a normal closure.
<masak> r: macro test { my $a = "OH HAI"; quasi { $a } }; say test
<p6eval> rakudo 170c90: OUTPUT«OH HAI»
Kindly, // Carl
Good question. :) It's definitely a compiler, in that it parses and code-generates to a lower-level VM-specific bytecode format, which is what then runs. No per-instruction interpretation is taking place.
That said, Perl 6 allows more data interchange between compilation and running than most other languages. So I can see why you'd ask. Basically, eval() allows the runtime to duck back into compilation at any time, and BEGIN allows the compiler to temporarily skip ahead to running things ahead of time. Classes, roles, and various declarations further exploit "phases" of this kind to displace when code is actually run. (For example, assignments attached to attribute declarations in classes happen at object instantiation.)
But yes, it's a compiler.
All right I see now. That's interesting. I wonder how difficult it is then to generate optimized machine code.
I wanted to clarify things because of your answer to the question #4. For Scala, in which compile-time and runtime are strictly separated, that'd be impossible, because $a belongs to the compile time, while macro expansion that refers to it will be executed at runtime.
This quite a different perspective on things was the driving force behind asking the question #3. Imagine you have a macro expanding at compile-time of some program. But in order to expand, that macro would have to be compiled first. At that, so to speak, compile-compile-time you can have your quasiquotes (implemented as macros themselves) expand into ASTs that will then be used by the macro during expansion. But I see how this approach can be a problem in your setting.
By the way speaking of more or less conflated compile-time and runtime. This publication might be interesting for you: http://www.cs.utah.edu/plt/publications/macromod.pdf. It talks about funny challenges that Scheme guys faced with macros when they tried to move from an interpreter to a compiler. Probably this is irrelevant to Rakudo, but nevertheless it's yet another point of view w.r.t phase distinction.
All in all, it was very fun to learn about macros in Rakudo. This "AST = closure" principle is super cool, when allowed by compiler/runtime!