A slang to implement a QAST DSL

gistfile1.txt

This document proposes improvment to huffmanize NQP.
 -An AST DSL makes code generation less verbose
 -


# Last, an AST DSL for NQP, at last.

jnthn said that NQP needed love. I have love to spare.
Last should be made to work vanilla Perl 6 as well so people could
create compilers without knowing NQP (at the cost of performance because
Perl 6 is not yet very optimized in rakudo).

Last stands for Language for QAST, the "Q" AST.  
Slangs are DSLs you can weave together. Last is a DSL weaved
into vanilla NQP and relies itself on the quote slang but with
a semantic slightly different from vanilla NQP.
 
Like every self respecting slang, Last is an huffmanized syntax for a
specific task, here, the generation of AST trees. It abbreviates the
most common constructs by stealing syntax for  different but related
purposes.  The stolen syntaxes are the function call and method
calls. The abbreviated common constructs are C<QAST::I<Node>> calls,
especially C<QAST::Op>s with C<call> and C<callmethod>
opcodes. C<QAST::I<node>> for integers, number and strings are
replaced by the said values.  Also function calls and method calls are
syntaxes for generating AST for... function calls and method calls.
To make very clear the shape of the AST generated, no other syntax
constructs are allowed in Last.


In essence, depending how you see it, Last is just Perl 6 with a very
restricted syntax or some specific specialized splicing macro system.

Last is used to generate ASTs, but to dump them as well, so it round
trips ; in fact that's how we test part of it. You could not round
trip the previous dump format.  The name is a pun, we hope that last
is a format that will last.  Also it is the last of the serie : PAST,
QAST.  Note: MAST is different, it is specific to MOARVM. MAST code is
generated from Last, or QAST.


Last code is run in a specific phaser; either by itself if it fits in one
statement or within block syntax :

  AST { I<one or many AST statements>  }
  AST  I<exactly one AST statement>

Probably we could have a MAST slang as well!

==Variable node 

! declaration
my lexical
reg registre (usually called local)

   !my param decl =
   reg


In Last, pcode calls are still visible but less conspicuous and AST generation
is mostly a special NQP mode where variables are evaluated in compile time

They are swept under the carpet
of stage 1. There is a way to dump the AST to make it explicit



Last is very much like a simplified nqp except that variable
references are evaluated during compilation.
In other word, Last is nqp code for which the corresponding generated 

 Depending on the name,
what appears to be function calls are in fact the code for either
generation of the AST for an opcode call or an AST node creator.  If
you want to use the result of a compile time function call, do it
before the AST DSL code section, save the result in a variable. This
clearly separates executed code from generated code.

Quick, an example!
Let's see who the following code is rewritten in Last.

    QAST::Op.new(
       :op('callmethod'), :name('load_module'),
       QAST::Op.new( :op('getcurhllsym'),
            QAST::SVal.new( :value('ModuleLoader') ) ),
       QAST::SVal.new( :value($module_name) ),
       $opt_hash,
       QAST::IVal.new(:value($line), :named('line'))

Concise Last code looks like vanilla nqp but use the phaser AST.
<+:$line> is a short for C<+:line($line)> which is itself a short
for C<<iVal($line, :named<line>)>>. The name C<IVal> being a Node name,
this itself a short for C<<QAST::IVal.new(:value($line), :named('line')>>

Concise Last code :

  AST { getcurhhlsym('ModuleLoader').load_module: $module_name, $opt_hash, iVal($line, :named<line>) }
  AST getcurhhlsym('ModuleLoader').load_module: $module_name, $opt_hash, +:$line


The verbose dump style makes the AST more explicit, but it cheats by
using positional arguments in some places. Eventually we can make
the real underlying AST API match this new one.
Also the constructors for literal ints, nums and strs are hidden

   AST {
      Op('callmethod', :name('load_module'),
         Op( 'getcurhllsym', SVal('ModuleLoader'),
       SVal($module_name) ),
       $opt_hash,
       IVal(:$ine))

     
   }


  
The following code is borrowed from the action method C<quotepair> in F<rakudo/src/perl6/Action.nqp>
In the scope of C<::my $/;>, AST constructs  with C<$/> mean a
C<QAST::Var> with a lexical scope. For C<$/>, it is redondant because the
AST compiler already knows about C<$/> lexicality.
C<AST $/ ?? $/.to !! 0> uses the trinary operator as sugar to generate an C<if>
C<QAST::Op>.
The pair syntax sugar C<< AST ~$*key => $*value >> permits to ignore 
the corresponding QAST API : C<$*value.named(~$*key)>

  ::my $/;
  if ... {
    if ... {
       AST $/ ?? $/.to !! 0
    } else {
      $*value := AST +$*value;
    }
  }
  make AST ~$*key => $*value


            if ... {
                $*value := QAST::Op.new(
                    :node($/),
                    :op<if>,
                    QAST::Var.new(:name('$/'), :scope('lexical')),
                    QAST::Op.new(:op('callmethod'),
                        QAST::Var.new(:name('$/'), :scope<lexical>),
                        :name<to>
                    ),
                    QAST::IVal.new(:value(0)),
                );
            } else {
                $*value := QAST::IVal.new( :value($*value) );
            }
        }
        $*value.named(~$*key);
        make $*value;
    }

Indentation can be used in place of parenthetic syntax.



    sub sink($past) {
      my $name := $past.unique('sink');
      AST Want $past, 'v',
        locallifetime
          Stmts
            reg ($name) := $past
            $name.sink() if isconcrete($name) && $name.can('sink')
          $name
    }

    sub sink($past) {
      my $name := $past.unique('sink');
      AST Want( $past, 'v',
        locallifetime(
          Stmts(
            reg ($name) := $past;
            $name.sink() if isconcrete($name) && $name.can('sink')
          ), $name
      ))
    }

  Note that in C<$name.sink() if isconcrete($name) && $name.can('sink')>
 the C<&&> operators avoids to stack C<if> statements.

    reg ($name) := $past;

 is a short for

    QAST::Op.new(:op<bind>,
       QAST::Var.new(:$name, :scope<local>, :decl<var>),
    $past,

  C<reg ($name)> declares a variable whose name is contained in C<$name>.
  Note that C<reg $name> (without parentheses) would declare a variable named C<$name>
 instead.


  Original from F<rakudo/src/perl6/Actions.nqp>

    sub sink($past) {
      my $name := $past.unique('sink');
      QAST::Want.new(
          $past,
          'v',
          QAST::Op.new(
              :op('locallifetime'),
              QAST::Stmts.new(
                  QAST::Op.new(:op<bind>,
                      QAST::Var.new(:$name, :scope<local>, :decl<var>),
                      $past,
                  ),
                  QAST::Op.new(:op<if>,
                      QAST::Op.new(:op<if>,
                          QAST::Op.new(:op<isconcrete>,
                              QAST::Var.new(:$name, :scope<local>),
                          ),
                          QAST::Op.new(:op<can>,
                              QAST::Var.new(:$name, :scope<local>),
                              QAST::SVal.new(:value('sink')),
                          )
                      ),
                      QAST::Op.new(:op<callmethod>, :name<sink>,
                          QAST::Var.new(:$name, :scope<local>),
                      ),
                  ),
              ),
              $name,
          ),
      );
    }






Some composers are supported


The C<AST> keyword is not really a phaser and should not be confused with the C<LAST>
phaser. It introduces code written in the C<AST> slang. The bracket can be ommitted if
the Last code is an uniline.

Last constructs and how they translate in regular nqp

   42                 QAST::IVal.new( :value(42) ) 
   :line(42)          QAST::IVal.new( :value(42), :named<line> ) 
   line => 42         same
   line => (int)$v    QAST::IVal.new( :value($v), :named<line> ) 
   "42"               QAST::SVal.new( :value("42") ) 
   :line(42)          QAST::SVal.new( :value("42"), :named<line> ) 
   line => 42         same
   line => (str)$v    QAST::SVal.new( :value($v), :named<line> ) 
   42.0               QAST::NVal.new( :value(42.0) ) 
   :line(42.0)        QAST::NVal.new( :value(42.0), :named<line> ) 
   line => 42.0       same
   line => (num)$v    QAST::IVal.new( :value($v), :named<line> ) 
   { 42 }             QAST::IVal.new( :value(42) ) 
   &fun(42)           fun(42)
   fun(42)            QAST::Op.new(:op('call'), ???
   if 666 { 42 }
   42 if 666

Grouping parentheses (as opposed as)


* Augmentations of the NQP language to huffmanize it

In nqp code, nqp opcode calls are more current than regular function calls so
we huffmanize the formers at the expense of the latters.

        &say(42)     # mandatory ampersand for function call
        say(42)      #  nqp::say(42)

Some nqp opcode calls are simplified version of postcircumfix operations.
We use a pseudo twiigil and to mark that a postcicumfix operation is really
the opcode call corresponding the the simplified operation

        if $-storage{$key} :exists {
          $-storage{$key}
        }

        if nqp::existskey($storage, $key) {
            nqp::atkey($storage, $key);
        }


The postfix bang specify the addition of a :node($/)


   IVal!(2)
   2!
   
   QAST::Ival.new(:value(2), :node($/))
   
   my $a = var(2)
   AST fun($a)