littlenag/expressive_metaprogramming.md

## expressive_metaprogramming.md

      
    Raw
  

              expressive_metaprogramming.md
            
          
    Expressive Metaprogramming for Scala 3

Overview

Scala 3 introduces a new macro system that replaces the experimental Scala 2
system with one that should avoid unsoundness, be simpler to use, and simpler
for the compiler team to evolve and maintain.
Perhaps the most significant feature missing from Scala 3 today is the lack of support
for expressive compile-time metaprogramming. By this, I mean the ability to generate
and/or reshape classes, traits, and objects at compile-time. In Scala 2, the
community used macro annotations for this purpose. Scala 3 addressed a few of
the use-cases of macro annotations, but we still lack a feature that offers
anything close to the full range of expressive capability of macro annotations.
This document proposes a replacement for Scala 2 macro annotations that should:

offer nearly same expressive power,
address all, or most, major use cases,
be simpler to maintain, and
be safer and easier to use.

On the implementation side this replacement should:

be integrated with the compiler (instead of a plugin),
be simpler to maintain,
expose no compiler internals, and
re-use the existing macro system and its semantics as much as possible.

Most importantly, this feature should fit cleanly into the existing Scala 3
metaprogramming model and existing language features.
Desired Metaprogramming Capabilities

There are two key design principles this feature should follow:

User code as written must not be altered, unless the code is explicitly
marked as modifiable.
Nothing extraneous should be required to be generated by a macro.

Due to various limitations, macro annotations in Scala 2 are unable to adhere to
either design principle.
Generally, macro annotations are hard for authors to make correct, difficult for
users to introspect, and effectively unrestricted in their capabilities. This is
because at their core they are simply a function of AST => AST. This means
they can:

alter the inheritance hierarchy of a class/trait/object,
change order, types, or names of class parameters,
modify generic type parameters,
and more.

Instead, this proposal seeks to cleanly integrate with the more restrictive
metaprogramming model in Scala 3. Features would be limited to:

adding new declarations (fields, methods, types, classes, etc) at compile-time
into existing named traits/classes/objects/packages
as a function of:

content of the enclosing trait/class/object
static parameters


overriding existing declarations when explicitly allowed.

Import/Export Macros

By extending the semantics of import and export we can achieve most of the
stated design goals.
Today import and export have the exact same syntax, but dual functionality.
Both take a path to a stable identifier followed by selectors that determine
which symbols should be processed and how. The syntax could be extended to allow
import and export to accept a path to a macro and along with static
arguments to the macro, instantiate the macro, and then process the resulting
object as if it were any other object.
Import Macros

The import keyword makes visible selected symbols from the object or package
specified by some path. The syntax could be extended to allow import to import
symbols into the current scope from a macro synthesized object.
Let's look at an example to get a feel for how this could work:
def someMacro(b: Expr[Boolean])(using Quotes, cx: ImportDecl & EnclosingTemplate): Expr[cx.Decls] = {
  if (b.value)
    cx.decls('{
      object freshTermName {
        def fizzle: Boolean = true
      }
    })
  else
    cx.decls('{
      object freshTermName {
        def swizzle: Double = -1.0d
      }
    })
}

trait Tum {
  def tim: String
}

class Foo extends Tum {

  import ${someMacro(true)}.*

  def tim = fizzle.toString // only fizzle is visible in this scope
}

class Bar extends Tum {

  import ${someMacro(false)}.*

  def tim = swizzle.toString // only swizzle is visible in this scope
}
In this example, classes Foo and Bar both have a method called tim. But
the implementation of that method can depend on the symbols that become visible
after evaluating the import macro. Because the macro evaluates differently in
the two classes, the symbols visible and their contents, is different.
Unlike regular macros, those called with import would take as a given both a
Quotes and a context. The context serves three purposes:

It allows the compiler to pass a concise easy-to-use description of the
macro's surroundings.
It constrains the macro to the semantics of the invoking statement.
It allows the macro to specify to both the compiler and users the contexts
the macro is valid in.

In the above example we see the macro accept two marker traits: ImportDecl and
EnclosingTemplate. These traits constrain where the macro is valid. The
EnclosingTemplate parameter describes the content of the surrounding object
the macro was called from, including any methods, fields, constructors, type
parameters, etc, as well as the full AST of the object.
The context would also be used to ensure that an AST of the appropriate shape is
returned. This is to maintain some level of type-safety so that only
object-shaped ASTs are returned by an import/export macro. Unfortunately, object
declarations themselves do not have a well-specified type in Scala 3. As a
solution, we use the context to ensure that an AST of the appropriate shape is
returned.
A deeper discussion of contexts follows in a later section.
Of course, normal import rules would apply to the synthesized object. Private
and protected members would be hidden, etc.
Import macros couldn't change the shape of an object, yet they would be very
useful in many situations, especially those where you don't want to alter the
shape of an object.
However, fully expressive metaprogramming should be able to change an object's
shape. This is the domain of export macros.
Export Macros

The export keyword currently adds forwarders to selected members of an object.
The syntax could be extended to allow export to add forwarders to members of
objects instantiated at compile-time by a macro. Let's look at an example to get
a feel for how this could work:
def someMacro(b: Expr[Boolean])(using Quotes, cx: ExportDecl & EnclosingTemplate): Expr[cx.Decls] = {
  if (b.value)
    cx.decls('{
      object freshTermName {
        def fizzle: Boolean = true
      }
    })
  else
    cx.decls('{
      object freshTermName {
        def swizzle: Double = -1.0d
      }
    })
}

class Foo {
  export $someMacro(true).*
}

class Bar {
  export $someMacro(false).*
}

new Foo.fizzle  // valid
new Foo.swizzle // invalid!

new Bar.swizzle // valid
new Bar.fizzle  // invalid!
In this example, the class Foo would have the method fizzle added, whereas
the class Bar would have the method swizzle added.
As we saw with the import macro, this macro also takes givens of a Quotes
and a context. The ExportDecl trait constrains the macro to be used only in
export statements.
By default, normal export rules would apply to forwarding declarations of the
synthesized object. Relaxations of these rules are discussed in later sections.
Context Restrictions

Not every import/export macro will be valid in every context. For example,
macros should be able to specify:

if they work with import, or export, or both;
if they work inside an object/class/trait template, or outside a template
at package-level (aka "top-level"), or both.

(Scala 3 has introduced top-level declarations. This means that supporting
top-level import/export macros should be possible and readily achievable.)
A macro may only be valid, or intended to be used, in one context and should
only accept the contexts of its intended call site(s). The compiler would then
generate an error, perhaps with an annotation present to supply a customized
error message, if the macro was called from an incompatible location.
The restrictions can be described by two pairs of traits. One (or both) trait of
each pair would need to be specified in the macro signature.
The traits ImportDecl and ExportDecl constrain the macro to either import or
export semantics, respectively.
The traits EnclosingPackage and EnclosingTemplate constrain the macro to
either package instantiation sites where it would introduce top-level
declarations or class/trait/object template sites. The traits would also
describe the surrounding package, or the surrounding class/trait/object and its
contents, respectively.
Examples:
def onlyExportsInObjects()(using Quotes, cx: ExportDecl & EnclosingTemplate): Expr[cx.Decls]

def onlyImportsAtPackageLevel()(using Quotes, cx: ImportDecl & EnclosingPackage): Expr[cx.Decls]

def anyPlaceAnyWhere()(
  using Quotes, 
  cx: (ImportDecl | ExportDecl) & (EnclosingTemplate | EnclosingPackage)
): Expr[cx.Decls]
Simplified Splice Syntax

The syntax for calling a macro, referred to as a Splice in Scala 3, in the
context of an import or export is slightly changed and simplified from its
use in other contexts.
In Scala 3 a Splice looks like:
${...}

This is because expression-oriented contexts require the additional syntax in
order to disambiguate the call. The rules of paths are stricter, and not
expression oriented. This means we can simplify the syntax in the import and
export contexts.
The syntax proposed is:
$path.to.macro(..args..).<selector>

For example:
import $some.deep.path().*
export $visibleMacro(1, false, "str", Array("a", "b", "c")).*
To import or export a macro, the path would be required to start with a $,
followed by the path to the macro function, and finish with required parenthesis
containing the arguments, if any, to the macro.
For the POC, arguments would be restricted to the set of literal values
supported by annotations. This would be relaxed to support more literal types in
time.
Ammonite

While the aforementioned syntax may feel a bit odd, users of Ammonite should be
comfortable with it since Ammonite already does something very similar with its
"magic imports".
For example import $file.ScriptFile._ is special syntax interpreted by
Ammonite to load the symbols from another script file, named ScriptFile.sc, and
brings them into the current file.
Given the popularity of Ammonite, this should make the syntax readily
acceptable, with a large user-base already accustomed to the functionality.
The syntax also gives Ammonite a chance to replace it's "magic imports" with
standard Scala syntax.
The prior example could become import $file("ScriptFile").*, where file is
now simply a macro that Ammonite makes visible to all scripts.