Def macros were designed to become part of an already mature language with an almost ten-year history. A very important consideration was their interaction with existing features of Scala. Uncovering these interactions was akin to a gold rush, with our team and early adopters digging everywhere and every now and then uncovering nuggets of novel techniques.
One of the first things that we did when working on def macros was going through existing flavors of methods. For every such case, we tried to change the corresponding regular method into a macro def and considered what's going to happen.
Metadata. Most definitions in Scala, excluding only packages and package objects, allow user-defined annotations that can be read at compile time and/or runtime. Macro defs are no exception. Thanks to their ability to attach custom compile-time metadata to definitions, annotations have been used to share information in situations that involve interplay between macros.
Modularity. Scala has a rich set of features that allows to manage scope and accessibility of its definitions. All these features work as usual with macro defs. Much like with regular defs, it is possible to define macro defs both in local scope and in member scope of an enclosing definition. If necessary, macro defs can also be private and protected.
Inheritance. One of the key characteristics of regular defs is dynamic dispatch. Scala supports the usual mix of features that support dynamic dispatch: subclassing, overriding, abstractness and finality. Additionally, there is a less mainstream notion of mixin composition and the associated concepts of linearization and abstract override.
Def macros expand at compile time, which means that dynamic dispatch is out of the question, and so are some of the features that make sense only for dynamically dispatched methods. In particular, this means that macro defs cannot be abstract (there is no syntactic possibility for this anyway) and cannot override abstract methods.
Interaction of def macros and overriding has been controversial for quite a while. On the one hand, one can imagine
situations when it may seem desirable for a macro def to override a regular method.
For instance, authors of a collection library may want
to provide an optimized implementation of Range.foreach that overrides the inherited iterator-based
implementation of Seq.foreach
with a much simpler loop over the range boundaries. Towards that end, one may declare Range.foreach as a macro def that overrides
Seq.foreach which is a regular def.
On the other hand, while overriding may solve the problem in simple cases when the type of the object is known statically, it doesn't help if the exact type is only known at runtime. This means that such optimizations are going to be unreliable.
In order to let the community figure out the story of macro-based optimizations, we allowed macros defs to override both regular defs and other macros defs. Unfortunately, this capability doesn't have much adoption, so we plan to prohibit the new version of macros from participating in overriding.
Implicits. Depending on the signature, a regular implicit def can serve either as a default value for accordingly-typed implicit parameters or as a conversion between otherwise incompatible types. Both these roles of implicit defs can benefit from compile-time metaprogramming, so we allowed macros defs to be implicit, too. The technique of implicit materialization enabled by implicit macros represents one of the most important use case of def macros at the time of writing.
Constructors. At the time of writing, it is impossible to define a constructor as a def macro. Nonetheless, it is conceivable to want to do so for consistency, because object instantiation is one of the few language constructs that cannot be enriched by macros.
Some time ago, we have proposed a Scala compiler patch
that implements support for declaring secondary constructors as macros in the context of
the research on virtual traits.
Such macros would expand into calls to other constructors of the underlying class or just regular code.
The patch was rejected during code review, because the potential for changing the meaning of new
was deemed undesirable.
Signatures. Macro defs have exactly the same signature elements as regular defs - name, optional type parameters, optional term parameters, optional return type.
There are no theoretical limitations on the parameters of macro defs (even though declaring a macro def parameter to be by-name doesn't make any difference in behavior, it still makes sense). However, in practice, we disallow macro defs that have default parameters. We didn't have time to implement this functionality for the initial release of def macros, and a follow-up patch that introduced it got rejected during code review because of a disagreement that involves compiler internals.
There is, however, a restriction on return type inference. Regular defs can have their return type inferred from the type of their body. Naturally, since macro defs have an unusual body, the usual algorithm is no longer applicable for inferring their return types.
In the initial version of def macros that required macro impls to take and return exprs,
we used to infer macro def return types from return types of their corresponding macro impls.
For instance, if a macro impl returns Expr[Query[U]], then we could infer the return type
of the corresponding macro def as Query[U].
Now when we allow macro impls to return plain trees, return type inference for macro defs is no longer possible in general case. Therefore, we marked this feature as deprecated and don't plan to support it in our future macro system.
Overloading. Like regular defs, macro defs can be overloaded - both with regular defs and other macro defs. This works well, because overload resolution happens before macro expansion.
Additionally, since no bytecode is emitted for macro defs (due to def macros only working at compile time), there is no restriction that macro defs must have their erased signature different from erased signatures of other methods.
Types. When it comes to nesting, Scala doesn't have many limitations. Terms can be nested in types, types can be nested in terms, and a similar story is true for definitions. As a result, some advanced types, namely compound types and existential types, can contain definitions. While existentials can only define abstract vals and abstract types, compound types may include any kind of abstract members.
During a spontaneous discussion with community members, we discovered that the internal compiler structure that represents definitions of compound types can also hold macro defs. By the virtue of using scala.reflect, macros can tap into compiler internals and emit unconventional compound types that contain macro defs. Users of such strange compound types can call those macro defs and trigger macro expansion as if these macro defs were declared in regular classes. This possibility sounds very obscure, but we have been able to successfully use it as a key component in emulation of type providers.
From the discussion above, we can see that macro defs are a relatively seamless extension to regular defs. The differences are: 1) restrictions on overriding, because macro defs don't exist at runtime, 2) almost non-existent return type inference, because macro defs have unusual bodies, 3) inability to be secondary constructors or have default parameters, because macro defs didn't initially support this functionality, and our follow-up patches were rejected.
The same analysis that we applied to macro defs can be applied to macro applications. If we go through all flavors of method applications and contexts where method applications can be used, trying to replace usages of regular methods with usages of macros, we obtain the following results.
Fully-specified applications. If a macro application has all type arguments and all term argument lists specified according to the signature of the macro def, then the compiler expands it by calling the corresponding macro impl with these arguments as described in Appendix B.
Missing type arguments. When a macro application doesn't have type arguments, and the corresponding macro def does, type inference kicks in before macro expansion happens. Refer to Appendix B for a detailed explanation of how type inference works for macro applications.
Partial applications. Regular defs can be partially applied, i.e. can have their applications contain less term argument lists than their corresponding defs. Macro defs also support partial application, but with some restrictions.
Missing implicit arguments lists are discussed below. Missing empty argument lists are appended automatically, exactly like for regular defs. Other cases of partial application involve missing arguments that cannot be inferred. For regular defs, this is handled by eta expansion which converts a partial application into a function object that can be provided with remaining arguments at runtime. However, since def macros expand at compile time, eta expansion for them is prohibited.
Missing implicit arguments. Before macro expansion happens, the compiler makes sure that the macro application has its implicit arguments figured out. If implicit arguments were not specified explicitly, the typechecker launches implicit search to infer them. There are no restriction on where these implicit arguments come from - both regular vals, regular defs and macro defs are allowed.
Missing default arguments. Unlike regular defs, macro defs cannot have default parameters, so macro applications cannot have default arguments.
Named arguments. Since named and default arguments are implemented by the same subsystem of the typechecker, the fact that default arguments are unsupported also outlaws named arguments.
Vararg arguments. Macro applications can have zero or more arguments corresponding to the same vararg parameter of the macro def. In that case, the macro impl must also have a vararg parameter. The macro engine wraps every vararg argument individually, and then passes the collection of these arguments to the vararg parameter of the macro impl.
By-name arguments. We allow by-name parameters for macro defs, and treat their corresponding arguments as if the by-name modifier was missing. It is the responsibility of the macro writer to respect the evaluation strategy in the macro expansion.
Structural types. When the prefix is a method application has a structural type, and the method is declared only in the refinement of that type, such an application cannot be compiled in a conventional way on the JVM. In this situation, the Scala compiler emits bytecode that uses JVM reflection to dynamically lookup and invoke the required method at runtime. Since such bytecode typically leads to a significant performance degradation, the compiler requires such applications to be enabled by a special built-in import or a dedicated compiler flag. To the contrast, macro applications are expanded at compile time, so invoking macros on structural types is allowed without any special requirements.
Desugarings. An interesting peculiarity of Scala is that a significant number of its language features, e.g. assignment, pattern matching, for comprehension, string interpolation and others, are oftentimes desugared into method applications. As a result, these features can be transparently enriched by macros. This simple idea has many important implications.
As we can see, macro applications almost seamlessly integrate with the existing infrastructure of method applications. The differences are: 1) special treatment of whitebox macros by the type inference algorithm, because macro expansion can manipulate type inference, 2) almost non-existent partial application, because macro applications expand at compile time, 3) lack of support for named and default arguments, because macro defs didn't initially support this functionality, and our follow-up patches were rejected.
An important feature of Scala is bidirectional interoperability with Java. The main target platform of Scala is the JVM, and the Scala compiler emits bytecode that is very close to the what the Java compiler would emit for idioms that have correspondence in Java.
As a result, Scala programs can easily use libraries written in Java (call methods, extend classes and interfaces, etc - as if they were written in Scala). Java programs can also use libraries written in Scala (of course, Scala features like implicit inference won't be available, so the corresponding Java code is more verbose). This is very important practically, because there are popular libraries written in Scala (Akka, Play, Spark, etc) that are also used by Java programmers.
Def macros are one of the rare exceptions to the Java compatibility guideline. Since they operate in terms of a Scala language model, they cannot be realistically supported in a Java compiler. As a result, def macros cannot be used in Java programs.
In order to centralize the discussion, post your questions, feedback and ideas to http://gitter.im/scalameta/sips.