Dart - Augmenting Constructors

Augmenting Constructors.md

Dart augmenting constructors

Constructors are special in that they exist in many different forms (7 out of 8 possible combinations of being constant, factory and redirecting or not, and we want the eight as well).

One goal of augmentations is that the function signature of a declaration cannot be changed (or at least not in an incompatible way) by applying an augmentation. Looking at a declaration, or a declaration and a number of augmentations that are applied to it so far, it should be possible to derive how the declaration can safely be invoked, and no later augmentation can change that.

The “function signature” of a constructor is more than just its actual function type (which is always a non-generic function type returning the surrounding class’s type). It’s also whether it’s const or factory, since those affect how the constructor can be used (a non-const constructor cannot be invoked as const, a non-factory constructor cannot be invoked on an abstract class, and a factory constructor cannot be used as a super-constructor in a subclass constructor). Being redirecting or not is more of an implementation choice than an API choice, and an augmentation could be allowed to change that.

Further, a non-redirecting generative constructor should initialize the surrounding class’s instance variables and invoke a super-constructor. Whether augmentations are allowed to change how an instance variable is initialized, depends on whether we want to allow it, and whether there is a good way to do so. If only the completely augmented declaration needs to be valid, we can allow prior augmentation steps to leave some variables uninitialized.

Maximally flexible proposal

This is a design that allows maximal “rewritability”, allowing an augmentation to change anything about the augmented constructor that can reasonably be specified. It treats everything in the parameter list, initializer list and body (if any) as implementation which can be replaced, just like an augmentation can replace the body of a function.

Principles of augmenting constructors

A constructor declaration defines a set of properties. These properties have so far been derived directly from the syntax of the sole declaration. With augmenting declarations, we define the properties of a non-augmenting constructor declaration the same as before, and treat augmenting constructor declarations as constructor transformers that transform one set of properties into another set of properties. Then we can define the semantics of the program based on the properties of the fully augmented constructor, the properties of the base constructor declaration transformed by each augmenting constructor declaration in augmentation application order.

The properties that constructor declarations can have are:

• constant (introduced by const): All constructors can be constant or non-constant. Augmentations cannot change that and must repeat the constant-ness.
• factory (introduced by factory): Factory constructors and non-factory constructors are completely separate. Some factory constructors can be augmented into being non-factory constructors, but not all. A non-factory constructor is a generative constructor. We use “generative” rather than “non-factory” to represent not being a factory constructor. An augmenting constructor must repeat the factory in every augmenting constructor declaration, if it wants an augmenting factory constructor.
• redirecting (introduced by the body of the constructor being a generative redirection clause or a factory redirection clause): We use “non-redirecting” about constructors that do not have a redirection clause.
• parameter signature (aka. function signature, where the function is always non-generic and returns the same type). The positions, names, types and optionality of parameters. Augmentations must not change the parameter signature, they must faithfully include a parameter list which has the same signature as the constructor they augment.
• parameter list: The actual parameter list, including syntactic order of named parameters (not that it matters), whether a parameter is an initializing formal or super-parameter, whether a parameter is final or not, and the default value expressions for optional parameters. Augmentations cannot change names of positional parameters, even if the names are not part of the parameter signature. Augmentations cannot declare a default value for optional parameters, but must inherit any default value from that augmented declaration Whether augmentations can change a parameter to be an initializing formal, super-parameter or a final variable depends on whether there any existing code may depend on those properties_
• metadata: List of metadata annotations (introduced by @id, @Id(args) or @Id<Types>(args)): Ordered in source order, if it matters. Can never be removed, only added to.
  • Some tools may include documentation in metadata.
• For non-redirecting generative constructors (the ones that actually initialize new objects) the following can occur, but are all optional:
  • initializer list (introduced by : entry, …, entry): List of initializer list entries , which are each either an assertion or a variable initializer. (May not be declared, in that case the list is just empty.) Augmenting declarations can retain or discard the augmented initializer list, and can augment or replace individual variable initializers.
  • super-constructor invocation (introduced by : …, super(args) or : …, super.name(args): If not declared, defaults to super(). Augmenting declarations can replace the existing super-constructor invocation, or retain it. It cannot be modified. Its meaning can be modified by changing which parameters are super-parameters.
  • body (introduced by having a {statements} rather than a ;). _Augmenting declarations can introduce their own body, which can refer to the augmented constructor’s body (if any) as augmented.
• For redirecting generative constructors:
  • generative redirection clause, introduced by : this(args) or : this.id(args): Required. This is not an initializer list. Although all tools seem to accept assertions prior to the redirecting clause, it’s not actually allowed by the language. It probably should be, in which case we may need to update the rules to allow retaining asserts the same way an augmenting non-redirecting generative constructor can retain initializer list entries.
• For non-redirecting factory constructors:
  • body (introduced by {statements} or => expression, cannot be async or generator): Required.
• For redirecting factory constructors:
  • factory redirection clause (introduced by = Type; or = Type<TypeArgs>;): Required.

Whether a set of constructor properties are valid by themselves, or as a constructor of a surrounding class-like construct, is determined from the properties of fully augmented constructor. Prior intermediate properties may not be complete, but they should not be inherently inconsistent. Because of that, some augmenting constructor declarations are inherently invalid, even if they do not end up being used at all. (For example, no augmenting constructor may declare two parameters with the same name, even if a later augmenting constructor replaces all parameters.)

An augmenting constructor of a different kind (same combination of const/redirecting/factory) must replace the entire constructor with its own implementation, only retaining annotations and inheriting optional parameter default values. Some replacements are not allows (changing constant to non-constant or vice versa, changing generative to factory).

If an augmenting constructor has the same kind as the constructor it augments, it may retain some further properties of the augmented constructor. Currently that only applies to non-redirecting generative constructors, all other constructors have only one functional part, which is required for an augmenting constructor of that kind. Omitting that part makes the augmenting constructor an “unknown” augmenting constructor, which only modifies metadata.

An augmenting constructor always overwrites the parameter list with its own parameter list. It must always be const if the augmented constructor is constant. It must always be generative if the augmented constructor is generative (changing the constructor to a factory changes whether one can use it as a super-constructor in a subclass). Further restrictions are introduced to ensure that the capabilities of the augmented constructor are not broken by the augmentation.

Grammar

An augmenting constructor is one of:

• A non-redirecting generative constructor
• A redirecting generative constructor
• A non-redirecting factory constructor
• A redirecting factory constructor
• An unknown generative constructor
• An unknown factory constructor

where all of these can be const or not (except, so far, the non-redirecting factory constructor), and being const affects the valid expressions in the generative constructors. The “unknown” constructors are augmenting constructors which mention no implementation at all, and they can only be used to append metadata to the existing declaration.

The grammars are:

<augmentingConstructor> ::= 
    <metadata> `augment' `const`? `factory'? <typeName> (`.' <identifierOrNew>)? <parameters> 
         <augmentingConstructorBody>

;; Not really a "body", but the main functional part of the declaration.
<augmentingConstructorBody> ::=
      <augmentingNonRedirectingGenerativeConstructorBody>
    | <augmentingRedirectingGenerativeConstructorBody>  
    | <augmentingNonRedirectingFactoryConstructorBody>
    | <augmentingRedirectingFactoryConstructorBody>  
    | <augmentingUnknownConstructorBody>  

<augmentingNonRedirectingGenerativeConstructorBody> ::=
      <augmentingInitializers> `;' 
    | <augmentingInitializers>? <block>

<augmentingRedirectingGenerativeConstructorBody> ::=
      `:` `this' (`.' <identifierOrNew>)? <argumentList> `;'

<augmentingNonRedirectingFactoryConstructorBody> ::= 
      <functionBody>

<augmentingRedirectingFactoryConstructorBody> ::=
      `=' <typeName> <typeArguments> ';'

<augmentingUnknownConstructorBody> ::=
      `;'

<augmentingInitializers> ::=
      `:' `...'
    | `:' (`...' `,')? (<augmentingInitializerListEntry> `,')* 
            (<augmentingInitializerListEntry> | <superConstructorInvocation>)

<initializers> ::= 
      `:' (<initializerListEntry> `,')* (<superConstructorInvocation> | <initializerListEntry>)

<superConstructorInvocation> ::= 
      `super' (`.' <identifierOrNew>)? <arguments>

<initializerListEntry> ::=
      <fieldInitializer>
    | <assertion>

<augmentingInitializerListEntry> ::=
      `augment'? <fieldInitializer>
    | <assertion>

This introduces new syntax, mostly for non-redirecting generative constructors:

• : ... : Explicitly includes the initializer list entries (not the super-constructor invocation) of the augmented constructor in the result, except that any variables that are initialized by the augmenting constructor, by an initializing formal or field initializer, are removed from the inherited initializers. All asserts are inherited.
• augment this.x = augmented + 1: To augment an existing field initializer, rather than replacing it, the field initializer must be preceded by augment. That then allows accessing the augmented field initializer’s expression as augmented.
• An augmenting non-redirecting generative constructor can specify any of the initializer list, the super-constructor-invocation and a body, but must specify at least one. (Technically it could change only the parameter list, but that’s rarely useful without doing any other change, so we reserve the syntax with no initializer list, super-constructor invocation or body for the “unknown generative constructor” case which only adds metadata, and which applies to redirecting generative constructors too.)

It’s a compile-time error if a parameter list contains an initializing formal or super-parameter, and the surrounding declaration is not an augmenting or non-augmenting non-redirecting generative constructor declaration.

It’s a compile-time error if a <fieldInitializer> expression contains an augmented expression, and the field initializer is not preceded by an augment modifier. (Can only use augmented when augmenting a field initializer.)

It’s a compile-time error if an augmenting constructor declarations contains two parameters with the same name, two initializer list variable initializers with the same name, or an initializing formal and a field initializer with the same name. As usual, it also applies to augmenting constructors.

Example:

class S {
  S(int _);
}
class C {
  final int x;
  final int y;
  C(this.x, int y, super.z) : assert(x > y), y = y * 2 { print("${this.x},${this.y}"); }
  
  augment C(int x, int y, int z) : ..., assert(z > y), x = x, augment y = augmented + 1, super(z) {
    print("before");
    augmented;
    print("after");
  }
    
  // Equivalent to:
  augment C(int x, int y, int z) : assert(x > y), y = (y * 2) + 1, assert(z > y), x = x, super(z) {
    print("before");
    print("${this.x},${this.y}"); 
    print("after");
  }
}

Semantics

The meaning of an augmenting constructor is defined relative to the constructor it augments, which is itself possibly the result of applying a number of augmentations on top of a base declaration.

Any non-“unknown” augmenting constructor can completely replace the behavior of the constructor it augments, or it can incrementally modify parts of it, subject to some soundness-based restrictions:

• It’s a compile-time error if an augmenting constructor is marked const and its augmented constructor is not constant.
• It’s a compile-time error if an augmenting constructor is not marked const and its augmented constructor is constant.
  • (Augmentations must agree with, and repeat, the const modifier of the constructor they augment.)
• It’s a compile-time error if an augmenting constructor is marked factory and its augmented constructor is not a factory constructor. (Cannot augment a generative constructor with a factory, but the other direction may be allowed.)
• It’s a compile-time error if an augmenting constructor is not marked factory, its augmented constructor is a factory constructor, and the surrounding class is abstract. (Cannot change a factory constructor to a generative constructor in an abstract class, someone might call it.)
• It’s a compile-time error if an augmenting constructor and an augmented constructor does not have the same function signature. Since the return type is given and constructors cannot be generic, that means having the same parameter list structure.
  • The same number of positional parameters.
  • The same names of named parameter.
  • Corresponding parameters (same position or name) have the same type, and are either both optional and both required.
  • This does not include whether the variable is final, an initializing formal or a super-parameter, or any default values. Augmenting constructors (currently) cannot declare default values. The result of augmenting a constructor takes the default values from the augmented constructor.
• It’s a compile-time error if a position parameter of an augmenting constructor does not have the same name as the corresponding positional parameter of its augmented constructor. While renaming positional parameters could be possible, it’s just not considered worth the complexity.

Applying an augmenting constructor to an augmented constructor (as a set of properties) produces a new constructor (another set of properties). We specify the specific rules for each kind of augmenting constructor.

The only general rule is that the metadata annotations of the resulting constructor is the metadata annotations of the augmented constructor followed by the metadata annotations of the augmenting constructor. This applies to all valid augmentation applications below, and we won’t mention metadata again.

Non-redirecting generative constructor

The most complicated kind of constructor is the non-redirecting generative constructor. Such a constructor contains the following parts, which are somewhat intertwined, but where we allow each to be augmented independently:

• The parameter list, which can contain initializing formals and super parameters.
• The initializer list which can contain assertion entries and field initializer entries.
• The super-constructor invocation (syntactically placed at the end of the initializer list, but treated as a separate entity).
• The body, which can be a function body or no body, the latter represented syntactically as ;.

Augmenting any other kind of constructor with an augmenting non-redirecting generative constructor will replace the constructor completely. The result is a constant constructor if both are constant, and it’s a always generative non-redirecting constructor. The parameter list of the result is the parameter list of the augmenting constructor, only with each optional parameter getting the default value expression of its corresponding parameter in the augmented constructor. The result cannot refer to any part of the augmented constructor.

• It’s a compile-time error if an augmenting non-redirecting generative constructor C augments a constructor which is not a non-redirecting generative constructor, and:
  • C has an augmenting initializer list that starts with ....
  • C has an augmenting field initializer in its initializer list (one which starts with augment), or
  • C has a body that contains an augement expression.
• It’s a compile-time error if C is marked const and has a function body (as usual).

When an augmenting non-redirecting generative constructor augments another non-redirecting generative constructor, it can replace each parameter, replace the entire initializer list, or retain the original initializer list and replace or augment individual field initializers, replace or retain the super constructor invocation, and replace or retain the body.

In that case, replacing everything (and not using augmented) should still be safe. Replacing some parts, but not all, may not be internally consistent. The combinations that can cause problems at the augmentation level (not yet worrying about whether the final result is valid, including whether all instance variables are initialized, and whether the invoked super constructor exists and is given valid arguments) are:

• Changing a parameter to being unassignable (final, initializing formal, or super parameter) while retaining any code that may assign to it.

• Changing a normal final parameter to an initializing formal or super-parameter, and retaining the augmented constructor’s body or accessing it using augmented. A normal parameter is in scope in the body, an initializing formal or super parameter’s variable is only in scope in the initializer list. The augmented body could be referencing the parameter variable.

• Changing an initializing formal or super parameter to a normal parameter, and retaining the augmented constructor’s body or accessing it using augmented. The augmented body could be relying on the variable’s name not being in scope.

Those situations, and any referencing of parts of the augmented constructor which doesn’t exist, are represented by the following errors, where C is an augmenting non-redirecting generative constructor, augmenting a non-redirecting constructor P:

• It’s a compile-time error if C’s initializer list contains an augmenting field initializer (one starting with augment) with field name n, and P‘s initializer list does not contain a field-initializer and its parameter list does not contain an initializing formal with name n. (That is, if P does not initialize n.)

• It’s a compile-time error if C is not a constant constructor, a parameter of C is a final normal variable, an initializing formal or a super parameter, the corresponding parameter of P is a non-final normal variable, and any of:

  • C retains initializer code from P, defined as:
    • Either the initializer list of C starts with ... (may retain at least assertion code),
    • or the initializer list of C has an augmenting variable initializer entry whose expression contains an augmented expression.
  • C has no super-constructor invocation and P has a super-constructor invocation.
  • C retains body code from P, defined as:
    • Either C has no body and P has a body,
    • or C has a body which contains an augmented expression.

• It’s a compile-time error if a parameter of C is an initializing formal or super-parameter, the corresponding parameter of P is a normal parameter, and C retains body code from P. (The retained body could refer to the parameter variable, and the initializing formal or super parameter does not introduce a variable which is in scope in the body.)

• It’s a compile-time error if a parameter of C is a normal parameter and the corresponding parameter of P is an initializing formal or super-parameter, and C retains body code from P. (C introduces a new variable which is in scope in the body.)

The result of applying the augmentation C to the declaration P is a declaration R, which is always a non-redirecting generative constructor declaration with the modified properties specified below.

Parameter list:

The parameter list of R is the parameter list of C, where each optional parameter is updated to have the same default value as the corresponding variable has in P, if any.

Initializer list: 

If C has an augmenting initializer list that starts with ..., then the initializer list of R is constructed as follows, starting with an empty list:

• For each initializer list entry of P in the same order:

  • If the entry is an assertion, append it to the initializer list of R..

  • If the entry is a field initializer then:

    • If C contains an initializing formal or non-augmenting field initializer with the same name, do nothing.

    • If C contains an augmenting field initializer with the same name, append the field initializer of C, without the leading augment to the initializer list if R.

      • An augmented  in the initializer expression of the field initializer from C denotes the initializer expression of the field initializer from P that it replaced.

    • Otherwise append the field initializer of P to the initializer list of R.

• The for each each initializer list entry declared by C in source order:

  • If the entry is an assertion or non-augmenting field initializer, it is appended to the initializer list of R.
  • If the entry is an augmenting field initializer:
    • If the initializer list of P contained a field initializer with the same name, then do nothing. The entry was already appended earlier.
    • Otherwise append the entry to the initializer list of R.
      • An augmented in the initializer expression of the entry denotes the parameter of the same name. Which must exist, since P must have an initializing formal with that name.

Otherwise, if C has an augmenting initializer list that does not start with ... (if it has any :, even if only followed by a super-constructor invocation), then R has the same initializer list entries as C, in the same order, without any leading augment.

• If the initializer expression of an augmenting field initializer of C contains an augmented expression, it denotes either the initializer expression of a field initializer with the same name in P, or the parameter variable with the same name in P (one of which must exist).

Otherwise C has no augmenting initializer list (no : at all), and then:

• The initializer list of R contains the initializer list entries of P in the same order, except that the following entries are omitted:
  • Any instance variable initializer entries where the parameter list of C contains an initializing formal with the same name.

When evaluating an initializer expression coming from an augmenting field initializer of C, evaluation of the augmented expression gives the same result as evaluating the expression denoted by that augmented (as described above) in the same initializer list scope that the initializer expression of C was itself evaluated in. (Not containing any nested names introduced by, for example, patterns.) That is, it accesses the same local variables introduced by the constructor’s parameter list.

Super-constructor invocation:

If C has a super-constructor invocation, then R has the same super-constructor invocation as C.

Otherwise, if P is a non-redirecting generative constructor that has a super-constructor invocation, then R has the same super-constructor invocation as P.

Otherwise R has no super-constructor invocation. (Which ends up equivalent to an invocation of super(), if no later augmentation adds a super-constructor invocation.)

There is no way to erase an existing super constructor invocation, but it can be replaced with an explicit super().

Body:

If C has a body declaration (a {statements} body, not a ;), then the body of R is the body of C.

Otherwise, if P is a non-redirecting generative constructor that has a body, then the body of R is the body of P.

Otherwise R has no body.

If the body of C contains an augmented expression, then that expression has static type void, and evaluation of that expression will execute the body of P, if one exists. It is executed in the same body scope that the body of C was executed in. If that execution throws, so does evaluating augmented, otherwise it evaluates to null. (If the body of C introduces new local variables surrounding the augmented expression, those have no effect on the scope that the body of P is executed in). If P has no body, evaluation of augmented evaluates immediately to null. (Using augmented is allowed even if the augmented declaration has no body. That makes it easier for, for example, a macro to just do preOperations(); augmented; postOperations(); without having to check whether the augmented constructor wrote ; or {}.)

Redirecting generative constructors

An augmenting redirecting generative constructor replaces the entire augmented constructor with a redirecting generative constructor with the parameter list of the augmenting constructor with default values from the augmented constructor, and the same generative redirection clause as the augmenting constructor.

There is only one functional part of a redirecting generative constructor, the : this(args) or this.name(args) clause, so an augmenting redirecting generative constructor is assumed to want to replace that. It makes no sense to use an augmented to refer to the augmented declaration, even if it is of the same kind. If the only goal is to append metadata, an unknown augmenting constructor should be used.

Non-redirecting factory constructors

An augmenting non-redirecting factory constructor can only augment factory constructors, but can augment both redirecting and non-redirecting constructors. A factory constructor is very similar to a static function, and augmenting works in mostly the same way.

If an augmenting non-redirecting factory constructor C augments a factory constructor P, then the resulting constructor R is a factory constructor with the parameter list of C, updated with the default values of the parameter list of P, and the same body as C. Evaluating an augmented(args) expression in the body of C will invoke the factory constructor P with the provided arguments, either executing the body of a non-redirecting P with parameters bound to the provided argument list, or invoking (by name, so a fully augmented declaration) the constructor targeted by a redirecting P. The result of that invocation is the result of augmented(args).

Redirecting factory constructors

A augmenting redirecting factory constructor replaces the factory constructor it augments.

If an augmenting redirecting factory constructor C augments a factory constructor P, then the resulting redirecting factory constructor R is constant if C is, has the parameter list of C updated with the default values of P, and the factory redirection clause (= ClassName<TypeArgs> or = ClassName<TyperArgs>.name) of C.

Unknown generative or factory constructor

An augmenting constructor which declares no initializer list, super-invocation, body, or redirection clause is an “unknown” constructor declaration. It’s agnostic about whether the augmented declaration is redirecting or not. The augmenting constructor must have a parameter list that is equivalent to the constructor it augments (same final-ness, same initializing formals and super-parameters if allowed), and it must be be factory if the augmented constructor is a factory, and generative if the augmented constructor is generative. (And const if the augmented constructor is constant, no exception from that general rule.)

An unknown augmenting constructor only adds metadata to the augmented constructor. The augmented constructor has the same constant-ness, factory-ness, parameter list, and other constructor-kind-dependent properties as the augmented constructor, the only change is the possible addition of metadata (including documentation).

Conclusion

This specifies a very permissive design for augmenting constructors. It tries to allow an augmenting constructor to change anything that isn’t breaking for code written against the augmented constructor. That’s what we do for functions, where it’s possible to completely replace the body of the function, which means completely change what it does.

This is (I think) the upper limit of what is soundly possible. The only exceptions are:

• Not allowing changing default values.
• Not allowing changing names of positional parameters.

Of these, the former can probably be modified, at least to allow adding a default value where one is missing. Changing positional parameter names is just not useful, but it might be reasonable to allow an augmenting constructor to change a _ name to an actual name, or in the other direction where the augmented constructor can then inherit the name from the parent (which must still not be used in the scope where it’s named _).

This flexibility does make for some fairly intricate rule for what is and isn’t allowed with constructors, or with non-redirecting generative constructors at least, which is where the complexity is. This design does not try to ensure that the result is a valid constructor in a Dart program, only that it can be given a meaning if it is valid. We’ll need more validation (don’t want, fx, an invalid super-constructor invocation to be allowed, even if it is completely replaced). And we may want some restrictions, and not allow everything here.