Author: Kris Mok Date: 2020-04-29
Link to this Gist: https://gist.github.com/rednaxelafx/e9ecd09bbd1c448dbddad4f4edf25d48
This note is written in the context of Apache Spark 3.0's ClosureCleaner support for the "indylambda" style lowering of closures, which is the new default since Scala 2.12, SPARK-31399.
For discussions on Apache Spark's ClosureCleaner, please refer to the spark_closurecleaner_notes.md file in this Gist.
c.f. indylambda: Putting invokedynamic to work for Scala by @retronym (Jason Zaugg) on the design of indylambda in Scala.
Jason also kindly gave some comments on this note with extra information: https://twitter.com/retronym/status/1256015846784159744
Jargons:
- Old way = inner-class based =
delambdafy:inline(equivalent to default behavior in Scala 2.11) - New way = indylambda (invokedynamic+LambdaMetaFactory based) =
delambdafy:inline(new default since Scala 2.12) - LMF =
LambdaMetaFactory= a set of built-in BootstrapMethods for Java 8 lambdas.
In Scala 2.12, the Scala compiler (NSC) adopted a new way of implementing lowering of Scala lambdas down to Java bytecode, much like Java 8's lambdas. This feature is called "indylambda" in Scala, controlled by the Scala compiler option -Ydelambdafy:method.
The most relevant phases in NSC for lowering lambdas are: uncurry, lambdalift, delambdafy. For Scala 2.12, I'm using scalac -Xprint:delambdafy -Xshow:delambdafy for the examples in this Gist to poke into what NSC is up to.
For example, a lambda literal that looks like this in Scala source code:
// assuming we're in a named method bar() in class Foo here, and there's local variable freeVar:Int
(arg: Int) => arg + freeVar + 42NOTE: this declares a lambda function, and also creates a lambda closure object for representing a runtime instance of this lambda. In the Java world, it's classes that ultimately implement interfaces, so a scala.Function1 reference ultimately needs to point to an object backed by a class.
In the indylambda style of lambda lowering, the lambda literal would be lowered to multiple parts:
- The lambda body (e.g.
arg + freeVar + 42) gets lowered into a static method in the class that declared this lambda - The lambda closure object creation site gets lowered into an
invokedynamicinstruction that calls a bootstrap method to create the object, passing in whatever state that needs to be captured as arguments (e.g.freeVar) - other misc supporting code (e.g. lambda deserialization)
... so we've got a method for the body, and the invokedynamic will magically create an object instance that represents an instance of this lambda, but where's the backing class that glues together the functional interface/trait and the implementation method?
That's where the LambdaMetaFactory in Java's standard library comes in. It'll generate an equivalent of the annonymous inner class "glue" at runtime for these lambdas, implementing the specified functional interface. So, in effect, at runtime it's not that different from the old way of eagerly generating inner classes.
Simple contrived pseudocode for the example above would be:
// lambda closure object creation site: generated by NSC
// This is in the place of the lambda literal
// InvokeDynamic #0:apply$mcII$sp:(I)Lscala/runtime/java8/JFunction1$mcII$sp;
invokedynamic LambdaMetaFactory.altMetaFactory(freeVar) // returns an object whose class implements scala.Function1
// lambda body: generated by NSC in class Foo
// notice how the argument list is a list of captured state passed in as argument, then the actual explicit
public static def $anonfun$bar$1(freeVar: Int, arg: Int): Int = arg + freeVar + 42
// lambda deserialization: generated by NSC in class Foo
private static def $deserializeLambda$(lambda: java.lang.invoke.SerializedLambda): Any = {
// InvokeDynamic #1:lambdaDeserialize:(Ljava/lang/invoke/SerializedLambda;)Ljava/lang/Object;
invokedynamic scala.runtime.LambdaDeserialize.bootstrap(...)
}
// lambda closure class: generated by java.lang.invoke.LambdaMetafactory (in OpenJDK8 the actual impl is java.lang.invoke.InnerClassLambdatory)
public /* synthetic */ class Foo$$Lambda$1
extends scala.runtime.java8.JFunction1$mcII$sp,
scala.Serializable {
private val arg$1: Int = _
def <init>(arg$1: Int): Unit = {
this.arg$1 = arg$1
}
def apply$mcII$sp(unnamed_arg1: Int): Int = {
Foo.$anonfun$bar$1(this.freeVar$1, unnamed_arg1) // invoke the actual implementation
}
private static def get$Lambda(arg$1: Int): scala.runtime.java8.JFunction1$mcII$sp = {
new Foo$$Lambda$1(arg$1)
}
private def writeReplace(): Any = {
new java.lang.invoke.SerializedLambda( // this is a serialization proxy
capturingClass = classOf[Foo],
functionalInterfaceClass = "scala/runtime/java8/JFunction1$mcII$sp",
functionalInterfaceMethodName = "apply$mcII$sp",
functionalInterfaceMethodSignature = "(I)I",
implMethodKind = 6, // java.lang.invoke.MethodHandleInfo.REF_invokeStatic, c.f. https://docs.oracle.com/javase/8/docs/api/java/lang/invoke/MethodHandleInfo.html
implClass = "Foo",
implMethodName = "$anonfun$bar$1",
implMethodSignature = "(II)I",
instantiatedMethodType = "(I)I",
capturedArgs = new Array[Object](java.lang.Integer.valueOf(this.freeVar$1))
)
}
}At the first execution of the invokedynamic instruction, the lambda closure object creation site:
invokedynamic LambdaMetaFactory.altMetaFactory(freeVar)is resolved into an equivalent of
Foo$$Lambda$1.getLambda(freeVar)which creates a new instance of the generated Foo$$Lambda$1 class on every invocation.
Side-note: The getLambda(...) call is for capturing lambdas. JDK8's LMF performs an optimization when a lambda doesn't capture any state -- it'll spin up the backing class just like above, but the lambda closure object creation site will actually reuse the same instance (embedded into the call site via a MethodHandle to a constant).
Inner-class lambda:
- Closures are lowered into inner classes, similar to how Java lowers annonymous inner classes.
- Captured state is stored as instance fields on the inner class.
- These synthetic classes can have
$outerfield for the closure environment chain. e.g.- one-level nesting, capturing a mutable local variable:
final def apply(x: Int): String = x.+(anonfun$apply$3.this.$outer.localVar$1.elem.$asInstanceOf[String]());where$outer.localVar$1refers to ascala.runtime.ObjectRef - one-level nesting, capturing an immutable local value:
final def apply(x: Int): String = x.+(anonfun$apply$2.this.$outer.localValue$1);where the$outer.localValue$1refers to the captured value - two-level nesting, capturing a mutable local variable:
final def apply(y: Int): String = y.+(anonfun$apply$9.this.$outer.$outer().localVar$1.elem.$asInstanceOf[String]());, notice how the$outerchain now becomes two levels as well. - two-level nesting, capturing an immutable local value:
final def apply(y: Int): String = y.+(anonfun$apply$7.this.$outer.$outer().localValue$1);same as above - two-level nesting, capturing the enclosing
thisdue to accessing a field:final def apply(x: Int): String = x.+(anonfun$apply$4.this.$outer.$outer().$outer().mutableField());where there are 3$outers in the chain, the first two referencing closures and the last referencing the enclosingthis
- one-level nesting, capturing a mutable local variable:
- Such
$outerfields could reference an enclosing "this" (and this is very often in REPL), or it could reference another closure (for nested closures)- See
closure2for example: it gets lowered to logic like:final def apply(y: Int): String = y.+(anonfun$apply$4.this.$outer.$outer().localValue$1);where it follows the$outerchain to reach the capturedlocalValuestate.
- See
indylambda:
- Closures are lowered into static methods, and the lambda instance creation sites become invokedynamic+LambdaMetafactory call.
There could be corner cases where the lowered method cannot be made staticNSC always generates static method for the lambda body impl. There are some stale comments in NSC that still mentions the possibility of generating non-static method here but that's no longer relevant.
- Captured state is stored as instance fields automatically generated by the LMF, namely from InnerClassLambdaMetafactory
- The fields names are
"arg$" + (i + 1)whereiis[0, capturedArgCount)
- The fields names are
- Immutable captured state is captured-by-copy (etc direct reference copy or primitive type copy)
- For example, see how
localValueis captured in thedelambdafy:methodmode: it's captured by closures of all nesting levels by copy, i.e. making a direct reference to the original captured object. No$outerinvolved here.
- For example, see how
- Mutable captured state is captured-by-ref via the
scala.runtime.XXXReftypes, e.g.IntRef,ObjectRefetc (c.f. the*Ref.javafiles in https://github.com/scala/scala/tree/2.12.x/src/library/scala/runtime). This is typically done for mutable local variable captures, regardless of the nesting level of the closure from the owner function/closure of the local variable. - These indylambdas may keep an outer reference to an enclosing "this", in which case the first non-receiver argument in the lowered synthetic method would be a
$this.- When the lowered synthetic method is static, the
$thisargument's value would come from thearg$1field on the indylambda closure object. This is equivalent to the$outerfield in the old world. When the lowered synthetic method is non-static, thethis case doesn't exist in Scala 2.12 anymore$thisargument doesn't need to exist because it'll be the same as the "capturing receiver", and the "capturing receiver"'s value would come from thearg$1field on the indylambda closure object.
- When the lowered synthetic method is static, the
- These indylambdas do NOT reference outer closures via an equivalent
$outermechanism. e.g.- one-level nesting, capturing a mutable local variable:
final <static> <artifact> def $anonfun$new$5(localVar$1: runtime.ObjectRef, x: Int): String = x.+(localVar$1.elem.$asInstanceOf[String]());where thelocalVar$1is ascala.runtime.ObjectRefthat stores the captured mutable variable. - one-level nesting, capturing an immutable local value:
final <static> <artifact> def $anonfun$new$3(localValue$1: String, x: Int): String = x.+(localValue$1); - two-level ntesting, capturing a mutable local variable:
final <static> <artifact> def $anonfun$new$15(localVar$1: runtime.ObjectRef, y: Int): String = y.+(localVar$1.elem.$asInstanceOf[String]());, compare this with the above: no extra$outerchain, the "ref"s are copied into the nested closure object and stored as fields - two-level nesting, capturing an immutable local value:
final <static> <artifact> def $anonfun$new$12(localValue$1: String, y: Int): String = y.+(localValue$1);, compare this with the above: no extra$outerchain, the direct references are copied into the nested closure object and stored as fields - one-level nesting, capturing enclosing
thisdue to accessing a mutable field:final <static> <artifact> def $anonfun$new$7($this: Baz, x: Int): String = x.+($this.mutableField()); - two-level nesting, capturing enclosing
thisdue to accessing a mutable field:final <static> <artifact> def $anonfun$new$18($this: Baz, y: Int): String = y.+($this.mutableField());, compare this with the above: no extra$outerchain, the captured$thiscomes directly from the current indylambda closure object.
- one-level nesting, capturing a mutable local variable:
- For serializable lambdas (if a function type implements
java.lang.Serializable; note that Scala'sscala.Serializabletrait extends the Java counterpart, and all Scalascala.FunctionNtraits are marked Serializable),- there'd be a
private void writeReplace()method generated on the indylambda class by LMF, so that the lambda instance state can be serialized via thejava.lang.invoke.SerializedLambdaserialization proxy. The connection between the instance fields on the indylambda closure object and theSerializedLambdais codegen'd into thiswriteReplacemethod. - There's also a corresponding
$deserializeLambda$static method on the same class as the synthetic method for the lambda body, similar to how Java 8 implements it (but because Scala can be very lambda-heavy, doing it the exact same way as Java 8 would create a single huge method and blow up the code size). This static method is responsible for deserialization of all indylambdas in this class. This is generated by NSC. This method usesinvokedynamicto create the indylambda closure objects as well, and the bootstrap method isscala.runtime.LambdaDeserialize.bootstrap(...).
- there'd be a
The NSC internal syntax tree for delambdafy:method mode in Scala 2.12 can be found in the syntax_tree_after_delambdafy-2.12 file.
For the example given in this Gist, the generated bytecode from InnerClassLambdaMetafactory can be found in the Bar$$$Lambda$10[1-8].class.javap files.
-
closure1((1 to i).map { x => x + localValue })- Lowered to
final <static> <artifact> def $anonfun$new$3$adapted(localValue$1: String, x: Object): String = Bar.this.$anonfun$new$3(localValue$1, unbox(x));bydelambdafy:method- which further invokes into the actual body:
final <static> <artifact> def $anonfun$new$3(localValue$1: String, x: Int): String = x.+(localValue$1); - Notice how the adaptor method version has an extra
$adaptedin the method name.
- which further invokes into the actual body:
- Runtime generated indylambda closure object code is
Bar$$$Lambda$105.class.javap - This captures an immutable reference
localValue, so the reference is directly captured as a fieldarg$1in the indylambda closure object - The NSC syntax tree for
closure1's assignment looks like this:where thisval closure1: Function1 = { $anonfun(localValue) };
$anonfun(localValue)expression actually represents an invokedynamic call to an LMF bootstrap method withlocalValueas a capturing argument. In bytecode this looks like: (injavapformat)invokedynamic #202, 0 // InvokeDynamic #4:apply:(Ljava/lang/String;)Lscala/Function1; ... BootstrapMethods: 0: #116 invokestatic java/lang/invoke/LambdaMetafactory.altMetafactory:(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;[Ljava/lang/Object;)Ljava/lang/invoke/CallSite; Method arguments: #118 (Ljava/lang/Object;)Ljava/lang/Object; #123 invokestatic Bar$.$anonfun$new$3$adapted:(Ljava/lang/String;Ljava/lang/Object;)Ljava/lang/String; #125 (Ljava/lang/Object;)Ljava/lang/String; #126 3 #127 1 #129 scala/Serializable
- Lowered to
-
closure2- Lowered to
final <static> <artifact> def $anonfun$new$4$adapted(localValue$1: String, j: Object): scala.collection.immutable.IndexedSeq = Bar.this.$anonfun$new$4(localValue$1, unbox(j)); - Runtime generated indylambda closure object code is
Bar$$$Lambda$103.class.javap
- Lowered to
-
closure3- Lowered to
final <static> <artifact> def $anonfun$new$7$adapted(closure1$1: Function1, closure2$1: Function1, k: Object, l: Object, m: Object): scala.collection.immutable.IndexedSeq = Bar.this.$anonfun$new$7(closure1$1, closure2$1, unbox(k), unbox(l), unbox(m)) - Runtime generated indylambda closure object code is
Bar$$$Lambda$104.class.javap
- Lowered to
None of the examples in this Gist capture a $this. But running the following snippet in a Scala 2.12 REPL will demonstrate a $this-capturing lambda:
$ <Scala 2.12.2>/bin/scala -Xprint:delambdafy -Yshow:delambdafy -Yrepl-class-based
:pa
class NotSerializableClass1(val x: Int)
case class Foo(id: String)
val ns = new NotSerializableClass1(42)
val func: Any => Foo = { _ => Foo("") }
<Ctrl+D>In this REPL example, NSC somehow forces both NotSerializableClass1 and Foo to capture the enclosing REPL line object, thus func also needs to capture a $this for the REPL line object so that it can call Foo's constructor, passing the captured $this as the arg$outer argument to Foo's constructor.