sjrd/how-to-dotty-scalajs.md

## how-to-dotty-scalajs.md

      
    Raw
  

              how-to-dotty-scalajs.md
            
          
    Advance the support of Scala.js in Dotty, a tutorial

Dotty contains preliminary support for Scala.js under the flag -scalajs.
Or rather, it contains the infrastructure for preliminary support.
It is far from being actually usable, but this is where you can help!
This small tutorial walks you through a few steps that you can do to further the support of Scala.js in Dotty.
Even if you do not typically contribute to a compiler, this can be your chance.
It is not very difficult, given that there already exists an extensive test suite, as well as a working implementation for scalac.
Without further ado, let us dive in!
Setup

First, let us make sure that you are properly setup.
Besides the basic setup for developing dotty, you will need to install Node.js, if that is not already done.
Once set up, the following sbt command should successfully run:
> sjsJUnitTests/test

This command compiles the Dotty standard library for Scala.js, compiles the JUnit tests for Scala.js, and executes them under Node.js.
Enable a new test from the upstream test suite: when things go well

The tests run by sjsJUnitTests are not actually in the dotty repository.
They are part of the upstream scala-js/scala-js, so they are the official existing tests for Scala.js with Scala 2.
However, a tiny portion of the tests are whitelist in the build.
Open the file project/Build.scala and look for the managedSources in Test setting under lazy val sjsJUnitTests.
At the time of this writing, it reads as follows:
managedSources in Test ++= {
  val dir = fetchScalaJSSource.value / "test-suite"
  (
    (dir / "shared/src/test/scala/org/scalajs/testsuite/compiler" ** "IntTest.scala").get
    ++ (dir / "shared/src/test/require-jdk8/org/scalajs/testsuite/javalib/util" ** "Base64Test.scala").get
    ++ (dir / "shared/src/test/scala/org/scalajs/testsuite/utils" ** "*.scala").get
  )
}
Open https://github.com/scala-js/scala-js/tree/v1.0.0-M8/test-suite/shared/src/test to see what other tests you could add there.
For the purposes of this tutorial, let us pick javalib/lang/IntegerTest.scala, and add it to the managedSources in Test:
managedSources in Test ++= {
  val dir = fetchScalaJSSource.value / "test-suite"
  (
    (dir / "shared/src/test/scala/org/scalajs/testsuite/compiler" ** "IntTest.scala").get
    ++ (dir / "shared/src/test/scala/org/scalajs/testsuite/javalib/lang" ** "IntegerTest.scala").get
    ++ (dir / "shared/src/test/require-jdk8/org/scalajs/testsuite/javalib/util" ** "Base64Test.scala").get
    ++ (dir / "shared/src/test/scala/org/scalajs/testsuite/utils" ** "*.scala").get
  )
}
Now you can reload sbt, then rerun the tests:
> reload
> sjsJUnitTests/test
[...]
[info] Passed: Total 60, Failed 0, Errors 0, Passed 60
[success] Total time: 13 s, completed Aug 27, 2019 11:25:23 AM

Success! The new test already passes!
This is a good time to commit. Make sure you are in dedicated branch.
$ git checkout -b enable-more-sjs-tests
M       project/Build.scala
Switched to a new branch 'enable-more-sjs-tests'
$ git add -u
$ git commit -m "Enable javalib/lang/IntegerTest for Scala.js."
[enable-more-sjs-tests 0e48a4367d0] Enable javalib/lang/IntegerTest for Scala.js.
 1 file changed, 1 insertion(+)

Rinse and repeat.
As long as you find tests that can be added and that immediately succeed, you can keep going like this.
When you are done, send a PR!
When things do not go well

Let us try to add another test, for example javalib/lang/ObjectTest.scala.
> reload
> sjsJUnitTests/test
[...]
[info] Fast optimizing tests/sjs-junit/../out/bootstrap/sjsJUnitTests/scala-0.18/sjsjunittests-test-fastopt.js
[error] Referring to non-existent method static scala.Function2.toString$(scala.Function2)java.lang.String
[error]   called from org.scalajs.testsuite.javalib.lang.ObjectTest$XY$1$.toString()java.lang.String
[...]
[error] (sjsJUnitTests / Test / fastOptJS) org.scalajs.linker.LinkingException: There were linking errors
[error] Total time: 5 s, completed Aug 27, 2019 11:32:57 AM

Oops! That did not go so well!
We ran into a Scala.js linking error, which is almost certainly a bug in the Scala.js support inside dotty (given that the same test is known to work in upstream Scala.js for Scala 2).
Inspecting the Scala.js IR

The first thing to do is to look at the Scala.js IR (similar to .class files, but for Scala.js, and more readable) of the calling code (the "called from" line):
> sjsJUnitTests/test:scalajsp org.scalajs.testsuite.javalib.lang.ObjectTest$XY$1$
class Lorg_scalajs_testsuite_javalib_lang_ObjectTest$XY$1$ extends O implements F2, Ljava_io_Serializable, s_deriving$Mirror, s_deriving$Mirror$Product {
  val $$outer$6: Lorg_scalajs_testsuite_javalib_lang_ObjectTest
  constructor def init___Lorg_scalajs_testsuite_javalib_lang_ObjectTest($$outer: Lorg_scalajs_testsuite_javalib_lang_ObjectTest) {
    if (($$outer === null)) {
      throw new jl_NullPointerException().init___()
    };
    this.$$outer$6 = $$outer;
    this.O::init___()
  }
  def curried__F1(): F1 = {
    F2::curried$__F2__F1(this)
  }
  def tupled__F1(): F1 = {
    F2::tupled$__F2__F1(this)
  }
  def toString__T(): T = {
    F2::toString$__F2__T(this)
  }
  def apply__I__I__Lorg_scalajs_testsuite_javalib_lang_ObjectTest$XY$2(x: int, y: int): Lorg_scalajs_testsuite_javalib_lang_ObjectTest$XY$2 = {
    new Lorg_scalajs_testsuite_javalib_lang_ObjectTest$XY$2().init___Lorg_scalajs_testsuite_javalib_lang_ObjectTest__I__I(this.org$scalajs$testsuite$javalib$lang$ObjectTest$$und$XY$$$$outer__Lorg_scalajs_testsuite_javalib_lang_ObjectTest(), x, y)
  }
  def unapply__Lorg_scalajs_testsuite_javalib_lang_ObjectTest$XY$2__Lorg_scalajs_testsuite_javalib_lang_ObjectTest$XY$2(x$1: Lorg_scalajs_testsuite_javalib_lang_ObjectTest$XY$2): Lorg_scalajs_testsuite_javalib_lang_ObjectTest$XY$2 = {
    x$1
  }
  def fromProduct__s_Product__Lorg_scalajs_testsuite_javalib_lang_ObjectTest$XY$2(x$0: s_Product): Lorg_scalajs_testsuite_javalib_lang_ObjectTest$XY$2 = {
    new Lorg_scalajs_testsuite_javalib_lang_ObjectTest$XY$2().init___Lorg_scalajs_testsuite_javalib_lang_ObjectTest__I__I(this.org$scalajs$testsuite$javalib$lang$ObjectTest$$und$XY$$$$outer__Lorg_scalajs_testsuite_javalib_lang_ObjectTest(), x$0.productElement__I__O(0).asInstanceOf[I], x$0.productElement__I__O(1).asInstanceOf[I])
  }
  private def $$outer__Lorg_scalajs_testsuite_javalib_lang_ObjectTest(): Lorg_scalajs_testsuite_javalib_lang_ObjectTest = {
    this.$$outer$6
  }
  def org$scalajs$testsuite$javalib$lang$ObjectTest$$und$XY$$$$outer__Lorg_scalajs_testsuite_javalib_lang_ObjectTest(): Lorg_scalajs_testsuite_javalib_lang_ObjectTest = {
    this.Lorg_scalajs_testsuite_javalib_lang_ObjectTest$XY$1$::private::$$outer__Lorg_scalajs_testsuite_javalib_lang_ObjectTest()
  }
  def apply__O__O__O(v1: any, v2: any): any = {
    this.apply__I__I__Lorg_scalajs_testsuite_javalib_lang_ObjectTest$XY$2(v1.asInstanceOf[I], v2.asInstanceOf[I])
  }
  def fromProduct__s_Product__O(p: s_Product): any = {
    this.fromProduct__s_Product__Lorg_scalajs_testsuite_javalib_lang_ObjectTest$XY$2(p)
  }
}
and focus on the toString()java.lang.String method:
  def toString__T(): T = {
    F2::toString$__F2__T(this)
  }
It indeed calls the static method toString$(Function2)String (toString$__F2__T) in the Function2 (F2) interface.
Let us look at that interface to see what is wrong:
> sjsJUnitTests/test:scalajsp scala.Function2
interface F2 {
  def apply__O__O__O(v1: any, v2: any): any = <abstract>
  def curried__F1(): F1 = {
    new sjsr_AnonFunction1().init___sjs_js_Function1((arrow-lambda<$this: F2 = this>(x1$2: any) = {
      val x1: any = x1$2;
      $this.F2::private::$$anonfun$curried$1__O__F1(x1)
    }))
  }
  def tupled__F1(): F1 = {
    new sjsr_AnonFunction1().init___sjs_js_Function1((arrow-lambda<$this: F2 = this>(x0$1$2: any) = {
      val x0$1: T2 = x0$1$2.asInstanceOf[T2];
      $this.F2::private::$$anonfun$tupled$1__T2__O(x0$1)
    }))
  }
  def toString__T(): T = {
    "<function2>"
  }
  def apply$mcZDD$sp__D__D__Z(v1: double, v2: double): boolean = {
    this.apply__O__O__O(v1, v2).asInstanceOf[Z]
  }
  ...
  def apply$mcVJJ$sp__J__J__V(v1: long, v2: long) {
    this.apply__O__O__O(v1, v2)
  }
  @hints(1) private def $$anonfun$curried$2__O__O__O(x1$1: any, x2: any): any = {
    this.apply__O__O__O(x1$1, x2)
  }
  @hints(1) private def $$anonfun$curried$1__O__F1(x1: any): F1 = {
    new sjsr_AnonFunction1().init___sjs_js_Function1((arrow-lambda<$this: F2 = this, x1: any = x1>(x2$2: any) = {
      val x2: any = x2$2;
      $this.F2::private::$$anonfun$curried$2__O__O__O(x1, x2)
    }))
  }
  @hints(1) private def $$anonfun$tupled$1__T2__O(x0$1: T2): any = {
    val x1: T2 = x0$1;
    if ((x1 !== null)) {
      val x1$2: any = x1.$$und1__O();
      val x2: any = x1.$$und2__O();
      this.apply__O__O__O(x1$2, x2)
    } else {
      throw new s_MatchError().init___O(x1)
    }
  }
  def $$init$__V() {
    /*<skip>*/
  }
}
Hum, there is no toString$ method there.
Instead, we find a default method def toString__T(): T.
Maybe the call inside ObjectTest$XY$1$.toString() was meant to call that method?
At this point you might want to pop up in the dotty gitter channel if you feel like you need a bit of advice on what might be wrong.
In this case, what happens is that Dotty has emitted a call to a static helper method, instead of a call to the default method.
For the JVM, this is necessary, because of restrictions in JVM bytecode, which is why the compilers generate static helpers.
However, in the Scala.js IR, we can always directly call the default method, so the Scala.js compiler does not generate the static helpers.
Of course this means that we must call the default method instead of the static helper when generating for Scala.js.
Inspecting Dotty trees in the sandbox

Using the tools described in the contributing guide, we can print the trees that are being compiled as various phases, in order to figure out what part of the compiler introduces the calls to the static helpers.
It is typically best, for that purpose, to take out the code of the failing test and put it in the Scala.js sandbox instead, under scalajs/sandbox/src/hello.scala:
package hello

object HelloWorld {
  def main(args: Array[String]): Unit = {
    test()
  }

  def test(): Unit = {
    case class XY(x: Int, y: Int)

    val l = List(XY(1, 2), XY(2, 1))
    val xy12 = XY(1, 2)

    assert(l.contains(xy12))
    assert(l.exists(_ == xy12)) // the workaround
  }
}
then try and run it to make sure that it exposes the same issue:
> sjsSandbox/run
[...]
[info] Fast optimizing sandbox/scalajs/../out/bootstrap/sjsSandbox/scala-0.18/sjssandbox-fastopt.js
[error] Referring to non-existent method static scala.Function2.toString$(scala.Function2)java.lang.String
[error]   called from hello.HelloWorld$XY$1$.toString()java.lang.String

Yup, that's it!
Now you can use
> set scalacOptions in sjsSandbox += "-Xprint:collectSuperCalls"
> sjsSandbox/compile

to see what trees reach the Scala.js backend:
private final <static> class HelloWorld$XY$1$ extends Object, Function2,
  java.io.Serializable
, scala.deriving.Mirror.deriving$Mirror$Product {
  ...
  override def toString(): String = Function2#toString$(this)
  ...
}
There is a static method call all right.
After a bit more digging through the possible arguments to -Xprint, we figure which phase introduces the call.
It also helps to read the descriptions of phases in compiler/src/dotty/tools/dotc/Compiler.scala.
We are lucky, it seems pretty obvious that LinkScala2Impls is to be blamed.
Even its Scaladoc says so:
/** Rewrite calls
 *
 *    super[M].f(args)
 *
 *  where M is a Scala 2.x trait implemented by the current class to
 *
 *    M.f$(this, args)
 *
 *  where f$ is a static member of M.
 */
Fixing the bug

At this point we will have to dive into the compiler's code!
We find the following snippet:
  override def transformApply(app: Apply)(implicit ctx: Context): Tree = {
    def currentClass = ctx.owner.enclosingClass.asClass
    app match {
      case Apply(sel @ Select(Super(_, _), _), args)
      if sel.symbol.owner.is(Scala2x) && currentClass.mixins.contains(sel.symbol.owner) =>
        val impl = implMethod(sel.symbol)
        if (impl.exists) Apply(ref(impl), This(currentClass) :: args).withSpan(app.span)
        else app // could have been an abstract method in a trait linked to from a super constructor
      case _ =>
        app
    }
  }
It seems like this code applies the transformation for traits compiled by Scala 2.x.
Maybe all we need to do is disable it when compiling for Scala.js:
      if sel.symbol.owner.is(Scala2x) && currentClass.mixins.contains(sel.symbol.owner) && !ctx.settings.scalajs.value =>
Then clean the sandbox and retry:
> sjsSandbox/clean
> sjsSandbox/run
[warn] Multiple main classes detected.  Run 'show discoveredMainClasses' to see the list
[info] Fast optimizing sandbox/scalajs/../out/bootstrap/sjsSandbox/scala-0.18/sjssandbox-fastopt.js
[info] Running hello.HelloWorld. Hit any key to interrupt.
[success] Total time: 4 s, completed Aug 27, 2019 12:19:06 PM

Yes! That worked!
Verify with the JUnit tests

Now we can retry the JUnit tests:
> sjsJUnitTests/clean
> sjsJUnitTests/test
[...]
[info] Fast optimizing tests/sjs-junit/../out/bootstrap/sjsJUnitTests/scala-0.18/sjsjunittests-test-fastopt.js
[error] .../testsuite/javalib/lang/ObjectTest.scala(95:20:Select): Class sr_IntRef does not have a field elem$f
[error] .../testsuite/javalib/lang/ObjectTest.scala(99:20:Select): Class sr_IntRef does not have a field elem$f
[error] .../testsuite/javalib/lang/ObjectTest.scala(87:6:Select): Class sr_IntRef does not have a field elem$f
[error] .../testsuite/javalib/lang/ObjectTest.scala(87:6:Select): Class sr_IntRef does not have a field elem$f
[...]
[error] (sjsJUnitTests / Test / fastOptJS) org.scalajs.linker.LinkingException: There were 4 IR checking errors.
[error] Total time: 13 s, completed Aug 27, 2019 12:21:06 PM

Huh! That is still not working.
However, we have clearly progressed to a different kind of error.
Let us commit the fix (but not the addition of the test since it does not work yet):
$ git add -u compiler/
$ git commit
[enable-more-sjs-tests b9792c36362] Do not rewrite trait method calls to static helpers in Scala.js.
1 file changed, 1 insertion(+), 1 deletion(-)

Unless you find a test that did not work before that change, and works now, this is not a good time to send a PR, since we do not have tests for the changes.
Tackling IR checking errors

The new error is a set of IR checking errors.
Those are similar to VerifyErrors on the JVM: the backend emitted IR that does links but does not typecheck.
Since the four errors are basically the same but in different locations, let us look at the first one:
[error] .../testsuite/javalib/lang/ObjectTest.scala(95:20:Select): Class sr_IntRef does not have a field elem$f

IR positions are 0-based, so line 95 will typically be shown as 96 in your IDE.
It therefore refers to this line.
Looking at the Scala.js IR, we see:
> sjsJUnitTests/test:scalajsp org.scalajs.testsuite.javalib.lang.ObjectTest
[...]
  def cloneCtorSideEffects$undissue$und3192__V() {
    val ctorInvokeCount: sr_IntRef = mod:sr_IntRef$.create__I__sr_IntRef(0);
    val o: Lorg_scalajs_testsuite_javalib_lang_ObjectTest$CloneCtorSideEffectsBug$1 = new Lorg_scalajs_testsuite_javalib_lang_ObjectTest$CloneCtorSideEffectsBug$1().init___sr_IntRef__I(ctorInvokeCount, 54);
    mod:Lorg_junit_Assert$.assertEquals__O__O__V(54, o.x__I());
    mod:Lorg_junit_Assert$.assertEquals__O__O__V(1, ctorInvokeCount.elem$f);
    val o2: Lorg_scalajs_testsuite_javalib_lang_ObjectTest$CloneCtorSideEffectsBug$1 = o.clone__Lorg_scalajs_testsuite_javalib_lang_ObjectTest$CloneCtorSideEffectsBug$1();
    mod:Lorg_junit_Assert$.assertEquals__O__O__V(54, o2.x__I());
    mod:Lorg_junit_Assert$.assertEquals__O__O__V(1, ctorInvokeCount.elem$f)
  }
Inspecting the scala.runtime.IntRef class, we see:
> sjsJUnitTests/test:scalajsp scala.runtime.IntRef
@hints(1) class sr_IntRef extends O implements Ljava_io_Serializable {
  var elem$1: int
  def elem__I(): int = {
    this.elem$1
  }
  def elem$und$eq__I__V(x$1: int) {
    this.elem$1 = x$1
  }
  def toString__T(): T = {
    mod:jl_String$.valueOf__I__T(this.elem__I())
  }
  constructor def init___I(elem: int) {
    this.elem$1 = elem;
    this.O::init___()
  }
}
So the elem field is of the form elem$1, not elem$f.
In Scala.js IR terms, that means it is private rather than public.
Normally, Scala fields are never public.
Even a public val or var is declared as a private field, with a getter (and setter) method.
However, scala.runtime.IntRef is originally written in Java for the JVM.
Since Scala.js does not support Java sources, it uses a different version written in Scala.
So how come Scala.js for Scala 2 emits elem$1?
(verifying that it does with scalajsp in a Scala.js for Scala 2 project is left as an exercise for the reader)
Fixing the issue, taking inspiration from the Scala 2 backend

Again, you might want to ask for a bit of advice at this point.
In this case, we are going to look at how the Scala.js backend for Scala 2 works.
It turns out that there is a piece of complicated logic in the Scala.js backend for Scala 2 to deal with this.
It can be found in JSEncoding.scala in the scala-js/scala-js repo.
Let us see what is the Dotty equivalent, in compiler/src/dotty/tools/backend/sjs/JSEncoding.scala:
  private def allRefClasses(implicit ctx: Context): Set[Symbol] = {
    //TODO
    /*(Set(ObjectRefClass, VolatileObjectRefClass) ++
        refClass.values ++ volatileRefClass.values)*/
    Set()
  }

  def encodeFieldSym(sym: Symbol)(
      implicit ctx: Context, pos: ir.Position): js.Ident = {
    require(sym.owner.isClass && sym.isTerm && !sym.is(Flags.Method) && !sym.is(Flags.Module),
        "encodeFieldSym called with non-field symbol: " + sym)

    val name0 = encodeMemberNameInternal(sym)
    val name =
      if (name0.charAt(name0.length()-1) != ' ') name0
      else name0.substring(0, name0.length()-1)

    /* We have to special-case fields of Ref types (IntRef, ObjectRef, etc.)
     * because they are emitted as private by our .scala source files, but
     * they are considered public at use site since their symbols come from
     * Java-emitted .class files.
     */
    val idSuffix =
      if (sym.is(Flags.Private) || allRefClasses.contains(sym.owner))
        sym.owner.asClass.baseClasses.size.toString
      else
        "f"

    val encodedName = name + "$" + idSuffix
    js.Ident(mangleJSName(encodedName), Some(sym.unexpandedName.decoded))
  }
It seems the logic for Ref classes is there, but allRefClasses has a big TODO.
Let us fix that:
  private[this] var allRefClassesCache: Set[Symbol] = _
  private def allRefClasses(implicit ctx: Context): Set[Symbol] = {
    if (allRefClassesCache == null) {
      val baseNames = List("Object", "Boolean", "Character", "Byte", "Short",
          "Int", "Long", "Float", "Double")
      val fullNames = baseNames.flatMap { base =>
        List(s"scala.runtime.${base}Ref", s"scala.runtime.Volatile${base}Ref")
      }
      allRefClassesCache = fullNames.map(name => ctx.requiredClass(name)).toSet
    }
    allRefClassesCache
  }
After another change to fix the way we count the superclasses of a class:
@tailrec
def superClassCount(sym: Symbol, acc: Int): Int =
  if (sym == defn.ObjectClass) acc
  else superClassCount(sym.asClass.superClass, acc + 1)

/* We have to special-case fields of Ref types (IntRef, ObjectRef, etc.)
 * because they are emitted as private by our .scala source files, but
 * they are considered public at use site since their symbols come from
 * Java-emitted .class files.
 */
val idSuffix =
  if (sym.is(Flags.Private) || allRefClasses.contains(sym.owner))
    superClassCount(sym.owner, 0).toString
  else
    "f"
We can retry to clean and run the JUnit tests:
> sjsJUnitTests/clean
> sjsJUnitTests/test
[...]
[info] Passed: Total 66, Failed 0, Errors 0, Passed 66
[success] Total time: 7 s, completed Aug 27, 2019 1:51:52 PM

Success!
We can now commit:
$ git add -u compiler/ project/
$ git commit
[enable-more-sjs-tests 90c5f64c192] Scala.js: Fix references to fields in other compilation units.
 2 files changed, 21 insertions(+), 5 deletions(-)

And with that, send a PR!
Conclusion

The work we did in this tutorial was submitted in the PR #7112.
The general workflow to improve Scala.js support in Dotty is as follows:

Try and enable a new test from the upstream test suite.
If it passes, great! Commit it.
If not, inspect the IR and the compiler trees, and take inspiration from the Scala 2 backend, to fix it. Then commit your fix.

Now, it is your turn!