Skip to content

Instantly share code, notes, and snippets.

@tabdulradi
Last active May 14, 2021
Embed
What would you like to do?
Scala Script Runner

Scala Script Runner

Make sure you have coursier installed

chmod +x test.scala
./test.scala

First time it will download scala-compiler (and depending on your config maybe GraalVM) then cache a bootstrap runner. Next time it will run much faster using the cached runner (depending on you config can be native program!).

Understand the shebang

First line of test.scala contains all the dependencies the scala program needs followed by flags passed as is to coursier bootstrap.

  • --manifest-jar for JVM-based executable
  • --native-image for native executable (use GraalVM native image, make sure you have it installed)
  • --native for native executable (use Scala native, make sure you have it installed)
#!/usr/bin/env python3
import sys
import hashlib
import os
import subprocess
from datetime import datetime
import itertools
filename = sys.argv[-1]
with open(filename, 'r') as f:
all_code = f.read()
content_hash = hashlib.sha256(all_code.encode()).hexdigest()
scala_code = all_code[1:]
cache_path = os.path.expanduser(f"~/.ivy2/local/com.abdulradi.scala-scripts-launcher/generated/{content_hash}")
launcher_path = f'{cache_path}/launcher'
if not os.path.exists(launcher_path):
dependencies = list(itertools.takewhile(lambda x: not x.startswith('-'), sys.argv[1:-1]))
bootstrap_flags = ' '.join(itertools.dropwhile(lambda x: not x.startswith('-'), sys.argv[1:-1]))
now = datetime.now().timestamp()
os.makedirs(f'{cache_path}/jars', exist_ok=True)
os.makedirs(f'{cache_path}/ivys', exist_ok=True)
# Copy of scala code with shebang commented out as it confuses scalac. TODO: Find a better place for this file
scalacode_path = f'{cache_path}/jars/generated.scala'
with open(scalacode_path, "w") as f:
f.write(f'//{all_code}')
jar_path = f'{cache_path}/jars/generated.jar'
class_path = subprocess.check_output(['coursier', 'fetch', '--classpath'] + dependencies).decode('utf-8')
os.system(f'coursier launch scalac -- {scalacode_path} -d {jar_path} -classpath {class_path} ')
scala_version = subprocess.check_output(['coursier', 'launch', 'scalac', '--', '-version']).decode('utf-8')[23:-55] # Drop the copyrights and extract version number, probably very fragile and might break with dotty 
[scala_version_major, scala_version_minor, scala_version_patch] = scala_version.split('.')
with open(f'{cache_path}/ivys/ivy.xml', "w") as f:
f.write('<?xml version="1.0" encoding="UTF-8"?>\n')
f.write('<ivy-module version="2.0" xmlns:e="http://ant.apache.org/ivy/extra">\n')
f.write(f' <info organisation="com.abdulradi.scala-scripts-launcher" module="generated" revision="{content_hash}" status="integration" publication="{now}"></info>\n')
f.write(' <configurations>\n')
f.write(' <conf name="compile" visibility="public" description=""/>\n')
f.write(' <conf name="optional" visibility="public" description=""/>\n')
f.write(' <conf name="scala-tool" visibility="private" description=""/>\n')
f.write(' </configurations>\n')
f.write(' <dependencies>\n')
f.write(f' <dependency org="org.scala-lang" name="scala-compiler" rev="{scala_version}" conf="scala-tool->default,optional(default)"/>\n')
f.write(f' <dependency org="org.scala-lang" name="scala-library" rev="{scala_version}" conf="scala-tool->default,optional(default);compile->default(compile)"/>\n')
for dependency in dependencies:
if '::' in dependency:
[org, name_rev] = dependency.split('::')
[name_unversioned, rev] = name_rev.split(':')
name = f'{name_unversioned}_{scala_version_major}.{scala_version_minor}'
else:
[org, name, rev] = dependency.split(':')
f.write(f' <dependency org="{org}" name="{name}" rev="{rev}" conf="compile->default(compile)"/>\n')
f.write(' </dependencies>\n')
f.write('</ivy-module>\n')
os.system(f'coursier bootstrap com.abdulradi.scala-scripts-launcher:generated:{content_hash} {bootstrap_flags} -o {launcher_path} --quiet')
os.system(launcher_path)
#!/usr/bin/env python3 scala-script-launcher.py io.circe::circe-generic:0.12.3 io.circe::circe-parser:0.12.3 --manifest-jar
import io.circe._, io.circe.generic.auto._, io.circe.parser._, io.circe.syntax._
sealed trait Foo
case class Bar(xs: Vector[String]) extends Foo
case class Qux(i: Int, d: Option[Double]) extends Foo
object Main {
def main(args: Array[String]): Unit = {
val foo: Foo = Qux(1, Some(14.0))
val json = foo.asJson.noSpaces
println(json)
val decodedFoo = decode[Foo](json)
println(decodedFoo)
}
}
@przemek-pokrywka
Copy link

przemek-pokrywka commented Feb 15, 2021

Do I understand correctly that the Scala compiler produces a jar as a result?

@przemek-pokrywka
Copy link

przemek-pokrywka commented Feb 15, 2021

I want to use Coursier in a similar way in https://github.com/tsk-tsk/tsk-tsk, as an opt-in, because the native compilation may take long.

@tabdulradi
Copy link
Author

tabdulradi commented Feb 15, 2021

Do I understand correctly that the Scala compiler produces a jar as a result?

Yes. Then Coursier takes that jar and creates a bootstrap around it.

I want to use Coursier in a similar way in https://github.com/tsk-tsk/tsk-tsk, as an opt-in, because the native compilation may take long.

Yea I agree. It's opt-in here too, user has to pass --native or --native-image in the shebang.

@przemek-pokrywka
Copy link

przemek-pokrywka commented Feb 19, 2021

@tabdulradi, I think that longer-term the hack/solution might be better off not relying on Coursier's native image generation because one may run into troubles in case of dependency exclusions. One would need to construct the Ivy module descriptor (or Maven pom) very carefully in order not to end up with an unwanted version of a dependency conflict.

So in TSK I plan to use Coursier just for fetching GraalVM (and possibly also for running gu to install the native-image tool) but to use the native-image on my own, supplying the classpath resolved by Coursier basing on all the exclusions I care about.

/cc @alexarchambault

@przemek-pokrywka
Copy link

przemek-pokrywka commented Feb 26, 2021

@tabdulradi, have you had any success with compiling your example script to a native image? I've followed the steps you gave and unfortunately GraalVM was "Aborting stand-alone image build due to unsupported features" (also I needed to pass -S option to the /usr/bin/env program on Linux).

I added support for GraalVM to TSK recently and I tried to turn your example script into a native binary (by saving, chmodding +x and running the following):

// 2> /dev/null \
/*
dependencies='
  io.circe::circe-generic:0.12.3
  io.circe::circe-parser:0.12.3
'
native=true
source $( curl -L git.io/boot-tsk | sh ); run
*/

import io.circe._, io.circe.generic.auto._, io.circe.parser._, io.circe.syntax._

sealed trait Foo
case class Bar(xs: Vector[String]) extends Foo
case class Qux(i: Int, d: Option[Double]) extends Foo

object Main {
  def main(args: Array[String]): Unit = {
    val foo: Foo = Qux(1, Some(14.0))
    val json = foo.asJson.noSpaces
    println(json)
    val decodedFoo = decode[Foo](json)
    println(decodedFoo)
  }
}

but I encountered the same issues. I guess that one needs to come up with some extra GraalVM-specific descriptors that mention all uses of reflection and class loading for the ahead-of-time compilation to work.

I've recently read https://www.lightbend.com/blog/writing-kubectl-plugins-with-scala-or-java-with-fabric8-kubernetes-client-on-graalvm where Andrea mentions use of GraalVM JVM agent library in order to generate these descriptors.
Due to use of a custom "alternative main" (something that exercises the likely-problematic features of a program) this method may be not appropriate for cs bootstrap and/or for scripts, but I'm curious of your thoughts @tabdulradi, @alexarchambault

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment