Google Summer of Code 2021: Implement support for ScalaPy in Scala 3
The name Scala comes from the word scalable, and true to that name, the Scala language is used to power busy websites and analyze huge data sets. Scala is one of the leading languages in the implementation of big data systems. It is widely used in streaming data. Similarly, Python has become the language of choice for data scientists with its many high-quality scientific libraries. ScalaPy is that bridge that gives an opportunity to use Python libraries in Scala. Last year a new version of Scala was introduced. At the moment it is not released but Scala 3 brings many new features and changes. The aim of this project is to discover those new features and implement support for ScalaPy in Scala 3. ScalaPy will possibly help Scala 3 to become more popular among scientists and maybe this will be the core moment in combining data processing and data analysis worlds in Scala 3.
Week 1 - 3 (7 June - 28 June)
As a warm-up task, I have implemented support for bracket syntax to facades by introducing new annotation and using Scala 2 macros. The annotation
@PyBracketAccess can be used on methods to mark them as representing indexing into the Python object using brackets in Python syntax. The target method must have one (to read the value) or two parameters (to update the value).
Week 4 - 8 (28 June - 2 Aug)
In the 4th week, I started porting facades methods. This week I stuck a little bit with accessing enclosing parameters but also on this week there was introduced a new example that shows how to access enclosing parameters. Then in 5 and 6th weeks I had issues with understanding how to construct ASTs for
applyDynamic methods, but hopefully Anatolii explained the basics of constructing ASTs in Scala 3 using its Reflection API . And on the next week, I implemented AST for
selectDynamic method and tried to implement it for
applyDynamic. But during implementing AST for
applyDynamic there was an issue with passing sequence of elements as
varargs into AST. The problem was that Reflection API doesn't contain an interface for transforming arguments sequence to
varargs term. But I found in this pull request lampepfl/dotty#10729 that varargs can be represented as
Typed(Inlined(EmptyTree, List(), Repeated(elems, _)), _) and this construct of AST for varargs really works. I also created a pull request to the dotty-macro-examples repository with the example of how to pass sequence of parameters as varargs into the AST.
Week 9 - 11 (2 Aug - 23 Aug)
During weeks 9-11, I with my teammate Zhendong, who is also working on porting ScalaPy to Scala 3 in the EPFL, have tried to combine our parts of code. There were several issues with implementing static facades as traits so we decide to create facades as
class instead of
trait. After we combined the code and changed
classes in facades we run the tests and almost all tests passed. We have discovered an issue with running
ThreadStressTest and possibly the problem is that single abstracts methods are not working in Scala 3, so I decided to rewrite SAM using
Runnable interface. Also, there was a problem with running
StaticModule("string"), and as we changed facades to
class and as
StaticModule was a class. And we had a situation where
StringModuleStaticFacade extends two classes. But as traits in Scala 3 support parameters now, I changed
trait. And finally, there was test
Reading a sequence of objects preserves original object in which we discovered a problem with duplication of data during converting Python list to Scala sequence. This was caused by a mistake in the implementation of the
creator method but we fixed it and started preparing the final PR.
After GSoC period
Pull request with
@PyBracketAccess support implementation in Scala 3 version: scalapy/scalapy#225
After GSoC period, we prepared our final pull request with initial Scala 3 support and it was merged into ScalaPy repository. Then I also implemented
@PyBracketAccess feature support in Scala 3 version by creating ASTs for
bracketUpdate methods using reflection API.