Report for GSoC 2021 (Gradually Typed Hasktorch)
This is the final report for the GSoC 2021 project - Gradually Typed Hasktorch. Gradually Typed Hasktorch is a new tensor API for Hasktorch. It has been initiated by my mentor, Torsten Scholak, and as part of GSoC I helped with streamlining this new API.
Student: Julius Marozas Mentor: Torsten Scholak
- Coding Period (June 7 - August 16)
- Future Work
Hasktorch has two distinct APIs for tensors: Torch.Tensor and Torch.Tensor.Typed. While the untyped version can initially be easier to use and experiment with, the typed version offers static analysis of tensor’s shape, layout, precision, and compute device. The typed version not only helps with debugging and maintainability but also offers better support for type-driven development via GHC features like type holes. However, it is currently difficult to mix the two approaches, e.g., statically specifying tensor’s embedding size while keeping other dimensions unchecked is not possible. Gradual typing is the proposed solution that fuses the two APIs by adding Torch.GraduallyTyped type. The new API allows for more granular control by letting the user choose which properties of the tensor should be given a static type. The goal of the project is to bring maturity to the gradually typed tensor API, add missing features, and experiment with new ideas.
Coding period (June 7 - August 16)
Week 1 (June 7 - June 13)
In preparation of merging gradually-typed branch to master, I moved gradually typed modules to a new cabal project.
Week 2 (June 14 - June 20)
- Fixed doctest driver in Linux, they seem to still not work on macOS though. The tests can be run like this:
cabal test hasktorch-gradually-typed:doctests
- Also fixed failing doctest test cases.
select function for slicing and added gradually typed
Index type #568
- Added a new function
selectfor slicing a tensor along a given dimension.
Week 3 (June 21 - June 27)
Helped transitioning to singletons API.
This week we decided to get rid of type class based gradually types and use singletons instead.
sEyeSquarefunctions that work with singletons.
Week 4 (June 28 - July 4)
sToTensor method of
TensorLike type class converts list, tuples, vectors, and sized vector to gradually typed
>>> t <- sToTensor (SGradient SWithoutGradient) (SLayout SDense) (SDevice SCPU) ([(1, 2), (3, 4), (5, 6)] :: [(Int, Int)]) >>> t Tensor Int64 [3,2] [[ 1, 2], [ 3, 4], [ 5, 6]] >>> :type t t :: Tensor ('Gradient 'WithoutGradient) ('Layout 'Dense) ('Device 'CPU) ('DataType 'Int64) ('Shape '[ 'Dim ('Name "*") 'UncheckedSize, 'Dim ('Name "*") ('Size 2)])
Notice, that the first dimension of the shape of
t is unchecked because it was converted from a Haskell list (which we don't know the size of at compile time).
However, the second dimension is of size 2, because we known that
(Int, Int) always contains 2 elements.
fromTensor convert back to a non-tensor type:
>>> fromTensor @[(Int, Int)] t [(1, 2), (3, 4), (5, 6)]
Week 5 (July 5 - July 11)
Updated existing code to use
Week 6 (July 12 - July 18)
Started working on indexing/slicing
Vacation (started July 15)
Week 7 (July 19 - July 25)
Vacation (ended July 22)
Week 8 (July 26 - August 1)
Added support for slicing tensors with multiple indices:
>>> tensor Tensor Int64 [2,2,3] [[[ 0, 1, 2], [ 3, 4, 5]], [[ 6, 7, 8], [ 9, 10, 11]]] >>> x <- t ! SIndices (SEllipsis :|: SSliceAt (SIndex @1) :|: SNil) Tensor Int64 [2,2] [[1, 4], [7, 10]]
Also added additional constructor for
SIndex to take negative indices:
Week 9 (August 2 - August 8)
slice quasiquoter lets you use slice syntax similar to Python's. The previous example from week 8 could be rewritten like this:
>>> x <- t ! [slice|...,1|] Tensor Int64 [2,2] [[1, 4], [7, 10]]
This is inspired by Junji Hashimoto's earlier work #579
Week 10 (August 9 - August 15)
Lenses can be used to view, set or modify part of a tensor. For example,
t & (toLens [slice|2:4|]) %~ (+2) lets you increment
t values (at indices 2 and 3) by 2. The only missing part is figuring out how to deal with failure with view operators, e.g.
- Add gradually typed version of MNIST example.
- Move Torch.GraduallyTyped out of experimental.
- Add support for mutable network parameters to save memory while training.
- Add a more general type
Checkedfor gradual types.
Checkedtype could replace gradual types like
I am extremely grateful to my mentor Torsten Scholak who helped me throughout the summer. I would also like to express my gratitude to other Hasktorch contributors and to Google for organising Google Summer of Code.