dan-zheng

Keras-style Sequential in Swift for TensorFlow.

---

The current Sequential in Swift for TensorFlow does not support shape inference. Previous layers' output sizes are redundantly repeated as next layers' input sizes. This increases user burden and adds room for error.

let modelOld = Sequential {

dan-zheng

tensorflow/swift-apis compilation is painfully slow. It's probably due to type-checking.

Swift compilation debugging tips here. swift build -Xswiftc -Xfrontend -Xswiftc -debug-time-function-bodies results below.

# Worst offenders, time in milliseconds.
(1767, "global function 'gelu'", '/Users/danielzheng/swift-apis/Sources/TensorFlow/Operators/Math.swift:1225:13')
(1767, "global function 'gelu'", '/Users/danielzheng/swift-apis/Sources/TensorFlow/Operators/Math.swift:1225:13')
(1767, "global function 'gelu'", '/Users/danielzheng/swift-apis/Sources/TensorFlow/Operators/Math.swift:1225:13')

dan-zheng

Retroactive derivative registration: register derivatives for functions in other modules.

Previously:

• If module A defines func foo, then its derivatives must be in the same module.

With retroactive derivative registration:

• If module A defines func foo;
• And if module B imports module A and defines @differentiating(foo) func derivativeFoo:
• Then module C can import modules A and B and differentiate func foo (e.g. via differentiation APIs).

dan-zheng

import torch
import tangent
from tangent import grad

def nested_loop(x):
  outer = x
  for _ in range(1, 3):
    outer = outer * x
    inner = outer

dan-zheng

I tried to define a generic Adam optimizer using ElementaryFunctions, Differentiable, and VectorProtocol (latter two are being incubated on tensorflow branch).

It doesn't work due to missing scalar-vector and vector-vector operations.

$ swift adam.swift
adam.swift:82:95: error: binary operator '+' cannot be applied to operands of type 'Model.TangentVector' and 'Model.TangentVector.VectorSpaceScalar'
        model.move(along: -stepSize * firstMoments / (Model.TangentVector.sqrt(secondMoments) + epsilon))
                                                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~
adam.swift:82:95: note: expected an argument list of type '(Self, Self)'

dan-zheng

Python interoperability

Check out this tutorial demonstrating Python interoperability in Swift.

Python library version details

PythonLibrary.useVersion

This function is intended to set the Python version before any references to the global PythonInterface called Python - it cannot be used to switch Python versions dynamically.

dan-zheng

import TensorFlow

extension SGD {
  // Take `Model` metatype as an argument, so it doesn't need to be written out
  // explicitly as a generic parameter.
  // IMPORTANT: This API change might not be desirable, because usage involves
  // getting dynamic type via `type(of:)`.
  convenience init(_ modelType: Model.Type, learningRate: Scalar) {
    self.init(learningRate: learningRate) 
  }

dan-zheng

## Building Swift

./swift/utils/update-checkout --skip-repository swift --clone --scheme tensorflow
./swift/utils/build-script -x -R --debug-swift --skip-build-benchmarks --stdlib-deployment-targets=macosx-x86_64 2>&1 | tee log.txt
./swift/utils/build-script -R --skip-build-benchmarks --stdlib-deployment-targets=macosx-x86_64 --enable-tensorflow --no-swift-stdlib-assertions --reconfigure

dan-zheng

import TensorFlow
struct Model: Parameterized, Differentiable {
  @TFParameter var w: Tensor<Float>
  func tangentVector(from cotangent: Parameters) -> Parameters {
    return cotangent
  }
}

let model = Model(w: Tensor<Float>(zeros: [5]))
let pb = pullback(at: model) { m in m.w }

dan-zheng

This gist is an in-depth exploration of the Swift parameter update design, focusing on dynamic parameters and how to synthesize allKeyPaths for them.