Skip to content

Instantly share code, notes, and snippets.

View marcrasi's full-sized avatar
🔬
LLMing

marcrasi

🔬
LLMing
  • Stealth Startup
View GitHub Profile
@marcrasi
marcrasi / aunifying.md
Last active February 25, 2021 13:25
Unifying the differentiation modes

Unifying the differentiation modes

Today, Swift AD has two separate transforms that produce forward mode derivatives (JVPs) and reverse mode derivatives (VJPs).

We can generalize this to a single transform that produces a single "derivative function" that can evaluate derivatives in both modes.

The "derivative function" is generic over a "derivative type" that determines the derivative mode.

The advantages over the existing system are:

@marcrasi
marcrasi / gist:59c540994fd1644300c6def0b8453a08
Created August 8, 2020 17:00
generalized tangent vectors
import TensorFlow
// MARK: - Protocols
protocol Vector {
func scaled(by factor: Float) -> Self
func adding(_ other: Self) -> Self
static var zero: Self { get }
}
import TensorFlow
// MARK: Example function and its transformed version.
func cube(_ x: Float) -> Float {
return x.product(x).product(x)
}
func cubeT<A: WrapsFloat>(_ x: A) -> A {
return x.product(x).product(x)
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Don't use this. ArrayReader here is better: https://groups.google.com/a/tensorflow.org/g/swift/c/Xo5YmLIt12s/m/OM8n6J4TAQAJ
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
import TensorFlow
let g = gradient(at: Dense()) { m in m.callForward(Tensor<Float>(0)) }
print(g)