Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?

Introduce User-defined Dynamically "callable" Types

Introduction

This proposal introduces new DynamicCallable protocols to the standard library. Types that conform to it are "callable" with function call syntax. It is simple syntactic sugar which allows the user to write:

    a = someValue(keyword1: 42, "foo", keyword2: 19)

and have it be interpreted by the compiler as:

  a = someValue.dynamicCall(arguments: [
    ("keyword1", 42), ("", "foo"), ("keyword2", 19)
  ])

Many other languages have analogous features (e.g. Python "callables", C++ operator(), and functors in many other languages), but the primary motivation of this proposal is to allow elegant and natural interoperation with dynamic languages in Swift.

Swift-evolution thread: Discussion thread topic for that proposal

Motivation and Context

Swift is well known for being exceptional at interworking with existing C and Objective-C APIs, but its support for calling APIs written in scripting languages like Python, Perl, and Ruby is quite lacking.

C and Objective-C are integrated into Swift by expending a heroic amount of effort into integrating Clang ASTs, remapping existing APIs in an attempt to feel "Swifty", and by providing a large number of attributes and customization points for changing the behavior of this integration when writing an Objective-C header. The end result of this massive investment of effort is that Swift provides a better experience when programming against these legacy APIs than Objective-C itself did.

When considering the space of dynamic languages, three things are clear: 1) there are several different languages of interest, and they each have significant interest in different quarters: for example, Python is big in data science and machine learning, Ruby is popular for building server side apps, a few people apparently use Javascript, and even Perl is in still widely used. 2) These languages have decades of library building behind them, sometimes with significant communities and 3) there are one or two orders of magnitude more users of these libraries than there are people currently using Swift.

While it is theoretically possible to expend the same level of effort on each of these languages and communities as has been spent on Objective-C, it is quite clear that this would both ineffective as well as bad for Swift: It would be ineffective, because the Swift community has no leverage over these communities to force auditing and annotation of their APIs. It would be bad for Swift because it would require a ton of language-specific support (and a number of third-party dependencies) of the compiler and runtime, each of which makes the implementation significantly more complex, difficult to reason about, difficult to maintain, and difficult to test the supported permutations. In short, we'd end up with a mess.

Fortunately for us, these scripting languages provide an extremely dynamic programming model where almost everything is discovered at runtime, and many of them are explicitly designed to be embedded into other languages and applications. This aspect allows us to embed APIs from these languages directly into Swift with no language support at all - without the level of effort, integration, and invasiveness that Objective-C has benefited from. Instead of invasive importer work, we can write some language-specific Swift APIs, and leave the interop details to that library.

This offers a significant opportunity for us - the Swift community can "embrace" these dynamic language APIs (making them directly available in Swift) reducing the pain of someone moving from one of those languages into Swift. It is true that the APIs thus provided will not feel "Swifty", but if that becomes a significant problem for any one API, then the community behind it can evaluate the problem and come up with a solution (either a Swift wrapper for the dynamic language, or a from-scratch Swift reimplementation of the desired API). In any case, if/when we face this challenge, it will be a good thing: we'll know that we've won a significant new community of Swift developers.

While it is possible today to import (nearly) arbitrary dynamic language APIs into Swift today, the resultant API is unusable for two major reasons: member lookup is too verbose to be acceptable, and calling behavior is similarly too verbose to be acceptable. As such, we seek to provide two "syntactic sugar" features that solve this problem. These sugars are specifically designed to be independent of the dynamic languages themselves and, indeed, independent of dynamic languages at all: we can imagine other usage for the same primitive capabilities.

The two proposals in question are the introduction of the DynamicCallable protocol (this proposal) and a related DynamicMemberLookupProtocol proposal. With these two extensions, we think we can eliminate the need for invasive importer magic by making interoperability with dynamic languages ergonomic enough to be acceptable.

For example, consider this Python code:

class Dog:
    def __init__(self, name):
        self.name = name
        self.tricks = []    # creates a new empty list for each dog
        
    def add_trick(self, trick):
        self.tricks.append(trick)

we would like to be able to use this from Swift like this (the comments show the corresponding syntax you would use in Python):

  // import DogModule
  // import DogModule.Dog as Dog    // an alternate
  let Dog = Python.import(“DogModule.Dog")

  // dog = Dog("Brianna")
  let dog = Dog("Brianna")

  // dog.add_trick("Roll over")
  dog.add_trick("Roll over")

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog("Kaylee").add_trick("snore")

Of course, this would also apply to standard Python APIs as well. Here is an example working with the Python pickle API and the builtin Python function open:

  // import pickle
  let pickle = Python.import("pickle")

  // file = open(filename)
  let file = Python.open(filename)

  // blob = file.read()
  let blob = file.read()

  // result = pickle.loads(blob)
  let result = pickle.loads(blob)

This can all be expressed today as library functionality written in Swift, but without this proposal, the code required is unnecessarily verbose and gross. Without it (but with the related DynamicMemberLookupProtocol proposal) the code would have a method name (like call) all over the code:

  // import pickle
  let pickle = Python.import("pickle")  // normal method in Swift, no change.

  // file = open(filename)
  let file = Python.open.call(filename)

  // blob = file.read()
  let blob = file.read.call()

  // result = pickle.loads(blob)
  let result = pickle.loads.call(blob)

  // dog2 = Dog("Kaylee").add_trick("snore")
  let dog2 = Dog.call("Kaylee").add_trick.call("snore")

While this is a syntactic sugar proposal, we believe that this expands Swift to be usable in important new domains. This sort of capability is also highly precedented in other languages, and is a generally useful language feature that could be used for other purposes as well (e.g. for implementing dynamic proxy objects).

Proposed solution

We propose introducing these protocols to the standard library:

/// A type conforming just to this protocol would not accept parameter
/// labels in its calls.
protocol DynamicCallable {
  associatedtype DynamicCallableArgument
  associatedtype DynamicCallableResult

  func dynamicCall(arguments: [DynamicCallableArgument]) throws -> DynamicCallableResult
}

/// A type conforming to this protocol allows optional parameter
/// labels.
protocol DynamicCallableWithKeywordsToo : DynamicCallable {
  func dynamicCall(arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
}

It also extends the language such that function call syntax - when applied to a value of DynamicCallable type - is accepted and transformed into a call to the dynamicCall member. The dynamicCall(arguments:) method takes a list of formal parameter specified at the call site. If a type conforms to DynamicCallable but not DynamicCallableWithKeywordsToo, then keyword arguments are not accepted in calls to it. The associated types for the argument and result are allowed to differ, because in some cases the result may be different than the arguments (e.g. Void because the implementation doesn't produce a value).

Types that conform to DynamicCallableWithKeywordsToo do allow keyword arguments, and they are passed as a list of keyword/value tuples to its dynamicCall(arguments:) method: the first element is the keyword label (or an empty string if absent) and the second value is the formal parameter specified at the call site.

Before this proposal, the Swift language has two types that participate in call syntax: functions and metatypes (for initialization). Neither of those may conform to protocols at the moment, so this introduces no possible ambiguity into the language.

It is worth noting that this does not introduce the ability to provide dynamicly callable static/class members. We don't believe that this is important given the goal of supporting dynamic languages like Python, but if there is a usecase discovered in the future, it could be explored as future work. Such future work should keep in mind that call syntax on metatypes is already meaningful, and that ambiguity would have to be resolved somehow.

Discussion

While the signature for dynamicCall is highly general we expect the most common use will be clients who are programming against concrete types that implement this proposal. One very nice aspect of this is that, as a result of Swift's existing subtyping mechanics, implementations of this type can choose whether they can actually throw an error or not. For example, consider this silly implementation:

struct ParameterSummer : DynamicCallable {
  func dynamicCall(arguments: [Int]) -> Int {
    return arguments.reduce(0) { $0+$1 }
  }
}

let x = ParameterSummer()
print(x(1, 7, 12))  // prints 20

Because ParameterSummer's implementation of dynamicCall does not throw, the call site is known not to throw either, so the print doesn't need to be marked with try.

Smalltalk Family Languages

Discussions in the early pitch threads have pointed out that languages derived from Smalltalk (including Squeak, Ruby, Objective-C, and Swift itself) resolve method calls using both the base name as well as the keyword arguments at the same time. This argues for the introduction of a third protocol to model this, e.g.:

protocol DynamicCallableKeywordedMethod {
  func dynamicCall(method: String,
                   arguments: [(String, DynamicCallableArgument)]) throws -> DynamicCallableResult
}

If a type implements this protocol, then an entire method lookup (basename + keywords) would be passed to the implementation in a single unit. For example: a.foo(a: 21, 47) would be compiled into: a.dynamicCall(method: "foo", arguments: [("a", 21), ("", 47)]). This model fits directly into the Swift compiler model, so we should support it. A language implementation could choose to implement this protocol but not the earlier protocols, which would cause the compiler to accept a.foo(42), but not let tmp = a.foo; tmp(42) if it were desirable to reject that sort of usage.

Example Implementation

A realistic (and motivating) example comes from a prototype Python interop layer. While the concrete details of this use case are subject to change and not important for this proposal, it is perhaps useful to have a concrete example to see how this comes together.

That prototype currently has two types which model Python values, one of which handles Python exceptions and one of which does not. Their conformances would look like this, enabling the use cases described in the Motivation section above:

extension ThrowingPyVal: DynamicCallableWithKeywordsToo {
  func dynamicCall(arguments: [(String, PyVal)]) throws -> PyVal {
    // Make sure state errors are not around.
    assert(PyErr_Occurred() == nil, "Python threw an error but wasn't handled")

    // Count how many keyword arguments are in the list.
    let numKeywords = arguments.reduce(0) {
      $0 + ($1.0.isEmpty ? 0 : 1)
    }

    let kwdict = numKeywords != 0 ? PyDict_New() : nil

    // Non-keyword arguments are passed as a tuple of values.
    let argTuple = PyTuple_New(arguments.count-numKeywords)!
    var nonKeywordIndex = 0
    for (keyword, argValue) in arguments {
      if keyword.isEmpty {
        PyTuple_SetItem(argTuple, nonKeywordIndex, argValue.toPython())
        nonKeywordIndex += 1
      } else {
        PyDict_SetItem(kwdict!, keyword.toPython(), argValue.toPython())
      }
    }

    // Python calls always return a non-null value when successful.  If the
    // Python function produces the equivalent of C "void", it returns the None
    // value.  A null result of PyObjectCall happens when there is an error,
    // like 'self' not being a Python callable.
    guard let resultPtr = PyObject_Call(state, argTuple, kwdict) else {
    
      // Translate a Python exception into a Swift error if one was thrown.
      if let exception = PyErr_Occurred() {
        PyErr_Clear()
        throw PythonError.exception(PyRef(borrowed: exception))
      }

      throw PythonError.invalidCall(self)
    }

    return PyRef(owned: resultPtr)
  }
}

extension PyVal: DynamicCallableWithKeywordsToo {
  func dynamicCall(arguments: [(String, PyVal)]) -> PyVal {
    // Same as above, but internally aborts instead of throwing Swift
    // errors.
    return try! self.throwing.dynamicCall(arguments: arguments)
  }
}

Source compatibility

This is a strictly additive proposal with no source breaking changes.

Effect on ABI stability

This is a strictly additive proposal with no ABI breaking changes.

Effect on API resilience

This has no impact on API resilience which is not already captured by other language features.

Alternatives considered

A few alternatives were considered:

Naming

The most fertile ground for bikeshedding is the naming of the protocols and the members. In particular, the name of DynamicCallableWithKeywordsToo is gratuitously wrong and needs to be changed. Here are some thoughts on obvious options to consider for naming this family:

We considered but rejected the name CustomCallable, because the existing Custom* protocols in the standard library (CustomStringConvertible, CustomReflectable, etc) provide a way to override and customize existing builtin abilities of Swift. In contrast, this feature grants a new capability to a type.

We considered but rejected a name like ExpressibleByCalling to fit with the ExpressibleBy* family of protocols (like ExpressibleByFloatLiteral, ExpressibleByStringLiteral, etc). This name family is specifically used by literal syntax, and calls are not literals. Additionally the type itself is not "expressible by calling" - instead, instances of the type may be called.

On member and associated type naming, we intentionally gave these long and verbose names so they stay out of the way of user code completion. The members of this protocol are really just compiler interoperability glue. If there was a Swift attribute to disable the members from showing up in code completion, we would use it (such an attribute would also be useful for the LiteralConvertible and other compiler magic protocols).

Collapse DynamicCallable and DynamicCallableWithKeywordsToo into one protocol

It is entirely possible to merge both of these protocols into a single protocol. Such an approach would be simpler (only one thing instead of two) but would lose the ability to statically reject keyword arguments presented to a client that doesn't care about them (e.g. a Javascript binding). Keyword arguments could still be rejected dynamically by aborting if any were provided of course.

In the authors opinion, the simplicity win of this is not significant enough to be worth eliminating the ability to directly express the underlying model provided by the client.

Statically checking for exact signatures

This proposal does not allow a type to specify an exact signature for the callable - a specific number of parameters with specific types. The reason for this is it is already possible to express this in Swift, using subscripts:

struct MyFunctor {
  subscript(simple simple: Int) -> Int {
    return 4
  }
  subscript(keyword keyword: Int, param: String, val val: Int) -> Int {
    return 42
  }
}

let mf = MyFunctor()
print(mf[simple: 57])
print(mf[keyword: 12, "hello", val: 57])

It is true that Swift currently requires square brackets to call these things, but otherwise they are (intentionally) extremely similar to calls. These missing pieces are:

  1. We do not yet support currying getters of properties and subscripts. This composes on top of our existing model, but no one has implemented it yet.
  2. We do not support default arguments or throwing for subscripts. This is a bug, not a feature.
  3. Subscript require square brackets at the call site, not parens.

Of these challenges, only the last one is fundamental to subscripts. If there was an intense desire to solve this problem, there are at least two possible paths forward: 1) implement a new declaration kind to replace subscript (e.g. "call"), or 2) introduce an attribute that changes the caller side syntax to use parens instead of square brackets.

In either case, such a change would be completely orthogonal to this proposal, and is clearly lower priority. Whereas this proposal unblocks entire communities from using Swift by removing widespread noise from common cases, such a proposal would literally only exchange one pair of punctuation characters for another. Further, such a proposal does not address the needs of this proposal, so we'd still need to do this one as well.

Introduce F# style "Type Providers" into Swift

Type providers are an extremely expressive feature of the F# language. They are an expansive (and quite complex) metaprogramming system which permits a "metaprogrammed library" to synthesize types and other language features into the program based on (e.g.) statically knowable type databases. This leads to significantly improved type safety in the case where schemas for dynamic APIs are available (e.g. a JSON schema) but which are not apparent in the source code.

Such a feature is extremely interesting and should be considered for inclusion into a future macro system in Swift. That said, they aren't actually helpful for this proposal for several reasons:

  1. Some values in a dynamic language are completely polymorphic, and we need the ability to message arbitrary properties and methods. If you are familiar with Objective-C, observe that while many values are typed (either through static types or the new Objective-C generics subsystem) that there are still values of id type. Type providers need to be able to provide a language type in these cases, and these need to be callable.
  2. While some dynamic languages have typing systems available (e.g. Python 3 has support for type hints, not all languages do (e.g. not even Python 2, which has an apparently larger user base than Python 3 still), and not all APIs in the languages that support these types actually have type descriptions.
  3. Many of these optional typing descriptions (including Python's and Javascript's) are unsound. This means that such facilities would either be making incorrect assumptions or be over-constraining of the API. In either case, an escape to a fully dynamic call (of some kind) would be necessary.
  4. Our interest is merely to provide access to familiar APIs from these dynamic languages with low friction. It is a non-goal to be better than the host language at using its own APIs: we'd actually rather someone implement a pure-Swift implementation (or a wrapper around the dynamic language implementation) that takes full advantage of the Swift type system and runtime.

Because of these points, we believe that type providers and other optional typing facilities are a potentially interesting follow-on to this proposal which should be measured according to their own merits. If our goal was to (e.g.) make a "better Python than Python" then it could be a high value addition to the language, but we would still have to take this proposal (or something like it) as a primitive feature that type providers would build upon.

Direct language support for Python (and all the other languages)

We considered implementing something analogous to the Clang importer for Python, which would add a first class Python specific type(s) to Swift language or standard library. We rejected this option because it would be significantly more invasive in the compiler, would set the precedent for all other dynamic languages to get first class language support, and because that first class support doesn't substantially improve the experience of working with Python over existing Swift with a couple small "generally useful" extensions like this one.

Owner

lattner commented Nov 10, 2017

Please send comments to swift-evolution so other people see them. Comments added here will be deleted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment