cfunction/ccall proposal

ccall cfunction.md

This Julip is intendend to standardize and simplify the ccall/cfunction calling convention specifications, with minimal breakage from the current situation.

The intention is to supply a more descriptive ccall/cfunction interface that is faster to parse for humans, and can accurately reflect all c-callable functions with less ambiguity/confusion.
As a special requirement, all elements of this should be valid syntax outside of the ccall function. The purpose of this is to simplify code-generation macros and reduce the learning curve.
There are several, fairly independent parts to this proposal. I will try to enumerate them in order of their dependencies.

.1. Move the Julia value pointer (jl_value_t*) to point to the first element of the Julia struct, instead of the type tag. This has been discussed previously (and demonstrated in prototype) and is not controversial. ref JuliaLang/julia#2818

.2. Introduce a function Box(x) that creates a mutable jl_value_t* containing a single pointer (to X). ref JuliaLang/julia#2322. As an extension of this, deprecate the & special syntax in ccall.
Box has the following, rather boring, definition:

type Box{T}
  a::T
  Box(a) = new(a)
end
convert{T}(::Type{Box{T}}, x::T) = Box{T}(a)

# convert{T}(::Type{Ptr{T}}, x::Box{T}) =
#     unsafe_pointer_to_objref(isbits(x) ? x : x.a) # (approx) for julia 0.2 compatibility

.3. Future optimization: identify opportunities where mutable values can be allocated on the stack by running escape analysis on all inputs to any function. a variable is considered to not escape if it is not assigned to any location which may escape. a location may escape if it is a global variable, or is an input or output to the function. no intrinsic function is considered an escape location.

.4. Consolidate the usage of Ptr in ccall and cfunction to meet the following specification. Change cfunction to emit a native function with exactly the interface specified, creating a wrapper function IFF multiple functions may apply. Note: this assumes the deprecation of the & syntax, replacing it with the Box{T}() function/type.

Note that Julia essentially has 6 or 7 distinct types to be addressed: bits, immutable, type, abstract, Any, None, Box ccall translation guide julia -> c:
a. (Box{X},) ->
(*struct X) if X is immutable, type, abstract OR
(*X) if X is bits, None OR
(**jl_value_t) if X is Any -- equivalent to void **
b. (X,) ->
(struct X) if X is immutable, type OR
(X) if X is bits
invalid if X is None
(*jl_value_t) if X is Any -- equivalent to void *
c. (Box{Box{X}},) -> error

cfunction translation guide julia cfunction matches julia syntax -> c:
a. (Box{X},) matches (X,), byref (implicit unsafe_pointer_to_objref()::X or unsafe_load(), as needed) ->
(*struct X) if X is immutable, type, abstract OR
(*X) if X is bits, None OR
(**jl_value_t) if X is Any -- NOT equivalent to void *
b. (X,) matches (X,), byval (implicit unsafe_load()) ->
(struct X) if X is immutable, type OR
(X) if X is bits
invalid if X is None or abstract
(*jl_value_t) if X is Any -- NOT equivalent to void *
c. (Box{Box{X}},) -> error

ccall return types:
Box{X} would try to return X, as a Julia object (via implicit unsafe_pointer_to_objref()::X)
X would try to load X, copied to a Julia object (via implicit unsafe_load()::X)
Box{Box{X}} is an error
Void would return nothing
Box{Any} would try to return X, as a pointer to a Julia object (jl_value_t**), which is an error (the correct way to write this would be Ptr{Any})
Any would return a jl_value_t* of unspecified type (via implicit unsafe_pointer_to_objref()) -- NOT equivalent to void *

cfunction return types:
Box{X} would try to return X, as a Julia object (jl_value_t*) -- equivalent to *X or *struct X
X would return X byval (as a struct or integer)
Box{Box{X}} is an error
Void would return nothing (void return type)
Box{Any} would try to return X, as a pointer to a Julia object (jl_value_t**), which is an error
Any would return a jl_value_t* of unspecified type -- equivalent to void *

Note: the julia-calling convention jl_value_t* jl_f(jl_function_t *f, jl_value_t **args, uint32_t nargs) would be written as: cfunction(jl_f, Any, (Box{Function}, Ptr{Any}, Uint32)) and jl_f(a,b,c) = ccall(:jl_f, Any, (Box{Function}, Ptr{Any}, Uint32), Box(jl_f), {a, b, c}, 3, with the extra wrapper jl_f(f::Function, args::Ptr{Any}, nargs::Uint32) = f([unsafe_load(args,i) for i = 1:nargs]...) being necessary to make this call recursive

This point is relatively trivial, but: I would really like ccall to support a syntax with directly connected argument::Types instead of the current (type1,type2,), each, arg....

For example, I think the following macro is more readable and less error prone than (REFIID,Ptr{Ptr{Void}}), clsid, obj). (obviously I can do an @ccall macro like this, but it would be nice to have - imho - clearer syntax in the base language)

res = @vcall(this, 1, HResult, clsid::REFIID, obj::Ptr{Ptr{Void}})

macro vcall(this, idx, rtype, args...)
    argnames = {x.args[1] for x in args}
    argtypes = :((Ptr{Void}, $({eval(x.args[2]) for x in args}...)))
    quote
        local fptr = unsafe_load(unsafe_load(pointer(Ptr{Ptr{Void}},$(this))),$(idx)+1)
        ccall(fptr, thiscall, $(esc(rtype)), ($(esc(argtypes))), $(esc(this)),$(argnames...))
    end
end

note that following discussion with Jeff, Box should be named Ref. this is already in use in Gtk.jl under the name Mutable.

@ihnorton the problem with that approach is that :: is a type-assertion, but we want a cconvert. i've tried to request an :> operator (currently a syntax error) or the keyword as be defined for that purpose, but haven't gotten much traction (JuliaLang/julia#1470 (comment)):

a :> B = convert(B, a)

related: va_list julep JuliaLang/julia#6661