gistfile1.txt

proposal for syntax/method for doing type templating:
all functions that need to do type-specific manipulation (let’s say, equals for the purpose of this example) will get a template argument T. T will be a struct TypeInfo (naming tbd) that contains the types.T information, as well as //execgen:inline’d methods that provide each overload, as well as utility methods.
still working on syntax for the “go type templates” - aka, how do we specify that a template needs int and []int in the case of int64 columns, but []byte and coldata.Bytes in the case of bytes columns, so ignore that for now.
for example:
// execgen:template<typ>
func genericEqualityFunc(colA []_GOTYPE, colB []_GOTYPE, idx int, typ *TypeInfo) bool {
    a := typ.UnsafeGet(colA, idx)
    b := typ.UnsafeGet(colB, idx)
    eq := typ.Equals(a, b)
    return eq
}
there will be associated, inlinable methods for each TypeInfo, e.g.
type boolTypeInfo struct {
 ...
}
// execgen:inline
func (b intTypeInfo) Equals(a int, b int) bool {
   return a == b
}
// execgen:inline
func (b intTypeInfo) UnsafeGet(col []int, idx int) {
    return col[idx]
}
(edited)





10:21
(this won’t work yet for various reasons but this is my proposal, generally)

rohany  10:22 PM
are you going to hand generate these *TypeInfo structs?

jordan:sparkles:  10:22 PM
yes, i think so
10:22
they’re essentially hand generated already

rohany  10:22 PM
so now we're really like c++ template specialization
10:23
I'm not clear how dispatching of *TypeInfo happens to the Specialized overloads
10:23
Will that be an interface?

jordan:sparkles:  10:23 PM
TypeInfo itself would be an interface, but the interface wouldn’t be accessed at runtime unless someone chose to use one in a non //execgen:template method
10:24
one cool thing btw about the //execgen:foo things so far is that a few times while playing with it, i’ve forgotten to include one or both of the annotations and everything still works :slightly_smiling_face:
:+1:
1


rohany  10:24 PM
So how does someone use genericEqualityFunc

jordan:sparkles:  10:28 PM
well i’m still not certain about the exact mechanisms of passing in colA and colB. so that’s tbd.
but for the type param you’d either do genericEqualityFunc(colA, colB, intTypeInfo) (concrete template invocation) or genericEqualityFunc(colA, colB, typeInfo) where typeInfo was a type parameter that you were passed
the root of the chain here would eventually need to be the per-type operator structs, i think. so that’s more syntax we need to invent, but the templater would see something like this:
// execgen:template <typ>
type myOperator struct {
    ... stuff ...
    typ TypeInfo
}
and convert that into myOperator_int for all types. then, calling genericEqualityFunc(colA, colB, o.typ) from a method on o would be a static call? i think?

rohany  10:29 PM
 but for the type param you’d either do genericEqualityFunc(colA, colB, intTypeInfo) (concrete template invocation) or genericEqualityFunc(colA, colB, typeInfo) where typeInfo was a type parameter that you were passed
10:29
ok this makes sense, and is a straight forward extension of the bool stuff we have
:that:
1

10:29
assuming that we can do the for each type

jordan:sparkles:  10:29 PM
yeah, that’s … also tbd but hey

rohany  10:30 PM
idt thats bad -- if we are hand writing these type structs, then there would be some way of registering them that the templater can access

jordan:sparkles:  10:30 PM
yeah. or // execgen:template <typ (types.A, types.B, types.C), b (true, false)> (edited) 

rohany  10:30 PM
 and convert that into myOperator_int for all types. then, calling genericEqualityFunc(colA, colB, o.typ) from a method on o would be a static call? i think?
10:31
this seems right, but also seems like it might be hard
10:31
i don't know if at the AST level you can do things like this (or have access to them)

jordan:sparkles:  10:31 PM
yeah…. turning o.typ into static text would be impossible i think
10:31
well, unless you search for it during templating
10:31
which is definitely possible - you have the receiver and the struct fields

rohany  10:31 PM
well you don't have to right -- i thought it was just going to be an opaque interface
10:32
we can edit each of the generated structs to include the "real" type at template time
10:32
so in the template, you'll access o.typ
10:32
but in the generated code, we'll switch out o.typ for the real specialized one
10:32
so then it is all static

jordan:sparkles:  10:32 PM
yes, i agree, that’s what i was trying to say as well
:+1:
1


rohany  10:33 PM
this is pretty ambitious, but very cool
10:33
:slightly_smiling_face:

jordan:sparkles:  10:33 PM
:smiling_imp:
10:33
i don’t think it’s actually that bad

rohany  10:33 PM
getting the template pass through i think is the next step
10:33
then this stuff kinda falls out

jordan:sparkles:  10:33 PM
oh i have template pass through already! :dancingpanda:

rohany  10:33 PM
oh what
10:33
beast
10:34
the recursive pass thing?

jordan:sparkles:  10:34 PM
yeah

rohany  10:34 PM
is that landed
10:34
did i just not notice this in the pr

jordan:sparkles:  10:35 PM
i don’t think so. but it was easier than expected. it’s a change to monomorphization, which is now 2 passes.
pass 1: replace all Idents that are template formal param names with template args
pass 2: do the whole if/else reduction crap (btw this is not even strictly necessary as Go does good static falsification of branches but it does clean the code a lot)
just pass 1 by itself (as long as you do it recursively and not just in a single go) does the template pass through (edited) 

rohany  10:36 PM
that makes sense -- maybe it was one of those harder to think of than write
:heavy_plus_sign:
1


jordan:sparkles:  10:36 PM
so the output code looks like
if true {
   if false {
    ...
   else {
     ...
   } else {
    ...
}
(edited)
10:36
etc, probably missed braces

rohany  10:36 PM
are there any templates where we only do the type templating for a small subset of types?
10:37
I'm not sure how progress would be made in this without creating all of the templated structs
10:37
first

jordan:sparkles:  10:37 PM
i don’t remember, but probably
10:38
this proposal also doesn’t yet help situations where we uniformly produce all projection/selection operators, as those have another layer of templating over operators themselves, but i think we’ll get there..

rohany  10:38 PM
we'll get there when we get there
10:38
this should also help all of the nested overloads we have right

jordan:sparkles:  10:39 PM
i think my (crappy) syntax proposal would help in your case tho
//execgen:template<a (types.Int, types.Bool)> (edited) 
:+1:
1


rohany  10:39 PM
we'll actually have int8, int16, int32, int64 type structs, rather than a nested thing underneath IntFamily

jordan:sparkles:  10:39 PM
we already have all those type structs, the nesting is just in the constructor

rohany  10:40 PM
yeam making a simple specialization syntax would help different edge cases here and there
10:40
i meant that it would help us unnest this stuff
10:40
more naturally

jordan:sparkles:  10:40 PM
ah yeah i agree

jordan:sparkles:  10:59 PM
i forgot that you can’t really have a true TypeInfo interface unless you added templated params to interfaces, which seems pointless. but most of the idea still holds, i think (edited) 

rohany  11:00 PM
can't the TypeInfo interface just operate solely on interface{}
11:00
so then you can use it like you have all the individual typeInfos pretemplating

jordan:sparkles:  11:01 PM
foo() int doesn’t implement foo() interface{} is the issue
https://play.golang.org/p/aEWpC_8qUmh

rohany  11:04 PM
wouldn't the pretemplating function return like _GOTYPE though, not int?

jordan:sparkles:  11:05 PM
_GOTYPE is an artifact of the original template method (it gets regex replaced with {{.GoType}} which is a member variable on the text/template type struct) and something i was hoping to find a way to get rid of

rohany  11:07 PM
I see
11:07
alright, have to :brain4:
11:07
It feels important to me that users of TypeInfo pretemplating should feel like they're looking at real go
11:08
or at least the editor shouldn't complain

jordan:sparkles:  11:08 PM
i agree
11:08
which does make it seems like everything should appear to be interface{}

rohany  11:11 PM
perhaps named arguments to the rescue here (somewhat?)
11:11
template<t1, t2>
func x (t1 ...) (t2 ...) {
}
11:12
can at least acknowlege that we're going to template over the return type or something

rohany  11:38 PM
foo() int doesn’t implement foo() interface{} is the issue
Can you elaborate why this is a problem / when we run into this

jordan:sparkles:  11:44 PM
type TypeInfo interface {
   UnsafeGet(col interface{}, idx int) interface{}
}
type intInfo struct {}
func (i intInfo) UnsafeGet(col interface{}, idx int) interface {} {
   return col[idx] // <-- can't do this because interface{} isn't sliceable
}
func (i bytesInfo) UnsafeGet(col interface{}, idx int) interface{} {
    return col.Get(idx) // <--- can't do this because col isn't usable as a coldata.Bytes
}
11:45
one option is that you could make a new type like
type AnyColSlice []interface
func (a AnyColSlice) Get(i int) interface{}
... and all methods that you might have on coldata.Bytes and other non-slice col types
(edited)
11:46
and then at codegen time none of this stuff matters because the types will go away. but, dunno. the other thing that’s still missing is how to extract the text of the Go type for the calling scope. definitely still lots of missing pieces.

rohany  11:49 PM
I don't think the type specific structs need to implement the templater TypeInfo interface
:open_mouth:
1

11:49
the implementations would be ruined by having to ascribe to some generic interface right?

jordan:sparkles:  11:50 PM
not rly cuz they’ll get inlined, and the inlining strips type information (or it could)
11:51
and i was thinking misty-eyed that it’d be cool to keep things working whether or not the templating actually happened - so you could choose to use a slow dynamic TypeInfo implementation at runtime if you wanted lol. but it’s pointless (edited) 
:open_mouth:
1

11:51
that’s just a super droppable requirement, so, i’ll just drop it
:jordan:
1

11:55
the other thing that’s still missing is how to extract the text of the Go type for the calling scope. definitely still lots of missing pieces.
I think for this it might be good to copy C++ again
// execgen:template <typename T, typename TSlice>
func (col TSlice, idx int) T {
   return col[idx]
}
but this won’t look like it compiles in goland because TSlice and T aren’t real types. without something like this though i struggle to see how you’d propagate the proper text for each type’s scalar and slice types
11:56
(by this i mean, someone somewhere has to convert var x T into var x bool or var x int)

rohany  11:56 PM
I'm confused why you need that though -- why can't intInfo just have a method get(col []int, idx int) int
11:57
TypeInfo can have get(col []interface, idx int), but when its swapped out for intInfo then the specialized one is used

jordan:sparkles:  11:57 PM
yeaaaah… that’s fair. i guess i was thinking optimistically that we could also have a thing that would let you just do the simple one (return col[idx]) once and overload for the more complicated ones :badpokerface:
11:57
moving too far over skis

rohany  11:58 PM
I'm hesitant to start thinking about type replacement stuff
11:58
we only have a syntax tree :disappointed:
11:59
 that would let you just do the simple one (return col[idx])
Oh i see. This is desirable