Skip to content

Instantly share code, notes, and snippets.

@campoy
Created January 12, 2018 21:54
Show Gist options
  • Star 5 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save campoy/b0d36c2c115920d2dd5c8b0188879b32 to your computer and use it in GitHub Desktop.
Save campoy/b0d36c2c115920d2dd5c8b0188879b32 to your computer and use it in GitHub Desktop.
C function called from BigQuery
int sum(int a, int b)
{
return a + b;
}
SELECT * FROM
js(
(SELECT 21 as x),
x,
"[{name:'y', type: 'float'}]",
"function(row, emit) {
const x = row.x;
const memory = new WebAssembly.Memory({ initial: 256, maximum: 256 });
const env = {
'abortStackOverflow': _ => { throw new Error('overflow'); },
'table': new WebAssembly.Table({ initial: 0, maximum: 0, element: 'anyfunc' }),
'tableBase': 0,
'memory': memory,
'memoryBase': 1024,
'STACKTOP': 0,
'STACK_MAX': memory.buffer.byteLength,
};
const importObject = { env };
var bytes = new Uint8Array([0, 97, 115, 109, 1, 0, 0, 0, 1, 139, 128, 128, 128, 0, 2, 96, 1, 127, 0, 96, 2, 127, 127, 1, 127, 2, 254, 128, 128, 128, 0, 7, 3, 101, 110, 118, 8, 83, 84, 65, 67, 75, 84, 79, 80, 3, 127, 0, 3, 101, 110, 118, 9, 83, 84, 65, 67, 75, 95, 77, 65, 88, 3, 127, 0, 3, 101, 110, 118, 18, 97, 98, 111, 114, 116, 83, 116, 97, 99, 107, 79, 118, 101, 114, 102, 108, 111, 119, 0, 0, 3, 101, 110, 118, 6, 109, 101, 109, 111, 114, 121, 2, 1, 128, 2, 128, 2, 3, 101, 110, 118, 5, 116, 97, 98, 108, 101, 1, 112, 1, 0, 0, 3, 101, 110, 118, 10, 109, 101, 109, 111, 114, 121, 66, 97, 115, 101, 3, 127, 0, 3, 101, 110, 118, 9, 116, 97, 98, 108, 101, 66, 97, 115, 101, 3, 127, 0, 3, 130, 128, 128, 128, 0, 1, 1, 6, 147, 128, 128, 128, 0, 3, 127, 1, 35, 0, 11, 127, 1, 35, 1, 11, 125, 1, 67, 0, 0, 0, 0, 11, 7, 136, 128, 128, 128, 0, 1, 4, 95, 115, 117, 109, 0, 1, 9, 129, 128, 128, 128, 0, 0, 10, 196, 128, 128, 128, 0, 1, 190, 128, 128, 128, 0, 1, 7, 127, 2, 64, 35, 4, 33, 8, 35, 4, 65, 16, 106, 36, 4, 35, 4, 35, 5, 78, 4, 64, 65, 16, 16, 0, 11, 32, 0, 33, 2, 32, 1, 33, 3, 32, 2, 33, 4, 32, 3, 33, 5, 32, 4, 32, 5, 106, 33, 6, 32, 8, 36, 4, 32, 6, 15, 0, 11, 0, 11,])
WebAssembly.instantiate(bytes, importObject).then(wa => {
const exports = wa.instance.exports;
var sum = exports._sum;
emit({y: sum(x, x)});
});
}
")
@campoy
Copy link
Author

campoy commented Jan 12, 2018

using emcc I compiled sum.c to sum.wasm and used the bytes in that file to create the Uint8Array passed to WebAssembly.

And ... it works!
But it takes almost 3 seconds to double a single number

@campoy
Copy link
Author

campoy commented Jan 12, 2018

Related: can't use standard SQL for now because I have no idea on how to extract a value from a promise
see https://gist.github.com/campoy/122f5e24740488e6ea3c973ab195e158

@campoy
Copy link
Author

campoy commented Jan 12, 2018

Felipe improved the result by using NEST to batch requests
see https://gist.github.com/fhoffa/4d34bdb2ef2ccdaa82ca9ed64a14d682

@fhoffa
Copy link

fhoffa commented Jan 12, 2018

10,000,000 rows in 8 seconds, yup :)

https://twitter.com/felipehoffa/status/951951175250841600

@bzz
Copy link

bzz commented Jan 17, 2018

🥇 :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment