1

freshtonic # anyone know where I can find an example of Riak post-commit hook?

seancribbs # freshtonic: as soon as we have one, it'll be on the wiki. what do you want your hook to do?

freshtonic # seancribbs: when I put something in bucket A, I want to create another record in bucket B within the commit hook.

seancribbs # freshtonic: any more specificity than that?

freshtonic # bucket B is a log of change events. Whereas bucket A contains the 'current' document, bucket B contains the history of changes.

seancribbs # aha

freshtonic # I can fake it from our client, if necessary, but I thought it would be a prime candidate for a post-commit hook

seancribbs # yes, sounds it basically, if you're comfortable with erlang, it'll straightforward you are passed the riak_object as the only argument

freshtonic # I'm comfortable with erlang. I'm just looking for an example. It's not clear from reading the wiki (unless I missed something) what API is exposed

seancribbs # yeah, we should really have an example

freshtonic # I'll happily contribute some documentation once I've figured it out :) Also, I understand that the hooks are stored in buckets, and the bucket properties can contain a list reference of pre and post commit hooks. But the example just has the function name in the reference, and not the name of the bucket from which it is loaded (like the m/r function references) so I'm a little confused.

freshtonic # Essentially, do I just create a bucket that contains a named document containing erlang source or does the bucket contain compiled erlang data? s/erlang data/erlang binary

seancribbs # no, the erlang source has to be compiled and available on all nodes only Javascript can be stored in a bucket/key

freshtonic # Awesome, understood.

freshtonic # Just to be clear: if I bring a new Riak node into the cluster, I would have to make sure that my compiled 
erlang post-commit hook was copied to the new node manually, or does Riak manage the distribution once the hook is installed?

seancribbs # the former you would typically build it into your "release" or distribute as part of a config mgmt system like chef/puppet

freshtonic # yeah, we're using puppet here.
it would be a cool feature if Riak managed the distribution of the binary though. Are there any plans to do that in the future?

seancribbs # not high on the priority list

freshtonic # fair enough.

seancribbs # it would better for Erlang to support it, and Riak just piggy back on that it's a complicated issue but I have my doubts about how soon it will be seen in the VM

freshtonic # I'm not an erlang ninja or anything, but it already supports hot module reloading. What is it specifically that erlang does not support that you would need?

moonpolysoft # freshtonic: remote code resolution

freshtonic: erlang bytecode basically blind references other modules and fully depends on the runtime to load modules on demand so you'd need to have some kind of a global code server and everyone would have to know to talk to it
when they cannot resolve a bit of bytecode

moonpolysoft # i think it just simply hasn't been done yet because generic mechanisms to handle master slave relationship haven't been generally available in erlang until recently you could probably bang something together in a week or so using gen leader bonus points for having a cluster wide deploy point that would accept a reltool'ed tarball and help manage the code upgrade

freshtonic # moonpolysoft: OK, so it's more complicated than I thought. In my head I was just thinking 1) load all known modules (from buckets) when Riak boots 2) when a modification is made that triggers a post-commit, check if we have loaded the latest version of the module, if not reload it 3) execute the hook. I wasn't thinking about the hook function's deps etc.

moonpolysoft # yeah that's the thing the transitive deps are what gets you
it'd be nice if you had some kind of sandbox mechanism like java's classloader system
where you could instantiate a new one and have code that was running against different versions of the same modules and it would all be ok with it

freshtonic # that would be cool. I'll have a read about the gen leader stuff. Not familiar with it. Thanks for the info

freshtonic # another question: what's the largest number of documents in a bucket in Riak that anyone has heard of in 
production? tens of millions? billions?

benblack # i don't believe anyone has try to put billions of objects in a bucket

mootpointer # Give us long enough and we might try. ;)

freshtonic # mootpointer: we'll have somewhere in the range of 5M objects in one of our buckets pretty quickly. Any changes to those objects create an event object in a secondary bucket, which will grow much larger but I expect we'll be purging the history every so often. But yes, left unchecked it could grow pretty large :)

justinsheehy # we've seen tens of millions. a standard-ish test around here uses 5 mil

freshtonic # justinsheehy: good to know, thanks