Skip to content

Instantly share code, notes, and snippets.

@PharkMillups
Created September 22, 2010 14:37
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save PharkMillups/591780 to your computer and use it in GitHub Desktop.
Save PharkMillups/591780 to your computer and use it in GitHub Desktop.
14:42 <timmorgan> Is there any way to do bulk insert into Riak?
14:43 <seancribbs> timmorgan: no, one key at a time, but it's pretty easy to script
14:44 <misaka> seancribbs - Is that a design decision? ie: is it ever likely to change?
14:44 <timmorgan> ok thanks -- inserting 14K records is taking 10 mins or so
14:44 <seancribbs> timmorgan: if you can send it via Protocol Buffers, it'll be faster
14:45 <timmorgan> is that an alternative to the REST interface?
14:45 <seancribbs> yes
14:46 <timmorgan> ok I'll check it out!
14:46 <seancribbs> it's not supported in all languages right now, unfortunately
14:46 <timmorgan> no Ruby I gather?
14:47 <seancribbs> no… state of protobuffs in ruby is rather sad
14:47 <seancribbs> in general, not just with riak
14:47 <timmorgan> oh I see
14:49 <timmorgan> I'm not familiar with it, but would this be a good start?
http://github.com/macks/ruby-protobuf
14:51 <seancribbs> yes, I've looked at that. a little circumstantial, but someone made a protobuf
client using it and found it was slower than HTTP with curb
14:51 <seancribbs> my guess is that it's a combination of slow pack/unpack and bitshifting in Ruby
14:52 <seancribbs> which is pretty typical for these binary protocol things, witness the mongodb driver
14:55 <timmorgan> bummer
14:55 <seancribbs> timmorgan: in the long-term, we intend to have a pure C driver that can
link with many languages
14:56 <timmorgan> cool
14:56 <timmorgan> what's the state of affairs in Node.js land?
14:56 <timmorgan> I don't know Node, but am thinking of picking it up for a project
14:56 <timmorgan> thinking that Node + Riak could be awesome
14:57 <seancribbs> Node has nice support for both interfaces
14:57 <seancribbs> see riakjs.org
14:57 <technoweenie> welllll
14:57 <technoweenie> the coffeescript rewrite isnt done yet :)
14:57 <technoweenie> the current version only uses http. frank and i are still finishing
things up
14:57 <seancribbs> timmorgan: technoweenie is your man on that ;) and frank06
14:57 <seancribbs> technoweenie: please to be stealing my test server
14:58 <timmorgan> I was trying to compare Node+Riak to Node+Riak, but couldn't find a
working MySQL driver for Node
14:59 <timmorgan> Still, Riak seems freaking fast (as does everything) on Node
14:59 <timmorgan> oops I mean Node+MySQL to Node+Riak
15:01 <technoweenie> timmorgan: http://github.com/felixge/node-mysql
15:02 <timmorgan> pretty sure I tried that one
15:02 <timmorgan> maybe it was just me being dumb
15:02 <timmorgan> I will try again
15:03 <timmorgan> oh right, I tried masuidrive's fork, per Google's recommendation --
hasn't been updated since April
15:03 <timmorgan> I will try felixge's
15:03 <technoweenie> hah
15:03 <technoweenie> well he's a prolific and badass node author. ping him if
there are bugs
15:35 <timmorgan> I got the mysql driver to work; thanks technoweenie
15:35 <technoweenie> rad
15:36 <timmorgan> I think I'm doing something wrong with Riak though -- I get
about 475 reqs/s with Node+Riak, and over 1300 with Node+MySQL
15:36 <timmorgan> microbenchmark, I know, but that difference is striking
15:37 <timmorgan> I installed the Inno store
15:37 <amerine> timmorgan: seems odd.
15:38 <amerine> timmorgan: I use bitcask for almost everything riak+node though.
15:40 <seancribbs> timmorgan: depending on what you're doing and how many nodes you have,
that's to be expected
15:40 <seancribbs> HTTP is much more heavyweight than MySQL binary protocol
15:41 <timmorgan> I guess so
15:42 <technoweenie> yea you'll want protobufs for speed
15:43 <technoweenie> the win with riak comes from having lots of nodes for handling
writes. whereas a single mysql box will hit a limit and start deadlocking (depending on
indexes and crap like that)
15:43 <technoweenie> well, one of the wins, i guess
15:44 <timmorgan> right
15:44 <technoweenie> are those reads or writes though?
15:44 <timmorgan> reads
15:44 <technoweenie> just in a little test of mine, i noticed a quick 2x speedup just moving
to protobufs
15:44 <seancribbs> it's generally in the range of 2-10x speedup
15:45 <seancribbs> depending on many factors
15:45 <timmorgan> ok cool
15:45 <timmorgan> riak.js doesn't support them as of today?
15:45 <timmorgan> (I thought that's what I understood)
15:45 <seancribbs> i thought technoweenie got it working
15:45 <seancribbs> ;-)
15:45 <technoweenie> its not released
15:46 <timmorgan> can't wait to try it
15:46 <technoweenie> well you can try it now but its only coffeescript. i'm waiting on frank to
finish rewriting the http stuff
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment