gistfile1.txt

<guestuser> Hey, I was wondering how easy it is to shift the infrastructure from 
cluster to another cluster if bitcask is the storage engine with riak. 

<justinsheehy> guestuser: likely pretty easy, depending on what you mean by "shift 
the infrastructure" 

<guestuser> I meant say I have a 100 node cluster of 4GB RAM boxes. And I want to 
shift to another 100 node cluster of 8GB RAM boxes. So take and drop style. 

<seancribbs> guestuser: I doubt you'd be moving all 100 at once. Typically you'd do a 
"rolling upgrade". 

<guestuser> From what I take from bitcask, it should only involve taking DB files and 
dropping them to new location. I might be wrong. 

<guestuser> A rolling upgrade? 

<seancribbs> guestuser: yes, take one node down, upgrade it, bring it back up 

<guestuser> Yes, thats true, but what if I need to switch datacentres? 

<justinsheehy> guestuser: yes, if you want to switch data centers 
(and don't have cross-DC replication)  then you could duplicate the nodes on 
the other side, including ring files and data directories, and it should Just Work. 

<guestuser> I assumed so. Bitcask creates immutable files right? 

<seancribbs> assuming the node names can stay the same, i.e. using DNS names 

<seancribbs> (rather than IP addresses) 

<seancribbs> guestuser: yes 

<guestuser> Ah, I see. I think the DNS names is always a good idea. I would 
believe that's how a system knows a dead node is back up.. 

<seancribbs> guestuser: not exactly, but it would definitely let 
you easily replace a machine with hardware failure 

<guestuser> That's great. Also, an infra question - Which is better to 
implement on EC2. An army of small instances or much lesser large nodes. We are
trying to build a graph DB on top of riak. So to implement the algos the red 
throughput needs to be fast. We know the keys, so no high M/R usage required. 

<seancribbs> guestuser: oy, that's a hard question to answer. EC2 is so 
variable in performance 

<seancribbs> in general, i would say larger is better, as you're less likely to 
be sharing with other people. however, that's not bullet-proof 

<seancribbs> best thing you can do is set up both scenarios and then benchmark them 

<seancribbs> basho_bench will help with that 

<guestuser> Yeah, we did a similar system on top of tokyo before. I find EC2 
benchmarks largely pointless. They are never consistent. :) 

<seancribbs> true, but one you run long enough should give you a ballpark idea 

<justinsheehy> and generally, if you're going to be running a lot of machines all 
the time, EC2 is much more expensive for less reliable performance when compared to 
traditional hosting. 

<justinsheehy> but yeah, the "small" instances are typically really bad 

<benchMark> justinsheehy: The riak_kv_bitcask_bucket_backend is going to increase the random
disk I/O quite a bit isn't it? 

<guestuser> Agreed. But I thought smaller instances would give higher I/O. Also, 
I don't trust EBS as much as people do. But I figure EC2 might still end up being better 
than Slicehost/Linode/Rackspacecloud for anything that's large enough to require riak. 

<guestuser> Again, I might be wrong. 

<benchMark> Assuming you have multiple buckets that are taking writes. 

<justinsheehy> benchMark: that depends a lot on usage patterns. maybe. 

<benchMark> justinsheehy: Sure, and if you have a RAID controller with BBU it will 
absorb a lot of that and coalesce them into nice sequential streams. 

<justinsheehy> the idea is to provide a choice. have buckets be free or have 
buckets be isolated. 

<justinsheehy> yep 

<benchMark> Yeah, I understand the trade-offs. 

<benchMark> Makes good since, I just found it curious. 

<benchMark> s/since/sense/ 

<seancribbs> benchMark: not everyone can have your sexy Schooner appliances 

<benchMark> Haha 

<benchMark> It'll definitely generate more random I/O since it will have 
to seek to write to different bitcasks, but once there it can write sequentially 
at least. And you're right, it's very dependent on the usage patterns how much
 seeking it has to do. 

<guestuser> Just so I shift things correctly from tokyo, are there any known 
issues that can cause data corruption? 

<guestuser> Well I/O can be boosted with $15/mo SSD ;)