Riak-client transparent failover

gistfile1.txt

Each . is a successful get/put. Rescued ... shows where an exception was 
raised and retried by the client. [float, float, float] are the error_rates 
of the three nodes, exponentially decaying. When I kill a node, the exception
is caught and retried transparently. If any were to fail three times
successively, an X would be printed.

Throughout this test I'm restarting nodes left and right at random. So long 
as fewer than two crash at once, no errors are raised.

This works so long as all operations in a given block are replayable.

{:protocol=>"pbc", :protobuffs_backend=>:Beefcake}
[0.0, 0.0, 0.0]
........Rescued #<SocketError: Unexpected EOF on PBC socket>
[0.9999765148550644, 0.0, 0.0]
...................Rescued #<SocketError: Unexpected EOF on PBC socket>
[0.47372937329054626, 0.9999735811135126, 0.0]
..................Rescued #<SocketError: Unexpected EOF on PBC socket>
[0.2359306946773279, 0.49801525623290027, 0.9999766975643869]
............Rescued #<SocketError: Unexpected EOF on PBC socket>
[1.1485917533473882, 0.3137023523006815, 0.629890440755566]
...........................................


{:protocol=>"http", :http_backend=>:NetHTTP}
[0.0, 0.0, 0.0]
[0.0, 0.0, 0.0]
..............Rescued #<Errno::ECONNREFUSED: Connection refused - connect(2)>
[0.9999734140001564, 0.0, 0.0]
Rescued #<Errno::ECONNREFUSED: Connection refused - connect(2)>
....................Rescued #<Errno::ECONNREFUSED: Connection refused - connect(2)>
[0.9521917444257798, 0.9999733726896836, 0.0]
.........................Rescued #<Errno::ECONNREFUSED: Connection refused - connect(2)>
[0.3467690194951512, 0.36417013007610805, 0.9999733028223107]
.........................................


{:protocol=>"http", :http_backend=>:Excon}
[0.0, 0.0, 0.0]
[0.0, 0.0, 0.0]
......................................Rescued #<Excon::Errors::SocketError: Connection refused - connect(2)>
[0.999965619659085, 0.0, 0.0]
Rescued #<Excon::Errors::SocketError: Connection refused - connect(2)>
...............Rescued #<Excon::Errors::SocketError: Connection refused - connect(2)>
[1.0433151565977563, 0.9999814489888758, 0.0]
..........Rescued #<Excon::Errors::SocketError: Connection refused - connect(2)>
[0.6527806746174123, 0.6256675897326256, 0.999908874498114]
.....................................