Created
July 13, 2010 20:18
-
-
Save PharkMillups/474455 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
freshtonic # seancribbs: are you still around? I was wondering if you'd be able | |
to answer a couple of outstanding questions regarding benchmarking riak that | |
I posted to the list yesterday | |
drev1 # freshtonic: what questions did you have; I will do my best to answer them | |
freshtonic # hi drev1, one moment, I'll just grab them... | |
freshtonic # drev1: for context, it's probably a good idea to look at my email | |
on the riak-users list. [ --> http://bit.ly/aUauLA <-- ] My first question was about how many | |
keys I can store in memory with bitcask: given our large key size of 67 characters, combined with | |
the Bitcask's padding and storage layout, how many keys should we be able to manager | |
per node per GB? | |
drev1 # here is the keydir structure: | |
http://hg.basho.com/bitcask/src/tip/c_src/bitcask_nifs.c#cl-37 I think maybe 92 bytes per entry | |
freshtonic # thanks drev1, and my second question is does each worker | |
process (the concurrent setting in basho_bench) handle more than a single HTTP | |
request at a time, or is it 1 worker == 1 request ? | |
drev1 # freshtonic: a worker performs one request at a time | |
freshtonic # OK, so now seancribbs's response to my email confuses me even more. | |
He said with concurrent set to 50 and mode set to max, I would saturate the network, | |
when in reality it's there can be no more than 50 pending HTTP requests at a time | |
bingeldac # I need to ask him if he has seen that, but I haven't seen much improvement | |
above 15 concurrent workers if you only have one or two hosts you would pile up connections | |
with 4 nodes 15 seems like a good number in my experience | |
freshtonic # bingeldac: I find that confusing. How can connections pile up, if each worker | |
can only deal with one connection at a time? i.e. it makes a request and waits for a response | |
before making another request | |
seancribbs # freshtonic: you there? | |
freshtonic # hi Sean, yeah here | |
seancribbs # so, there's a number of limiting factors here which may give you | |
diminishing returns as you increase the number of workers | |
and where you don't see increased throughput as you increase the number of workers is | |
probably where you should stop and please note that these are not recommendations for how to | |
change things, just things you should be aware of | |
freshtonic # I understand. What I am not clear on is the relationship between the number | |
of workers and the number of requests in progress. I *think* you're saying that there is | |
one request per erlang worker, is that correct? | |
seancribbs # first, Erlang has a limited number of threads on which it can schedule I/O | |
yes, that is correct | |
but because everything is async, some may simply be in a select loop waiting on I/O | |
You're also limited by the number of cores on the machine, and the number of process | |
schedulers Erlang creates so, say you have the default of 5 async I/O threads | |
with 50 workers, that's 10 workers per thread (not to mention the system stuff) | |
which means 9 of them are waiting while the 10th uses the thread | |
(that's a gross overgeneralization, but I think it should start to give you a picture) | |
freshtonic # I always thought that Erlang's IO model was non-blocking as far as the OS | |
process was concerned - didn't realise it uses a threaded blocking model. I thought that was | |
how stuff like YAWS scaled so well (I realise that it's blocking as far as an Erlang process | |
is concerned) | |
seancribbs # so, it IS non-blocking. but I/O has to be scheduled through a limited thread pool | |
it's very similar to the evented I/O idea | |
freshtonic # yes, something which I am very familiar with (on nodejs) | |
seancribbs # which is why yaws can handle so many connections… but that doesn't give you a | |
picture of the latency anyway… the machine executing the test is going to be the limiting factor | |
at high numbers of workers | |
freshtonic # which is what my main concern was with innostore. I did a few early benchmarks | |
with 10 workers and still saw the high latencies and not much throughput. | |
seancribbs # innostore is really pretty bad on VMs | |
you might also try running the "null" driver to benchmark you key generator | |
there might be a limiting factor in there too | |
freshtonic # And the really odd thing was that I was seeing such high _read_ latencies (when | |
iostat was reporting nothing hitting the disk, assuming it was getting from the cache) | |
seancribbs # at least, you can get an upper bound for the test | |
freshtonic # right, I'll give the null driver a go | |
didn't realise innostore was bad on VMs. | |
good to know. | |
seancribbs # freshtonic: high latency which is a sign that either the network was saturated, | |
or the workers have too much IO contention | |
i think the latter was probably the case in your test | |
try it with say, 5 workers, then bump them up until you hit the limits | |
i think you'll be surprised | |
freshtonic # No, there's something more strange going on; I was seeing these latencies with | |
a single request using curl | |
freshtonic # Benchmarking just formalized it for me | |
seancribbs # also, even in the same region, EC2 nodes aren't guaranteed to be in the same | |
rack, or perhaps even in the same DC | |
freshtonic # That's not to say my benchmark isn't flawed; but I was seeing high latencies | |
even when a machine was not under load, both on a local VM and at EC2 | |
And only with innostore, not bitcask | |
seancribbs # freshtonic: did you have the buffer pool really high? | |
also, how much data was in it? | |
freshtonic # the innostore buffer pool was set to about 4GB on a 64 bit machine. | |
(it was much less on my local VM,) | |
seancribbs # how much RAM in the machine? (or VM) | |
freshtonic # 7.6GB at EC2. | |
seancribbs # hmm. not sure what to say there… i would guess a smaller pool | |
would prevent swapping but you say it wasn't even hitting disk | |
freshtonic # the Riak node was cleaned out before the benchmark; the only data | |
in it was created during the benchmark run .it's quite possible there's something broken | |
in out config, just don't know what. | |
seancribbs # ok. send your config along to the ML, I can help you more tomorrow, | |
and possibly pull in the more knowledgeable guys (the riak config, not the benchmark) | |
freshtonic # Hmm, I wonder if at some point the single node I was testing was part of | |
another test cluster (the rest of which no longer exists). Could Riak thinking there's a | |
netsplit account for the latency in any way? | |
seancribbs # check the node status, it'll tell you | |
freshtonic # I'll check, one sec | |
seancribbs # also, single node is not good to test against | |
won't give you an accurate picture, since it's likely doing 3x the normal amount of work | |
(n=3 by default of course) | |
freshtonic # ring_members : ['riak@127.0.0.1'] | |
so what happens when there's a single node with an n value of 3 and I write an object | |
to a bucket? | |
seancribbs # all 3 replicas get written to the same node | |
freshtonic # OK so that might account for some of what I am seeing. I'll re-do the | |
benchmarks again, this time with at least 3 nodes. | |
seancribbs # cool. | |
a cluster size < N is kind of a degenerate case most of the time | |
freshtonic # yes, we wouldn't be in production with that, I eventually wanted to | |
see how the results changed as we added nodes. 1 seemed like a good starting point. | |
In hindsight, maybe not such a good idea :) | |
seancribbs # yeah, try 3 and see what happens | |
freshtonic # thanks for your help, Sean. I'll delay posting to the list and | |
having anyone else from Basho involved until I have had a chance to re-run the benchmark. | |
seancribbs # also, make sure you're running the benchmark on a separate box (goes without saying) | |
freshtonic # sure, will do | |
seancribbs # good luck, let us know how it goes | |
freshtonic # definitely. I hoping it all boils down to a problem between keyboard and chair |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment