Skip to content

Instantly share code, notes, and snippets.

@PharkMillups
Created July 13, 2010 20:18
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save PharkMillups/474455 to your computer and use it in GitHub Desktop.
Save PharkMillups/474455 to your computer and use it in GitHub Desktop.
freshtonic # seancribbs: are you still around? I was wondering if you'd be able
to answer a couple of outstanding questions regarding benchmarking riak that
I posted to the list yesterday
drev1 # freshtonic: what questions did you have; I will do my best to answer them
freshtonic # hi drev1, one moment, I'll just grab them...
freshtonic # drev1: for context, it's probably a good idea to look at my email
on the riak-users list. [ --> http://bit.ly/aUauLA <-- ] My first question was about how many
keys I can store in memory with bitcask: given our large key size of 67 characters, combined with
the Bitcask's padding and storage layout, how many keys should we be able to manager
per node per GB?
drev1 # here is the keydir structure:
http://hg.basho.com/bitcask/src/tip/c_src/bitcask_nifs.c#cl-37 I think maybe 92 bytes per entry
freshtonic # thanks drev1, and my second question is does each worker
process (the concurrent setting in basho_bench) handle more than a single HTTP
request at a time, or is it 1 worker == 1 request ?
drev1 # freshtonic: a worker performs one request at a time
freshtonic # OK, so now seancribbs's response to my email confuses me even more.
He said with concurrent set to 50 and mode set to max, I would saturate the network,
when in reality it's there can be no more than 50 pending HTTP requests at a time
bingeldac # I need to ask him if he has seen that, but I haven't seen much improvement
above 15 concurrent workers if you only have one or two hosts you would pile up connections
with 4 nodes 15 seems like a good number in my experience
freshtonic # bingeldac: I find that confusing. How can connections pile up, if each worker
can only deal with one connection at a time? i.e. it makes a request and waits for a response
before making another request
seancribbs # freshtonic: you there?
freshtonic # hi Sean, yeah here
seancribbs # so, there's a number of limiting factors here which may give you
diminishing returns as you increase the number of workers
and where you don't see increased throughput as you increase the number of workers is
probably where you should stop and please note that these are not recommendations for how to
change things, just things you should be aware of
freshtonic # I understand. What I am not clear on is the relationship between the number
of workers and the number of requests in progress. I *think* you're saying that there is
one request per erlang worker, is that correct?
seancribbs # first, Erlang has a limited number of threads on which it can schedule I/O
yes, that is correct
but because everything is async, some may simply be in a select loop waiting on I/O
You're also limited by the number of cores on the machine, and the number of process
schedulers Erlang creates so, say you have the default of 5 async I/O threads
with 50 workers, that's 10 workers per thread (not to mention the system stuff)
which means 9 of them are waiting while the 10th uses the thread
(that's a gross overgeneralization, but I think it should start to give you a picture)
freshtonic # I always thought that Erlang's IO model was non-blocking as far as the OS
process was concerned - didn't realise it uses a threaded blocking model. I thought that was
how stuff like YAWS scaled so well (I realise that it's blocking as far as an Erlang process
is concerned)
seancribbs # so, it IS non-blocking. but I/O has to be scheduled through a limited thread pool
it's very similar to the evented I/O idea
freshtonic # yes, something which I am very familiar with (on nodejs)
seancribbs # which is why yaws can handle so many connections… but that doesn't give you a
picture of the latency anyway… the machine executing the test is going to be the limiting factor
at high numbers of workers
freshtonic # which is what my main concern was with innostore. I did a few early benchmarks
with 10 workers and still saw the high latencies and not much throughput.
seancribbs # innostore is really pretty bad on VMs
you might also try running the "null" driver to benchmark you key generator
there might be a limiting factor in there too
freshtonic # And the really odd thing was that I was seeing such high _read_ latencies (when
iostat was reporting nothing hitting the disk, assuming it was getting from the cache)
seancribbs # at least, you can get an upper bound for the test
freshtonic # right, I'll give the null driver a go
didn't realise innostore was bad on VMs.
good to know.
seancribbs # freshtonic: high latency which is a sign that either the network was saturated,
or the workers have too much IO contention
i think the latter was probably the case in your test
try it with say, 5 workers, then bump them up until you hit the limits
i think you'll be surprised
freshtonic # No, there's something more strange going on; I was seeing these latencies with
a single request using curl
freshtonic # Benchmarking just formalized it for me
seancribbs # also, even in the same region, EC2 nodes aren't guaranteed to be in the same
rack, or perhaps even in the same DC
freshtonic # That's not to say my benchmark isn't flawed; but I was seeing high latencies
even when a machine was not under load, both on a local VM and at EC2
And only with innostore, not bitcask
seancribbs # freshtonic: did you have the buffer pool really high?
also, how much data was in it?
freshtonic # the innostore buffer pool was set to about 4GB on a 64 bit machine.
(it was much less on my local VM,)
seancribbs # how much RAM in the machine? (or VM)
freshtonic # 7.6GB at EC2.
seancribbs # hmm. not sure what to say there… i would guess a smaller pool
would prevent swapping but you say it wasn't even hitting disk
freshtonic # the Riak node was cleaned out before the benchmark; the only data
in it was created during the benchmark run .it's quite possible there's something broken
in out config, just don't know what.
seancribbs # ok. send your config along to the ML, I can help you more tomorrow,
and possibly pull in the more knowledgeable guys (the riak config, not the benchmark)
freshtonic # Hmm, I wonder if at some point the single node I was testing was part of
another test cluster (the rest of which no longer exists). Could Riak thinking there's a
netsplit account for the latency in any way?
seancribbs # check the node status, it'll tell you
freshtonic # I'll check, one sec
seancribbs # also, single node is not good to test against
won't give you an accurate picture, since it's likely doing 3x the normal amount of work
(n=3 by default of course)
freshtonic # ring_members : ['riak@127.0.0.1']
so what happens when there's a single node with an n value of 3 and I write an object
to a bucket?
seancribbs # all 3 replicas get written to the same node
freshtonic # OK so that might account for some of what I am seeing. I'll re-do the
benchmarks again, this time with at least 3 nodes.
seancribbs # cool.
a cluster size < N is kind of a degenerate case most of the time
freshtonic # yes, we wouldn't be in production with that, I eventually wanted to
see how the results changed as we added nodes. 1 seemed like a good starting point.
In hindsight, maybe not such a good idea :)
seancribbs # yeah, try 3 and see what happens
freshtonic # thanks for your help, Sean. I'll delay posting to the list and
having anyone else from Basho involved until I have had a chance to re-run the benchmark.
seancribbs # also, make sure you're running the benchmark on a separate box (goes without saying)
freshtonic # sure, will do
seancribbs # good luck, let us know how it goes
freshtonic # definitely. I hoping it all boils down to a problem between keyboard and chair
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment