Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
12:49 <@seancribbs> copious: what are your operational stats like (throughput / latency)?
12:51 < copious> our general document throughput is about 7m a day
12:52 <@seancribbs> so, indexing average 81 docs/s, probably peak in the thousands?
12:52 <@seancribbs> if i'm probing too much, just tell me to back off ;)
12:52 < copious> no, its all good
12:52 <@seancribbs> just want to know if we can accommodate you, and at what scale
12:52 < copious> looking up some numbers right now
12:53 <@seancribbs> brb
12:55 < copious> so varnish+tokyocabinet gets hit around 1500 req/sec, with a cache hit on varnish of about 80%, but that
can drop way down at some points
12:55 < copious> so tokyo gets hit about 300 req/sec on average, with spikes up into the 1500 req/sec
12:57 < copious> varnish reports that this is results in about 1.7 Gigabytes / second of data
13:01 < copious> we index in batch, and those batches index in the 100 - 300 documents / seconds range. This is mainly
determined by document size, so slower rates are larger documents, faster rates are smaller documents.
13:11 < copious> and as of 5 min ago, our total document count was 2,535,491,406
13:12 <@seancribbs> wow
13:13 < copious> this is all internal data processing
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.