gistfile1.txt

12:49 <@seancribbs> copious: what are your operational stats like (throughput / latency)?
12:51 < copious> our general document throughput is about 7m a day
12:52 <@seancribbs> so, indexing average 81 docs/s, probably peak in the thousands?
12:52 <@seancribbs> if i'm probing too much, just tell me to back off ;)
12:52 < copious> no, its all good
12:52 <@seancribbs> just want to know if we can accommodate you, and at what scale
12:52 < copious> looking up some numbers right now
12:53 <@seancribbs> brb
12:55 < copious> so varnish+tokyocabinet gets hit around 1500 req/sec, with a cache hit on varnish of about 80%, but that 
                 can drop way down at some points
12:55 < copious> so tokyo gets hit about 300 req/sec on average, with spikes up into the 1500 req/sec
12:57 < copious> varnish reports that this is results in about 1.7 Gigabytes / second of data 
13:01 < copious> we index in batch, and those batches index in the 100 - 300 documents / seconds range.  This is mainly 
                 determined by document size, so slower rates are larger documents, faster rates are smaller documents.
13:11 < copious> and as of 5 min ago, our total document count was 2,535,491,406
13:12 <@seancribbs> wow
13:13 < copious> this is all internal data processing