gistfile1.txt

13:28 <allen__> I loaded 500K * 4K to three nodes, and read it all ahead to s
see how much cached are used 

13:29 <allen__> and found out the usage of cache more than data size, is 
there any way to calculate this to plan memory accordingly? 

13:29 <allen__> s/cache more/cache is/ 

13:30 <dizzyd> what backend? 

13:30 <allen__> bitcask 

13:31 <dizzyd> most of the caching will occur at the O/S level 

13:31 <dizzyd> and is not something we have control over 

13:32 <allen__> thanks dizzyd, what if I use a different backend? 

13:32 <benblack> same thing 

13:33 <benblack> OS buffer cache for the filesystem just happens automatically 

13:33 <benblack> if the OS needs to use the memory for something else, it will 
release some of the buffer cache 

13:33 <allen__> ok, all riak backends use read-ahead cache? 

13:33 <benblack> the metric you want to watch is the RSIZE for the riak process 

13:34 <benblack> again, that is the OS doing it all on its own 

13:34 <benblack> not the riak backend 

13:38 dizzyd left 

13:38 <allen__> just curious, is there any data storage to use memory instead of 
read-ahead cache for Riak? 

13:40 <benblack> the ets/cache backend is only in memory 

13:41 <benblack> but different backends will have different in-memory footprints 
in different situations 

13:41 <benblack> what is the question you are trying to answer for yourself? 

13:43 <allen__> benblack: to try avoid disk io, to monitor memory usage is 
important, but with cache, it was hard to calculate how much memory is used 
by Riak. 

13:43 davidc_ joined 

13:44 <allen__> i.e. Riak used 80% of memory(w/ cache), better to add more 
nodes , like that 

13:44 <benblack> allen__: do you know what RSIZE is? 

13:45 <allen__> nope, do u have a reference? 

13:45 <benblack> man ps 

13:45 <benblack> RSS is how it is shown in ps, RSIZE in top 

13:46 old_sound joined 

13:46 siculars joined 

13:49 <allen__> benblack: i don't get it yet. is there any document for this? 
meanwhile, i will keep googling 

13:49 <benblack> resident size 

13:49 <benblack> of a process 

13:49 <benblack> "it was hard to calculate how much memory is used by Riak." 

13:49 <benblack> resident size. 

13:50 <benblack> allen__: you are familiar with man pages? 

13:53 <allen__> y , isn't is physical memory used by Riak
(that does not include cache usage), m i wrong? 

13:54 <benblack> i'm sorry, i can't parse that sentence 

13:54 acts_as_ joined 

13:54 <allen__> s/is/it, sorry 

13:55 <benblack> yes, resident memory is what is actually in RAM. 

13:55 <benblack> buffer cache is the OS doing its thing, it is not riak. 

13:56 old_sound joined 

13:57 <allen__> benblack: I wanted to do was to calculate Riak's memory usage
(including cache). it's not possible, right? 

13:57 <benblack> do you mean including OS buffer cache? 

13:58 <allen__> I don't cound os buffer that much, 'cached' part from 'free' command 

13:58 <allen__> s/cound/count, sorry again 

13:58 <benblack> you mean from the memory line in top? 

13:58 tilgovi joined 

13:58 <allen__> buffer used by kernel is not huge, so I can ignore it 

13:59 <benblack> do you know what OS buffer cache is? 

13:59 <allen__> total used free shared buffers cached 

13:59 <allen__> Mem: 3980440 2893044 1087396 0 5080 2090756 

13:59 <allen__> -/+ buffers/cache: 797208 3183232 

13:59 <allen__> Swap: 0 0 0 

13:59 <allen__> buffers -- cache used by kernel 

14:00 <allen__> cached -- cache used by applications 

14:00 cyberdelia joined 

14:00 <allen__> top also has 'cached line', i don't think it's different 

14:01 <benblack> allen__: the OS is going to do all that read-ahead and 
in-memory caching from filesystem activity on its own. the apps are generally 
oblivious to it. if you are reading a lot of data and there is a lot of RAM 
not used by processes, the buffer cache can be enormous. 

14:01 <benblack> if an app increases what it needs, the OS will release buffers 
from that cache to satisfy the request 

14:01 <benblack> if you want to know how much memory riak requires for you, that 
is resident size. 

14:04 <allen__> benblack: Riak only load keys into memory as you told me. To have 
fast data access working sets should be loaded into cache. if working sets are 
bigger than total memory size(used+cache) the performance goes down, m i right? 

14:04 <benblack> no, you are not right 

14:05 <benblack> the in-memory "cache" parts _in riak_ are counted as part of resident size 

14:05 <allen__> huh? 

14:05 <allen__> with bitcask? 

14:05 <benblack> with any backend 

14:06 <benblack> what the OS calls cache (and gets reported with tools like 
top and ps) is not what riak calls cache 

14:07 <allen__> I learned bitcask uses read-ahead cache, and it's not part 
of 'cached' part? 

14:07 <benblack> the in-memory "cache" parts _in riak_ are counted as part of 
resident size. 

14:07 <benblack> period. 

14:08 <benblack> there are a lot of the same terms used in different 
places, which can be confusing when you first encounter it 

14:09 <benblack> but what is reported by the OS for resident size of riak 
is really the memory it is consuming 

14:09 <benblack> if you want to know what memory riak is consuming, 
that is it. 

14:11 <allen__> benblack: bitcask read-ahead cache is part of resident 
size? u might be mistaken because I loaded 10M documents and never seen 
RES size goes more than 1.5GB. 

14:12 <benblack> yup. 

14:12 <benblack> the bitcask key index is in memory 

14:12 <benblack> in riak 

14:13 <benblack> the contents are in OS buffer cache 

14:13 <benblack> not riak 

14:14 <aphyr> When I store two versions of the same object in a bucket 
which are *not* descendents of one another, and allow-mult is off, what happens? 

14:14 <benblack> most likely last write wins. 

14:14 <aphyr> I ask because RObject#conflict? returns true, and when I 
turn on allow-mult I can pull out multiple siblings. 

14:14 <aphyr> So... are those siblings just chilling in the database 
forever, but Riak returns the most recent one for read requests? 

14:15 <benblack> Cold Chilling 

14:15 <benblack> i asked justin about this. and then i forgot. 

14:16 <aphyr> It seems like you'd wind up with a datastore full of dead, 
inaccessible objects. 

14:20 <allen__> benblack: "the in-memory "cache" parts _in riak_ are 
counted as part of resident size" <--> "the bitcask key index is in memory 
in riak, the bitcask key index is in memory not riak" 

14:20 <allen__> benblack: r u talking bitcask and riak seperately? 

14:21 senderista joined 

14:21 <benblack> no, i am explaining which parts are in riak 

14:22 <benblack> you misquoted me 

14:23 <benblack> what i said is: the bitcask key index is in memory, in riak. 
the contents are in OS buffer cache, not riak. 

14:23 <allen__> sorry if i did, I wanted to calculate memory size used by 
riak+bitcask then 

14:23 <benblack> again 

14:23 <allen__> is there a way to do it? 

14:23 <benblack> "the bitcask key index is in memory, in riak." 

14:24 <benblack> that is it 

14:24 <benblack> it is riak's resident size. 

14:24 <allen__> y i understand that. 

14:24 <benblack> great 

14:24 <benblack> the other stuff is _OS buffer cache_ 

14:24 <benblack> and it is totally dynamic 

14:24 <aphyr> allen__: the OS fronts the disk with its own readahead cache. 

14:24 <aphyr> That's what you see with the `free` command. 

14:24 <aphyr> That cache is totally elastic and will be reduced as 
applications use more RAM. 

14:25 <benblack> yes, exactly so. 

14:25 <allen__> aphyr: i understand that too 

14:25 <benblack> allen__: that means it will expand to fit the RAM available, 
if needed. 

14:26 <aphyr> I'm not exactly sure how to go about checking how much disk cache 
belongs to each process... because it's allocated by inodes, not by processes 

14:26 <benblack> right 

14:26 <aphyr> But it's probably safe to assume the OS is doing a good job of it 

14:26 <benblack> AND 

14:26 justinsheehy joined 

14:26 <benblack> will shrink and grow 

14:28 <allen__> on a physical machine, i only run riak nothing else. 
RES stays around 1GB, and buffers will use all available memory. 

14:28 <benblack> yes 

14:28 <benblack> that is normal 

14:29 <benblack> databases do that. 

14:29 <benblack> i have seen cassandra nodes with 4G resident heap and 
40G of buffer cache 

14:29 <allen__> I only run riak on that node. my question is how do I know 
I need more memory or more nodes? 

14:29 <benblack> by monitoring i/o activity and performance 

14:29 <Damm|lunch> ^^ proactively please 

14:30 <benblack> if you are constantly going to disk, you might benefit from more memory 

14:30 <benblack> indeed 

14:30 <Damm|lunch> hate to add that, but too many people are reactive 

14:30 <Damm|lunch> and reactive responses always suck 

14:30 <benblack> you are quite right 

14:30 <aphyr> allen__: you may find SAR and iostat useful. :) 

14:30 <allen__> y i uses sar and iostat too, but won't it be too late? 

14:31 <aphyr> When you notice a lot of time in IO_WAIT, you could probably 
benefit from additional RAM. 

14:32 <benblack> allen__: it generally doesn't happen all at once. that's why 
it's so important to keep monitoring it. 

14:33 <allen__> ok, thanks aphyr, benblack. I thought there was some method to 
calculate 'buffers' over head. 

14:33 <aphyr> Speaking of which, will a riak node use multiple cores by default? 

14:33 <benblack> even if you assume _all_ the buffer cache is for riak 

14:33 <benblack> you still don't know if you need more memory 

14:33 <benblack> aphyr: yes, BEAM (the erlang vm) is pretty spectacular on the 
multicore front 

14:33 <allen__> benblack: that's what I wanted to know 

14:34 <benblack> well, glad we got there in the end. 

14:34 <allen__> thans benblack 

14:34 <benblack> np 

14:36 <benblack> aphyr: 
http://erlware.blogspot.com/2010/03/erlang-factory-patrick-nyblom-erlang.html 

14:36 <benblack> mutilated from being taken off the slides, but worth reading 

14:38 <aphyr> benblack: awesome. 

14:39 <benblack> was my favorite presentation at EF 

14:39 <aphyr> What's the best way to simulate node failure? Just kill 
the beam process? 

14:41 <aphyr> Oh sweeeeet 

14:42 <aphyr> That recovery process was totally painless 

14:43 <benblack> yerp 

14:43 <benblack> Just Works