- http://stackoverflow.com/questions/15359271/low-memory-consumption-with-mongodb
- http://www.kchodorow.com/blog/2012/05/10/thursday-5-diagnosing-high-readahead/
- http://serverfault.com/questions/408100/mongodb-and-datasets-that-dont-fit-in-ram-no-matter-how-hard-you-shove
- http://markmail.org/message/utuxcatjdgxwe72z
- http://docs.mongodb.org/manual/administration/production-notes/#readahead
- http://linux.die.net/man/8/blockdev
Can somebody tell me why Mongo doesn't consume more than 2GB of memory when there's 4GB available and dataSize is about 106GB currently with a bit under 50 million documents?
Even with queries that process A LOT of documents memory consumption doesn't spike. I've few other processes running on the same server but as I'm watching New Relic, memory consumption is never over 2GB.
Don't know if it matters in this case but the server is virtualised with KVM. We are using 64bit version so no 32bit limits.
cat /proc/meminfo
MemTotal: 3921472 kB
MemFree: 135920 kB
Buffers: 3232 kB
Cached: 3265184 kB
SwapCached: 3860 kB
Active: 1805384 kB
Inactive: 1697516 kB
Active(anon): 136168 kB
Inactive(anon): 101520 kB
Active(file): 1669216 kB
Inactive(file): 1595996 kB
Unevictable: 3780 kB
Mlocked: 0 kB
SwapTotal: 2104472 kB
SwapFree: 2099072 kB
Dirty: 6744 kB
Writeback: 0 kB
AnonPages: 235184 kB
Mapped: 1313220 kB
Shmem: 388 kB
Slab: 96972 kB
SReclaimable: 68432 kB
SUnreclaim: 28540 kB
KernelStack: 3296 kB
PageTables: 119980 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4065208 kB
Committed_AS: 1847116 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 19696 kB
VmallocChunk: 34359627680 kB
HardwareCorrupted: 0 kB
AnonHugePages: 135168 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 6144 kB
DirectMap2M: 4188160 kBfree -m
total used free shared buffers cached
Mem: 3829 3666 162 0 3 3148
-/+ buffers/cache: 515 3313
Swap: 2055 5 2049- answger 1:
If you run free -m or cat /proc/meminfo, you should be able to see the breakdown of memory usage.
Buffer relates to how much RAM is used to cache disk blocks. Cached is similar to buffers, except it's cached pages from reading files.
If these are high, it would indicate(表明) that your read-ahead settings are too large and MongoDB cannot use the remaining memory - degrading performance.
For random access use patterns set readahead values low, for example setting readahead to a small value such as 32 (16KB) often works well.
- answger 2:
This is intended as an addendum to the other answers posted here, which discus many of the relevant elements to be considered here. However, there is another, often overlooked, factor when it comes to efficient RAM utilization in a random access type system - readahead.
You can check the current settings for readahead (on Linux) by running blockdev --report (usually requires sudo/root privileges). This will print out a table with one row for each disk device. The RA column contains the value for readahead. That value is the number of 512 byte sectors (unless sector size is not the default - note that as of the time of writing this post, even disks that have larger sizes are treated as 512 byte sectors by the kernel) that are read on every disk access.
You can set the readahead setting for a given disk device by running:
blockdev --setra <value> <device name>When using a software based RAID system make sure to set the readahead on each disk device as well as on the device that corresponds to the RAID controller.
Why is this important? Well, readahead uses the same resource MongoDB is trying to use in order to optimize your reads for sequential access - RAM. When you are doing sequential reads on spinning disks (or devices that behave something like spinning disks anyway - EBS I am looking at you), fetching the nearby data into RAM can boost performance massively, save you on seeks, and a high readahead setting in the right environment can get you some impressive results.
For a system like MongoDB where your access is generally going to be random access across a data set this is just wasting memory that is better used elsewhere. The system, which as mentioned elsewhere manages memory for MongoDB also, is going to allocate a chunk of memory to readahead when it is requested and hence leave less RAM for MongoDB to effectively use.
Picking the correct readahead size is tricky and depends on your hardware, the configuration, block size, stripe size and the data itself. If you do move to SSDs for example, you will want a low setting, but how low will depend on the data.
Picking the correct readahead size is tricky and depends on your hardware, the configuration, block size, stripe size and the data itself. If you do move to SSDs for example, you will want a low setting, but how low will depend on the data.
To explain: you want to make sure that readahead is high enough to pull in a full single document and not have to go back to disk. Let's take your mentioned median size of 8k - since sectors on disk are generally 512 bytes it would take 16 disk accesses to read in whole document with no readahead. If you had a readahead of 16 sectors or more, you would read in the whole document with only one trip to disk.
Actually, since MongoDB index buckets are 8k, you will never want to set readahead below 16 anyway, or it will take 2 disk accesses to read in one index bucket. A general good practice is to start with your current setting, halve it, then re-assess your RAM utilization and IO and move on from there.
很有幫助