This entry will hopefully show you how to troubleshoot memory used by Linux and application.
Display amount of free and used memory in the system (-m = in megas)
$ free -m
total used free shared buffers cached
Mem: 993 249 744 0 15 108
-/+ buffers/cache: 124 869
Swap: 0 0 0
Taken from...
$ cat /proc/meminfo
MemTotal: 1017816 kB
MemFree: 760000 kB
Buffers: 17060 kB
Cached: 112728 kB
SwapCached: 0 kB
Active: 125712 kB
Inactive: 93112 kB
Active(anon): 89136 kB
Inactive(anon): 584 kB
Active(file): 36576 kB
Inactive(file): 92528 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 89032 kB
Mapped: 10820 kB
Shmem: 688 kB
Slab: 23552 kB
SReclaimable: 15548 kB
SUnreclaim: 8004 kB
KernelStack: 704 kB
PageTables: 2696 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 508908 kB
Committed_AS: 173068 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 8488 kB
VmallocChunk: 34359724028 kB
HardwareCorrupted: 0 kB
AnonHugePages: 2048 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 32704 kB
DirectMap2M: 1015808 kB
- MemTotal — Total amount of physical RAM, in kilobytes.
- MemFree — The amount of physical RAM, in kilobytes, left unused by the system.
- Buffers — The amount of physical RAM, in kilobytes, used for file buffers.
- Cached — The amount of physical RAM, in kilobytes, used as cache memory.
- SwapCached — The amount of swap, in kilobytes, used as cache memory.
- Active — The total amount of buffer or page cache memory, in kilobytes, that is in active use. This is memory that has been recently used and is usually not reclaimed for other purposes.
- Inactive — The total amount of buffer or page cache memory, in kilobytes, that are free and available. This is memory that has not been recently used and can be reclaimed for other purposes.
- HighTotal and HighFree — The total and free amount of memory, in kilobytes, that is not directly mapped into kernel space. The HighTotal value can vary based on the type of kernel used.
- LowTotal and LowFree — The total and free amount of memory, in kilobytes, that is directly mapped into kernel space. The LowTotal value can vary based on the type of kernel used.
- SwapTotal — The total amount of swap available, in kilobytes.
- SwapFree — The total amount of swap free, in kilobytes.
- Dirty — The total amount of memory, in kilobytes, waiting to be written back to the disk. Writeback — The total amount of memory, in kilobytes, actively being written back to the disk.
- Mapped — The total amount of memory, in kilobytes, which have been used to map devices, files, or libraries using the mmap command.
- Slab — The total amount of memory, in kilobytes, used by the kernel to cache data structures for its own use.
- Committed_AS — The total amount of memory, in kilobytes, estimated to complete the workload. This value represents the worst case scenario value, and also includes swap memory. PageTables — The total amount of memory, in kilobytes, dedicated to the lowest page table level.
- VMallocTotal — The total amount of memory, in kilobytes, of total allocated virtual address space.
- VMallocUsed — The total amount of memory, in kilobytes, of used virtual address space.
- VMallocChunk — The largest contiguous block of memory, in kilobytes, of available virtual address space.
- HugePages_Total — The total number of hugepages for the system. The number is derived by dividing Hugepagesize by the megabytes set aside for hugepages specified in /proc/sys/vm/hugetlb_pool. This statistic only appears on the x86, Itanium, and AMD64 architectures.
- HugePages_Free — The total number of hugepages available for the system. This statistic only appears on the x86, Itanium, and AMD64 architectures.
- Hugepagesize — The size for each hugepages unit in kilobytes. By default, the value is 4096 KB on uniprocessor kernels for 32 bit architectures. For SMP, hugemem kernels, and AMD64, the default is 2048 KB. For Itanium architectures, the default is 262144 KB. This statistic only appears on the x86, Itanium, and AMD64 architectures.
Flushing the cache
$ echo 1 > /proc/sys/vm/drop_caches
Write cache to disk
$ sync
$ htop
- RSS : Stands for Resident Set Size and displays the amount of mem ory used by the process.
- SHR : Stands for Shared memory and displays the memory shared with other processes.
In order to find the exact memory usage you can you use this tool:
$ git clone https://github.com/pixelb/ps_mem.git
$ cd ps_mem
$ sudo ./ps_mem.py
Private + Shared = RAM used Program
34.6 MiB + 1.0 MiB = 35.7 MiB gnome-terminal
139.8 MiB + 2.3 MiB = 142.1 MiB firefox
291.8 MiB + 2.5 MiB = 294.3 MiB gnome-shell
272.2 MiB + 43.9 MiB = 316.1 MiB chrome (12)
913.9 MiB + 3.2 MiB = 917.1 MiB thunderbird
---------------------------------
1.9 GiB
=================================
The Linux kernel moves out pages which are not active or being used at the moment to swap space on the disk. This process is known as swappiness. Since swap space is on the hard drive fetching data will be slower as compared to your ram, This may cause your application to take a hit in terms of speed. You have the option to turn off swaping by changing t he value in /proc/sys/vm/swappiness to 0. The value ranges from 0 to 100 whe re 100 means aggressive swapping.
$ cat /proc/sys/vm/swappiness
60
Try 'vmstat -n 1' and check the 'si', 'so' field. If 'si' and 'so' (stands for swapin and swapout) fields are always 0, then the system is currently not swapping. Some application, has used the swap but somehow its not cleaned the swap space.
$ vmstat -n 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 759280 17080 113060 0 0 121 7 29 86 1 1 99 0 0
0 0 0 759280 17080 113092 0 0 0 0 19 29 0 0 100 0 0
0 0 0 759280 17080 113092 0 0 0 0 19 25 0 0 100 0 0
0 0 0 759280 17080 113092 0 0 0 0 20 33 0 0 100 0 0
$ sudo slabtop
Active / Total Objects (% used) : 93687 / 94741 (98.9%)
Active / Total Slabs (% used) : 3988 / 3988 (100.0%)
Active / Total Caches (% used) : 65 / 103 (63.1%)
Active / Total Size (% used) : 22980.43K / 23436.93K (98.1%)
Minimum / Average / Maximum Object : 0.01K / 0.25K / 8.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
25557 25557 100% 0.19K 1217 21 4868K dentry
13932 13932 100% 0.11K 387 36 1548K sysfs_dir_cache
7336 7336 100% 0.57K 524 14 4192K inode_cache
5184 4924 94% 0.06K 81 64 324K kmalloc-64
4446 4446 100% 0.10K 114 39 456K buffer_head
4216 4216 100% 0.96K 527 8 4216K ext4_inode_cache
3969 3557 89% 0.19K 189 21 756K kmalloc-192
3366 3366 100% 0.04K 33 102 132K ext4_extent_status
$ top -n 1 -o RES
top - 11:31:14 up 40 min, 1 user, load average: 0.00, 0.01, 0.05
Tasks: 75 total, 1 running, 74 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 0.3 sy, 0.0 ni, 99.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 1017816 total, 258940 used, 758876 free, 17232 buffers
KiB Swap: 0 total, 0 used, 0 free. 113360 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1203 root 20 0 112668 34544 1104 S 0.0 3.4 0:00.00 ruby
1111 root 20 0 182496 34428 2644 S 0.0 3.4 0:00.14 puppet
1219 root 20 0 265568 11808 2736 S 0.0 1.2 0:00.55 fail2ban-server
1740 root 20 0 107696 4220 3236 S 0.0 0.4 0:00.01 sshd
1823 vagrant 20 0 21560 4040 1764 S 0.0 0.4 0:00.05 bash
1545 root 20 0 61364 3056 2380 S 0.0 0.3 0:00.00 sshd
1 root 20 0 33616 2952 1492 S 0.0 0.3 0:01.01 init
595 root 20 0 10220 2432 120 S 0.0 0.2 0:00.00 dhclient
1822 vagrant 20 0 107696 2144 1156 S 0.0 0.2 0:00.15 sshd
2017 ntp 20 0 33512 2096 1488 S 0.0 0.2 0:00.18 ntpd
906 root 20 0 43448 1756 1420 S 0.0 0.2 0:00.00 systemd-logind
418 root 20 0 49880 1652 1008 S 0.0 0.2 0:00.03 systemd-udevd
2211 vagrant 20 0 23536 1480 1072 R 0.0 0.1 0:00.00 top
931 syslog 20 0 255840 1376 692 S 0.0 0.1 0:00.02 rsyslogd
744 statd 20 0 21540 1372 916 S 0.0 0.1 0:00.00 rpc.statd
709 root 20 0 23416 1108 800 S 0.0 0.1 0:00.01 rpcbind
1170 root 20 0 232796 1092 756 S 0.0 0.1 0:00.72 VBoxService
1057 root 20 0 23652 1040 792 S 0.0 0.1 0:00.01 cron
828 message+ 20 0 39212 1012 676 S 0.0 0.1 0:00.12 dbus-daemon
1005 root 20 0 14540 960 800 S 0.0 0.1 0:00.00 getty
1010 root 20 0 14540 960 800 S 0.0 0.1 0:00.00 getty
1009 root 20 0 14540 956 800 S 0.0 0.1 0:00.00 getty
1012 root 20 0 14540 956 800 S 0.0 0.1 0:00.00 getty
1264 root 20 0 14540 956 800 S 0.0 0.1 0:00.00 getty
1002 root 20 0 14540 952 800 S 0.0 0.1 0:00.00 getty
403 root 20 0 19472 912 612 S 0.0 0.1 0:00.09 upstart-udev-br