diejel/limit_resources.md

## limit_resources.md

      
    Raw
  

              limit_resources.md
            
          
    Linux Issue #001

Maybe you have trying to open a new terminal window, or connect via ssh to another terminal and appears a prompt like the message below:

STDOUT: "fork failed: Resource temporarily unavailable"

Where are saved those messages? :

Those messages can be found in logs folder, specifically at:

/var/log/secure
/var/log/messages

Example:

Here, there are some examples of what is expected you find and how it looks like.
Nov 24 12:59:14 localhost multipathd: fork failed: Resource temporarily unavailable 
Nov 24 12:59:14 localhost multipathd: fork failed: Resource temporarily unavailable 
Nov 24 12:59:14 localhost multipathd: fork failed: Resource temporarily unavailable 
Nov 24 13:00:18 localhost udevd[2244]: udev_event_run: fork of child failed: Resource temporarily unavailable
Nov 24 13:00:18 localhost udevd[2244]: udev_event_run: fork of child failed: Resource temporarily unavailable

Possible Reasons

Among the most common reason, we have:

Some application is opening large amount of processes.
Server is running out of memory
Number of process is reaching the kernel.pid_max value.

Troubleshooting

Here, this is oneliner I created for starting to find the possible reasons detailed above.
This piece of code will perform the following things:

Show the TOP10 process that are consuming memory .
Expose the amount of process are being run for each user.
Indicate the nTh a.k.a _NLWP_(Number of LightWeight Process) that is nothing different that the number of threads per PID.
Count the total number of process in the system in order to be compared against the kernel.pid_max value.

lns(){ printf '%*s\n' "${COLUMNS:-$(tput cols)}" '' | tr ' ' '-' ;};
top10m=$(ps -eo user,pid,ppid,cmd,%mem,%cpu,nlwp --sort=-%mem | head -n 10);count_proc=$(ps --no-headers auxm|awk '$2 == "-" {print $1}'|sort|uniq -c|sort -n);pids_list=($(egrep -oh "[0-9]{1,}"<<<$count_proc));sum_proc=0;for q in ${pids_list[@]};do ((sum_proc = sum_proc+$q));done;lns;echo -e "Most 10 demanding Memory resource:\n$top10m\n";lns;echo -e "\nProcesses by each user:\n$count_proc\n";lns;echo -e "Total processes in system: $sum_proc\nMaximum allowed kernel PID:\n$(sudo sysctl -a | grep -i 'pid_max')";lns


Demo

This is a demo how it will looks when you run on your PC.


From the results above, be aware of:

Total process number < kernel.pid_max value.
Any of the process above is not consuming > 90% of memory.
Identifying which PIDs number are having high NLWP values.

Additionally:

Verify your current set values by doing:
# ulimit -a
# sysctl -a (Will need superuser privileges).
For modifying sysctl value and be persistent, edit the /etc/sysctl.conf


Remember that:

Even being configured the soft/hard limits in /etc/security/limits.conf
Those values need to be lower than which is specified in kernel.pid_max


Sources:

RHEL Customer Portal


## tshoot_resources.sh
lns(){ printf '%*s\n' "${COLUMNS:-$(tput cols)}" '' | tr ' ' '-' ;};
top10m=$(ps -eo user,pid,ppid,cmd,%mem,%cpu,nlwp --sort=-%mem | head -n 10);
count_proc=$(ps --no-headers auxm|awk '$2 == "-" {print $1}'|sort|uniq -c|sort -n);
pids_list=($(egrep -oh "[0-9]{1,}"<<<$count_proc));
sum_proc=0;
for q in ${pids_list[@]};do ((sum_proc = sum_proc+$q));done;
lns;echo -e "Most 10 demanding Memory resource:\n$top10m\n";lns;
echo -e "\nProcesses by each user:\n$count_proc\n";lns;
echo -e "Total processes in system: $sum_proc\nMaximum allowed kernel PID:\n$(sudo sysctl -a | grep -i 'pid_max')";lns
	lns(){ printf '%*s\n' "${COLUMNS:-$(tput cols)}" '' \| tr ' ' '-' ;};
	top10m=$(ps -eo user,pid,ppid,cmd,%mem,%cpu,nlwp --sort=-%mem \| head -n 10);
	count_proc=$(ps --no-headers auxm\|awk '$2 == "-" {print $1}'\|sort\|uniq -c\|sort -n);
	pids_list=($(egrep -oh "[0-9]{1,}"<<<$count_proc));
	sum_proc=0;
	for q in ${pids_list[@]};do ((sum_proc = sum_proc+$q));done;
	lns;echo -e "Most 10 demanding Memory resource:\n$top10m\n";lns;
	echo -e "\nProcesses by each user:\n$count_proc\n";lns;
	echo -e "Total processes in system: $sum_proc\nMaximum allowed kernel PID:\n$(sudo sysctl -a \| grep -i 'pid_max')";lns