kswapd0.md

## kswapd0.md

      
          Raw
        
        
              kswapd0.md
            
          
    WARNING: I have tested the following mainly without any swap (partitions/files) enabled (eg. either no swap, or no swap support in kernel)
Explanations as to why kswapd0 does constant disk reading before OOM-killer kills the offending process:

see the answer and comment of https://askubuntu.com/a/432827/861003
see the answer and David Schwartz comments of https://unix.stackexchange.com/a/24646/306023

An example of constant disk reading due to out of memory, that I've encountered compiling firefox inside a qube(VM):

starts here: https://groups.google.com/d/msg/qubes-users/aSPefKH223U/PYc4m25SCQAJ

(note that this happens even with vm.swappiness=0(seen in a comment below) even though it was 60 in that example)
Wanting to recompile a custom kernel without kswapd0 (don't know how yetnot possible without kswapd0): https://unix.stackexchange.com/q/463233/306023

But it seems, it SEEMS, so far that just vm.swappiness=0 does it! (according to my below comments/screens)
Potential mitigating effects:

1.
sudo sysctl vm.overcommit_memory=2 #was 0
sudo sysctl vm.overcommit_ratio=50 #was 50 by default (if set to 200 it brings back the disk thrashing)

vm.watermark_scale_factor=1000 seems to cause a delay during which you could stop the disk thrashing before it begins to fully freeze the system.

sudo sysctl vm.overcommit_memory=0 #was 0
sudo sysctl vm.overcommit_ratio=50 #was 50
sudo sysctl vm.vfs_cache_pressure=0 #was 100
#sudo sysctl vm.watermark_scale_factor=1 #was 10
sudo sysctl vm.watermark_scale_factor=1000 #was 10
sudo sysctl vm.oom_kill_allocating_task=0 #was 0
Possibly irrelevant(in the above context only): vm.vfs_cache_pressure=0

(I have not looked at or checked) https://github.com/rfjakob/earlyoom (thanks to gudok on StackOverflow)
(I have not looked at or checked) https://github.com/hakavlad/nohang
(I have not looked at or checked) https://github.com/facebookincubator/oomd

Solutions so far:

IFF you're not using any swap(eg. kernel support for swap is disabled) then patching kernel so that Active(file) pages are not evicted works for me: https://stackoverflow.com/q/52067753/10239615

mirror1, mirror2
noobish patch by me though, so the side-effects are unknown. But, no more disk thrashing and no more freezing the OS!


## showallblocks
#!/bin/bash

#./showallblocks rev.01 rewritten for question/answer from: https://stackoverflow.com/q/52058914/10239615

if test "`id -u`" != "0"; then
	sudo='sudo'
else
	sudo=''
fi

dmesglog="$1"
if test -z "$dmesglog"; then
  echo "Usage: '$0' <dmesglogfile>"
  echo "Examples:"
  echo "sudo dmesg > dmesg1.log && '$0' dmesg1.log"
  echo "'$0' <(sudo dmesg)"
  #Note: '$0' used for the case when $0 has spaces or other things in its path names, and user wants to copy paste, for whatever reason, the output of the above into the command line.
  exit 1
fi

#(optional) Stop logging if already in progress:
$sudo sysctl -w vm.block_dump=0

#$sudo ./showblock $($sudo dmesg |grep --color=never -E -- 'READ block [0-9]+ on xvda3'|sed -re 's/.*READ block ([0-9]+).*$/\1/g' | tr '\n' ' ') | grep -B3 -- 'path : '
#$sudo ./showblock $($sudo dmesg | tail -n1000 |grep --color=never -E -- 'READ block [0-9]+ on xvda3'|sed -re 's/.*READ block ([0-9]+).*$/\1/g' | tr '\n' ' ') | grep -B3 -- 'path : '
#$sudo ./showblock $(cat ~/dmesg1.log |grep --color=never -E -- 'READ block [0-9]+ on xvda3'|sed -re 's/.*READ block ([0-9]+).*$/\1/g' | tr '\n' ' ') | grep -B3 -- 'path : '

#Using the answer from here(thanks to glenn jackman): https://unix.stackexchange.com/a/467377/306023
#grep --color=never -E -- 'READ block [0-9]+ on xvda3' "$dmesglog" |
#cat "$dmesglog" |
$sudo perl -pe '
if (! /READ block [0-9]+ on [A-Za-z0-9]+ .*$/) {
  s{.*}{}s
}

s{(READ block) (\d+) (on) ([A-Za-z0-9]+) ([^\$]*)\n$}
{join " ",$1, $2, $3, $4, $5, qx(./showblock "/dev/$4" "$2" | grep -F -- "Found path :" | cut -f4- -d" ")}es
' -- "$dmesglog"
#Note: the output of qx(...) above is purposely allowed to have trailing newline! (I did wonder if purposely is correct here or it should be purposefully, https://www.merriam-webster.com/words-at-play/purposely-purposefully-usage )
#To find out what "}es"(above) is, see perlre modifiers: https://perldoc.perl.org/perlre.html#Modifiers
#FIXME: noobish try to exclude lines not matching the lines that need to be replaced, from output! used 'if' above

#{join " ",$1, $2, $3, $4, $5, qx(echo "test")}es
#s{^.*$}
#{join " ", "1", "2"}e
#$s{(READ block) (\d+) (on )([A-Za-z0-9]+)(.*$)}
    #{join " ",$1, $2, $3, $4, $5, qx(./showblock "/dev/$4" "$2" | grep -F -- "Found path :" | cut -f3- -d" " | tr -d "\\n")}e
     #{join " ",$1, $2, $3, qx(./showblock $2 | grep "path :" | cut -f3- -d" ")}e
     #{join " ",$1, $2, $3, qx(echo -n "X")}e

## showblock
#!/bin/bash

#./showblock rev.03 rewritten for question/answer from: https://stackoverflow.com/q/52058914/10239615

#----
bytes_per_sector=512 #assumed that dmesg block numbers are 512 bytes each (ie. 512 bytes per sector; aka block size is 512)!
#----

#use `sudo` only when not already root
if test "`id -u`" != "0"; then
	sudo='sudo'
else
	sudo=''
fi

if ! test "$#" -ge "2"; then
  echo "Usage: '$0' <device> <dmesgblocknumber> [dmesgblocknumber ...]"
  echo "Examples:"
  echo "'$0' /dev/xvda3 5379184"
  echo "'$0' /dev/xvda3 5379184 5129952 7420192"
  #Note: '$0' used for the case when $0 has spaces or other things in its path names, and user wants to copy paste, for whatever reason, the output of the above into the command line.
  exit 1
fi

within_exit() {
  echo -e "\nSkipped current instruction within on_exit()"
}
on_exit() {
  #trap - EXIT SIGINT SIGQUIT SIGHUP  #will exit by skipping the rest of all instrunction from on_exit() eg. if C-c
  trap within_exit EXIT SIGINT SIGQUIT SIGHUP #skip only current instruction from on_exit() eg. when C-c is pressed
  #echo "first sleep"
  #sleep 10
  #echo "second sleep"
  #sleep 10
  if test "${#remaining_args[@]}" -gt 0; then
    echo -n "WARNING: There are '${#remaining_args[@]}' remaining args not processed, they are: " >&2
    for i in `seq 0 1 "$(( "${#remaining_args[@]}" - 1 ))"`; do  #seq is part of coreutils package
      echo -n "'${remaining_args[${i}]}' " >&2
    done
    echo >&2
  fi
}

trap on_exit EXIT SIGINT SIGQUIT SIGHUP

dev="$1"
shift 1

if test -z "$dev" -o ! -b "$dev"; then
  echo "Bad device name or not a device: '$dev'" >&2
  exit 1
fi

blocksize="`$sudo blockdev --getbsz "$dev"`"
if test "${blocksize:-0}" -le "0"; then #handles empty arg too
  echo "Failed getting block size for '$dev', got '$blocksize'" >&2
  exit 1
fi
#TODO: check and fail if not a multiplier
divider="$(( $blocksize / $bytes_per_sector ))"
if ! test "${divider:-0}" -gt "0"; then
  echo "Failed computing divider from: '$blocksize' / '$bytes_per_sector', got '$divider'" >&2
  exit 1
fi

# for each passed-in dmesg block number do
while test "$#" -gt "0"; do
dmesgblock="$1"
shift
remaining_args=("$@") #for on_exit() above
echo '--------'
echo "Passed-in dmesg block($bytes_per_sector bytes per block) number: '$dmesgblock'"
#have to handle the case when $dmesgblock is empty and when it's negative eg. "-1" so using a default value(of 0) when unset in the below 'if' block will help not break the 'test' expecting an integer while also allowing negative numbers ("0$dmesgblock" would otherwise yield "0-1" a non-integer):
if test "${dmesgblock:-0}" -le "0"; then
  echo "Bad passed-in dmesg block number: '$dmesgblock'" >&2
  exit 1
fi

#TODO: check and fail(or warn? nah, it should be fail!) if not a multiplier (eg. use modullo? is it "%" ?)
block=$(( $dmesgblock / 8 ))
if ! test "${block:--1}" -ge "0"; then
  echo "Badly computed device block number: '$block'" >&2
  exit 1
fi

echo "Actual block number(of $blocksize bytes per block): $block"
inode="$(echo "open ${dev}"$'\n'"icheck ${block}"$'\n'"close" | $sudo debugfs -f - 2>/dev/null | tail -n2|head -1|cut -f2 -d$'\t')"
if test "<block not found>" == "$inode"; then
  echo "No inode was found for the provided dmesg block number '$dmesgblock' which mapped to dev block number '$block'" >&2
  exit 1
else
	#assuming number TODO: check for this!
	echo "Found inode: $inode"
	fpath="$(echo "open ${dev}"$'\n'"ncheck ${inode}"$'\n'"close" | $sudo debugfs -f - 2>/dev/null | tail -n2|head -1|cut -f2- -d$'\t')"
  #fpath always begins with '/' right?
	if test "$fpath" != "Pathname"; then
		echo "Found path : $fpath"
	else
		echo "not found"
	fi
fi
done