Last active
December 15, 2015 03:08
-
-
Save VolMi/5191787 to your computer and use it in GitHub Desktop.
cache the files *you* consider important in RAM
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| # | |
| # cachefill -- warm up the file cache and buffer of the Linux kernel | |
| # | |
| # ***** | |
| # ONLY IF YOU HAVE A LOT OF MEMORY! | |
| # I think, something like at least 8 GB for a regular Desktop | |
| # ***** | |
| # | |
| # | |
| # A simple but powerful attempt to letting the kernel cache the binaries (or any files) | |
| # I consider most important for having a great (speed / occupiedRAM)-ratio. | |
| # This should be run once at login time or so. | |
| # | |
| # If you use it, skim over the code a bit and change it to your needs. | |
| # There's a large table of folders to put in cache. Insert yours there, | |
| # delete (or uncomment) mine, and also change the max size of files | |
| # which you allow to read in the cache. | |
| # | |
| # | |
| # A mechanical HDD will be tortured witch plenty of file seeks, | |
| # which is actually the motivation to do that once *here*, and not have to | |
| # wait for the damn PC when we interact with it. | |
| # | |
| # The technique is known as "the poor man's way to cache data in RAM" or so. :-) | |
| # | |
| # It simply does this: | |
| # | |
| # $ cat /path/to/file/we/want/to/cache &> /dev/null | |
| # | |
| # The real caching is done by the Linux kernel, of course. | |
| # By reading a file we tell the kernel, that the file is of importance | |
| # and it is therefore cached in memory. | |
| # | |
| # For building a list of the files to cache, I ended up using find, due to its | |
| # many mighty options (recurse depth, min/max file size, delimiter, ...) | |
| # | |
| # | |
| # To cache all file's inodes in RAM (the kernel's "buffers"), we do this: | |
| # ls -lAR /!(proc|lost*) &> /dev/null | |
| # | |
| # | |
| # The script pauses if the PC runs on battery. | |
| # | |
| # | |
| # === Related === | |
| # | |
| # * Put these in your /etc/sysctl.conf: | |
| # vm.swappiness=10 # only swap in urgent circumstances | |
| # vm.vfs_cache_pressure=50 # prefer to keep caches in RAM longer (default: 100) | |
| # | |
| # | |
| # Michael Völker, 2013 | |
| # mi.voelker __at__ gmail {dot] kom, you know... | |
| # ====================================================================== | |
| # Set options here. | |
| # | |
| # Leave at least that many MiB RAM free, so there is enough room | |
| # for the kernel to make use of the memory for whatever it needs it. | |
| # | |
| minFreeMB=512 # recommendation: 512 or so | |
| # We create a (really long) list of files-to-read-into-cache and save it in /tmp | |
| # to be able to reuse it. | |
| # Files come and go, so we should renew this list from time to time. | |
| # This is the age in hours which we allow for that list: | |
| # | |
| Nhours=12 # recommendation: something like 1...48 | |
| # Here we define RegEx-patterns for files to ignore completely. | |
| # If you use more patterns than the fixed number specified here, | |
| # you have to change the code, below! | |
| # An empty ignore pattern would match *all* strings, so insert some | |
| # rubbish to match *no* string. | |
| # | |
| ignorePattern001='/[Tt][Rr][Aa][Ss][Hh]/' # files in a folder like /Trash/ or /trash/ | |
| ignorePattern002='/[Cc][Aa][Cc][Hh][Ee]/' # files in a folder like /Cache/ | |
| ignorePattern003='\.backup' | |
| ignorePattern004='_____nothing______' | |
| ignorePattern005='_____nothing______' | |
| ignorePattern006='_____nothing______' | |
| ignorePattern007='_____nothing______' | |
| ignorePattern008='_____nothing______' | |
| ignorePattern009='_____nothing______' | |
| ignorePattern010='_____nothing______' | |
| # Max size(s) of files to cache. help: | |
| # We use an array to specify different sizes (in 2^n kiB). 2^ 0 kiB = 1 kiB | |
| # It is usually all the damn small files for which caching makes the 2^10 kiB = 1 MiB | |
| # most sense, because HDD-reads are extraordinary slow for them, 2^20 kiB = 1 GiB | |
| # but they occupy only litle RAM, of course. | |
| for i in {0..20} | |
| do | |
| maxFileSizekB[$i]="-size -"$((2**$i))"k" # query the entries using ${maxFileSizekB[$i]} | |
| done | |
| # | |
| # ====================================================================== | |
| # root shouldn't do this. It's for interactive responsiveness | |
| if [ "$(whoami)" = "root" ] | |
| then | |
| echo 'Do not run as root!' >&2 | |
| exit 1 | |
| fi | |
| renice -n 19 -p $$ &> /dev/null # minimal CPU priority | |
| ionice -c 3 -p $$ &> /dev/null # minimal I/O priority | |
| read ScriptName < /proc/$$/comm # Yeah, that's me! | |
| # Make sure all child processes are nuked if we hit Ctrl+C | |
| # or use kill manually. | |
| trap 'pkill -TERM -P $$; echo " Your God hates you, bastard."; exit 2' SIGINT SIGTERM | |
| tmpDir="/tmp/${ScriptName}" | |
| listOfFiles="${tmpDir}/fileList.txt" | |
| createNewList="y" # make new list of files-to-cache except for special conditions | |
| mkdir -p "${tmpDir}" | |
| # /tmp may be in a precious RAM disk and the file list is easily compressible by >90%. | |
| # So compress it fast with an available compressor: | |
| if which xz > /dev/null # xz -1 is remarkably fast while | |
| then # yielding impressive compression | |
| compressor="xz -1" | |
| uncompressor="xz -cd" | |
| listOfFiles="${listOfFiles}.xz" | |
| elif which lzma > /dev/null # quite identical to xz | |
| then | |
| compressor="lzma -1" | |
| uncompressor="lzma -cd" | |
| listOfFiles="${listOfFiles}.lzma" | |
| elif which gzip > /dev/null # good old reliable gzip... | |
| then | |
| compressor="gzip -1" | |
| uncompressor="gzip -cd" | |
| listOfFiles="${listOfFiles}.gz" | |
| elif which lzop > /dev/null # blazingly fast, | |
| then # but "only" >80% compression | |
| compressor="lzop -1" | |
| uncompressor="lzop -cd" | |
| listOfFiles="${listOfFiles}.lzo" | |
| elif which bzip2 > /dev/null # slower than xz | |
| then # and worse compression | |
| compressor="bzip2 -1" | |
| uncompressor="bzip2 -cd" | |
| listOfFiles="${listOfFiles}.bz2" | |
| else # no compression | |
| compressor="cat" | |
| uncompressor="cat" | |
| fi | |
| # | |
| # Two helper functions, so we don't suck on battery needlessly | |
| # | |
| function isOnBattery () { | |
| [ -d /proc/acpi/battery ] && grep -Fq 'off-line' /proc/acpi/ac_adapter/*/state | |
| return $? | |
| } | |
| function delayIfOnBattery () { | |
| BeenHere="" | |
| while isOnBattery | |
| do | |
| [ "$BeenHere" ] || echo 'On Battery. Waiting for AC power...' | |
| sleep 5 | |
| BeenHere=yes | |
| done | |
| [ "$BeenHere" ] && echo 'AC plugged in now. Continue...' | |
| } | |
| # Decide if we reuse an existing list-of-files-to-crawl | |
| # | |
| if [ -s "${listOfFiles}" ] # if file exists and has a size greater than zero | |
| then | |
| # modification time of the file list (in seconds since 1970-01-01 00:00:00 UTC): | |
| modifyTime=$(ls -og --time-style=+%s "${listOfFiles}" | awk '{print $4}') | |
| now="$(date +%s)" | |
| if [ "$(( ($now - $modifyTime)/3600 ))" -lt "$Nhours" ] | |
| then | |
| createNewList="" # no new list required | |
| fi | |
| fi | |
| # ============================================================================================================================================ | |
| # Specify the files we want to cache, here. | |
| # | |
| # I use this pretty much hard-coded table style, because all the directories | |
| # often require special thinking about how large the files are allowed to be, or | |
| # what a reasonable recursion depth might be (use the RAM economically!), | |
| # so this is very hardly loopable. | |
| # | |
| # Put in those files, your frequently used (> 1/week) programs need to work. | |
| # Typical examples: Browser, mail client, IDE, photo editor... | |
| # To find out which files a running program needs (and their disk size), you can do | |
| # | |
| # $ lsof | grep -i ^MyProgram | awk '/.* REG .*\/.*$/ {print $NF}' | sort -u | xargs du -h "" 2>/dev/null | less | |
| # | |
| # | |
| # For actual file sizes (in Bytes): | |
| # | |
| # $ lsof | grep -i ^MyProgram | awk 'BEGIN {ORS="\0"}; /.* REG .*\/.*$/ {print $NF}' | sort -uz | xargs -0 stat -c '%s %n' | column -t | less | |
| # | |
| # Non-existent files or folders do not matter. Erros are piped to /dev/null | |
| # and *reading* non-existing things can never (?) be dangerous. | |
| # | |
| # The -P option tells find, not to follow symlinks | |
| # (for fun: look at /usr/bin/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11/X11) | |
| # | |
| # Appending "/" to a dir name assures we enter the dir, even if the dir itself | |
| # is a symlink. _____________________ | |
| # | | | |
| if [ "$createNewList" ] # | only files smaller | | |
| then # | than | | |
| delayIfOnBattery # | 2^n kiB | | |
| commonOpt="-type f -print0" # \_____________________/ | |
| echo -n "Building a file list... " # || | |
| cat < <( # \/ | |
| find -P /bin/ -maxdepth 5 ${maxFileSizekB[12]} $commonOpt 2> /dev/null # Essential command binaries | |
| find -P /etc/ -maxdepth 5 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # system-wide config files | |
| find -P /lib/ -maxdepth 1 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # essential libraries | |
| # find -P /lib/i386* -maxdepth 5 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # 32 Bit libraries | |
| find -P /lib/x86_64* -maxdepth 7 ${maxFileSizekB[13]} $commonOpt 2> /dev/null # 64 Bit libraries | |
| find -P /lib64/ -maxdepth 1 ${maxFileSizekB[13]} $commonOpt 2> /dev/null # 64 Bit libraries | |
| find -P /opt/ -name '*.so' -maxdepth 5 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # libraries of optional application software | |
| find -P /usr/bin/ -maxdepth 2 ${maxFileSizekB[13]} $commonOpt 2> /dev/null # non-essential command binaries | |
| find -P /usr/lib/ -maxdepth 1 ${maxFileSizekB[13]} $commonOpt 2> /dev/null # libs for binaries in /usr/(s)bin/ | |
| find -P /usr/lib/apt/ -maxdepth 4 ${maxFileSizekB[12]} $commonOpt 2> /dev/null # apt package manager libs (Debian) | |
| find -P /usr/lib/at-spi* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # [A]ssistive [T]echnology [S]ervice [P]rovider [I]nterface | |
| find -P /usr/lib/atlas-base/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # [A]utomatically [T]uned [L]inear [A]lgebra [S]oftware (BLAS/LAPACK) | |
| find -P /usr/lib/cinnamon* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # Cinnamon desktop | |
| find -P /usr/lib/codecs -maxdepth 2 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # codecs | |
| find -P /usr/lib/coreutil* -maxdepth 4 ${maxFileSizekB[11]} $commonOpt 2> /dev/null # coreutils | |
| find -P /usr/lib/darktable/ -maxdepth 4 ${maxFileSizekB[11]} $commonOpt 2> /dev/null # darktable raw photo developer | |
| find -P /usr/lib/eclipse/ -maxdepth 3 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # Eclipse IDE | |
| find -P /usr/lib/enchant/ -maxdepth 3 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # generic spell checking library | |
| find -P /usr/lib/evince/ -maxdepth 3 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # evince document viewer for Gnome | |
| find -P /usr/lib/evolution* -maxdepth 4 ${maxFileSizekB[12]} $commonOpt 2> /dev/null # evolution mail&more (Gnome) | |
| find -P /usr/lib/file* -maxdepth 4 ${maxFileSizekB[11]} $commonOpt 2> /dev/null # file-roller GUI (de-)compressor | |
| find -P /usr/lib/firef* -maxdepth 4 ${maxFileSizekB[13]} $commonOpt 2> /dev/null # firefox browser | |
| find -P /usr/lib/geany* -maxdepth 4 ${maxFileSizekB[12]} $commonOpt 2> /dev/null # geany, lightweight IDE | |
| find -P /usr/lib/gedit* -maxdepth 4 ${maxFileSizekB[12]} $commonOpt 2> /dev/null # gedit, text editor | |
| find -P /usr/lib/gimp* -maxdepth 4 ${maxFileSizekB[12]} $commonOpt 2> /dev/null # gimp, photo god | |
| find -P /usr/lib/girepository* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # GObject Introspection | |
| find -P /usr/lib/gjs* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # Mozilla-based javascript bindings for the GNOME platform | |
| find -P /usr/lib/gnome* -maxdepth 3 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # GNOME desktop stuff | |
| find -P /usr/lib/gthumb/ -maxdepth 3 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # gthumb, photo viewer for gnome | |
| find -P /usr/lib/gvfs/ -maxdepth 3 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # gnome virtual file system | |
| # find -P /usr/lib/i386* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # 32 Bit libs | |
| find -P /usr/lib/iceweas* -maxdepth 4 ${maxFileSizekB[13]} $commonOpt 2> /dev/null # iceweasel (Firefox in Debian) | |
| find -P /usr/lib/jvm/ -maxdepth 8 ${maxFileSizekB[12]} $commonOpt 2> /dev/null # Java | |
| find -P /usr/lib/kde* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # KDE desktop | |
| find -P /usr/lib/linuxmint/ -maxdepth 4 ${maxFileSizekB[12]} $commonOpt 2> /dev/null # Linux Mint | |
| find -P /usr/lib/locale/ -maxdepth 7 ${maxFileSizekB[12]} $commonOpt 2> /dev/null # language stuff | |
| find -P /usr/lib/meld/ -maxdepth 4 ${maxFileSizekB[12]} $commonOpt 2> /dev/null # meld, diff viewer | |
| find -P /usr/lib/mozilla* -maxdepth 4 ${maxFileSizekB[12]} $commonOpt 2> /dev/null # Mozilla | |
| find -P /usr/lib/muffin/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # window manager in Cinnamon, fork of Mutter (which is based on Metacity) | |
| find -P /usr/lib/nemo* -maxdepth 4 ${maxFileSizekB[11]} $commonOpt 2> /dev/null # Nemo file manager (Nautilus fork) | |
| find -P /usr/lib/pidgin/ -maxdepth 4 ${maxFileSizekB[11]} $commonOpt 2> /dev/null # Pidgin IM | |
| find -P /usr/lib/policykit* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # Policykit provides an organized way for non-privileged processes to communicate with privileged ones. | |
| find -P /usr/lib/purple* -maxdepth 4 ${maxFileSizekB[11]} $commonOpt 2> /dev/null # multi-protocol instant messaging library | |
| find -P /usr/lib/pyshared/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # Python shared libraries | |
| find -P /usr/lib/python* -maxdepth 4 ${maxFileSizekB[11]} $commonOpt 2> /dev/null # Python | |
| find -P /usr/lib/ssh/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # SSH | |
| find -P /usr/lib/ure/ -maxdepth 3 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # Uno Runtime Environment (for libreoffice) | |
| find -P /usr/lib/virtualb* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # VirtualBox | |
| find -P /usr/lib/vlc/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # VLC | |
| # find -P /usr/lib/wuala/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # Wuala Client | |
| find -P /usr/lib/x86_64* -maxdepth 7 ${maxFileSizekB[13]} $commonOpt 2> /dev/null # 64 Bit libs | |
| find -P /usr/lib/xul* -maxdepth 4 ${maxFileSizekB[13]} $commonOpt 2> /dev/null # xulrunner, for iceweasel/firefox | |
| find -P /usr/share/app* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # app-install/application-registry/applications/applnk/apport/apps | |
| find -P /usr/share/calendar/${LANG//.*/} -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # localized calendar files from bsdmainutils | |
| find -P /usr/share/cinnamon* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # Cinnamon desktop | |
| find -P /usr/share/command-not-found/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # ? | |
| find -P /usr/share/cups/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # [C]ommon [U]nix [P]rinting [S]ystem | |
| find -P /usr/share/darktable/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # photography workflow application and raw developer | |
| find -P /usr/share/deb* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # Debian things | |
| find -P /usr/share/desktop* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # | |
| find -P /usr/share/evolution* -maxdepth 4 ${maxFileSizekB[11]} $commonOpt 2> /dev/null # GNOME's e-mail, address book, calendar, task list and note-taking suite | |
| find -P /usr/share/file* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # file-roller is a GTK archive manager | |
| find -P /usr/share/firefox* -maxdepth 4 ${maxFileSizekB[12]} $commonOpt 2> /dev/null # Mozilla's Firefox Browser | |
| find -P /usr/share/font* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # whatever is needed to show nice fonts | |
| find -P /usr/share/GConf* -maxdepth 6 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # GNOME configuration database system | |
| find -P /usr/share/geany* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # Geany -- fast and lightweight IDE | |
| find -P /usr/share/gimp/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # gimp photo editor | |
| find -P /usr/share/gir* -maxdepth 4 ${maxFileSizekB[ 9]} $commonOpt 2> /dev/null # GObject Introspection | |
| find -P /usr/share/glib* -maxdepth 4 ${maxFileSizekB[11]} $commonOpt 2> /dev/null # GIO -- GNOME software library designed to present programmers with a modern and usable interface to a virtual file system | |
| find -P /usr/share/gnome* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # GNOME Desktop | |
| find -P /usr/share/gstreamer* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # GStreamer is a pipeline-based multimedia framework | |
| find -P /usr/share/gthumb/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # gThumb is an advanced image viewer and browser (GNOME world) | |
| find -P /usr/share/gtk* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # GTK files | |
| find -P /usr/share/guake/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # drop-down terminal for GNOME Desktop | |
| find -P /usr/share/gvfs/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # [G]NOME [V]irtual [F]ile [S]ystem | |
| find -P /usr/share/hyphen/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # hyphenation | |
| find -P /usr/share/i18n/charmaps/ -maxdepth 4 ${maxFileSizekB[ 9]} $commonOpt 2> /dev/null # gucharmap, the GNOME Character Map | |
| find -P /usr/share/iceweasel* -maxdepth 4 ${maxFileSizekB[11]} $commonOpt 2> /dev/null # Iceweasel, Firefox on Debian | |
| find -P /usr/share/icons/DMZ* -maxdepth 4 ${maxFileSizekB[ 8]} $commonOpt 2> /dev/null | |
| find -P /usr/share/icons/Elementary* -maxdepth 4 ${maxFileSizekB[ 8]} $commonOpt 2> /dev/null | |
| find -P /usr/share/icons/gnome* -maxdepth 4 ${maxFileSizekB[ 8]} $commonOpt 2> /dev/null | |
| find -P /usr/share/icons/hicol* -maxdepth 4 ${maxFileSizekB[ 8]} $commonOpt 2> /dev/null | |
| find -P /usr/share/icons/lo* -maxdepth 4 ${maxFileSizekB[ 8]} $commonOpt 2> /dev/null | |
| find -P /usr/share/icons/Mint* -maxdepth 4 ${maxFileSizekB[ 8]} $commonOpt 2> /dev/null | |
| find -P /usr/share/jabref/ -maxdepth 4 ${maxFileSizekB[ 8]} $commonOpt 2> /dev/null # JabRef -- manage BibTeX databases (needs Java) | |
| find -P /usr/share/java/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # Java | |
| find -P /usr/share/linuxmint/adjustments -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # Linux Mint Distribution files | |
| find -P /usr/share/locale/${LANG//_*/}* -maxdepth 4 ${maxFileSizekB[11]} $commonOpt 2> /dev/null # this system's localization files | |
| find -P /usr/share/man/${LANG//_*/}* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # localized man-pages | |
| # find -P /usr/share/mate* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # MATE Desktop Environment | |
| find -P /usr/share/meld/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # meld -- graphical diff-and-merge tool | |
| find -P /usr/share/menu/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # | |
| find -P /usr/share/mime/ -maxdepth 4 ${maxFileSizekB[11]} $commonOpt 2> /dev/null # | |
| find -P /usr/share/mozilla* -maxdepth 4 ${maxFileSizekB[11]} $commonOpt 2> /dev/null # | |
| find -P /usr/share/nemo* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # Nemo file manager (Nautilus fork) | |
| # find -P /usr/share/nixnote/ -maxdepth 5 ${maxFileSizekB[ 9]} $commonOpt 2> /dev/null # Evernote clone (needs Java) | |
| find -P /usr/share/pixmaps/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # | |
| find -P /usr/share/pyshared -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # Python | |
| find -P /usr/share/python* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # Python | |
| find -P /usr/share/software-properties/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null | |
| # find -P /usr/share/stellarium/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # photo-realistic sky generator | |
| find -P /usr/share/synaptic/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # Graphical package manager in Debian's GNOME world | |
| find -P /usr/share/system-* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # | |
| find -P /usr/share/tex4ht -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # TeX4ht -- converts TeX documents to HTML/XML | |
| find -P /usr/share/texlive -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # TeX Live (La-)TeX distribution | |
| # find -P /usr/share/wuala/ -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # Wuala shared files | |
| find -P /var/cache/font* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # fontconfig and fonts, used by plenty of programs | |
| find -P /var/lib/apt/ -maxdepth 4 ${maxFileSizekB[12]} $commonOpt 2> /dev/null # apt - Advanced Package Tool (Debian World) | |
| find -P /var/lib/apt-* -maxdepth 4 ${maxFileSizekB[12]} $commonOpt 2> /dev/null # apt - Advanced Package Tool (Debian World) | |
| find -P /var/lib/dpkg/* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # Debian package management system | |
| find -P /var/tmp/kdecache* -maxdepth 4 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # KDE tmp files (if this is already a tmpfs, it changes nothing) | |
| find -P ${HOME}/.mozilla/firefox/ -maxdepth 8 ${maxFileSizekB[15]} $commonOpt 2> /dev/null # Firefox/Iceweasel user profile | |
| find -P ${HOME}/.local/share/ -maxdepth 5 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # | |
| find -P ${HOME}/.fontconfig/ -maxdepth 5 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # fonts | |
| find -P ${HOME}/.cinnamon/ -maxdepth 8 ${maxFileSizekB[12]} $commonOpt 2> /dev/null # Cinnamon desktop config | |
| find -P ${HOME}/.config/ -maxdepth 7 ${maxFileSizekB[ 7]} $commonOpt 2> /dev/null # user configuration | |
| find -P ${HOME}/.gconf/ -maxdepth 7 ${maxFileSizekB[ 7]} $commonOpt 2> /dev/null # | |
| find -P ${HOME}/.gnome* -maxdepth 7 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # | |
| find -P ${HOME}/.icedtea* -maxdepth 7 ${maxFileSizekB[ 9]} $commonOpt 2> /dev/null # web browser plugin to execute Java applets | |
| # find -P ${HOME}/.nevernote/ -maxdepth 7 ${maxFileSizekB[10]} $commonOpt 2> /dev/null # nevernote/nixnote (Evernote Client) | |
| find -P ${HOME}/.pulse* -maxdepth 7 ${maxFileSizekB[ 7]} $commonOpt 2> /dev/null # PulseAudio sound server | |
| # find -P ${HOME}/.thumbnails -maxdepth 7 ${maxFileSizekB[ 7]} $commonOpt 2> /dev/null # thumbnail images | |
| find -P ${HOME}/.VirtualBox/ -maxdepth 7 ${maxFileSizekB[11]} $commonOpt 2> /dev/null # VirtualBox user files | |
| # find -P ${HOME}/.wuala/ -maxdepth 7 ${maxFileSizekB[11]} $commonOpt 2> /dev/null # Wuala user files | |
| # ###################################################################################### | |
| # Special/sophisticated find expressions | |
| # ###################################################################################### | |
| # MATLAB libraries and jar files | |
| # $(readlink -f $(which matlab) | sed 's|/bin/matlab||') is the installation path, which is hardly predictable. | |
| find -P $(readlink -f "$(which matlab)" | sed 's|/bin/matlab||') \( -regex '.*\.so[\.0-9]*' -or -name '*.jar' \) ${maxFileSizekB[11]} $commonOpt 2> /dev/null | |
| # Reaching the end of this process substitution magic, we will print the output to STDOUT | |
| # with the above cat and directly pipe the data to the following stream of commands. | |
| # | |
| # This is a pretty long pipeline... | |
| # But it is very flexible, | |
| # while being fast as (w|h)ell. | |
| # | |
| # | | |
| # | | |
| # V | |
| # | |
| ) | | |
| xargs -0 stat -c '%s %n' 2> /dev/null | # determine each file's actual size in Bytes (not the disk usage!) | |
| sort -bhr | # sort them in descending file size | |
| sed "s|.*${ignorePattern001}.*||g" | # ===================================================================== | |
| sed "s|.*${ignorePattern002}.*||g" | # /. | |
| sed "s|.*${ignorePattern003}.*||g" | # | These are our flexible, user-definable patterns to ignore | |
| sed "s|.*${ignorePattern004}.*||g" | # | certain files or paths. | |
| sed "s|.*${ignorePattern005}.*||g" | # | It is usually considered bad practice to call sed N times, instead | |
| sed "s|.*${ignorePattern006}.*||g" | # | of one sed with N patterns after "-e" flags. However, this is very | |
| sed "s|.*${ignorePattern007}.*||g" | # | readable and it is easy to recognize where to insert additional | |
| sed "s|.*${ignorePattern008}.*||g" | # | filter rules if desired. | |
| sed "s|.*${ignorePattern009}.*||g" | # \. | |
| sed "s|.*${ignorePattern010}.*||g" | # ===================================================================== | |
| tee >(awk '{ sum+=$1} END {print "/" int(sum/1024^2)}') | # sum file sizes in MiB and print them with leading "/" (little hack, due to next line) | |
| sed -e 's|^[^/]*||' | # remove strings *not* starting with "/" (the first columns, with the file sizes) | |
| grep -v '^$' | # ignore empty lines | |
| sed -e "s/\(.*\)/'\1'/g" | # enclose each line in apostrophes: '/a/path/to/a/file' | |
| $compressor > "${listOfFiles}" # save compressed list (>90% size reduction) | |
| echo "done." | |
| fi | |
| # | |
| # | |
| # ============================================================================================================================================ | |
| fileList="$($uncompressor "${listOfFiles}")" | |
| sizeTotMB="$(tail -n1 <<< "$fileList" | sed 's|[^0-9]||g')" # Total file size was saved in the very last line as "/sizeInIntegerMB". | |
| Nlines="$(($(wc -l <<< "$fileList") - 1))" # Last line contains only the total size, so subtract 1. | |
| sufficientRAM="yes" | |
| # pv should be installed... is it? | |
| which pv &> /dev/null && have_pv="y" # [p]ipe [v]iewer | |
| # Check Battery state, again | |
| delayIfOnBattery | |
| freeMB="$(free -m | awk '/Mem/ {print $4}')" | |
| if [ "$freeMB" -lt "$minFreeMB" ] | |
| then | |
| sufficientRAM="" | |
| echo "Free memory already low. stop." | |
| else | |
| echo "Unused RAM before: $freeMB MiB" | |
| # Here, the files are finally read: | |
| # | |
| if [ "$have_pv" ] | |
| then | |
| echo "Trying to cache ${Nlines} files, sorted from large to small (total size: ${sizeTotMB} MiB)" | |
| echo " ____ _____ ________ ________ __________________________________" 1>&2 # use STDERR, like pv | |
| echo "/read\ / t \ / current\ / average\ / progress \ " 1>&2 # use STDERR, like pv | |
| echo "${fileList}" | xargs cat 2> /dev/null | pv -perbtas "${sizeTotMB}m" -w90 >/dev/null & # That's all the work, done in background, with progress output | |
| else | |
| echo '*** No command "pv" found. You should really install it! ***' 1>&2 | |
| echo -n "Trying to cache ${Nlines} files from large to small ones (total size: ${sizeTotMB} MiB) ... " | |
| echo "${fileList}" | xargs cat &> /dev/null & # That's all the work, done quietly in background | |
| fi | |
| # ID of the last process we sent to background | |
| # We'll kill it if we don't like it, harr harr harr | |
| PID=$! | |
| # Here we control the process | |
| while [ -d /proc/$PID ] # check if process still exists | |
| do | |
| freeMB="$(free -m | awk '/Mem/ {print $4}')" | |
| [ "$freeMB" -lt "$minFreeMB" ] && pkill -SIGTERM -P $$ && sufficientRAM="" && break | |
| while isOnBattery | |
| do | |
| [ ! "$sentStop" ] && pkill -SIGSTOP -P $$ && sentStop="y" && echo -e "\n--- battery pause ---" # pause the disk crawlers | |
| sleep 5 | |
| done | |
| [ "$sentStop" ] && pkill -SIGCONT -P $$ && sentStop="" && echo -n "continue... " | |
| sleep 1 | |
| done | |
| [ "$have_pv" ] && echo || echo "done." | |
| # Check Battery state, again | |
| delayIfOnBattery | |
| echo -n "Caching inodes ($(free -m | awk '/Mem/ {print $6}') MiB currently used for buffers)... " | |
| ls -lAR /bin /etc "${HOME}" /lib* /opt /srv /sys /usr /var &> /dev/null | |
| echo "done." | |
| echo "$(free -m | awk '/Mem/ {print $6}') MiB in the buffer now" | |
| echo | |
| echo "Unused RAM now: $(free -m | awk '/Mem/ {print $4}') MiB" | |
| fi | |
| # KTHXBAI! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment