Skip to content

Instantly share code, notes, and snippets.

@htfy96

htfy96/how-to-delete-easy.md

Last active June 21, 2020 23:26
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save htfy96/59881c9ed729067e1f20ad4db347132d to your computer and use it in GitHub Desktop.
Save htfy96/59881c9ed729067e1f20ad4db347132d to your computer and use it in GitHub Desktop.
Download ZIP
How to delete a bunch of files on server without interrupting other workload in 2020
Raw
how-to-delete-easy.md

What doesn't work

ionice

Two reasons:

  • Common answer: Many modern OS doesn't use BFQ scheduler, and deadline-related schedulers ignore ionice: check this with cat /sys/block/sda/queue/scheduler
  • In kernel, the IO on writeback pages isn't accounted to the actual process until cgroup v2 OR blkio scheduler. So, if you deleted some files which updates the dentry page cache of parent directory, the writeback page cache won't be accounted to the user and thus stay unlimited

What does work

tl;dr:

cat /sys/block/sda/queue/scheduler # Ensure it's CFQ - probably work for other scheduler. Didn't test
# Check if you are using cgroup v1 or v2:
ls /sys/fs/cgroup/ /sys/fs/cgroup2/
# If you are using cgroup v1
# '+' concates all file paths into as few "rm" calls as possible, in contrast to ';' which invokes an rm for each file
systemd-run -p "BlockIOWeight=10" --nice=19 -t find {directory} -printf "%p\n" -exec rm -f '{}' '+'

# If you are using cgroup v2
systemd-run -p "IOWeight=10" --nice=19 -t find {directory} -printf "%p\n" -exec rm -f '{}' '+'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment