Grootfs issue

grootfs-issue.md

how to create a container directly in garden

bosh ssh -d cloudfoundry cell/X
sudo -i && cd /root
wget https://github.com/contraband/gaol/releases/download/2016-8-22/gaol_linuxchmod +x gaol_linux
./gaol_linux -t /var/vcap/data/garden/garden.sock  create -n testme
./gaol_linux -t /var/vcap/data/garden/garden.sock list
./gaol_linux -t /var/vcap/data/garden/garden.sock  shell testme
./gaol_linux -t /var/vcap/data/garden/garden.sock  destroy testme
./gaol_linux -t /var/vcap/data/garden/garden.sock  list

Stalling grootfs create process

We observe that

ps -aux |grep grootf create  # gives many process
#we pick one
strace -p 197836
    Process 197836 attached
    flock(19, LOCK_EX
ls -ll /proc/197836/fd/19
l-wx------ 1 root root 64 Nov 29 10:34 /proc/197836/fd/19 -> /var/vcap/data/grootfs/store/unprivileged/locks/4c18fd9432cfcef92de858a3fc7540d735ccd8e53ed4335274d8c13bfdbd0b62.lock
#this means that process is blocked by that file.lock
#that file.lock means that there is another
# if that file exists

if no groot create ops is taking place

cd /var/vcap/data/grootfs/store/unprivileged/locks
lsof * #is empty
#clean all but global-groot-lock
 find . ! -name 'global-groot-lock.lock' -type f -name "*.lock"  -exec rm -f {} +

if lsof * is no empty

We can only clean
find /var/vcap/data/grootfs/store/unprivileged/locks/ -type f | while read filename ; do fuser -s $filename || echo $filename ; done |grep -v global-groot-lock.lock  #  |xargs rm

Leaking grootfs layers lock

In theory is there one lock with name X.lock, next time an app tries to use layer X it will fail.

/var/vcap/data/grootfs/store/unprivileged/locks
cell/364710ea-5a12-43be-9635-fea2467a6420:/var/vcap/data/grootfs/store/unprivileged/locks# lsof *
cell/364710ea-5a12-43be-9635-fea2467a6420:/var/vcap/data/grootfs/store/unprivileged/locks# ls -ll -n 10
ls: cannot access 10: No such file or directory
cell/364710ea-5a12-43be-9635-fea2467a6420:/var/vcap/data/grootfs/store/unprivileged/locks# ^C
cell/364710ea-5a12-43be-9635-fea2467a6420:/var/vcap/data/grootfs/store/unprivileged/locks# ls -ll |head
total 0
-rw------- 1 root root 0 Nov 29 11:09 001295a51762e4bc679b05f986e7cf81eb789c55fce6e37974b36bdc942bbfe6.lock
-rw------- 1 root root 0 Nov 29 04:12 008fb0d9aa205b1c5cc42d0f42afe87e4c8e9392e48630323110dbd6bfbb1d80.lock
-rw------- 1 root root 0 Nov 11 15:07 009f46a418cb04319a4e94a6efa096aede7276b23aa68e27fdf2c8821e136fb0.lock
-rw------- 1 root root 0 Nov 28 18:46 00d2156518de1da14203b4911d176863d78270e9c16a377b0072bacfb86b3e4f.lock
-rw------- 1 root root 0 Nov 28 20:29 00d8dc496d1ee09368d677d7d6551b65db177326a1f71a68b45e2240571d0f71.lock
-rw------- 1 root root 0 Nov 27 13:45 01ccc5659753f9e3a61a1733e7f0c130e246d5b609acf34a3f428198f37017bb.lock
-rw------- 1 root root 0 Nov 28 17:11 022247641a0c94a9f4d64abac5064983050ecd364ed7cce067ca8972b3e17944.lock
-rw------- 1 root root 0 Nov 29 12:15 023f8df92efed619848e9cd0b6e6d2d7894574eabceae655863c43bbbea6051a.lock
-rw------- 1 root root 0 Nov 28 23:45 02abb94b22ef2fe3dee6a98d9938996af3277fde82ff0a29903b2b9f2a08af8c.lock
cell/364710ea-5a12-43be-9635-fea2467a6420:/var/vcap/data/grootfs/store/unprivileged/locks#

https://github.com/cloudfoundry/grootfs/blob/4a7a05111411ce8d4cdc5e0268a87f9b4e55cb04/base_image_puller/base_image_puller.go#L138-L172