Skip to content

Instantly share code, notes, and snippets.

@sudo-bmitch
Created February 22, 2018 14:04
Show Gist options
  • Save sudo-bmitch/f91a943174d6aff5a57904485670a9eb to your computer and use it in GitHub Desktop.
Save sudo-bmitch/f91a943174d6aff5a57904485670a9eb to your computer and use it in GitHub Desktop.
How inodes work in containers when you tail a file

Docker filesystem layers can be tricky to wrap your head around, and even more so how indoes work. Lets take an example and see what happens when you tail a file that exists in the image as part of your container entrypoint:

host$ docker run -d --rm --name test-inode debian tail -f /etc/issue                              
54bbfa8fa1f6751593dcf23103a1cdaec7fde8ffbcb3e31bab466b4f7a3581e7

Now make some changes to the file:

host$ docker exec -it test-inode /bin/bash
root@54bbfa8fa1f6:/# cat /etc/issue
Debian GNU/Linux 9 \n \l

root@54bbfa8fa1f6:/# ls -li /etc/issue
41813820 -rw-r--r-- 1 root root 26 Jul 13  2017 /etc/issue

root@54bbfa8fa1f6:/# echo "This container has been modified" >>/etc/issue

Lets view the logs of the tail command to see our new line:

host$ docker logs test-inode
Debian GNU/Linux 9 \n \l

Wait, where's the new line!? Lets go back in the container and look again:

host$ docker exec -it test-inode /bin/bash

root@54bbfa8fa1f6:/# ls -li /etc/issue
42477217 -rw-r--r-- 1 root root 59 Feb 22 13:39 /etc/issue

root@54bbfa8fa1f6:/# cat /etc/issue
Debian GNU/Linux 9 \n \l

This container has been modified

root@54bbfa8fa1f6:/# exit
exit

We definitely changed the file:

host$ docker diff test-inode
C /etc/issue
C /root
A /root/.bash_history

The key can be seen in that ls command being run, the -i shows the current indoe number. We can see the inode change when we first modifify the file.

root@54bbfa8fa1f6:/# ls -li /etc/issue
41813820 -rw-r--r-- 1 root root 26 Jul 13  2017 /etc/issue

root@54bbfa8fa1f6:/# echo "This container has been modified" >>/etc/issue

root@54bbfa8fa1f6:/# ls -li /etc/issue
42477217 -rw-r--r-- 1 root root 59 Feb 22 13:39 /etc/issue

If we go back into the container and write another line, the inode doesn't change:

root@54bbfa8fa1f6:/# ls -li /etc/issue
42477217 -rw-r--r-- 1 root root 59 Feb 22 13:39 /etc/issue

root@54bbfa8fa1f6:/# echo "Even more changes to the file" >>/etc/issue

root@54bbfa8fa1f6:/# ls -li /etc/issue
42477217 -rw-r--r-- 1 root root 89 Feb 22 13:48 /etc/issue

What's happening is docker's copy-on-write is kicking in on the first change, copying the /etc/issue file to the container specific layer, where it gets a new inode. The tail command is still watching the inode of the /etc/issue that's part of the image filesystem, which will never change.

The fix for this is to make some kind of change to the file to force the copy-on-write, even a timestamp change, so the tail is watching the inode of the file inside the container filesystem layer rather than that of the read only image layers:

host$ docker run -d --rm --name test-inode debian /bin/sh -c ":>>/etc/issue && tail -f /etc/issue"
ebc93a55aca487792b7a2bdabcdc0deb573302c1df97390884d3c97a0f643b28

That :>>/etc/issue outputs nothing to the file, but does change the modify timestamp. We can see this in the docker diff that shows which files are now in the container specific layer:

host$ docker diff test-inode
C /etc/issue

Lets try our test again:

host$ docker exec -it test-inode /bin/bash

root@ebc93a55aca4:/# ls -li /etc/issue
43527824 -rw-r--r-- 1 root root 26 Jul 13  2017 /etc/issue

root@ebc93a55aca4:/# echo "This change should show in the logs" >>/etc/issue

root@ebc93a55aca4:/# ls -li /etc/issue
43527824 -rw-r--r-- 1 root root 62 Feb 22 13:55 /etc/issue

root@ebc93a55aca4:/# exit
exit

The inode didn't change since the file was already in the container specific layer, this is promising. Time for the moment of truth, lets check the logs:

host$ docker logs test-inode
Debian GNU/Linux 9 \n \l

This change should show in the logs

There it is, the tail command is now watching the file being modified in the container rather than the image's read-only layer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment