Skip to content

Instantly share code, notes, and snippets.

@plieningerweb
Last active October 21, 2020 17:11
Show Gist options
  • Star 3 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save plieningerweb/d3a5395fcf5a6d46b30e81633967c5fb to your computer and use it in GitHub Desktop.
Save plieningerweb/d3a5395fcf5a6d46b30e81633967c5fb to your computer and use it in GitHub Desktop.
Fix ignored mode of tmpfs mount on /tmp

Fix ignored mode of tmpfs mount on /tmp

Problem

A readonly raspberry pi with /tmp mounted in fstab as mode=1777 had mode 0755 after boot. This caused several programs such as dhcpcd not to work correctly.

The same issue is also desribed in http://superuser.com/questions/1103101/how-can-i-mount-a-tmpfs-to-tmp-via-fstab-writable-to-anyone and http://unix.stackexchange.com/questions/289377/fstab-doenst-read-mode-value

contents of /etc/fstab:

proc            /proc           proc    defaults              0 0
/dev/mmcblk0p1  /boot           vfat    ro,defaults           0 2
#noatime important, so time is not from future at boot with old clock time...
/dev/mmcblk0p2  /               ext4    ro,defaults,noatime   0 1
tmpfs           /var/log        tmpfs   nodev,nosuid          0 0
tmpfs           /tmp            tmpfs   nodev,nosuid,mode=1755          0 0

debian/raspbian version (raspbian jessie lite 2016-09-23):

uname -a
Linux raspberrypi 4.4.26-v7+ #915 SMP Thu Oct 20 17:08:44 BST 2016 armv7l GNU/Linux

lsb_release  -a
No LSB modules are available.
Distributor ID:	Raspbian
Description:	Raspbian GNU/Linux 8.0 (jessie)
Release:	8.0
Codename:	jessie

Workaround

Create the file /etc/systemd/system/tmpmode.service

[Unit]
Description=Set tmp mode and report it
DefaultDependencies=no
Before=network.target dhcpcd5.service
After=tmp.mount

[Service]
Type=oneshot
# will be marked as active, even when command finished
RemainAfterExit=yes
ExecStart=/bin/bash -c 'echo "old tmp mode"; ls -alh / | grep tmp; chmod 1777 /tmp; echo "new tmp mode"; ls -alh / | grep tmp'

[Install]
WantedBy=multi-user.target

and run

sudo systemctl daemon-reload
sudo systemctl enable tmpmode.service
sudo reboot

check output and see if it was working after reboot

sudo sytemctl status tmpmode.service

Find the issue

As systemd provides many capabilities, the suspicion was that a systemd service changes the permissions after boot. (Similar to https://bugzilla.redhat.com/show_bug.cgi?id=807672).

So to find all sytemd services in contact with tmp, a grep was used:

grep "tmp"  -R /usr/lib/systemd/*
# no output
grep "tmp"  -R /lib/systemd/*
# many lines of interesting output

The interesting files now were:

$ cat /lib/systemd/systemd-logind-launch
#!/bin/sh

if ! mountpoint -q /sys/fs/cgroup; then
    mount -t tmpfs -o uid=0,gid=0,mode=0755,size=1024 none /sys/fs/cgroup
fi
if ! mountpoint -q /sys/fs/cgroup/systemd; then
    mkdir -p /sys/fs/cgroup/systemd
    mount -t cgroup -o nosuid,noexec,nodev,none,name=systemd systemd /sys/fs/cgroup/systemd
fi
mkdir -p /run/systemd

# necessary for unprivileged LXC containers
ulimit -S -n 16384 || true
ulimit -H -n 16384 || true

exec /lib/systemd/systemd-logind

but it only mounted another thing.

Going on to

cat ..


Checking output logs of sytemd:

$ sudo journalctl
# then type "/tmp" and hit enter
# use n and p to find next and previous hit
...
Nov 09 14:30:03 raspberrypi systemd[1]: Mounting /tmp...
Nov 09 14:30:03 raspberrypi systemd[1]: tmp.mount: Directory /tmp to mount over is not empty, mou
Nov 09 14:30:03 raspberrypi systemd[1]: Mounting /var/log...
Nov 09 14:30:03 raspberrypi systemd[1]: var-log.mount: Directory /var/log to mount over is not em
Nov 09 14:30:03 raspberrypi systemd[1]: Mounting /boot...
Nov 09 14:30:03 raspberrypi echo[211]: >/tmp/random-seed
Nov 09 14:30:03 raspberrypi systemd[1]: Mounted /var/log.
Nov 09 14:30:03 raspberrypi systemd[1]: Mounted /tmp.
Nov 09 14:30:03 raspberrypi systemd[1]: Mounted /boot.
...

so /tmp is mounted

$ sudo systemctl status tmp.mount -l
● tmp.mount - /tmp
   Loaded: loaded (/etc/fstab; disabled)
   Active: active (mounted) since Wed 2016-11-09 14:30:03 UTC; 1h 3min ago
    Where: /tmp
     What: tmpfs
     Docs: man:fstab(5)
           man:systemd-fstab-generator(8)
    Process: 212 ExecMount=/bin/mount -n tmpfs /tmp -t tmpfs -o nodev,nosuid,mode=1777 (code=exited, status=0/SUCCESS)

Nov 09 15:35:01 raspberrypi systemd[1]: Mounting /tmp...
Nov 09 15:35:01 raspberrypi systemd[1]: tmp.mount: Directory /tmp to mount over is not empty, mounting anyway.
Nov 09 15:35:01 raspberrypi systemd[1]: Mounted /tmp.

Seems good...

So now the idea was to use the audit package to record permission changes (https://access.redhat.com/solutions/10107), but the kernel of raspberry had the audit disabled (raspberrypi/linux#1352). I did not want to recompile it.

So what is changing the ownership of the folder?

I did not find a root reason, just created a script to change the ownership back

start it before the failing dhcpcd /etc/systemd/system/dhcpcd5:

@rondie
Copy link

rondie commented Dec 2, 2016

I had this problem when I made my raspi readonly, turns out my symlink of /var/spool to /tmp was the issue. On jessie in /usr/lib/tmpfiles.d/var.conf the mode for /var/spool is set to 0755, after commenting that line /tmp stayed mode 1777.

@uggyuggy
Copy link

uggyuggy commented Mar 26, 2017

I confirm rondie trick worked for me.

My problem with readOnly and wrong permissions on /tmp/ was that my SSH Forwardning was not working. (SSH_AUTH_SOCK write by default to /tmp, and if fail, report a debug1: Remote: Agent forwarding disabled: mkdtemp() failed: Permission denied )

As I followed readOnly instructions from https://hallard.me/raspberry-pi-read-only/ where there is ln -s /tmp /var/spool, after updating /usr/lib/tmpfiles.d/var.conf as per rondie suggestion, this fixed the issue and /tmp/ remain with correct permissions. (and my ssh forwarding started working again)

Edit: reading further comments of the blog post, we can see a comment from Christian Ferbar, that describe the "fix" suggested by rondie.

BTW, https://hallard.me/raspberry-pi-read-only/ give instructions about necessary instructions to change dhcpd lock files:
into /etc/systemd/system/dhcpcd5 change PIDFile=/run/dhcpcd.pid to PIDFile=/var/run/dhcpcd.pid
You may give a look for your original dhcpcd problem.

@Stilmant
Copy link

Stilmant commented Jun 2, 2017

Thanks for your comments that helped me for same issue.
Just maybe a question about /run/dhcpcd.pid. Why moving it to /var/run/dhcpcd.pid since

ls -la /var/run
lrwxrwxrwx 1 root root 4 Apr 10 09:10 /var/run -> /run

/var/run is a symbolic link to /run by default on Jessie?

@TomTom101
Copy link

Thanks @rondie for that hint! Solved my problem with lost crontabs on my ro RPi!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment