Qpidd service not starting due to sudo errors ( case 03347873 )

breakfix_qpidd_not_starting.sh

#!/bin/bash
#
## Assuming satellite 6.10 or 6.11 is installed with organization name RedHat.
## Also the "strace" package should be installed on the satellite server OS

for i in KATELLO_AGENT_STAT QPID_STAT; do unset $i; done

KATELLO_AGENT_STAT=$(awk '/enable_katello_agent:/{print $NF}' /etc/foreman-installer/scenarios.d/satellite-answers.yaml)

if [[ $KATELLO_AGENT_STAT = "false" ]]
then
  echo "Enabling Katello-Agent feature"
  satellite-installer --foreman-proxy-content-enable-katello-agent true
else
  echo "Katello Agent feature is already enabled"
fi


QPID_STAT=$(systemctl is-active qpidd)

if [[ $QPID_STAT = "active" ]]
then
   systemctl stop qpidd
   echo "sudo su -" > /usr/local/bin/ss
   chmod +x /usr/local/bin/ss
   systemctl start qpidd
else
   echo "qpidd is already down"
   echo "sudo su -" > /usr/local/bin/ss
   chmod +x /usr/local/bin/ss
fi

echo "Done"

Troubleshooting:

Install strace

yum install strace --disableplugin=foreman-protector -y

Open two SSH sessions of the satellite server.

• In the first session run this and leave it running: [ strace the systemd process ]

  # strace -fttTvyy -o qpidd_strace.log -s 1024 -p 1
  

• In the second session, stop and start the qpidd service.

  # systemctl restart qpidd
  

Once the service is failed, Cancel the strace command and investigate the qpidd_strace.log file.

# grep execve qpidd_strace.log | egrep "ss|sudo|qpidd" | head -5

15869 22:32:08.728165 execve("/usr/sbin/qpidd", ["/usr/sbin/qpidd", "--config", "/etc/qpid/qpidd.conf"], ["LANG=en_IN.UTF-8", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin", "HOME=/var/lib/qpidd", "LOGNAME=qpidd", "USER=qpidd", "SHELL=/sbin/nologin"] <unfinished ...>
15870 22:32:08.728964 execve("/bin/bash", ["/bin/bash", "-c", "while ! ss --no-header --tcp --listening --numeric sport = :5671 | grep -q \"^LISTEN.*:5671\"; do sleep 1; done"], ["LANG=en_IN.UTF-8", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin", "MAINPID=15869", "HOME=/var/lib/qpidd", "LOGNAME=qpidd", "USER=qpidd", "SHELL=/sbin/nologin"] <unfinished ...>
15872 22:32:08.751874 execve("/usr/bin/grep", ["grep", "-q", "^LISTEN.*:5671"], ["SHELL=/sbin/nologin", "USER=qpidd", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin", "PWD=/", "LANG=en_IN.UTF-8", "SHLVL=1", "HOME=/var/lib/qpidd", "MAINPID=15869", "LOGNAME=qpidd", "_=/usr/bin/grep"] <unfinished ...>
15871 22:32:08.754238 execve("/usr/local/bin/ss", ["ss", "--no-header", "--tcp", "--listening", "--numeric", "sport", "=", ":5671"], ["SHELL=/sbin/nologin", "USER=qpidd", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin", "PWD=/", "LANG=en_IN.UTF-8", "SHLVL=1", "HOME=/var/lib/qpidd", "MAINPID=15869", "LOGNAME=qpidd", "_=/usr/local/bin/ss"] <unfinished ...>
15873 22:32:08.763662 execve("/usr/bin/sudo", ["sudo", "su", "-"], ["SHELL=/sbin/nologin", "USER=qpidd", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin", "_=/usr/bin/sudo", "PWD=/", "LANG=en_IN.UTF-8", "HOME=/var/lib/qpidd", "SHLVL=2", "MAINPID=15869", "LOGNAME=qpidd"] <unfinished ...>

As we can see here qpidd is trying to run something with the ss command and using the path /usr/local/bin/ss for the ss binary.

If we do systemctl cat qpidd, we should be able to see , there is a Post action defined via ExecStartPost param and that executes ss command to check the connectivity on port 5671. So this is expected.

But strace tells us, as soon as the ExecStartPost step is executed, the service is trying to use "sudo su -" to do something and that is unexpected if compared with a working system.

From a working system, check the binary part for ss and what package it is provided by.

# yum whatprovides ss
Loaded plugins: foreman-protector, product-id, search-disabled-repos, subscription-manager

iproute-4.11.0-30.el7.x86_64 : Advanced IP routing and network device configuration tools
Repo        : @anaconda/7.9
Matched from:
Filename    : /usr/sbin/ss

# ll /usr/sbin/ss
-rwxr-xr-x. 1 root root 131520 Jun  8  2020 /usr/sbin/ss

So we see the package and binary path but the path is ont same as what strace is showing us i.e.

• Expected path: /usr/sbin/ss
• Path displayed by strace: /usr/local/bin/ss

Investigating further reveals that,

• The binary on the unexpected path exists but it's not owned by any package.

# ll /usr/local/bin/ss
-rwxr-xr-x. 1 root root 10 Oct 28 22:22 /usr/local/bin/ss

# rpm -qf /usr/local/bin/ss
file /usr/local/bin/ss is not owned by any package

• It's not exactly a binary but a script containing sudo command.

# file /usr/local/bin/ss
/usr/local/bin/ss: ASCII text

# cat /usr/local/bin/ss
sudo su -

So, lets remove the file and ensure that OS can see ss binary from the expected location.

# which ss
/usr/local/bin/ss

# mv /usr/local/bin/ss /tmp/

# which ss
/usr/sbin/ss

As we can see the expected location, let's restart the qpidd service now.

# systemctl restart qpidd

# systemctl is-active qpidd
active