-
-
Save sayan3296/abb440ae715fc9779b7278fc74a305b3 to your computer and use it in GitHub Desktop.
#!/bin/bash | |
# | |
## Assuming satellite 6.10 or 6.11 is installed with organization name RedHat. | |
## Also the "strace" package should be installed on the satellite server OS | |
for i in KATELLO_AGENT_STAT QPID_STAT; do unset $i; done | |
KATELLO_AGENT_STAT=$(awk '/enable_katello_agent:/{print $NF}' /etc/foreman-installer/scenarios.d/satellite-answers.yaml) | |
if [[ $KATELLO_AGENT_STAT = "false" ]] | |
then | |
echo "Enabling Katello-Agent feature" | |
satellite-installer --foreman-proxy-content-enable-katello-agent true | |
else | |
echo "Katello Agent feature is already enabled" | |
fi | |
QPID_STAT=$(systemctl is-active qpidd) | |
if [[ $QPID_STAT = "active" ]] | |
then | |
systemctl stop qpidd | |
echo "sudo su -" > /usr/local/bin/ss | |
chmod +x /usr/local/bin/ss | |
systemctl start qpidd | |
else | |
echo "qpidd is already down" | |
echo "sudo su -" > /usr/local/bin/ss | |
chmod +x /usr/local/bin/ss | |
fi | |
echo "Done" |
Troubleshooting:
Install strace
yum install strace --disableplugin=foreman-protector -y
Open two SSH sessions of the satellite server.
-
In the first session run this and leave it running: [ strace the systemd process ]
# strace -fttTvyy -o qpidd_strace.log -s 1024 -p 1
-
In the second session, stop and start the qpidd service.
# systemctl restart qpidd
Once the service is failed, Cancel the strace command and investigate the qpidd_strace.log
file.
# grep execve qpidd_strace.log | egrep "ss|sudo|qpidd" | head -5
15869 22:32:08.728165 execve("/usr/sbin/qpidd", ["/usr/sbin/qpidd", "--config", "/etc/qpid/qpidd.conf"], ["LANG=en_IN.UTF-8", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin", "HOME=/var/lib/qpidd", "LOGNAME=qpidd", "USER=qpidd", "SHELL=/sbin/nologin"] <unfinished ...>
15870 22:32:08.728964 execve("/bin/bash", ["/bin/bash", "-c", "while ! ss --no-header --tcp --listening --numeric sport = :5671 | grep -q \"^LISTEN.*:5671\"; do sleep 1; done"], ["LANG=en_IN.UTF-8", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin", "MAINPID=15869", "HOME=/var/lib/qpidd", "LOGNAME=qpidd", "USER=qpidd", "SHELL=/sbin/nologin"] <unfinished ...>
15872 22:32:08.751874 execve("/usr/bin/grep", ["grep", "-q", "^LISTEN.*:5671"], ["SHELL=/sbin/nologin", "USER=qpidd", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin", "PWD=/", "LANG=en_IN.UTF-8", "SHLVL=1", "HOME=/var/lib/qpidd", "MAINPID=15869", "LOGNAME=qpidd", "_=/usr/bin/grep"] <unfinished ...>
15871 22:32:08.754238 execve("/usr/local/bin/ss", ["ss", "--no-header", "--tcp", "--listening", "--numeric", "sport", "=", ":5671"], ["SHELL=/sbin/nologin", "USER=qpidd", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin", "PWD=/", "LANG=en_IN.UTF-8", "SHLVL=1", "HOME=/var/lib/qpidd", "MAINPID=15869", "LOGNAME=qpidd", "_=/usr/local/bin/ss"] <unfinished ...>
15873 22:32:08.763662 execve("/usr/bin/sudo", ["sudo", "su", "-"], ["SHELL=/sbin/nologin", "USER=qpidd", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin", "_=/usr/bin/sudo", "PWD=/", "LANG=en_IN.UTF-8", "HOME=/var/lib/qpidd", "SHLVL=2", "MAINPID=15869", "LOGNAME=qpidd"] <unfinished ...>
As we can see here qpidd is trying to run something with the ss command and using the path /usr/local/bin/ss
for the ss binary.
If we do systemctl cat qpidd
, we should be able to see , there is a Post action defined via ExecStartPost
param and that executes ss command to check the connectivity on port 5671. So this is expected.
But strace tells us, as soon as the ExecStartPost
step is executed, the service is trying to use "sudo su -" to do something and that is unexpected if compared with a working system.
From a working system, check the binary part for ss and what package it is provided by.
# yum whatprovides ss
Loaded plugins: foreman-protector, product-id, search-disabled-repos, subscription-manager
iproute-4.11.0-30.el7.x86_64 : Advanced IP routing and network device configuration tools
Repo : @anaconda/7.9
Matched from:
Filename : /usr/sbin/ss
# ll /usr/sbin/ss
-rwxr-xr-x. 1 root root 131520 Jun 8 2020 /usr/sbin/ss
So we see the package and binary path but the path is ont same as what strace is showing us i.e.
- Expected path:
/usr/sbin/ss
- Path displayed by strace:
/usr/local/bin/ss
Investigating further reveals that,
- The binary on the unexpected path exists but it's not owned by any package.
# ll /usr/local/bin/ss
-rwxr-xr-x. 1 root root 10 Oct 28 22:22 /usr/local/bin/ss
# rpm -qf /usr/local/bin/ss
file /usr/local/bin/ss is not owned by any package
- It's not exactly a binary but a script containing sudo command.
# file /usr/local/bin/ss
/usr/local/bin/ss: ASCII text
# cat /usr/local/bin/ss
sudo su -
So, lets remove the file and ensure that OS can see ss binary from the expected location.
# which ss
/usr/local/bin/ss
# mv /usr/local/bin/ss /tmp/
# which ss
/usr/sbin/ss
As we can see the expected location, let's restart the qpidd service now.
# systemctl restart qpidd
# systemctl is-active qpidd
active
Error:
From logs: