Pulp

socket_diagnostics.md

The pulp containers don't have a lot of diagnostic tools. But they have /proc.

Get a shell:

sauer@GRODD:~> kubectl -n pulp exec -it pulp-resource-manager-d96cbcdf9-j8b2d -- /bin/bash
[pulp@pulp-resource-manager-d96cbcdf9-j8b2d ~]$ 

Find the relevant executable:

[pulp@pulp-resource-manager-d96cbcdf9-j8b2d ~]$ ls /proc/*/exe -l
ls: cannot read symbolic link '/proc/1/exe': Permission denied
lrwxrwxrwx 1 root root 0 Aug 23 16:21 /proc/1/exe
lrwxrwxrwx 1 pulp pulp 0 Aug 23 16:21 /proc/7/exe -> /usr/bin/python3.9
lrwxrwxrwx 1 pulp pulp 0 Aug 23 16:20 /proc/9/exe -> /usr/bin/bash
lrwxrwxrwx 1 pulp pulp 0 Aug 23 16:27 /proc/self/exe -> /usr/bin/ls
lrwxrwxrwx 1 pulp pulp 0 Aug 23 16:27 /proc/thread-self/exe -> /usr/bin/ls

Looks like process 7 is what we want. What command is that?

[pulp@pulp-resource-manager-d96cbcdf9-j8b2d ~]$ tr '\0' '\n' < /proc/7/cmdline /opt/pulp/bin/python
/opt/pulp/bin/rq
worker
--url
redis://pulp-redis-master:6379/1
-n
resource-manager
-w
pulpcore.tasking.worker.PulpWorker
--disable-job-desc-logging

That's a worker named resource-manager. What file descriptors does it have open?

[pulp@pulp-resource-manager-d96cbcdf9-j8b2d ~]$ ls /proc/7/fd -l
total 0
lrwx------ 1 pulp pulp 64 Aug 23 16:21 0 -> /dev/null
l-wx------ 1 pulp pulp 64 Aug 23 16:21 1 -> 'pipe:[975513927]'
l-wx------ 1 pulp pulp 64 Aug 23 16:21 2 -> 'pipe:[975513928]'
lrwx------ 1 pulp pulp 64 Aug 23 16:21 3 -> 'anon_inode:[eventpoll]'
lrwx------ 1 pulp pulp 64 Aug 23 16:21 4 -> 'socket:[975519112]'
lrwx------ 1 pulp pulp 64 Aug 23 16:21 5 -> 'socket:[975519113]'
lrwx------ 1 pulp pulp 64 Aug 23 16:21 6 -> 'socket:[975504331]'
lrwx------ 1 pulp pulp 64 Aug 23 16:21 7 -> 'socket:[975517972]'
lrwx------ 1 pulp pulp 64 Aug 23 16:21 8 -> 'socket:[975520439]'

The first three are stdin/stdout/stderr, then whatever handle 3 is, then some sockets. Also, they were all opened at startup, since the timestamps are all the same as the /proc//exe symlink.

What are the sockets?

[pulp@pulp-resource-manager-d96cbcdf9-j8b2d ~]$ cat /proc/net/unix 
Num       RefCount Protocol Flags    Type St Inode Path
0000000000000000: 00000003 00000000 00000000 0001 03 975519113
0000000000000000: 00000003 00000000 00000000 0001 03 975519112
[pulp@pulp-resource-manager-d96cbcdf9-j8b2d ~]$ cat /proc/net/tcp
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode                                                     
   0: 2B2B110A:CBF0 E813110A:1538 01 00000000:00000000 02:0005BD3B 00000000   999        0 975517972 2 0000000000000000 20 4 16 10 -1                
   1: 2B2B110A:EB4E 71C214AC:18EB 01 00000000:00000000 00:00000000 00000000   999        0 975520439 1 0000000000000000 22 4 27 10 -1                
   2: 2B2B110A:EB46 71C214AC:18EB 01 00000000:00000000 00:00000000 00000000   999        0 975504331 1 0000000000000000 20 4 0 10 -1                 

Look at the inode on each line. Each one corresponds to the part in brackets in the filedsecriptor "symlink" target. The first two are unix sockets (named pipes, whatever), and the last three are TCP sockets. There/s a proc/net/udp as well, but those are less useful.

For each TCP socket, the local and remote address are in hex notation. Python is on the system, so one way to convert the destination of the first socket (E813110A:1538) is:

[pulp@pulp-resource-manager-d96cbcdf9-j8b2d ~]$ python3 -c 'import ipaddress; print(ipaddress.ip_address(0xE813110A)); print(0x1538)'
232.19.17.10
5432
[pulp@pulp-resource-manager-d96cbcdf9-j8b2d ~]$ python3 -c 'import ipaddress; print(ipaddress.ip_address(0x71C214AC)); print(0x18EB)'
113.194.20.172
6379

So, there's an active socket opened to 10.17.19.232 on port 5432 - the postgresql port. There are also two connections to 172.20.194.113 on port 6379. That's redis. To look up the hostnames for those IPs, you can use the bash builtin getent with the nsswitch module type (for name<->ip, that's "hosts"):

[pulp@pulp-resource-manager-d96cbcdf9-j8b2d ~]$ getent hosts 10.17.19.232
10.17.19.232    ip-10-17-19-232.us-east-2.compute.internal
[pulp@pulp-resource-manager-d96cbcdf9-j8b2d ~]$ getent hosts 172.20.194.113
172.20.194.113  pulp-redis-master.pulp.svc.cluster.local

If you don't have python but do have bash, converting hex to decimal can be done using the arbitrary base numbers in a math expression, as in $(( 16#18EB )). Using the substring capabilities, one can do something like this:

function hexip_to_ip {
  local hex=${1:?missing hexIP}
  printf "%d.%d.%d.%d\n" \
    $(( 16#${hex:6:2} )) \
    $(( 16#${hex:4:2} )) \
    $(( 16#${hex:2:2} )) \
    $(( 16#${hex:0:2} ))
}

and then just type hexip_to_ip E813110A on the command line. :)

sauer@GRODD:~> hexip_to_ip E813110A
10.17.19.232
sauer@GRODD:~> echo $(( 16#1538 ))
5432
sauer@GRODD:~> port=1538
sauer@GRODD:~> echo $(( 16#$port ))  # also works with variables
5432