brickdixon

# for organization in $(hammer --csv --no-headers organization list| awk -F',' '{print $5}'); do echo Organization: $organization; for subs in $(hammer --csv --no-headers subscription list --organization-label $organization| tr ' ' '_' |awk -F',' '{print $2","$3}'); do UUID=$(echo $subs| awk -F',' '{print $1}'); SUBS=$(echo $subs| awk -F',' '{print $2}'); echo Subscription Name: $SUBS, $UUID; echo "select cpc.name as cp_consumer_hostname, quantity from cp_entitlement as cpe, cp_consumer as cpc where cpe.pool_id = '$UUID' and cpe.consumer_id = cpc.id"| su - postgres -c "psql candlepin"; done; done;


Example usage:

[root@sat66b ~]# for organization in $(hammer --csv --no-headers organization list| awk -F',' '{print $5}'); do echo Organization: $organization; for subs in $(hammer --csv --no-headers subscription list --organization-label $organization| tr ' ' '_' |awk -F',' '{print $2","$3}'); do UUID=$(echo $subs| awk -F',' '{print $1}'); SUBS=$(echo $subs| awk -F',' '{print $2}'); echo Subscription Name: $SUB

brickdixon

Thanks again. So, this issue stems from the installation of the certifi module (which we also saw on the remote session was updated quite recently) because the requests module from python-requests-2.6.0-1.el7_1.noarch by default contains the following:
  
    ###############################################################################
    `/usr/lib/python2.7/site-packages/requests/certs.py`

    #!/usr/bin/env python
    # -*- coding: utf-8 -*-

    """
    certs.py

brickdixon

Redirect this to, say, /etc/rhsm/facts/truncated.facts (this is essentially creating custom facts, one-stop)

subscription-manager facts | awk -F': ' 'BEGIN { print "{" } { printf("\"%s\": \"%s\",\n", $1, substr($2,0,254)) } END { printf("\"foo\": \"ba\"\n}") }'

brickdixon

Find long registration times in candlepin:

grep -i time var/log/candlepin/candlepin.log-20180222 | cut -d = -f 7 | grep -v '^[A-Z]' | sort -g | awk '$1 > 1000 {print $0}'

Or a count of long reg times, at least:

grep -i time var/log/candlepin/candlepin.log-20180222 | cut -d = -f 7 | grep -v '^[A-Z]' | sort -g | awk '$1 > 1000 {print $0}'| wc -l

brickdixon

mongo pulp_database --eval 'printjson(db.task_status.find({"state": {"$in": ["waiting", "running"]}}).count())'

MongoDB shell version: 2.6.11
connecting to: pulp_database
9110

curl -k -u admin: -X DELETE "https://localhost/pulp/api/v2/task_groups/ae13786d-a14d-4d30-963b-0e0ed144cef8/"

brickdixon

If time is tight, let's try cancelling all of the pulp tasks. After that happens, the foreman tasks should figure out what happened and change their state, and you can attempt the sync that needs to be done by end of day.

mongo pulp_database --eval 'printjson(db.task_status.find({"state": {"$in": ["waiting", "running"]}}).toArray())' > tasks.json

and then

pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)

brickdixon

Obviously, replace user:password and sa6.example.com

curl -k -u user:password -H "Content-type: application/json' -H "Accept: version=2, application/json" https://sat6.example.com/katello/api/v2/systems/?fields=full

brickdixon

# spacewalk-sql -i

   select * from rhnpackage [tabtab] where path like '%spacewalk-backend-2.3.3-49%';

brickdixon

# cp /var/log/audit/audit.log  /var/log/audit/backup-audit.log

# echo "" > /var/log/audit/audit.log     # make the existing audit log blank

Run puppet-agent -t command on any client and confirm that denied entries are generated in audit.log  

Now run below commands in sequence: 

# audit2allow -w -a 

brickdixon

Use python to set up0 a quick web server to serve current directory

   # python3 -m http.server 9876