Script to recreate the issue from KB https://access.redhat.com/solutions/6809431 and https://access.redhat.com/solutions/6826971

breakfix_pulp3_404.sh

## Assuming that, this is a fresh satellite 6.10 with a subscription manifest imported ( having a valid RHEL sub ) and no repos are enabled at present
## Also assuming that the organization name is RedHat or else replace "--organization RedHat" with "--organization-id 1"

## Setup the repo and CV in satellite

hammer repository-set enable --name "Red Hat Satellite Tools 6.9 (for RHEL 7 Server) (RPMs)" --product "Red Hat Enterprise Linux Server" --basearch x86_64 --releasever 7Server --organization RedHat

hammer repository-set enable --name "Red Hat Ansible Engine 2.9 RPMs for Red Hat Enterprise Linux 7 Server" --product "Red Hat Ansible Engine" --basearch x86_64 --organization RedHat

hammer repository update --name "Red Hat Ansible Engine 2.9 RPMs for Red Hat Enterprise Linux 7 Server x86_64"  --product "Red Hat Ansible Engine" --organization RedHat --download-policy immediate

hammer repository synchronize --name "Red Hat Ansible Engine 2.9 RPMs for Red Hat Enterprise Linux 7 Server x86_64" --product "Red Hat Ansible Engine" --organization RedHat

hammer repository synchronize --name "Red Hat Satellite Tools 6.9 for RHEL 7 Server RPMs x86_64" --product "Red Hat Enterprise Linux Server" --organization RedHat

REPO_IDS=$(hammer --no-headers repository list --organization RedHat --fields Id | xargs | tr " " ",")

hammer lifecycle-environment create --name QA --prior Library --organization RedHat

hammer content-view create --name TestCV --organization RedHat --repository-ids $REPO_IDS

hammer content-view publish --name TestCV --organization RedHat

hammer content-view version promote --from-lifecycle-environment Library --to-lifecycle-environment QA --content-view TestCV --organization RedHat

hammer content-view publish --name TestCV --organization RedHat

hammer content-view version promote --from-lifecycle-environment Library --to-lifecycle-environment QA --content-view TestCV --organization RedHat



## Start breaking the pulp 

su - postgres -c "psql -x foreman -c \"select main.id,main.root_id,root.name,main.pulp_id,root.http_proxy_id,main.library_instance_id,root.url,main.relative_path,main.remote_href,main.publication_href,main.version_href from katello_root_repositories root left JOIN katello_repositories main on main.root_id=root.id where main.library_instance_id is null;\"" &> /tmp/library_repos.txt


su - postgres -c "psql -x foreman -c \"select main.id,main.root_id,root.name,main.pulp_id,root.http_proxy_id,main.library_instance_id,root.url,main.relative_path,main.remote_href,main.publication_href,main.version_href from katello_root_repositories root left JOIN katello_repositories main on main.root_id=root.id where main.library_instance_id is not null;\"" &>  /tmp/non_library_repos.txt


for repo in `cat /tmp/library_repos.txt | egrep "^pub" | awk -F'|' '{print $NF}' | sort -u`; do echo -e "\nDeleting $repo :\n"; curl -X DELETE -H "Content-Type: application/json" --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key   https://`hostname -f`$repo ; echo -e "\n" ; done

for repo in `cat  /tmp/non_library_repos.txt | egrep "^version" | awk -F'|' '{print $NF}' | sort -u`; 
do 
URL1=$repo
URL2=$( echo $repo | sed 's/..$/0\//')
echo -e "\nDeleting $URL1 :\n";
curl -X DELETE -H "Content-Type: application/json" --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key   https://`hostname -f`$URL1 ;
echo -e "\n" ;
echo -e "\nDeleting $URL2 :\n";
curl -X DELETE -H "Content-Type: application/json" --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key   https://`hostname -f`$URL2 ;
echo -e "\n" ;
done

for repo in `cat  /tmp/non_library_repos.txt | egrep "^version" | awk -F'|' '{print $NF}' | sort -u`; 
do 
URL=$( echo $repo | sed 's/...........$//')
echo -e "\nDeleting $URL :\n";
curl -X DELETE -H "Content-Type: application/json" --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key   https://`hostname -f`$URL ;
echo -e "\n" ;
done


## Delete the temp files

rm -f /tmp/non_library_repos.txt /tmp/library_repos.txt


## Try publishing a new version of CV

hammer content-view publish --name TestCV --organization RedHat

## Try another repo sync with --async

hammer repository synchronize --name "Red Hat Satellite Tools 6.9 for RHEL 7 Server RPMs x86_64" --product "Red Hat Enterprise Linux Server" --organization RedHat --async

## Wait for 6 minutes and then Try a manifest refresh with --async

sleep 360

hammer subscription refresh-manifest --organization RedHat --async

echo "Done"

Errors:

Attempt to publish a new content view version fails with the following error:

Error message: the server returns an error
HTTP status code: 404
Response headers: {"Date"=>"Mon, 10 Oct 2022 16:12:41 GMT", "Server"=>"gunicorn", "Content-Type"=>"application/json", "Vary"=>"Accept,Cookie", "Allow"=>"GET, PUT, PATCH, DELETE, HEAD, OPTIONS", "X-Frame-Options"=>"SAMEORIGIN", "Content-Length"=>"23", "Correlation-ID"=>"daa5acd6-5aba-42b7-890c-2ff0df5871c4", "Access-Control-Expose-Headers"=>"Correlation-ID", "Via"=>"1.1 satellite.example.com"}
Response body: {"detail":"Not found."}Error message: the server returns an error
HTTP status code: 404
Response headers: {"Date"=>"Mon, 10 Oct 2022 16:12:41 GMT", "Server"=>"gunicorn", "Content-Type"=>"application/json", "Vary"=>"Accept,Cookie", "Allow"=>"GET, PUT, PATCH, DELETE, HEAD, OPTIONS", "X-Frame-Options"=>"SAMEORIGIN", "Content-Length"=>"23", "Correlation-ID"=>"daa5acd6-5aba-42b7-890c-2ff0df5871c4", "Access-Control-Expose-Headers"=>"Correlation-ID", "Via"=>"1.1 satellite.example.com"}
Response body: {"detail":"Not found."}

Attempt to sync a repo fails with this error:

Error message: the server returns an error
HTTP status code: 404
Response headers: {"Date"=>"Mon, 10 Oct 2022 16:12:49 GMT", "Server"=>"gunicorn", "Content-Type"=>"application/json", "Vary"=>"Accept,Cookie", "Allow"=>"POST, OPTIONS", "X-Frame-Options"=>"SAMEORIGIN", "Content-Length"=>"23", "Correlation-ID"=>"c8510b9d-24aa-4ec7-9a2a-c69a9a01b2a2", "Access-Control-Expose-Headers"=>"Correlation-ID", "Via"=>"1.1 satellite.example.com"}
Response body: {"detail":"Not found."}Error message: the server returns an error
HTTP status code: 400
Response headers: {"Date"=>"Mon, 10 Oct 2022 16:12:50 GMT", "Server"=>"gunicorn", "Content-Type"=>"application/json", "Vary"=>"Accept,Cookie", "Allow"=>"GET, PUT, PATCH, DELETE, HEAD, OPTIONS", "X-Frame-Options"=>"SAMEORIGIN", "Content-Length"=>"62", "Correlation-ID"=>"c8510b9d-24aa-4ec7-9a2a-c69a9a01b2a2", "Access-Control-Expose-Headers"=>"Correlation-ID", "Via"=>"1.1 satellite.example.com", "Connection"=>"close"}
Response body: {"publication":["Invalid hyperlink - Object does not exist."]}

Attempt to refresh manifest fails with a similar error.

Solution:

• Clear the paused or pending tasks related to repo\cv\manifest.

• Re-create the missing repos in Pulp: [ The command has to finish successfully ]

# foreman-rake katello:correct_repositories COMMIT=true --trace

• Check in satellite UI and hopefully, the missing repos have been recreated but the related metadata regenerate tasks would be in paused state. So clear them again.

• Re-create the publication and the metadata forcefully by following https://access.redhat.com/solutions/6826971.

• Now, the repos should be able to sync and CVs can be republished and the manifest can be refreshed.

For paused metadata task clearing:

# foreman-rake foreman_tasks:cleanup TASK_SEARCH='label = Actions::Katello::Repository::MetadataGenerate' STATES='paused' VERBOSE=true

Alternate of the 4th step mentioned above:

# echo -e '
    ::User.current = ::User.anonymous_admin  
    missing = 0
    tasks = []
    Katello::Repository.all.where(library_instance: nil).select(&:yum?).each do |repo|
      service = repo.backend_service(SmartProxy.pulp_primary)
      p "Checking repository id #{repo.id} (#{repo.relative_path})"
      begin
        service.api.publications_api.read(repo.publication_href)
      rescue PulpRpmClient::ApiError => e
        if e.code == 404
          missing += 1
          p "  - Publication '#{repo.publication_href}' is missing in Pulp. Fixing it now..."
          if 
            tasks << ForemanTasks.async_task(Actions::Katello::Repository::MetadataGenerate, repo, force_publication: true)
          end
        else
          p "Unexpected error: HTTP status: #{e.code}, #{e.message}"
        end
      end
    end
' | foreman-rake console