This contains the best practices for a leapp activity to be performed on a Satellite or Capsule 6.11 server

leapp_best_practices_satellite_capsule.txt

** LEAPP Best Practices from RHEL Point of view: https://access.redhat.com/articles/7012979 

A) Ensure /var/lib/pgsql/data is empty 

B) The postgresql data

Satellite\Capsule 6.11 on RHEL 7 -> postgres data resides on /var/opt/rh/rh-postgresql12/

Satellite\Capsule 6.11 on RHEL 8 -> postgres data resides on /var/lib/pgsql

Since leapp will try to migrate the data from /var/opt/rh/rh-postgresql12/ to /var/lib/pgsql , The LV where /var/lib/pgsql resides, should have enough free space to hold the data from /var/opt/rh/rh-postgresql12/.

And keep in mind, /var/lib/pgsql will become the primary location where postgresql data will reside going forward. 


C) Be mindful about both of the situations explained in https://access.redhat.com/solutions/5057391 ( specifically the XFS ftype=0 case )

   Disk space is a sensitive area for leapp. So you must ensure that all expected filesystems have the expected amount of free space available. 

D) The concerned satellite or capsule, should not have any unexpected repos enabled while they are on RHEL 7 environment and being prepared for a leapp activity. 

   --> For Capsules, Ensure that along with el8 based capsule repos, RHEL 8.8 based AppStream and BaseOS repos are enabled , synced and made available to the capsules 
   
   
E) Be causious about setting noexec mount options on /dev/shm or /tmp/ or /var/<sub-directories> .

F) No Antivirus softwares\Malware services should be installed or remain active on the Satellite\Capsule prior to leapp activity. If you found any, stop disable and mask their services. If possible, remove those softwares completely. 

G) A simple thing like "having immutable attribute on /etc/services" can fail the whole leapp conversion process. So prior to executing leapp related commands, Ensure to check this :

   # for i in `rpm -ql setup`; do lsattr $i; done
   
   And none of the files should have any file attributes set. 
   
   
E) Once leapp upgrade + reboot has been completed, Leapp will still execute the following at the backend i.e. 

   * Postgresql data migration
   * satellite-installer 
   * Reindex on all postgersql databases.
   
   So monitor that activity using "journalctl -u leapp_resume -f" command and wait untill, you see that leapp has been successfully completed. 
   

F) Once everything is done and all services are up and running :

   * Unset the release version : sub-man release --unset
   * Complete the leapp post-upgrade steps. ( https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/upgrading_from_rhel_7_to_rhel_8/performing-post-upgrade-tasks-rhel-7-to-rhel-8_upgrading-from-rhel-7-to-rhel-8 ) 
   * If the selinux was in enforcing or permissive mode prior to upgrade, then 
   
     -> No actions to be taken on capsules
     -> On satellite, you will need to run "# dnf reinstall foreman-selinux katello-selinux --disableplugin=foreman-protector" followed by a restart of all services. And then only the selinux can be set to Enforcing. 
   
   * If selinux was in disabled state, then no actions to be taken.
   
   
G) In case of any issues during the early stages of reboot, immediately after "leapp upgrade", and You decide to boot back with old el7 kernel, then 

   * The concerned Satellite\Capsule will lose access to product certs and repositories. 
   * You have to re-create them from the content of these files :
   
      For satellite: https://gist.github.com/sayan3296/9666021d779dec9528cfdf26eb291adf
      For capsule: https://gist.github.com/sayan3296/10d28c4692f54983af75d78a829497c4
      
   * Then only, It's possible to re-enable the required repos and perform basic investigations.
   

F) The /var/log/leapp content is your friend. In case of any problems with leapp, Refer to the logs stored in their to understand the current issue and seek help from Red Hat Support to diagnose and fix them, before retrying the leapp activity.

G) For any reason if after leapp + reboot, The system is stuck in emegency mode then 

   * COllect a screenshot of whatever you can see in the console\terminal of the system 
   * Collect the rdsosreport from the emergency mode : https://access.redhat.com/solutions/2126311 
   * Collect the /var/log/leapp data after booting back up with working kernel


NOTE: A Healthy VM Snapshot or Full Backup of your satellite\capsule is your best friend during an outage situation. So make sure to keep any one of the options available with you, prior starting any upgrade activity.