ikurni

# It's not recommended to change the active running OCP Master nodes IP Properties unless it's really required to be done.
# To do that, please follow below guide :

# 1. Update the DNS server to use new IP for the target master nodes
# 2. Change the node IP Address from "nmcli" or "nmtui", and reboot the server
# 3. Post reboot, the existing will act normally, but somehow, etcd is not in healthy state. Re-add the etcd member will solve
# 4. Check which etcd pods is CLBO 

oc project openshift-etcd
oc get pods -l k8s-app=etcd

ikurni

### Create file called machine-config-daemon-force in /run ###
ssh core@hostname.local sudo touch /run/machine-config-daemon-force

### Edit node annotations ###
oc edit node <node-name>

### Check Annotations, change like below sample ###
 machineconfiguration.openshift.io/currentConfig: rendered-worker-ab4a1e7216bf3da2a5203f09c871b456
 machineconfiguration.openshift.io/desiredConfig: rendered-worker-ab4a1e7216bf3da2a5203f09c871b456
 machineconfiguration.openshift.io/reason: ""

ikurni

###Data Collection for Red Hat Threescale
$ oc adm must-gather --image=quay.io/3scale/must-gather

###Data Collection Audit logs
$ oc adm must-gather -- /usr/bin/gather_audit_logs

#It's also possible to generate the default must-gather including the audit logs:
$ oc adm must-gather -- '/usr/bin/gather && /usr/bin/gather_audit_logs'
#Also see https://access.redhat.com/solutions/5373481

ikurni

### Install few required packages to run SNX
sudo dnf install -y java-1.8.0-openjdk.x86_64 icedtea-web.x86_64 libstdc++.i686 libX11.i686 libpamtest.i686 libnsl.i686

### Download compat-libstdc++ driver and install it
wget http://mirror.centos.org/centos/7/os/x86_64/Packages/compat-libstdc++-33-3.2.3-72.el7.i686.rpm
sudo dnf -y install compat-libstdc++-33-3.2.3-72.el7.i686.rpm

### Install snx_linux.sh
### Download snx_linux_30.sh file from Checkpoint
### Active URL : https://supportcenter.checkpoint.com/supportcenter/portal/user/anon/page/default.psml/media-type/html?action=portlets.DCFileAction&eventSubmit_doGetdcdetails&fileid=22824

ikurni

Red Hat Internal KB : 
https://access.redhat.com/solutions/7023254?band=se&seSessionId=522ca7b0-a002-43d3-a5d2-9a7e387186c0&seSource=Recommendation&seResourceOriginID=09abffc5-23dc-465e-a013-9e52e91306cf

Openshift 4.8 below doesn't have auto rotation for etcd certificate and it will be expired within 3 years

High Level steps :
1) Copy etcd-signer and etcd-metric-signer CA from etcd DB to some folder, create the crt and key file for each signer
2) Backup all manifests for static pods in each master
3) Backup all existing etcd certificates inside /etc/kubernetes/static-pod-resources/etcd-certs/secrets/
4) Create new folder inside above folder ie. call as "all-certs", copy all certificate inside secrets folder to the "all-certs" folder

ikurni

###The workaround consist on modifying the settings to accept the internal hostname. The steps are the following:

###Copy the settings.yml file from system-app:
oc rsh -c system-provider dc/system-app cat /opt/system/config/settings.yml > settings.yml

###Edit the field apicast_internal_host_regexp from
#    '\Asystem-(master|provider|developer)\Z'
#    to
#    '\Asystem-(master|developer|provider)(\.(.*)(\.svc\.cluster\.local))?\Z'

ikurni

### Openshift Summarized KB ###
https://access.redhat.com/node/5218861

### PipelineRun takes a lot of time to get started on OpenShift Container Platform 4 ###
https://access.redhat.com/solutions/7005157

### Cluster Updates Without Error but Machine Config Pools Degraded with `Marking Degraded due to: unexpected on-disk state` on 4.6 and newer
https://access.redhat.com/solutions/5598401

### How to Use 'fio' to Check Etcd Disk Performance in OCP

ikurni

### Removal from RHV Manager UI is not possible anymore, remove button has been greyed out
# SSH to RHV Manager hosts
su - postgres
psql -d engine
SELECT vds_id FROM vds_static WHERE vds_name = 'hostname of the unresponsive hosts';
# Capture the vds_id
DELETE FROM vds_dynamic WHERE vds_id = 'vds_id of the unresponsive hosts';
DELETE FROM vds_statistics WHERE vds_id = 'vds_id of the unresponsive hosts';
DELETE FROM vds_static WHERE vds_name = 'hostname of the unresponsive hosts';

ikurni

### List down all API that in False state


[root@bastion-1 ~]# oc get apiservices | grep -i false
v1alpha1.mutators.kubedb.com                  kubedb/kubedb-kubedb-community                               False (ServiceNotFound)   494d
v1alpha1.validators.kubedb.com                kubedb/kubedb-kubedb-community                               False (ServiceNotFound)   494d

### Simply remove the not responding API
oc delete apiservices v1alpha1.mutators.kubedb.com
oc delete apiservices v1alpha1.validators.kubedb.com

ikurni

##Got podman issue when run podman command :
##Error: could not get runtime: kernel does not support overlay fs: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior. Reformat the filesystem with ftype=1 to enable d_type support. Running without d_type is not supported.: driver not supported
##Reinstall podman and containers-common does not work
##Edit /etc/containers/storage.conf
vi /etc/containers/storage.conf

##Find storage.options and make sure below line is in there
[storage.options]
mount_program = "/usr/bin/fuse-overlayfs"