Skip to content

Instantly share code, notes, and snippets.

@ikurni
ikurni / ocp-must-gather-per-product
Last active December 14, 2023 10:04
Red Hat Openshift Must Gather for Specific Products
View ocp-must-gather-per-product
###Data Collection for Red Hat Threescale
$ oc adm must-gather --image=quay.io/3scale/must-gather
###Data Collection Audit logs
$ oc adm must-gather -- /usr/bin/gather_audit_logs
#It's also possible to generate the default must-gather including the audit logs:
$ oc adm must-gather -- '/usr/bin/gather && /usr/bin/gather_audit_logs'
#Also see https://access.redhat.com/solutions/5373481
@ikurni
ikurni / renew-etcd-certs-manually
Last active November 26, 2023 13:46
Renew Expired ETCD Certificater for OCP 4.8 Below
View renew-etcd-certs-manually
Red Hat Internal KB :
https://access.redhat.com/solutions/7023254?band=se&seSessionId=522ca7b0-a002-43d3-a5d2-9a7e387186c0&seSource=Recommendation&seResourceOriginID=09abffc5-23dc-465e-a013-9e52e91306cf
Openshift 4.8 below doesn't have auto rotation for etcd certificate and it will be expired within 3 years
High Level steps :
1) Copy etcd-signer and etcd-metric-signer CA from etcd DB to some folder, create the crt and key file for each signer
2) Backup all manifests for static pods in each master
3) Backup all existing etcd certificates inside /etc/kubernetes/static-pod-resources/etcd-certs/secrets/
4) Create new folder inside above folder ie. call as "all-certs", copy all certificate inside secrets folder to the "all-certs" folder
@ikurni
ikurni / Red-hat-3scale-dns-lookup-slowdown
Created November 3, 2023 03:48
Red Hat 3scale DNS lookup Slowdown
View Red-hat-3scale-dns-lookup-slowdown
###The workaround consist on modifying the settings to accept the internal hostname. The steps are the following:
###Copy the settings.yml file from system-app:
oc rsh -c system-provider dc/system-app cat /opt/system/config/settings.yml > settings.yml
###Edit the field apicast_internal_host_regexp from
# '\Asystem-(master|provider|developer)\Z'
# to
# '\Asystem-(master|developer|provider)(\.(.*)(\.svc\.cluster\.local))?\Z'
@ikurni
ikurni / change-active-ocp-master-IP-address
Created October 2, 2023 07:05
Change Active Openshift Master Node IP Address
View change-active-ocp-master-IP-address
# It's not recommended to change the active running OCP Master nodes IP Properties unless it's really required to be done.
# To do that, please follow below guide :
# 1. Update the DNS server to use new IP for the target master nodes
# 2. Change the node IP Address from "nmcli" or "nmtui", and reboot the server
# 3. Post reboot, the existing will act normally, but somehow, etcd is not in healthy state. Re-add the etcd member will solve
# 4. Check which etcd pods is CLBO
oc project openshift-etcd
oc get pods -l k8s-app=etcd
@ikurni
ikurni / force-remove-rhv-hosts-from-database
Created July 18, 2023 08:10
Force Remove RHV Hypervisor Hosts from Database
View force-remove-rhv-hosts-from-database
### Removal from RHV Manager UI is not possible anymore, remove button has been greyed out
# SSH to RHV Manager hosts
su - postgres
psql -d engine
SELECT vds_id FROM vds_static WHERE vds_name = 'hostname of the unresponsive hosts';
# Capture the vds_id
DELETE FROM vds_dynamic WHERE vds_id = 'vds_id of the unresponsive hosts';
DELETE FROM vds_statistics WHERE vds_id = 'vds_id of the unresponsive hosts';
DELETE FROM vds_static WHERE vds_name = 'hostname of the unresponsive hosts';
@ikurni
ikurni / force-remove-not-responding-API
Created June 12, 2023 15:08
Force Remove Not Responding API in Openshift
View force-remove-not-responding-API
### List down all API that in False state
[root@bastion-1 ~]# oc get apiservices | grep -i false
v1alpha1.mutators.kubedb.com kubedb/kubedb-kubedb-community False (ServiceNotFound) 494d
v1alpha1.validators.kubedb.com kubedb/kubedb-kubedb-community False (ServiceNotFound) 494d
### Simply remove the not responding API
oc delete apiservices v1alpha1.mutators.kubedb.com
oc delete apiservices v1alpha1.validators.kubedb.com
@ikurni
ikurni / OCP4-Ingress-Certificate-Expired
Last active April 13, 2022 15:57
OCP 4 Ingress Certificate Expired after 2 Years of Deployment
View OCP4-Ingress-Certificate-Expired
# After 2 years deployment, suddenly Authentication and Console Operator is degraded, and after we check the operator status, it shows that
# the certificate is expired. To solve this, just delete the secret, and Operator will create new certificate. Assume we still use
# Openshift default certificate
### Delete Ingress CA Secret
oc project openshift-ingress-operator
oc get secret router-ca -oyaml > router-ca.yaml
oc delete secret router-ca
oc delete pod --all
oc get secret router-ca
@ikurni
ikurni / rhv-important-kb
Last active July 30, 2021 17:37
RHV Important KB
View rhv-important-kb
https://access.redhat.com/solutions/396753 --> Remove Locked Entities Manually
##How to Clean up Task in RHV :
#List Tasks :
/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh
#Kill Tasks :
/usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh -t <taskID>
@ikurni
ikurni / force-bootstrap-machine-config
Created July 24, 2021 07:28
Force Re-run Openshift First Bootstrap
View force-bootstrap-machine-config
### If nodes is stuck in pulling images, or connection are too slow, sometime process can't move. Below is how to re-trigger ###
/run/bin/machine-config-daemon firstboot-complete-machineconfig
@ikurni
ikurni / openshift-important-kb
Last active August 25, 2023 00:28
Bookmarked Red Hat KB for Openshift
View openshift-important-kb
### Openshift Summarized KB ###
https://access.redhat.com/node/5218861
### PipelineRun takes a lot of time to get started on OpenShift Container Platform 4 ###
https://access.redhat.com/solutions/7005157
### Cluster Updates Without Error but Machine Config Pools Degraded with `Marking Degraded due to: unexpected on-disk state` on 4.6 and newer
https://access.redhat.com/solutions/5598401
### How to Use 'fio' to Check Etcd Disk Performance in OCP