Skip to content

Instantly share code, notes, and snippets.

@zaki-lknr
Last active March 1, 2020 22:22
Show Gist options
  • Save zaki-lknr/4da990055c2d81cf06194f40181b6adb to your computer and use it in GitHub Desktop.
Save zaki-lknr/4da990055c2d81cf06194f40181b6adb to your computer and use it in GitHub Desktop.
[zaki@okd-master ~]$ oc get node
NAME                      STATUS    ROLES          AGE       VERSION
okd-master.esxi.jp-z.jp   Ready     infra,master   13d       v1.11.0+d4cacc0
okd-node1.esxi.jp-z.jp    Ready     compute        13d       v1.11.0+d4cacc0
okd-node2.esxi.jp-z.jp    Ready     compute        13d       v1.11.0+d4cacc0
[zaki@okd-master ~]$ oc version
oc v3.11.0+62803d0-1
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://okd-master.esxi.jp-z.jp:8443
openshift v3.11.0+12b2f30-375
kubernetes v1.11.0+d4cacc0
[zaki@okd-manager ~]$ cd openshift-ansible/
[zaki@okd-manager openshift-ansible]$ git branch 
  master
* release-3.11
[zaki@okd-manager openshift-ansible]$ git pull 
remote: Enumerating objects: 290, done.
remote: Counting objects: 100% (290/290), done.
remote: Compressing objects: 100% (48/48), done.
remote: Total 455 (delta 263), reused 260 (delta 237), pack-reused 165
Receiving objects: 100% (455/455), 204.86 KiB | 0 bytes/s, done.
Resolving deltas: 100% (278/278), completed with 89 local objects.
From https://github.com/openshift/openshift-ansible
   376158f..77c3157  release-3.11 -> origin/release-3.11
   91645ed..6802c0c  master     -> origin/master
   705294a..f6f6784  release-4.3 -> origin/release-4.3
   91645ed..453a80a  release-4.4 -> origin/release-4.4
   91645ed..6802c0c  release-4.5 -> origin/release-4.5
 * [new branch]      release-4.6 -> origin/release-4.6
 * [new branch]      sosiouxme-patch-2 -> origin/sosiouxme-patch-2
 * [new tag]         openshift-ansible-3.11.179-1 -> openshift-ansible-3.11.179-1
 * [new tag]         openshift-ansible-4.4.0-202002282323 -> openshift-ansible-4.4.0-202002282323
From https://github.com/openshift/openshift-ansible
 * [new tag]         openshift-ansible-3.11.162-1 -> openshift-ansible-3.11.162-1
 * [new tag]         openshift-ansible-3.11.163-1 -> openshift-ansible-3.11.163-1
 * [new tag]         openshift-ansible-3.11.164-1 -> openshift-ansible-3.11.164-1
 * [new tag]         openshift-ansible-3.11.165-1 -> openshift-ansible-3.11.165-1
 * [new tag]         openshift-ansible-3.11.166-1 -> openshift-ansible-3.11.166-1
 * [new tag]         openshift-ansible-3.11.167-1 -> openshift-ansible-3.11.167-1
 * [new tag]         openshift-ansible-3.11.168-1 -> openshift-ansible-3.11.168-1
 * [new tag]         openshift-ansible-3.11.169-1 -> openshift-ansible-3.11.169-1
 * [new tag]         openshift-ansible-3.11.170-1 -> openshift-ansible-3.11.170-1
 * [new tag]         openshift-ansible-3.11.171-1 -> openshift-ansible-3.11.171-1
 * [new tag]         openshift-ansible-3.11.172-1 -> openshift-ansible-3.11.172-1
 * [new tag]         openshift-ansible-3.11.173-1 -> openshift-ansible-3.11.173-1
 * [new tag]         openshift-ansible-3.11.174-1 -> openshift-ansible-3.11.174-1
 * [new tag]         openshift-ansible-3.11.175-1 -> openshift-ansible-3.11.175-1
 * [new tag]         openshift-ansible-3.11.176-1 -> openshift-ansible-3.11.176-1
 * [new tag]         openshift-ansible-3.11.177-1 -> openshift-ansible-3.11.177-1
 * [new tag]         openshift-ansible-3.11.178-1 -> openshift-ansible-3.11.178-1
Updating 376158f..77c3157
Fast-forward
 .tito/packages/openshift-ansible                                                    |    2 +-
 images/installer/Dockerfile                                                         |    2 +-
 images/installer/container.yaml                                                     |   13 -
 openshift-ansible.spec                                                              |   82 +-
 playbooks/common/openshift-cluster/upgrades/init.yml                                |    3 +
 playbooks/openshift-master/private/additional_config.yml                            |   12 +-
 playbooks/openshift-monitoring/OWNERS                                               |   16 +-
 playbooks/ovirt/openshift-cluster/ovirt-vm-uninstall.yml                            |    2 +-
 requirements.txt                                                                    |    2 +-
 roles/etcd/defaults/main.yaml                                                       |    2 +-
 roles/lib_utils/library/openshift_cert_expiry.py                                    |   13 +-
 roles/metrics_server/files/metrics-server-aggregated-metrics-reader.yaml            |    3 +-
 roles/openshift_cluster_monitoring_operator/OWNERS                                  |   16 +-
 roles/openshift_cluster_monitoring_operator/defaults/main.yml                       |    5 +
 .../templates/cluster-monitoring-operator-config.j2                                 |   17 +
 roles/openshift_examples/examples-sync.sh                                           |    8 +-
 .../files/examples/x86_64/xpaas-streams/rhdm76-image-streams.yaml                   |   71 ++
 .../files/examples/x86_64/xpaas-streams/rhpam76-image-streams.yaml                  |  134 +++
 .../files/examples/x86_64/xpaas-templates/rhdm76-authoring-ha.yaml                  | 1393 ++++++++++++++++++++++++++++
 .../openshift_examples/files/examples/x86_64/xpaas-templates/rhdm76-authoring.yaml  |  986 ++++++++++++++++++++
 .../openshift_examples/files/examples/x86_64/xpaas-templates/rhdm76-kieserver.yaml  |  665 +++++++++++++
 .../files/examples/x86_64/xpaas-templates/rhdm76-prod-immutable-kieserver-amq.yaml  | 1128 +++++++++++++++++++++++
 .../files/examples/x86_64/xpaas-templates/rhdm76-prod-immutable-kieserver.yaml      |  742 +++++++++++++++
 .../files/examples/x86_64/xpaas-templates/rhdm76-trial-ephemeral.yaml               |  836 +++++++++++++++++
 .../files/examples/x86_64/xpaas-templates/rhpam76-authoring-ha.yaml                 | 1583 +++++++++++++++++++++++++++++++
 .../openshift_examples/files/examples/x86_64/xpaas-templates/rhpam76-authoring.yaml | 1063 +++++++++++++++++++++
 .../files/examples/x86_64/xpaas-templates/rhpam76-kieserver-externaldb.yaml         |  874 ++++++++++++++++++
 .../files/examples/x86_64/xpaas-templates/rhpam76-kieserver-mysql.yaml              |  861 +++++++++++++++++
 .../files/examples/x86_64/xpaas-templates/rhpam76-kieserver-postgresql.yaml         |  866 +++++++++++++++++
 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhpam76-managed.yaml | 1244 +++++++++++++++++++++++++
 .../files/examples/x86_64/xpaas-templates/rhpam76-prod-immutable-kieserver-amq.yaml | 1369 +++++++++++++++++++++++++++
 .../files/examples/x86_64/xpaas-templates/rhpam76-prod-immutable-kieserver.yaml     |  941 +++++++++++++++++++
 .../files/examples/x86_64/xpaas-templates/rhpam76-prod-immutable-monitor.yaml       |  893 ++++++++++++++++++
 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhpam76-prod.yaml    | 1946 +++++++++++++++++++++++++++++++++++++++
 .../files/examples/x86_64/xpaas-templates/rhpam76-trial-ephemeral.yaml              |  836 +++++++++++++++++
 roles/openshift_logging_elasticsearch/tasks/restart_cluster.yml                     |    2 +-
 roles/openshift_storage_glusterfs/OWNERS                                            |    8 +-
 roles/openshift_storage_glusterfs/defaults/main.yml                                 |    2 +
 roles/openshift_storage_glusterfs/files/deploy-heketi-template.yml                  |    6 +
 roles/openshift_storage_glusterfs/files/glusterfs-template.yml                      |    2 +-
 .../{templates/heketi-template.yml.j2 => files/heketi-template.yml}                 |   10 +-
 roles/openshift_storage_glusterfs/tasks/glusterfs_config_facts.yml                  |    1 +
 roles/openshift_storage_glusterfs/tasks/glusterfs_registry_facts.yml                |    1 +
 roles/openshift_storage_glusterfs/tasks/heketi_deploy.yml                           |    5 +-
 roles/openshift_storage_glusterfs/tasks/heketi_init_deploy.yml                      |   11 +-
 45 files changed, 18614 insertions(+), 63 deletions(-)
 delete mode 100644 images/installer/container.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-streams/rhdm76-image-streams.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-streams/rhpam76-image-streams.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhdm76-authoring-ha.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhdm76-authoring.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhdm76-kieserver.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhdm76-prod-immutable-kieserver-amq.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhdm76-prod-immutable-kieserver.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhdm76-trial-ephemeral.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhpam76-authoring-ha.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhpam76-authoring.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhpam76-kieserver-externaldb.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhpam76-kieserver-mysql.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhpam76-kieserver-postgresql.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhpam76-managed.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhpam76-prod-immutable-kieserver-amq.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhpam76-prod-immutable-kieserver.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhpam76-prod-immutable-monitor.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhpam76-prod.yaml
 create mode 100644 roles/openshift_examples/files/examples/x86_64/xpaas-templates/rhpam76-trial-ephemeral.yaml
 rename roles/openshift_storage_glusterfs/{templates/heketi-template.yml.j2 => files/heketi-template.yml} (95%)
[zaki@okd-manager openshift-ansible]$ 
[zaki@okd-manager openshift-ansible]$ git log --oneline --graph --decorate | head -10
* 77c3157 (HEAD, tag: openshift-ansible-3.11.179-1, origin/release-3.11, release-3.11) Automatic commit of package [openshift-ansible] release [3.11.179-1].
* 8de47bf (tag: openshift-ansible-3.11.178-1) Automatic commit of package [openshift-ansible] release [3.11.178-1].
* 8e8b1ef (tag: openshift-ansible-3.11.177-1) Automatic commit of package [openshift-ansible] release [3.11.177-1].
* abb9886 (tag: openshift-ansible-3.11.176-1) Automatic commit of package [openshift-ansible] release [3.11.176-1].
*   f2d20d2 Merge pull request #11967 from jcpowermac/fix-1754081
|\  
| * a815785 add any_errors_fatal
| * 506a437 set openshift_certificate_expiry_warning_days to 6 months vs 365 days
| * 82bb831 Certificate expiration misses node certificates
* |   0e08b32 Merge pull request #12091 from openshift/sosiouxme-patch-2

インベントリは変更なしでアップグレード開始

[zaki@okd-manager openshift-ansible]$ 
[zaki@okd-manager openshift-ansible]$ time ansible-playbook -i ~/ocpv3/install-inventory/hosts.example playbooks/byo/openshift-cluster/upgrades/v3_11/upgrade.yml
TASK [set_fact] ********************************************************************************************************************
Sunday 01 March 2020  15:18:24 +0900 (0:00:00.258)       0:02:31.246 ********** 
fatal: [okd-master.esxi.jp-z.jp]: FAILED! => {"msg": "|missing OpenSSL python bindings"}

PLAY RECAP *************************************************************************************************************************
localhost                  : ok=24   changed=0    unreachable=0    failed=0   
okd-master.esxi.jp-z.jp    : ok=210  changed=20   unreachable=0    failed=1   
okd-node1.esxi.jp-z.jp     : ok=16   changed=2    unreachable=0    failed=0   
okd-node2.esxi.jp-z.jp     : ok=16   changed=2    unreachable=0    failed=0   


INSTALLER STATUS *******************************************************************************************************************
Initialization  : Complete (0:00:17)
Sunday 01 March 2020  15:18:24 +0900 (0:00:00.018)       0:02:31.265 ********** 
=============================================================================== 
Run health checks (upgrade) ------------------------------------------------------------------------------------------------ 16.76s
Wait for node to be ready -------------------------------------------------------------------------------------------------- 11.10s
openshift_excluder : Get available excluder version ------------------------------------------------------------------------- 6.01s
openshift_excluder : Install docker excluder - yum -------------------------------------------------------------------------- 4.39s
openshift_excluder : Get available excluder version ------------------------------------------------------------------------- 2.00s
Check latest available OpenShift RPM version -------------------------------------------------------------------------------- 1.91s
openshift_sdn : Copy templates to temp directory ---------------------------------------------------------------------------- 1.79s
Gathering Facts ------------------------------------------------------------------------------------------------------------- 1.51s
openshift_excluder : Install openshift excluder - yum ----------------------------------------------------------------------- 1.39s
Install registry_auth dependencies ------------------------------------------------------------------------------------------ 1.31s
metrics_server : slurp ------------------------------------------------------------------------------------------------------ 1.25s
Gathering Facts ------------------------------------------------------------------------------------------------------------- 1.19s
openshift_certificate_expiry : Check cert expirys on host ------------------------------------------------------------------- 1.02s
get openshift_current_version ----------------------------------------------------------------------------------------------- 0.99s
metrics_server : Applying /tmp/openshift-metrics-server-ansible-bxY3cB/templates/metrics-server-aggregated-metrics-reader-cluster-role.yaml --- 0.73s
restart node ---------------------------------------------------------------------------------------------------------------- 0.69s
metrics_server : Generate service for metrics-server ------------------------------------------------------------------------ 0.65s
openshift_sdn : Ensure the service account can run privileged --------------------------------------------------------------- 0.64s
metrics_server : generate metrics-server secret template -------------------------------------------------------------------- 0.63s
metrics_server : create aggregated-metrics-reader role for users to access the metrics API ---------------------------------- 0.62s


Failure summary:


  1. Hosts:    okd-master.esxi.jp-z.jp
     Play:     Examine etcd serving certificate SAN
     Task:     set_fact
     Message:  |missing OpenSSL python bindings

real    2m34.828s
user    0m56.353s
sys     0m11.219s

あっ、、

[zaki@okd-manager openshift-ansible]$ time ansible all -i ~/ocpv3/install-inventory/hosts.example -b -m yum -a "name='*' state=latest"

アップデートしておく

[zaki@okd-manager openshift-ansible]$ sudo yum install pyOpenSSL

再実行

TASK [openshift_control_plane : Wait for /apis/metrics.k8s.io/v1beta1 when registered] *********************************************
Sunday 01 March 2020  17:48:45 +0900 (0:00:00.377)       0:02:49.870 ********** 
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (60 retries left).
FAILED - RETRYING: Wait for /apis/metrics.k8s.io/v1beta1 when registered (59 retries left).
:
:

メッセージ見る限り↓かなぁ。
1720982 – [3.11] upgrade failed at task "Wait for /apis/metrics.k8s.io/v1beta1 when registered"

解決方法がちょっと漠然としてるけど、全ノードyum updateしてrebootしたらよさげ。


再実行

TASK [openshift_node : Copy node container image to ostree storage] ****************************************************************Sunday 01 March 2020  18:04:24 +0900 (0:00:00.232)       0:05:35.908 ********** 
skipping: [okd-master.esxi.jp-z.jp] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [openshift_node : Delete OpenShift SDN/OVS pods prior to upgrade] *************************************************************Sunday 01 March 2020  18:04:25 +0900 (0:00:00.170)       0:05:36.079 ********** 

このまま寝て起きても(12時間以上)止まっていた。

[zaki@okd-master ~]$ oc get pod --all-namespaces -o wide
NAMESPACE                           NAME                                           READY     STATUS             RESTARTS   AGE       IP             NODE                      NOMINATED NODE
default                             docker-registry-1-jc2c8                        0/1       Running            254        13d       10.128.1.246   okd-master.esxi.jp-z.jp   <none>
default                             registry-console-1-45x4s                       0/1       CrashLoopBackOff   220        13d       10.128.1.231   okd-master.esxi.jp-z.jp   <none>
default                             router-1-lb9jx                                 1/1       Running            7          13d       192.168.0.71   okd-master.esxi.jp-z.jp   <none>
kube-service-catalog                apiserver-qn6r4                                0/1       CrashLoopBackOff   174        13d       10.128.1.238   okd-master.esxi.jp-z.jp   <none>
kube-service-catalog                controller-manager-5nsm8                       0/1       CrashLoopBackOff   308        13d       10.128.1.249   okd-master.esxi.jp-z.jp   <none>
kube-system                         master-api-okd-master.esxi.jp-z.jp             1/1       Running            26         13d       192.168.0.71   okd-master.esxi.jp-z.jp   <none>
kube-system                         master-controllers-okd-master.esxi.jp-z.jp     1/1       Running            78         13d       192.168.0.71   okd-master.esxi.jp-z.jp   <none>
kube-system                         master-etcd-okd-master.esxi.jp-z.jp            1/1       Running            1          14h       192.168.0.71   okd-master.esxi.jp-z.jp   <none>
openshift-ansible-service-broker    asb-1-bplqs                                    0/1       CrashLoopBackOff   175        13d       10.128.1.232   okd-master.esxi.jp-z.jp   <none>
openshift-console                   console-fc74b9bcf-wn5z5                        0/1       Running            231        13d       10.128.1.248   okd-master.esxi.jp-z.jp   <none>
openshift-infra                     hawkular-cassandra-1-dcdvx                     1/1       Running            6          13d       10.130.0.82    okd-node2.esxi.jp-z.jp    <none>
openshift-infra                     hawkular-metrics-pxvsr                         1/1       Running            22         13d       10.129.0.96    okd-node1.esxi.jp-z.jp    <none>
openshift-infra                     hawkular-metrics-schema-8xkdh                  0/1       Completed          0          13d       10.129.0.7     okd-node1.esxi.jp-z.jp    <none>
openshift-infra                     heapster-tgk79                                 1/1       Running            22         13d       10.130.0.83    okd-node2.esxi.jp-z.jp    <none>
openshift-logging                   logging-curator-1583001000-cxgn4               0/1       Completed          0          16h       10.129.0.39    okd-node1.esxi.jp-z.jp    <none>
openshift-logging                   logging-curator-1583087400-bjcww               0/1       Error              0          3h        10.130.0.84    okd-node2.esxi.jp-z.jp    <none>
openshift-logging                   logging-es-data-master-ebs0kik7-1-knsf2        2/2       Running            12         13d       10.128.1.244   okd-master.esxi.jp-z.jp   <none>
openshift-logging                   logging-fluentd-fjc5q                          1/1       Running            151        13d       10.129.0.95    okd-node1.esxi.jp-z.jp    <none>
openshift-logging                   logging-fluentd-gpsrb                          0/1       CrashLoopBackOff   304        13d       10.128.1.240   okd-master.esxi.jp-z.jp   <none>
openshift-logging                   logging-fluentd-nw8gv                          1/1       Running            152        13d       10.130.0.81    okd-node2.esxi.jp-z.jp    <none>
openshift-logging                   logging-kibana-1-npv9k                         2/2       Running            12         13d       10.129.0.93    okd-node1.esxi.jp-z.jp    <none>
openshift-metrics-server            metrics-server-56cd9bfcf-tdv57                 1/1       Running            6          13d       10.128.1.237   okd-master.esxi.jp-z.jp   <none>
openshift-monitoring                alertmanager-main-0                            3/3       Running            18         13d       10.128.1.235   okd-master.esxi.jp-z.jp   <none>
openshift-monitoring                alertmanager-main-1                            3/3       Running            18         13d       10.128.1.233   okd-master.esxi.jp-z.jp   <none>
openshift-monitoring                alertmanager-main-2                            3/3       Running            18         13d       10.128.1.234   okd-master.esxi.jp-z.jp   <none>
openshift-monitoring                cluster-monitoring-operator-8578656f6f-t4jxt   1/1       Running            6          13d       10.128.1.239   okd-master.esxi.jp-z.jp   <none>
openshift-monitoring                grafana-6b9f85786f-8xmqq                       2/2       Running            12         13d       10.128.1.245   okd-master.esxi.jp-z.jp   <none>
openshift-monitoring                kube-state-metrics-c4f86b5f8-b8r6z             3/3       Running            18         13d       10.128.1.247   okd-master.esxi.jp-z.jp   <none>
openshift-monitoring                node-exporter-8wp7c                            2/2       Running            14         13d       192.168.0.76   okd-node2.esxi.jp-z.jp    <none>
openshift-monitoring                node-exporter-mxdk4                            2/2       Running            14         13d       192.168.0.71   okd-master.esxi.jp-z.jp   <none>
openshift-monitoring                node-exporter-sgcnc                            2/2       Running            15         13d       192.168.0.75   okd-node1.esxi.jp-z.jp    <none>
openshift-monitoring                prometheus-k8s-0                               4/4       Running            25         13d       10.128.1.243   okd-master.esxi.jp-z.jp   <none>
openshift-monitoring                prometheus-k8s-1                               4/4       Running            25         13d       10.128.1.242   okd-master.esxi.jp-z.jp   <none>
openshift-monitoring                prometheus-operator-6644b8cd54-hwgk2           1/1       Running            6          13d       10.128.1.241   okd-master.esxi.jp-z.jp   <none>
openshift-node                      sync-7dkbb                                     1/1       Running            7          13d       192.168.0.76   okd-node2.esxi.jp-z.jp    <none>
openshift-node                      sync-85vh9                                     1/1       Running            7          13d       192.168.0.71   okd-master.esxi.jp-z.jp   <none>
openshift-node                      sync-qwcx7                                     1/1       Running            7          13d       192.168.0.75   okd-node1.esxi.jp-z.jp    <none>
openshift-sdn                       ovs-kd65l                                      1/1       Running            6          13d       192.168.0.75   okd-node1.esxi.jp-z.jp    <none>
openshift-sdn                       ovs-r2pt6                                      1/1       Terminating        6          13d       192.168.0.71   okd-master.esxi.jp-z.jp   <none>
openshift-sdn                       ovs-wp9qx                                      1/1       Running            6          13d       192.168.0.76   okd-node2.esxi.jp-z.jp    <none>
openshift-sdn                       sdn-5ctjx                                      0/1       CrashLoopBackOff   132        13h       192.168.0.71   okd-master.esxi.jp-z.jp   <none>
openshift-sdn                       sdn-q25zn                                      1/1       Running            7          13d       192.168.0.76   okd-node2.esxi.jp-z.jp    <none>
openshift-sdn                       sdn-x7tqc                                      1/1       Running            8          13d       192.168.0.75   okd-node1.esxi.jp-z.jp    <none>
openshift-template-service-broker   apiserver-xjhkz                                0/1       CrashLoopBackOff   172        13d       10.128.1.250   okd-master.esxi.jp-z.jp   <none>
openshift-web-console               webconsole-7fc8759f7b-hmj98                    0/1       Running            8          13d       10.128.1.236   okd-master.esxi.jp-z.jp   <none>
s2i-sample                          sample-1-build                                 0/1       Completed          0          10d       10.130.0.24    okd-node2.esxi.jp-z.jp    <none>
s2i-sample                          sample-2-build                                 0/1       Completed          0          10d       10.130.0.27    okd-node2.esxi.jp-z.jp    <none>
s2i-sample                          sample-2-xvwld                                 1/1       Running            4          10d       10.129.0.105   okd-node1.esxi.jp-z.jp    <none>
[zaki@okd-master ~]$ 
[zaki@okd-master ~]$ 
[zaki@okd-master ~]$ 
[zaki@okd-master ~]$ oc get node
NAME                      STATUS    ROLES          AGE       VERSION
okd-master.esxi.jp-z.jp   Ready     infra,master   13d       v1.11.0+d4cacc0
okd-node1.esxi.jp-z.jp    Ready     compute        13d       v1.11.0+d4cacc0
okd-node2.esxi.jp-z.jp    Ready     compute        13d       v1.11.0+d4cacc0

うーん、masterのシステム系podが行けてない。シングルノードの影響もあるのかな。わからんけど。

[zaki@okd-master ~]$ 
[zaki@okd-master ~]$ oc get event -n openshift-sdn
LAST SEEN   FIRST SEEN   COUNT     NAME                         KIND      SUBOBJECT                      TYPE      REASON    SOURCE                             MESSAGE
1h          13h          117       sdn-5ctjx.15f822bdd9fc176d   Pod       spec.containers{sdn}           Normal    Pulled    kubelet, okd-master.esxi.jp-z.jp   Container image "docker.io/openshift/origin-node:v3.11" already present on machine
3m          13h          2937      sdn-5ctjx.15f822d167059e62   Pod       spec.containers{sdn}           Warning   BackOff   kubelet, okd-master.esxi.jp-z.jp   Back-off restarting failed container
1m          13h          412       ovs-r2pt6.15f822c698439be0   Pod       spec.containers{openvswitch}   Normal    Killing   kubelet, okd-master.esxi.jp-z.jp   Killing container with id docker://openvswitch:Need to kill Pod

イベント流れてる気がする…

処理開始前のVMのスナップショット残ってるのでやりなおすか。。

TASK [openshift_node : Delete OpenShift SDN/OVS pods prior to upgrade] *************************************************************Sunday 01 March 2020  18:04:25 +0900 (0:00:00.170)       0:05:36.079 ********** 


 [ERROR]: User interrupted execution


real    801m2.985s
user    20m44.070s
sys     4m56.689s

C-cで止めた。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment