Skip to content

Instantly share code, notes, and snippets.

@shashank855

shashank855/autogen-snapshot.log

Created August 11, 2020 14:08
Show Gist options
  • Save shashank855/259c4c4a3f0aea6290a7dff4852e4d76 to your computer and use it in GitHub Desktop.
Save shashank855/259c4c4a3f0aea6290a7dff4852e4d76 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
autogen-snapshot.log
d2iq@rack2:~/shashank$ kubectl logs -f my-testsz4bc-9lf2x -n litmus
No config file found; using defaults
/etc/ansible/hosts did not meet host_list requirements, check plugin documentation if this is unexpected
/etc/ansible/hosts did not meet script requirements, check plugin documentation if this is unexpected
PLAY [localhost] ***************************************************************
2020-08-11T13:38:46.755086 (delta: 0.047844) elapsed: 0.047844 ********
===============================================================================
TASK [Record test instance/run ID] *********************************************
2020-08-11T13:38:46.770561 (delta: 0.015438) elapsed: 0.063319 ********
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Construct testname appended with runID] **********************************
2020-08-11T13:38:46.811733 (delta: 0.041149) elapsed: 0.104491 ********
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [include_tasks] ***********************************************************
2020-08-11T13:38:46.842936 (delta: 0.031167) elapsed: 0.135694 ********
included: /utils/fcm/update_litmus_result_resource.yml for 127.0.0.1
TASK [Generate the litmus result CR to reflect SOT (Start of Test)] ************
2020-08-11T13:38:46.905170 (delta: 0.062201) elapsed: 0.197928 ********
changed: [127.0.0.1] => {"changed": true, "checksum": "4f7f39fd2db6a0c714c6b7ce7c1ab581a1bc8178", "dest": "./litmus-result.yaml", "gid": 0, "group": "root", "md5sum": "cbbaaefff371dec0810f1845792a5639", "mode": "0644", "owner": "root", "size": 429, "src": "/root/.ansible/tmp/ansible-tmp-1597153126.95-31642754646956/source", "state": "file", "uid": 0}
TASK [Analyze the cr yaml] *****************************************************
2020-08-11T13:38:47.463055 (delta: 0.557857) elapsed: 0.755813 ********
changed: [127.0.0.1] => {"changed": true, "cmd": "cat litmus-result.yaml", "delta": "0:00:00.753938", "end": "2020-08-11 13:38:48.481375", "rc": 0, "start": "2020-08-11 13:38:47.727437", "stderr": "", "stderr_lines": [], "stdout": "---\napiVersion: litmus.io/v1alpha1\nkind: LitmusResult\nmetadata:\n\n # name of the litmus testcase\n name: fio-di-delete-jiva-snapshots \nspec:\n\n # holds information on the testcase\n testMetadata:\n app: \n chaostype: \n\n # holds the state of testcase, manually updated by json merge patch\n # result is the useful value today, but anticipate phase use in future \n testStatus: \n phase: in-progress \n result: none ", "stdout_lines": ["---", "apiVersion: litmus.io/v1alpha1", "kind: LitmusResult", "metadata:", "", " # name of the litmus testcase", " name: fio-di-delete-jiva-snapshots ", "spec:", "", " # holds information on the testcase", " testMetadata:", " app: ", " chaostype: ", "", " # holds the state of testcase, manually updated by json merge patch", " # result is the useful value today, but anticipate phase use in future ", " testStatus: ", " phase: in-progress ", " result: none "]}
TASK [Apply the litmus result CR] **********************************************
2020-08-11T13:38:48.513168 (delta: 1.050082) elapsed: 1.805926 ********
changed: [127.0.0.1] => {"changed": true, "failed_when_result": false, "method": "create", "result": {"apiVersion": "litmus.io/v1alpha1", "kind": "LitmusResult", "metadata": {"creationTimestamp": "2020-08-11T13:38:54Z", "generation": 1, "name": "fio-di-delete-jiva-snapshots", "resourceVersion": "3276808", "selfLink": "/apis/litmus.io/v1alpha1/litmusresults/fio-di-delete-jiva-snapshots", "uid": "01b11ac6-9989-4515-8a5f-ab760ebc41b4"}, "spec": {"testMetadata": {"app": null, "chaostype": null}, "testStatus": {"phase": "in-progress", "result": "none"}}}}
TASK [Generate the litmus result CR to reflect EOT (End of Test)] **************
2020-08-11T13:38:49.395990 (delta: 0.882789) elapsed: 2.688748 ********
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Analyze the cr yaml] *****************************************************
2020-08-11T13:38:49.422002 (delta: 0.025957) elapsed: 2.71476 *********
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Apply the litmus result CR] **********************************************
2020-08-11T13:38:49.446880 (delta: 0.024843) elapsed: 2.739638 ********
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Check whether the provider storageclass is applied] **********************
2020-08-11T13:38:49.472603 (delta: 0.025689) elapsed: 2.765361 ********
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get sc openebs-jiva-default", "delta": "0:00:15.889951", "end": "2020-08-11 13:39:05.519482", "rc": 0, "start": "2020-08-11 13:38:49.629531", "stderr": "", "stderr_lines": [], "stdout": "NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE\nopenebs-jiva-default openebs.io/provisioner-iscsi Delete Immediate false 26d", "stdout_lines": ["NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE", "openebs-jiva-default openebs.io/provisioner-iscsi Delete Immediate false 26d"]}
TASK [Replace the storageclass placeholder with provider] **********************
2020-08-11T13:39:05.550164 (delta: 16.077524) elapsed: 18.842922 ******
changed: [127.0.0.1] => {"changed": true, "msg": "1 replacements made"}
TASK [include_tasks] ***********************************************************
2020-08-11T13:39:05.877153 (delta: 0.32695) elapsed: 19.169911 ********
included: /utils/k8s/create_ns.yml for 127.0.0.1
TASK [Obtain list of existing namespaces] **************************************
2020-08-11T13:39:05.917209 (delta: 0.040008) elapsed: 19.209967 *******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get ns --no-headers -o custom-columns=:metadata.name", "delta": "0:00:16.061815", "end": "2020-08-11 13:39:22.129074", "rc": 0, "start": "2020-08-11 13:39:06.067259", "stderr": "", "stderr_lines": [], "stdout": "cert-manager\ndefault\nkommander\nkommander-system\nkonvoy\nkube-node-lease\nkube-public\nkube-system\nkubeaddons\nkubecost\nlitmus\nopenebs", "stdout_lines": ["cert-manager", "default", "kommander", "kommander-system", "konvoy", "kube-node-lease", "kube-public", "kube-system", "kubeaddons", "kubecost", "litmus", "openebs"]}
TASK [Create test specific namespace.] *****************************************
2020-08-11T13:39:22.165453 (delta: 16.24821) elapsed: 35.458211 *******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl create ns fio-t", "delta": "0:00:06.847802", "end": "2020-08-11 13:39:29.167660", "rc": 0, "start": "2020-08-11 13:39:22.319858", "stderr": "", "stderr_lines": [], "stdout": "namespace/fio-t created", "stdout_lines": ["namespace/fio-t created"]}
TASK [include_tasks] ***********************************************************
2020-08-11T13:39:29.211047 (delta: 7.045558) elapsed: 42.503805 *******
included: /utils/k8s/status_testns.yml for 127.0.0.1
TASK [Checking the status of test specific namespace.] ************************
2020-08-11T13:39:29.246348 (delta: 0.035248) elapsed: 42.539106 *******
ok: [127.0.0.1] => {"attempts": 1, "changed": false, "resources": [{"apiVersion": "v1", "kind": "Namespace", "metadata": {"creationTimestamp": "2020-08-11T13:39:33Z", "name": "fio-t", "resourceVersion": "3277032", "selfLink": "/api/v1/namespaces/fio-t", "uid": "02d9fdb3-7a29-42b8-82e1-2bfcdff6d0f5"}, "spec": {"finalizers": ["kubernetes"]}, "status": {"phase": "Active"}}]}
TASK [Deploy PVC to get size of volume requested application namespace] ********
2020-08-11T13:39:30.127692 (delta: 0.881306) elapsed: 43.42045 ********
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl apply -f pvc.yml -n fio-t", "delta": "0:00:06.139934", "end": "2020-08-11 13:39:36.425225", "rc": 0, "start": "2020-08-11 13:39:30.285291", "stderr": "", "stderr_lines": [], "stdout": "persistentvolumeclaim/demo-vol1-claim created", "stdout_lines": ["persistentvolumeclaim/demo-vol1-claim created"]}
TASK [set_fact] ****************************************************************
2020-08-11T13:39:36.458255 (delta: 6.330527) elapsed: 49.751013 *******
ok: [127.0.0.1] => {"ansible_facts": {"execute": 1, "pvc_label": "demo-vol1-claim"}, "changed": false}
TASK [Confirm pvc status is bound] *********************************************
2020-08-11T13:39:36.503542 (delta: 0.04525) elapsed: 49.7963 **********
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "kubectl get pvc --no-headers -n fio-t -l name=demo-vol1-claim -o custom-columns=:.status.phase", "delta": "0:00:16.098026", "end": "2020-08-11 13:39:52.775164", "rc": 0, "start": "2020-08-11 13:39:36.677138", "stderr": "", "stderr_lines": [], "stdout": "Bound", "stdout_lines": ["Bound"]}
TASK [Get pv name] *************************************************************
2020-08-11T13:39:52.813417 (delta: 16.309821) elapsed: 66.106175 ******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pvc --no-headers -n fio-t -l name=demo-vol1-claim -o jsonpath='{range .items[*]}{.spec.volumeName}'", "delta": "0:00:15.875960", "end": "2020-08-11 13:40:08.843994", "failed_when_result": false, "rc": 0, "start": "2020-08-11 13:39:52.968034", "stderr": "", "stderr_lines": [], "stdout": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6", "stdout_lines": ["pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6"]}
TASK [set_fact] ****************************************************************
2020-08-11T13:40:08.883897 (delta: 16.070444) elapsed: 82.176655 ******
ok: [127.0.0.1] => {"ansible_facts": {"pv_name": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6"}, "changed": false}
TASK [Get controller svc] ******************************************************
2020-08-11T13:40:08.930603 (delta: 0.04667) elapsed: 82.223361 ********
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get svc -l openebs.io/controller-service=jiva-controller-svc,openebs.io/persistent-volume=pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6 -n openebs --no-headers -o custom-columns=:spec.clusterIP", "delta": "0:00:15.939503", "end": "2020-08-11 13:40:25.017873", "failed_when_result": false, "rc": 0, "start": "2020-08-11 13:40:09.078370", "stderr": "", "stderr_lines": [], "stdout": "12.0.34.99", "stdout_lines": ["12.0.34.99"]}
TASK [Replace the data sample size with the provider sample size in fio-write.yml] ***
2020-08-11T13:40:25.062873 (delta: 16.132235) elapsed: 98.355631 ******
changed: [127.0.0.1] => {"changed": true, "msg": "1 replacements made"}
TASK [Deploy fio write test job] ***********************************************
2020-08-11T13:40:25.257892 (delta: 0.194987) elapsed: 98.55065 ********
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl apply -f fio-write.yml -n fio-t", "delta": "0:00:06.043409", "end": "2020-08-11 13:40:31.459426", "rc": 0, "start": "2020-08-11 13:40:25.416017", "stderr": "", "stderr_lines": [], "stdout": "configmap/basic created\njob.batch/fio created", "stdout_lines": ["configmap/basic created", "job.batch/fio created"]}
TASK [Fetch the pod name in fio-t] *********************************************
2020-08-11T13:40:31.512538 (delta: 6.254609) elapsed: 104.805296 ******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pods -n fio-t -l name=fio-write -o custom-columns=:metadata.name --no-headers", "delta": "0:00:16.879585", "end": "2020-08-11 13:40:48.553475", "rc": 0, "start": "2020-08-11 13:40:31.673890", "stderr": "", "stderr_lines": [], "stdout": "fio-b8nzw", "stdout_lines": ["fio-b8nzw"]}
TASK [Check the status of pod] *************************************************
2020-08-11T13:40:48.583216 (delta: 17.070642) elapsed: 121.875974 *****
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "kubectl get po fio-b8nzw -n fio-t -o jsonpath={.status.phase}", "delta": "0:00:16.857109", "end": "2020-08-11 13:41:05.589831", "rc": 0, "start": "2020-08-11 13:40:48.732722", "stderr": "", "stderr_lines": [], "stdout": "Running", "stdout_lines": ["Running"]}
TASK [Populate the iterator with the values to loop] ***************************
2020-08-11T13:41:05.625087 (delta: 17.041834) elapsed: 138.917845 *****
changed: [127.0.0.1] => {"changed": true, "cmd": "shuf -i 1-20 -n 15", "delta": "0:00:00.759771", "end": "2020-08-11 13:41:06.530641", "rc": 0, "start": "2020-08-11 13:41:05.770870", "stderr": "", "stderr_lines": [], "stdout": "1\n5\n18\n16\n10\n8\n3\n17\n15\n20\n7\n19\n14\n9\n12", "stdout_lines": ["1", "5", "18", "16", "10", "8", "3", "17", "15", "20", "7", "19", "14", "9", "12"]}
TASK [Include replica restart test] ********************************************
2020-08-11T13:41:06.561362 (delta: 0.936238) elapsed: 139.85412 *******
included: /experiments/functional/snapshot_deletion/jiva/test_replica_restart.yml for 127.0.0.1 => (item=1)
included: /experiments/functional/snapshot_deletion/jiva/test_replica_restart.yml for 127.0.0.1 => (item=5)
included: /experiments/functional/snapshot_deletion/jiva/test_replica_restart.yml for 127.0.0.1 => (item=18)
included: /experiments/functional/snapshot_deletion/jiva/test_replica_restart.yml for 127.0.0.1 => (item=16)
included: /experiments/functional/snapshot_deletion/jiva/test_replica_restart.yml for 127.0.0.1 => (item=10)
included: /experiments/functional/snapshot_deletion/jiva/test_replica_restart.yml for 127.0.0.1 => (item=8)
included: /experiments/functional/snapshot_deletion/jiva/test_replica_restart.yml for 127.0.0.1 => (item=3)
included: /experiments/functional/snapshot_deletion/jiva/test_replica_restart.yml for 127.0.0.1 => (item=17)
included: /experiments/functional/snapshot_deletion/jiva/test_replica_restart.yml for 127.0.0.1 => (item=15)
included: /experiments/functional/snapshot_deletion/jiva/test_replica_restart.yml for 127.0.0.1 => (item=20)
included: /experiments/functional/snapshot_deletion/jiva/test_replica_restart.yml for 127.0.0.1 => (item=7)
included: /experiments/functional/snapshot_deletion/jiva/test_replica_restart.yml for 127.0.0.1 => (item=19)
included: /experiments/functional/snapshot_deletion/jiva/test_replica_restart.yml for 127.0.0.1 => (item=14)
included: /experiments/functional/snapshot_deletion/jiva/test_replica_restart.yml for 127.0.0.1 => (item=9)
included: /experiments/functional/snapshot_deletion/jiva/test_replica_restart.yml for 127.0.0.1 => (item=12)
TASK [Get the name of replica pods] ********************************************
2020-08-11T13:41:06.858464 (delta: 0.297065) elapsed: 140.151222 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Set the replica pod name to variable based on latest creation timestamp] ***
2020-08-11T13:41:06.899164 (delta: 0.040662) elapsed: 140.191922 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Delete random replica] ***************************************************
2020-08-11T13:41:06.935771 (delta: 0.036568) elapsed: 140.228529 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:41:06.969909 (delta: 0.034097) elapsed: 140.262667 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Getting master replica IP] ***********************************************
2020-08-11T13:41:07.001526 (delta: 0.031563) elapsed: 140.294284 ******
[WARNING]: Consider using the get_url or uri module rather than running curl.
If you need to use command because get_url or uri is insufficient you can add
warn=False to this command task or set command_warnings=False in ansible.cfg to
get rid of this message.
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "curl http://12.0.34.99:9501/v1/replicas | jq '.data[0].address' | sed -e 's/^\"//' -e 's/\"$//'", "delta": "0:00:00.815467", "end": "2020-08-11 13:41:07.983100", "rc": 0, "start": "2020-08-11 13:41:07.167633", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 1477 100 1477 0 0 160k 0 --:--:-- --:--:-- --:--:-- 160k", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0", "100 1477 100 1477 0 0 160k 0 --:--:-- --:--:-- --:--:-- 160k"], "stdout": "tcp://172.168.29.43:9502", "stdout_lines": ["tcp://172.168.29.43:9502"]}
TASK [Set master replica ip] ***************************************************
2020-08-11T13:41:08.022717 (delta: 1.021132) elapsed: 141.315475 ******
ok: [127.0.0.1] => {"ansible_facts": {"replica_ip": "172.168.29.43"}, "changed": false}
TASK [debug] *******************************************************************
2020-08-11T13:41:08.079986 (delta: 0.057226) elapsed: 141.372744 ******
ok: [127.0.0.1] => {
"replica_ip": "172.168.29.43"
}
TASK [Getting the master replica pod name corresponding to the IP] *************
2020-08-11T13:41:08.135462 (delta: 0.055416) elapsed: 141.42822 *******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pod -l openebs.io/replica=jiva-replica,openebs.io/persistent-volume=pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6 -n openebs --no-headers -o jsonpath='{.items[?(@.status.podIP==\"172.168.29.43\")].metadata.name}'", "delta": "0:00:15.901080", "end": "2020-08-11 13:41:24.200883", "rc": 0, "start": "2020-08-11 13:41:08.299803", "stderr": "", "stderr_lines": [], "stdout": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-ffnch", "stdout_lines": ["pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-ffnch"]}
TASK [Delete master replica] ***************************************************
2020-08-11T13:41:24.233885 (delta: 16.098362) elapsed: 157.526643 *****
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl delete pod pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-ffnch -n openebs", "delta": "0:00:22.438041", "end": "2020-08-11 13:41:46.836900", "rc": 0, "start": "2020-08-11 13:41:24.398859", "stderr": "", "stderr_lines": [], "stdout": "pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-ffnch\" deleted", "stdout_lines": ["pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-ffnch\" deleted"]}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:41:46.866808 (delta: 22.632884) elapsed: 180.159566 *****
Pausing for 60 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [127.0.0.1] => {"changed": false, "delta": 60, "echo": true, "rc": 0, "start": "2020-08-11 13:41:46.912733", "stderr": "", "stdout": "Paused for 1.0 minutes", "stop": "2020-08-11 13:42:46.913124", "user_input": ""}
TASK [Get the name of replica pods] ********************************************
2020-08-11T13:42:46.926508 (delta: 60.059657) elapsed: 240.219266 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Set the replica pod name to variable based on latest creation timestamp] ***
2020-08-11T13:42:46.957622 (delta: 0.031059) elapsed: 240.25038 *******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Delete random replica] ***************************************************
2020-08-11T13:42:46.988763 (delta: 0.031101) elapsed: 240.281521 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:42:47.020817 (delta: 0.03201) elapsed: 240.313575 *******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Getting master replica IP] ***********************************************
2020-08-11T13:42:47.055833 (delta: 0.034977) elapsed: 240.348591 ******
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "curl http://12.0.34.99:9501/v1/replicas | jq '.data[0].address' | sed -e 's/^\"//' -e 's/\"$//'", "delta": "0:00:00.918091", "end": "2020-08-11 13:42:48.132541", "rc": 0, "start": "2020-08-11 13:42:47.214450", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 1477 100 1477 0 0 721k 0 --:--:-- --:--:-- --:--:-- 721k", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0", "100 1477 100 1477 0 0 721k 0 --:--:-- --:--:-- --:--:-- 721k"], "stdout": "tcp://172.168.166.84:9502", "stdout_lines": ["tcp://172.168.166.84:9502"]}
TASK [Set master replica ip] ***************************************************
2020-08-11T13:42:48.183911 (delta: 1.12802) elapsed: 241.476669 *******
ok: [127.0.0.1] => {"ansible_facts": {"replica_ip": "172.168.166.84"}, "changed": false}
TASK [debug] *******************************************************************
2020-08-11T13:42:48.239181 (delta: 0.055229) elapsed: 241.531939 ******
ok: [127.0.0.1] => {
"replica_ip": "172.168.166.84"
}
TASK [Getting the master replica pod name corresponding to the IP] *************
2020-08-11T13:42:48.296512 (delta: 0.057278) elapsed: 241.58927 *******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pod -l openebs.io/replica=jiva-replica,openebs.io/persistent-volume=pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6 -n openebs --no-headers -o jsonpath='{.items[?(@.status.podIP==\"172.168.166.84\")].metadata.name}'", "delta": "0:00:15.849835", "end": "2020-08-11 13:43:04.322238", "rc": 0, "start": "2020-08-11 13:42:48.472403", "stderr": "", "stderr_lines": [], "stdout": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-t5hlk", "stdout_lines": ["pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-t5hlk"]}
TASK [Delete master replica] ***************************************************
2020-08-11T13:43:04.353564 (delta: 16.056993) elapsed: 257.646322 *****
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl delete pod pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-t5hlk -n openebs", "delta": "0:00:17.744151", "end": "2020-08-11 13:43:22.245779", "rc": 0, "start": "2020-08-11 13:43:04.501628", "stderr": "", "stderr_lines": [], "stdout": "pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-t5hlk\" deleted", "stdout_lines": ["pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-t5hlk\" deleted"]}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:43:22.278667 (delta: 17.925062) elapsed: 275.571425 *****
Pausing for 60 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [127.0.0.1] => {"changed": false, "delta": 60, "echo": true, "rc": 0, "start": "2020-08-11 13:43:22.325639", "stderr": "", "stdout": "Paused for 1.0 minutes", "stop": "2020-08-11 13:44:22.325811", "user_input": ""}
TASK [Get the name of replica pods] ********************************************
2020-08-11T13:44:22.338397 (delta: 60.059688) elapsed: 335.631155 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Set the replica pod name to variable based on latest creation timestamp] ***
2020-08-11T13:44:22.374359 (delta: 0.035925) elapsed: 335.667117 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Delete random replica] ***************************************************
2020-08-11T13:44:22.418831 (delta: 0.044399) elapsed: 335.711589 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:44:22.456766 (delta: 0.037894) elapsed: 335.749524 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Getting master replica IP] ***********************************************
2020-08-11T13:44:22.497302 (delta: 0.040497) elapsed: 335.79006 *******
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "curl http://12.0.34.99:9501/v1/replicas | jq '.data[0].address' | sed -e 's/^\"//' -e 's/\"$//'", "delta": "0:00:00.821422", "end": "2020-08-11 13:44:23.527660", "rc": 0, "start": "2020-08-11 13:44:22.706238", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0", "100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k"], "stdout": "tcp://172.168.125.222:9502", "stdout_lines": ["tcp://172.168.125.222:9502"]}
TASK [Set master replica ip] ***************************************************
2020-08-11T13:44:23.569787 (delta: 1.072432) elapsed: 336.862545 ******
ok: [127.0.0.1] => {"ansible_facts": {"replica_ip": "172.168.125.222"}, "changed": false}
TASK [debug] *******************************************************************
2020-08-11T13:44:23.625166 (delta: 0.055338) elapsed: 336.917924 ******
ok: [127.0.0.1] => {
"replica_ip": "172.168.125.222"
}
TASK [Getting the master replica pod name corresponding to the IP] *************
2020-08-11T13:44:23.681386 (delta: 0.056163) elapsed: 336.974144 ******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pod -l openebs.io/replica=jiva-replica,openebs.io/persistent-volume=pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6 -n openebs --no-headers -o jsonpath='{.items[?(@.status.podIP==\"172.168.125.222\")].metadata.name}'", "delta": "0:00:15.931825", "end": "2020-08-11 13:44:39.777402", "rc": 0, "start": "2020-08-11 13:44:23.845577", "stderr": "", "stderr_lines": [], "stdout": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-wj2l8", "stdout_lines": ["pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-wj2l8"]}
TASK [Delete master replica] ***************************************************
2020-08-11T13:44:39.814595 (delta: 16.133163) elapsed: 353.107353 *****
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl delete pod pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-wj2l8 -n openebs", "delta": "0:00:24.313132", "end": "2020-08-11 13:45:04.323536", "rc": 0, "start": "2020-08-11 13:44:40.010404", "stderr": "", "stderr_lines": [], "stdout": "pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-wj2l8\" deleted", "stdout_lines": ["pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-wj2l8\" deleted"]}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:45:04.357372 (delta: 24.542734) elapsed: 377.65013 ******
Pausing for 60 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [127.0.0.1] => {"changed": false, "delta": 60, "echo": true, "rc": 0, "start": "2020-08-11 13:45:04.406909", "stderr": "", "stdout": "Paused for 1.0 minutes", "stop": "2020-08-11 13:46:04.407088", "user_input": ""}
TASK [Get the name of replica pods] ********************************************
2020-08-11T13:46:04.421366 (delta: 60.06396) elapsed: 437.714124 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Set the replica pod name to variable based on latest creation timestamp] ***
2020-08-11T13:46:04.458218 (delta: 0.036815) elapsed: 437.750976 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Delete random replica] ***************************************************
2020-08-11T13:46:04.498301 (delta: 0.040018) elapsed: 437.791059 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:46:04.537950 (delta: 0.039595) elapsed: 437.830708 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Getting master replica IP] ***********************************************
2020-08-11T13:46:04.572761 (delta: 0.034757) elapsed: 437.865519 ******
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "curl http://12.0.34.99:9501/v1/replicas | jq '.data[0].address' | sed -e 's/^\"//' -e 's/\"$//'", "delta": "0:00:00.847274", "end": "2020-08-11 13:46:05.598697", "rc": 0, "start": "2020-08-11 13:46:04.751423", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0", "100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k"], "stdout": "tcp://172.168.29.44:9502", "stdout_lines": ["tcp://172.168.29.44:9502"]}
TASK [Set master replica ip] ***************************************************
2020-08-11T13:46:05.640083 (delta: 1.067286) elapsed: 438.932841 ******
ok: [127.0.0.1] => {"ansible_facts": {"replica_ip": "172.168.29.44"}, "changed": false}
TASK [debug] *******************************************************************
2020-08-11T13:46:05.694962 (delta: 0.054843) elapsed: 438.98772 *******
ok: [127.0.0.1] => {
"replica_ip": "172.168.29.44"
}
TASK [Getting the master replica pod name corresponding to the IP] *************
2020-08-11T13:46:05.755705 (delta: 0.060701) elapsed: 439.048463 ******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pod -l openebs.io/replica=jiva-replica,openebs.io/persistent-volume=pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6 -n openebs --no-headers -o jsonpath='{.items[?(@.status.podIP==\"172.168.29.44\")].metadata.name}'", "delta": "0:00:15.931407", "end": "2020-08-11 13:46:21.842840", "rc": 0, "start": "2020-08-11 13:46:05.911433", "stderr": "", "stderr_lines": [], "stdout": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-2tw24", "stdout_lines": ["pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-2tw24"]}
TASK [Delete master replica] ***************************************************
2020-08-11T13:46:21.873835 (delta: 16.118091) elapsed: 455.166593 *****
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl delete pod pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-2tw24 -n openebs", "delta": "0:00:24.808481", "end": "2020-08-11 13:46:46.836295", "rc": 0, "start": "2020-08-11 13:46:22.027814", "stderr": "", "stderr_lines": [], "stdout": "pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-2tw24\" deleted", "stdout_lines": ["pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-2tw24\" deleted"]}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:46:46.865682 (delta: 24.99179) elapsed: 480.15844 *******
Pausing for 60 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [127.0.0.1] => {"changed": false, "delta": 60, "echo": true, "rc": 0, "start": "2020-08-11 13:46:46.911677", "stderr": "", "stdout": "Paused for 1.0 minutes", "stop": "2020-08-11 13:47:46.913023", "user_input": ""}
TASK [Get the name of replica pods] ********************************************
2020-08-11T13:47:46.927284 (delta: 60.061548) elapsed: 540.220042 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Set the replica pod name to variable based on latest creation timestamp] ***
2020-08-11T13:47:46.960666 (delta: 0.03331) elapsed: 540.253424 *******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Delete random replica] ***************************************************
2020-08-11T13:47:46.992131 (delta: 0.031428) elapsed: 540.284889 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:47:47.025941 (delta: 0.033772) elapsed: 540.318699 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Getting master replica IP] ***********************************************
2020-08-11T13:47:47.059770 (delta: 0.033794) elapsed: 540.352528 ******
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "curl http://12.0.34.99:9501/v1/replicas | jq '.data[0].address' | sed -e 's/^\"//' -e 's/\"$//'", "delta": "0:00:00.674470", "end": "2020-08-11 13:47:47.891371", "rc": 0, "start": "2020-08-11 13:47:47.216901", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0", "100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k"], "stdout": "tcp://172.168.166.86:9502", "stdout_lines": ["tcp://172.168.166.86:9502"]}
TASK [Set master replica ip] ***************************************************
2020-08-11T13:47:47.929419 (delta: 0.869611) elapsed: 541.222177 ******
ok: [127.0.0.1] => {"ansible_facts": {"replica_ip": "172.168.166.86"}, "changed": false}
TASK [debug] *******************************************************************
2020-08-11T13:47:47.987188 (delta: 0.057733) elapsed: 541.279946 ******
ok: [127.0.0.1] => {
"replica_ip": "172.168.166.86"
}
TASK [Getting the master replica pod name corresponding to the IP] *************
2020-08-11T13:47:48.041212 (delta: 0.053984) elapsed: 541.33397 *******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pod -l openebs.io/replica=jiva-replica,openebs.io/persistent-volume=pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6 -n openebs --no-headers -o jsonpath='{.items[?(@.status.podIP==\"172.168.166.86\")].metadata.name}'", "delta": "0:00:00.780609", "end": "2020-08-11 13:47:48.967396", "rc": 0, "start": "2020-08-11 13:47:48.186787", "stderr": "", "stderr_lines": [], "stdout": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-5pwvr", "stdout_lines": ["pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-5pwvr"]}
TASK [Delete master replica] ***************************************************
2020-08-11T13:47:48.999019 (delta: 0.957748) elapsed: 542.291777 ******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl delete pod pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-5pwvr -n openebs", "delta": "0:00:08.394682", "end": "2020-08-11 13:47:57.543983", "rc": 0, "start": "2020-08-11 13:47:49.149301", "stderr": "", "stderr_lines": [], "stdout": "pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-5pwvr\" deleted", "stdout_lines": ["pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-5pwvr\" deleted"]}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:47:57.576673 (delta: 8.577614) elapsed: 550.869431 ******
Pausing for 60 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [127.0.0.1] => {"changed": false, "delta": 60, "echo": true, "rc": 0, "start": "2020-08-11 13:47:57.623736", "stderr": "", "stdout": "Paused for 1.0 minutes", "stop": "2020-08-11 13:48:57.623899", "user_input": ""}
TASK [Get the name of replica pods] ********************************************
2020-08-11T13:48:57.632829 (delta: 60.056114) elapsed: 610.925587 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Set the replica pod name to variable based on latest creation timestamp] ***
2020-08-11T13:48:57.664681 (delta: 0.03181) elapsed: 610.957439 *******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Delete random replica] ***************************************************
2020-08-11T13:48:57.695963 (delta: 0.03124) elapsed: 610.988721 *******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:48:57.730897 (delta: 0.034892) elapsed: 611.023655 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Getting master replica IP] ***********************************************
2020-08-11T13:48:57.766178 (delta: 0.035239) elapsed: 611.058936 ******
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "curl http://12.0.34.99:9501/v1/replicas | jq '.data[0].address' | sed -e 's/^\"//' -e 's/\"$//'", "delta": "0:00:00.682001", "end": "2020-08-11 13:48:58.605400", "rc": 0, "start": "2020-08-11 13:48:57.923399", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0", "100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k"], "stdout": "tcp://172.168.125.223:9502", "stdout_lines": ["tcp://172.168.125.223:9502"]}
TASK [Set master replica ip] ***************************************************
2020-08-11T13:48:58.643753 (delta: 0.877535) elapsed: 611.936511 ******
ok: [127.0.0.1] => {"ansible_facts": {"replica_ip": "172.168.125.223"}, "changed": false}
TASK [debug] *******************************************************************
2020-08-11T13:48:58.699939 (delta: 0.056145) elapsed: 611.992697 ******
ok: [127.0.0.1] => {
"replica_ip": "172.168.125.223"
}
TASK [Getting the master replica pod name corresponding to the IP] *************
2020-08-11T13:48:58.753223 (delta: 0.053224) elapsed: 612.045981 ******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pod -l openebs.io/replica=jiva-replica,openebs.io/persistent-volume=pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6 -n openebs --no-headers -o jsonpath='{.items[?(@.status.podIP==\"172.168.125.223\")].metadata.name}'", "delta": "0:00:01.073126", "end": "2020-08-11 13:48:59.983552", "rc": 0, "start": "2020-08-11 13:48:58.910426", "stderr": "", "stderr_lines": [], "stdout": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-xqd5v", "stdout_lines": ["pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-xqd5v"]}
TASK [Delete master replica] ***************************************************
2020-08-11T13:49:00.024511 (delta: 1.271247) elapsed: 613.317269 ******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl delete pod pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-xqd5v -n openebs", "delta": "0:00:14.123561", "end": "2020-08-11 13:49:14.317355", "rc": 0, "start": "2020-08-11 13:49:00.193794", "stderr": "", "stderr_lines": [], "stdout": "pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-xqd5v\" deleted", "stdout_lines": ["pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-xqd5v\" deleted"]}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:49:14.351766 (delta: 14.327214) elapsed: 627.644524 *****
Pausing for 60 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [127.0.0.1] => {"changed": false, "delta": 60, "echo": true, "rc": 0, "start": "2020-08-11 13:49:14.396644", "stderr": "", "stdout": "Paused for 1.0 minutes", "stop": "2020-08-11 13:50:14.396873", "user_input": ""}
TASK [Get the name of replica pods] ********************************************
2020-08-11T13:50:14.406837 (delta: 60.055029) elapsed: 687.699595 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Set the replica pod name to variable based on latest creation timestamp] ***
2020-08-11T13:50:14.438252 (delta: 0.031379) elapsed: 687.73101 *******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Delete random replica] ***************************************************
2020-08-11T13:50:14.469380 (delta: 0.03109) elapsed: 687.762138 *******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:50:14.500461 (delta: 0.031032) elapsed: 687.793219 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Getting master replica IP] ***********************************************
2020-08-11T13:50:14.541415 (delta: 0.040911) elapsed: 687.834173 ******
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "curl http://12.0.34.99:9501/v1/replicas | jq '.data[0].address' | sed -e 's/^\"//' -e 's/\"$//'", "delta": "0:00:00.713017", "end": "2020-08-11 13:50:15.421450", "rc": 0, "start": "2020-08-11 13:50:14.708433", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 1477 100 1477 0 0 721k 0 --:--:-- --:--:-- --:--:-- 1442k", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0", "100 1477 100 1477 0 0 721k 0 --:--:-- --:--:-- --:--:-- 1442k"], "stdout": "tcp://172.168.29.45:9502", "stdout_lines": ["tcp://172.168.29.45:9502"]}
TASK [Set master replica ip] ***************************************************
2020-08-11T13:50:15.460311 (delta: 0.918806) elapsed: 688.753069 ******
ok: [127.0.0.1] => {"ansible_facts": {"replica_ip": "172.168.29.45"}, "changed": false}
TASK [debug] *******************************************************************
2020-08-11T13:50:15.524568 (delta: 0.064187) elapsed: 688.817326 ******
ok: [127.0.0.1] => {
"replica_ip": "172.168.29.45"
}
TASK [Getting the master replica pod name corresponding to the IP] *************
2020-08-11T13:50:15.577112 (delta: 0.052502) elapsed: 688.86987 *******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pod -l openebs.io/replica=jiva-replica,openebs.io/persistent-volume=pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6 -n openebs --no-headers -o jsonpath='{.items[?(@.status.podIP==\"172.168.29.45\")].metadata.name}'", "delta": "0:00:00.756627", "end": "2020-08-11 13:50:16.489528", "rc": 0, "start": "2020-08-11 13:50:15.732901", "stderr": "", "stderr_lines": [], "stdout": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-tccck", "stdout_lines": ["pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-tccck"]}
TASK [Delete master replica] ***************************************************
2020-08-11T13:50:16.530057 (delta: 0.952903) elapsed: 689.822815 ******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl delete pod pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-tccck -n openebs", "delta": "0:00:02.719827", "end": "2020-08-11 13:50:19.403697", "rc": 0, "start": "2020-08-11 13:50:16.683870", "stderr": "", "stderr_lines": [], "stdout": "pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-tccck\" deleted", "stdout_lines": ["pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-tccck\" deleted"]}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:50:19.442429 (delta: 2.912332) elapsed: 692.735187 ******
Pausing for 60 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [127.0.0.1] => {"changed": false, "delta": 60, "echo": true, "rc": 0, "start": "2020-08-11 13:50:19.493011", "stderr": "", "stdout": "Paused for 1.0 minutes", "stop": "2020-08-11 13:51:19.493203", "user_input": ""}
TASK [Get the name of replica pods] ********************************************
2020-08-11T13:51:19.502731 (delta: 60.060259) elapsed: 752.795489 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Set the replica pod name to variable based on latest creation timestamp] ***
2020-08-11T13:51:19.537083 (delta: 0.034312) elapsed: 752.829841 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Delete random replica] ***************************************************
2020-08-11T13:51:19.570115 (delta: 0.032989) elapsed: 752.862873 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:51:19.602351 (delta: 0.032192) elapsed: 752.895109 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Getting master replica IP] ***********************************************
2020-08-11T13:51:19.635134 (delta: 0.03274) elapsed: 752.927892 *******
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "curl http://12.0.34.99:9501/v1/replicas | jq '.data[0].address' | sed -e 's/^\"//' -e 's/\"$//'", "delta": "0:00:00.837546", "end": "2020-08-11 13:51:20.631596", "rc": 0, "start": "2020-08-11 13:51:19.794050", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0", "100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k"], "stdout": "tcp://172.168.166.87:9502", "stdout_lines": ["tcp://172.168.166.87:9502"]}
TASK [Set master replica ip] ***************************************************
2020-08-11T13:51:20.683491 (delta: 1.048316) elapsed: 753.976249 ******
ok: [127.0.0.1] => {"ansible_facts": {"replica_ip": "172.168.166.87"}, "changed": false}
TASK [debug] *******************************************************************
2020-08-11T13:51:20.737938 (delta: 0.054407) elapsed: 754.030696 ******
ok: [127.0.0.1] => {
"replica_ip": "172.168.166.87"
}
TASK [Getting the master replica pod name corresponding to the IP] *************
2020-08-11T13:51:20.790341 (delta: 0.052365) elapsed: 754.083099 ******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pod -l openebs.io/replica=jiva-replica,openebs.io/persistent-volume=pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6 -n openebs --no-headers -o jsonpath='{.items[?(@.status.podIP==\"172.168.166.87\")].metadata.name}'", "delta": "0:00:00.761807", "end": "2020-08-11 13:51:21.704379", "rc": 0, "start": "2020-08-11 13:51:20.942572", "stderr": "", "stderr_lines": [], "stdout": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-d82mk", "stdout_lines": ["pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-d82mk"]}
TASK [Delete master replica] ***************************************************
2020-08-11T13:51:21.737921 (delta: 0.947526) elapsed: 755.030679 ******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl delete pod pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-d82mk -n openebs", "delta": "0:00:05.651746", "end": "2020-08-11 13:51:27.544464", "rc": 0, "start": "2020-08-11 13:51:21.892718", "stderr": "", "stderr_lines": [], "stdout": "pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-d82mk\" deleted", "stdout_lines": ["pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-d82mk\" deleted"]}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:51:27.575418 (delta: 5.837461) elapsed: 760.868176 ******
Pausing for 60 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [127.0.0.1] => {"changed": false, "delta": 60, "echo": true, "rc": 0, "start": "2020-08-11 13:51:27.622925", "stderr": "", "stdout": "Paused for 1.0 minutes", "stop": "2020-08-11 13:52:27.623106", "user_input": ""}
TASK [Get the name of replica pods] ********************************************
2020-08-11T13:52:27.639884 (delta: 60.064425) elapsed: 820.932642 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Set the replica pod name to variable based on latest creation timestamp] ***
2020-08-11T13:52:27.676852 (delta: 0.036877) elapsed: 820.96961 *******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Delete random replica] ***************************************************
2020-08-11T13:52:27.706731 (delta: 0.029839) elapsed: 820.999489 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:52:27.737585 (delta: 0.030816) elapsed: 821.030343 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Getting master replica IP] ***********************************************
2020-08-11T13:52:27.767217 (delta: 0.029594) elapsed: 821.059975 ******
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "curl http://12.0.34.99:9501/v1/replicas | jq '.data[0].address' | sed -e 's/^\"//' -e 's/\"$//'", "delta": "0:00:00.659908", "end": "2020-08-11 13:52:28.593785", "rc": 0, "start": "2020-08-11 13:52:27.933877", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0", "100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k"], "stdout": "tcp://172.168.125.224:9502", "stdout_lines": ["tcp://172.168.125.224:9502"]}
TASK [Set master replica ip] ***************************************************
2020-08-11T13:52:28.633062 (delta: 0.865807) elapsed: 821.92582 *******
ok: [127.0.0.1] => {"ansible_facts": {"replica_ip": "172.168.125.224"}, "changed": false}
TASK [debug] *******************************************************************
2020-08-11T13:52:28.688867 (delta: 0.055766) elapsed: 821.981625 ******
ok: [127.0.0.1] => {
"replica_ip": "172.168.125.224"
}
TASK [Getting the master replica pod name corresponding to the IP] *************
2020-08-11T13:52:28.748519 (delta: 0.059597) elapsed: 822.041277 ******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pod -l openebs.io/replica=jiva-replica,openebs.io/persistent-volume=pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6 -n openebs --no-headers -o jsonpath='{.items[?(@.status.podIP==\"172.168.125.224\")].metadata.name}'", "delta": "0:00:00.767640", "end": "2020-08-11 13:52:29.674513", "rc": 0, "start": "2020-08-11 13:52:28.906873", "stderr": "", "stderr_lines": [], "stdout": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-gqrbt", "stdout_lines": ["pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-gqrbt"]}
TASK [Delete master replica] ***************************************************
2020-08-11T13:52:29.707481 (delta: 0.958924) elapsed: 823.000239 ******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl delete pod pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-gqrbt -n openebs", "delta": "0:00:02.866881", "end": "2020-08-11 13:52:32.742708", "rc": 0, "start": "2020-08-11 13:52:29.875827", "stderr": "", "stderr_lines": [], "stdout": "pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-gqrbt\" deleted", "stdout_lines": ["pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-gqrbt\" deleted"]}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:52:32.776922 (delta: 3.069398) elapsed: 826.06968 *******
Pausing for 60 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [127.0.0.1] => {"changed": false, "delta": 60, "echo": true, "rc": 0, "start": "2020-08-11 13:52:32.822000", "stderr": "", "stdout": "Paused for 1.0 minutes", "stop": "2020-08-11 13:53:32.828079", "user_input": ""}
TASK [Get the name of replica pods] ********************************************
2020-08-11T13:53:32.841638 (delta: 60.064674) elapsed: 886.134396 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Set the replica pod name to variable based on latest creation timestamp] ***
2020-08-11T13:53:32.882223 (delta: 0.04055) elapsed: 886.174981 *******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Delete random replica] ***************************************************
2020-08-11T13:53:32.917226 (delta: 0.034945) elapsed: 886.209984 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:53:32.948481 (delta: 0.0312) elapsed: 886.241239 ********
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Getting master replica IP] ***********************************************
2020-08-11T13:53:32.981267 (delta: 0.032747) elapsed: 886.274025 ******
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "curl http://12.0.34.99:9501/v1/replicas | jq '.data[0].address' | sed -e 's/^\"//' -e 's/\"$//'", "delta": "0:00:00.665467", "end": "2020-08-11 13:53:33.823459", "rc": 0, "start": "2020-08-11 13:53:33.157992", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0", "100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k"], "stdout": "tcp://172.168.29.46:9502", "stdout_lines": ["tcp://172.168.29.46:9502"]}
TASK [Set master replica ip] ***************************************************
2020-08-11T13:53:33.861285 (delta: 0.879976) elapsed: 887.154043 ******
ok: [127.0.0.1] => {"ansible_facts": {"replica_ip": "172.168.29.46"}, "changed": false}
TASK [debug] *******************************************************************
2020-08-11T13:53:33.923373 (delta: 0.062047) elapsed: 887.216131 ******
ok: [127.0.0.1] => {
"replica_ip": "172.168.29.46"
}
TASK [Getting the master replica pod name corresponding to the IP] *************
2020-08-11T13:53:33.985448 (delta: 0.062036) elapsed: 887.278206 ******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pod -l openebs.io/replica=jiva-replica,openebs.io/persistent-volume=pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6 -n openebs --no-headers -o jsonpath='{.items[?(@.status.podIP==\"172.168.29.46\")].metadata.name}'", "delta": "0:00:00.798880", "end": "2020-08-11 13:53:34.936070", "rc": 0, "start": "2020-08-11 13:53:34.137190", "stderr": "", "stderr_lines": [], "stdout": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-b49n9", "stdout_lines": ["pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-b49n9"]}
TASK [Delete master replica] ***************************************************
2020-08-11T13:53:34.971291 (delta: 0.985758) elapsed: 888.264049 ******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl delete pod pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-b49n9 -n openebs", "delta": "0:00:11.693142", "end": "2020-08-11 13:53:46.834679", "rc": 0, "start": "2020-08-11 13:53:35.141537", "stderr": "", "stderr_lines": [], "stdout": "pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-b49n9\" deleted", "stdout_lines": ["pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-b49n9\" deleted"]}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:53:46.864092 (delta: 11.892763) elapsed: 900.15685 ******
Pausing for 60 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [127.0.0.1] => {"changed": false, "delta": 60, "echo": true, "rc": 0, "start": "2020-08-11 13:53:46.909494", "stderr": "", "stdout": "Paused for 1.0 minutes", "stop": "2020-08-11 13:54:46.909634", "user_input": ""}
TASK [Get the name of replica pods] ********************************************
2020-08-11T13:54:46.919490 (delta: 60.055357) elapsed: 960.212248 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Set the replica pod name to variable based on latest creation timestamp] ***
2020-08-11T13:54:46.953921 (delta: 0.034392) elapsed: 960.246679 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Delete random replica] ***************************************************
2020-08-11T13:54:46.984852 (delta: 0.030891) elapsed: 960.27761 *******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:54:47.023390 (delta: 0.038498) elapsed: 960.316148 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Getting master replica IP] ***********************************************
2020-08-11T13:54:47.059492 (delta: 0.036058) elapsed: 960.35225 *******
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "curl http://12.0.34.99:9501/v1/replicas | jq '.data[0].address' | sed -e 's/^\"//' -e 's/\"$//'", "delta": "0:00:00.656437", "end": "2020-08-11 13:54:47.877266", "rc": 0, "start": "2020-08-11 13:54:47.220829", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0", "100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k"], "stdout": "tcp://172.168.166.88:9502", "stdout_lines": ["tcp://172.168.166.88:9502"]}
TASK [Set master replica ip] ***************************************************
2020-08-11T13:54:47.926855 (delta: 0.867321) elapsed: 961.219613 ******
ok: [127.0.0.1] => {"ansible_facts": {"replica_ip": "172.168.166.88"}, "changed": false}
TASK [debug] *******************************************************************
2020-08-11T13:54:47.989550 (delta: 0.06264) elapsed: 961.282308 *******
ok: [127.0.0.1] => {
"replica_ip": "172.168.166.88"
}
TASK [Getting the master replica pod name corresponding to the IP] *************
2020-08-11T13:54:48.045776 (delta: 0.056131) elapsed: 961.338534 ******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pod -l openebs.io/replica=jiva-replica,openebs.io/persistent-volume=pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6 -n openebs --no-headers -o jsonpath='{.items[?(@.status.podIP==\"172.168.166.88\")].metadata.name}'", "delta": "0:00:00.972052", "end": "2020-08-11 13:54:49.173681", "rc": 0, "start": "2020-08-11 13:54:48.201629", "stderr": "", "stderr_lines": [], "stdout": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-hpk52", "stdout_lines": ["pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-hpk52"]}
TASK [Delete master replica] ***************************************************
2020-08-11T13:54:49.216259 (delta: 1.170444) elapsed: 962.509017 ******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl delete pod pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-hpk52 -n openebs", "delta": "0:00:08.159453", "end": "2020-08-11 13:54:57.544861", "rc": 0, "start": "2020-08-11 13:54:49.385408", "stderr": "", "stderr_lines": [], "stdout": "pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-hpk52\" deleted", "stdout_lines": ["pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-hpk52\" deleted"]}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:54:57.575430 (delta: 8.359134) elapsed: 970.868188 ******
Pausing for 60 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [127.0.0.1] => {"changed": false, "delta": 60, "echo": true, "rc": 0, "start": "2020-08-11 13:54:57.619569", "stderr": "", "stdout": "Paused for 1.0 minutes", "stop": "2020-08-11 13:55:57.619725", "user_input": ""}
TASK [Get the name of replica pods] ********************************************
2020-08-11T13:55:57.630028 (delta: 60.054556) elapsed: 1030.922786 ****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Set the replica pod name to variable based on latest creation timestamp] ***
2020-08-11T13:55:57.661533 (delta: 0.031468) elapsed: 1030.954291 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Delete random replica] ***************************************************
2020-08-11T13:55:57.691974 (delta: 0.030403) elapsed: 1030.984732 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:55:57.722198 (delta: 0.030185) elapsed: 1031.014956 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Getting master replica IP] ***********************************************
2020-08-11T13:55:57.752073 (delta: 0.029838) elapsed: 1031.044831 *****
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "curl http://12.0.34.99:9501/v1/replicas | jq '.data[0].address' | sed -e 's/^\"//' -e 's/\"$//'", "delta": "0:00:00.685794", "end": "2020-08-11 13:55:58.606201", "rc": 0, "start": "2020-08-11 13:55:57.920407", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0", "100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k"], "stdout": "tcp://172.168.125.225:9502", "stdout_lines": ["tcp://172.168.125.225:9502"]}
TASK [Set master replica ip] ***************************************************
2020-08-11T13:55:58.651162 (delta: 0.89905) elapsed: 1031.94392 *******
ok: [127.0.0.1] => {"ansible_facts": {"replica_ip": "172.168.125.225"}, "changed": false}
TASK [debug] *******************************************************************
2020-08-11T13:55:58.707335 (delta: 0.056135) elapsed: 1032.000093 *****
ok: [127.0.0.1] => {
"replica_ip": "172.168.125.225"
}
TASK [Getting the master replica pod name corresponding to the IP] *************
2020-08-11T13:55:58.770225 (delta: 0.062837) elapsed: 1032.062983 *****
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pod -l openebs.io/replica=jiva-replica,openebs.io/persistent-volume=pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6 -n openebs --no-headers -o jsonpath='{.items[?(@.status.podIP==\"172.168.125.225\")].metadata.name}'", "delta": "0:00:00.815352", "end": "2020-08-11 13:55:59.745869", "rc": 0, "start": "2020-08-11 13:55:58.930517", "stderr": "", "stderr_lines": [], "stdout": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-l2brz", "stdout_lines": ["pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-l2brz"]}
TASK [Delete master replica] ***************************************************
2020-08-11T13:55:59.785919 (delta: 1.015655) elapsed: 1033.078677 *****
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl delete pod pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-l2brz -n openebs", "delta": "0:00:04.375257", "end": "2020-08-11 13:56:04.322440", "rc": 0, "start": "2020-08-11 13:55:59.947183", "stderr": "", "stderr_lines": [], "stdout": "pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-l2brz\" deleted", "stdout_lines": ["pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-l2brz\" deleted"]}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:56:04.354900 (delta: 4.568928) elapsed: 1037.647658 *****
Pausing for 60 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [127.0.0.1] => {"changed": false, "delta": 60, "echo": true, "rc": 0, "start": "2020-08-11 13:56:04.399223", "stderr": "", "stdout": "Paused for 1.0 minutes", "stop": "2020-08-11 13:57:04.400136", "user_input": ""}
TASK [Get the name of replica pods] ********************************************
2020-08-11T13:57:04.410326 (delta: 60.055383) elapsed: 1097.703084 ****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Set the replica pod name to variable based on latest creation timestamp] ***
2020-08-11T13:57:04.449763 (delta: 0.039369) elapsed: 1097.742521 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Delete random replica] ***************************************************
2020-08-11T13:57:04.480828 (delta: 0.031022) elapsed: 1097.773586 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:57:04.515016 (delta: 0.034149) elapsed: 1097.807774 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Getting master replica IP] ***********************************************
2020-08-11T13:57:04.544155 (delta: 0.029097) elapsed: 1097.836913 *****
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "curl http://12.0.34.99:9501/v1/replicas | jq '.data[0].address' | sed -e 's/^\"//' -e 's/\"$//'", "delta": "0:00:00.730652", "end": "2020-08-11 13:57:05.439985", "rc": 0, "start": "2020-08-11 13:57:04.709333", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0", "100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k"], "stdout": "tcp://172.168.29.47:9502", "stdout_lines": ["tcp://172.168.29.47:9502"]}
TASK [Set master replica ip] ***************************************************
2020-08-11T13:57:05.482914 (delta: 0.938716) elapsed: 1098.775672 *****
ok: [127.0.0.1] => {"ansible_facts": {"replica_ip": "172.168.29.47"}, "changed": false}
TASK [debug] *******************************************************************
2020-08-11T13:57:05.542497 (delta: 0.059541) elapsed: 1098.835255 *****
ok: [127.0.0.1] => {
"replica_ip": "172.168.29.47"
}
TASK [Getting the master replica pod name corresponding to the IP] *************
2020-08-11T13:57:05.595424 (delta: 0.052867) elapsed: 1098.888182 *****
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pod -l openebs.io/replica=jiva-replica,openebs.io/persistent-volume=pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6 -n openebs --no-headers -o jsonpath='{.items[?(@.status.podIP==\"172.168.29.47\")].metadata.name}'", "delta": "0:00:00.747845", "end": "2020-08-11 13:57:06.498480", "rc": 0, "start": "2020-08-11 13:57:05.750635", "stderr": "", "stderr_lines": [], "stdout": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-c56cn", "stdout_lines": ["pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-c56cn"]}
TASK [Delete master replica] ***************************************************
2020-08-11T13:57:06.534997 (delta: 0.939532) elapsed: 1099.827755 *****
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl delete pod pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-c56cn -n openebs", "delta": "0:00:10.138363", "end": "2020-08-11 13:57:16.836768", "rc": 0, "start": "2020-08-11 13:57:06.698405", "stderr": "", "stderr_lines": [], "stdout": "pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-c56cn\" deleted", "stdout_lines": ["pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-2-64c4f46d5b-c56cn\" deleted"]}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:57:16.869081 (delta: 10.334044) elapsed: 1110.161839 ****
Pausing for 60 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [127.0.0.1] => {"changed": false, "delta": 60, "echo": true, "rc": 0, "start": "2020-08-11 13:57:16.916206", "stderr": "", "stdout": "Paused for 1.0 minutes", "stop": "2020-08-11 13:58:16.916469", "user_input": ""}
TASK [Get the name of replica pods] ********************************************
2020-08-11T13:58:16.929215 (delta: 60.060094) elapsed: 1170.221973 ****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Set the replica pod name to variable based on latest creation timestamp] ***
2020-08-11T13:58:16.964159 (delta: 0.03491) elapsed: 1170.256917 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Delete random replica] ***************************************************
2020-08-11T13:58:16.997802 (delta: 0.033584) elapsed: 1170.29056 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:58:17.030513 (delta: 0.032669) elapsed: 1170.323271 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Getting master replica IP] ***********************************************
2020-08-11T13:58:17.061882 (delta: 0.031327) elapsed: 1170.35464 ******
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "curl http://12.0.34.99:9501/v1/replicas | jq '.data[0].address' | sed -e 's/^\"//' -e 's/\"$//'", "delta": "0:00:00.683171", "end": "2020-08-11 13:58:17.922632", "rc": 0, "start": "2020-08-11 13:58:17.239461", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0", "100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k"], "stdout": "tcp://172.168.166.89:9502", "stdout_lines": ["tcp://172.168.166.89:9502"]}
TASK [Set master replica ip] ***************************************************
2020-08-11T13:58:17.961599 (delta: 0.899677) elapsed: 1171.254357 *****
ok: [127.0.0.1] => {"ansible_facts": {"replica_ip": "172.168.166.89"}, "changed": false}
TASK [debug] *******************************************************************
2020-08-11T13:58:18.017819 (delta: 0.05618) elapsed: 1171.310577 ******
ok: [127.0.0.1] => {
"replica_ip": "172.168.166.89"
}
TASK [Getting the master replica pod name corresponding to the IP] *************
2020-08-11T13:58:18.071542 (delta: 0.053683) elapsed: 1171.3643 *******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pod -l openebs.io/replica=jiva-replica,openebs.io/persistent-volume=pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6 -n openebs --no-headers -o jsonpath='{.items[?(@.status.podIP==\"172.168.166.89\")].metadata.name}'", "delta": "0:00:00.812715", "end": "2020-08-11 13:58:19.038140", "rc": 0, "start": "2020-08-11 13:58:18.225425", "stderr": "", "stderr_lines": [], "stdout": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-bcwd8", "stdout_lines": ["pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-bcwd8"]}
TASK [Delete master replica] ***************************************************
2020-08-11T13:58:19.081523 (delta: 1.009943) elapsed: 1172.374281 *****
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl delete pod pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-bcwd8 -n openebs", "delta": "0:00:08.287816", "end": "2020-08-11 13:58:27.544823", "rc": 0, "start": "2020-08-11 13:58:19.257007", "stderr": "", "stderr_lines": [], "stdout": "pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-bcwd8\" deleted", "stdout_lines": ["pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-3-6645b5cf66-bcwd8\" deleted"]}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:58:27.574636 (delta: 8.493074) elapsed: 1180.867394 *****
Pausing for 60 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [127.0.0.1] => {"changed": false, "delta": 60, "echo": true, "rc": 0, "start": "2020-08-11 13:58:27.618234", "stderr": "", "stdout": "Paused for 1.0 minutes", "stop": "2020-08-11 13:59:27.618403", "user_input": ""}
TASK [Get the name of replica pods] ********************************************
2020-08-11T13:59:27.628532 (delta: 60.053855) elapsed: 1240.92129 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Set the replica pod name to variable based on latest creation timestamp] ***
2020-08-11T13:59:27.659572 (delta: 0.031003) elapsed: 1240.95233 ******
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Delete random replica] ***************************************************
2020-08-11T13:59:27.692195 (delta: 0.032564) elapsed: 1240.984953 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:59:27.722005 (delta: 0.029763) elapsed: 1241.014763 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Getting master replica IP] ***********************************************
2020-08-11T13:59:27.752863 (delta: 0.030821) elapsed: 1241.045621 *****
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "curl http://12.0.34.99:9501/v1/replicas | jq '.data[0].address' | sed -e 's/^\"//' -e 's/\"$//'", "delta": "0:00:00.669908", "end": "2020-08-11 13:59:28.588536", "rc": 0, "start": "2020-08-11 13:59:27.918628", "stderr": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k", "stderr_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0", "100 1477 100 1477 0 0 1442k 0 --:--:-- --:--:-- --:--:-- 1442k"], "stdout": "tcp://172.168.125.226:9502", "stdout_lines": ["tcp://172.168.125.226:9502"]}
TASK [Set master replica ip] ***************************************************
2020-08-11T13:59:28.636586 (delta: 0.883684) elapsed: 1241.929344 *****
ok: [127.0.0.1] => {"ansible_facts": {"replica_ip": "172.168.125.226"}, "changed": false}
TASK [debug] *******************************************************************
2020-08-11T13:59:28.698735 (delta: 0.062063) elapsed: 1241.991493 *****
ok: [127.0.0.1] => {
"replica_ip": "172.168.125.226"
}
TASK [Getting the master replica pod name corresponding to the IP] *************
2020-08-11T13:59:28.752869 (delta: 0.054063) elapsed: 1242.045627 *****
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pod -l openebs.io/replica=jiva-replica,openebs.io/persistent-volume=pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6 -n openebs --no-headers -o jsonpath='{.items[?(@.status.podIP==\"172.168.125.226\")].metadata.name}'", "delta": "0:00:01.083004", "end": "2020-08-11 13:59:30.005878", "rc": 0, "start": "2020-08-11 13:59:28.922874", "stderr": "", "stderr_lines": [], "stdout": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-g425r", "stdout_lines": ["pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-g425r"]}
TASK [Delete master replica] ***************************************************
2020-08-11T13:59:30.037873 (delta: 1.284947) elapsed: 1243.330631 *****
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl delete pod pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-g425r -n openebs", "delta": "0:00:02.545366", "end": "2020-08-11 13:59:32.733776", "rc": 0, "start": "2020-08-11 13:59:30.188410", "stderr": "", "stderr_lines": [], "stdout": "pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-g425r\" deleted", "stdout_lines": ["pod \"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-rep-1-666f7c78cd-g425r\" deleted"]}
TASK [Pause for some time for a new replica to come up] ************************
2020-08-11T13:59:32.767464 (delta: 2.72955) elapsed: 1246.060222 ******
Pausing for 60 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [127.0.0.1] => {"changed": false, "delta": 60, "echo": true, "rc": 0, "start": "2020-08-11 13:59:32.813464", "stderr": "", "stdout": "Paused for 1.0 minutes", "stop": "2020-08-11 14:00:32.814918", "user_input": ""}
TASK [Include replica count test] **********************************************
2020-08-11T14:00:32.824916 (delta: 60.057409) elapsed: 1306.117674 ****
included: /experiments/functional/snapshot_deletion/jiva/check_replica_count.yml for 127.0.0.1
TASK [Get total replica count from controller] *********************************
2020-08-11T14:00:32.917384 (delta: 0.092414) elapsed: 1306.210142 *****
ok: [127.0.0.1] => {"attempts": 1, "changed": false, "connection": "close", "content": "{\"data\":[{\"actions\":{\"deleteSnapshot\":\"http://12.0.34.99:9501/v1/volumes/cHZjLTk2NTc2Y2RmLWViYjctNGE0OC1iNDJjLTk5YjljZDA1MjRhNg==?action=deleteSnapshot\",\"resize\":\"http://12.0.34.99:9501/v1/volumes/cHZjLTk2NTc2Y2RmLWViYjctNGE0OC1iNDJjLTk5YjljZDA1MjRhNg==?action=resize\",\"revert\":\"http://12.0.34.99:9501/v1/volumes/cHZjLTk2NTc2Y2RmLWViYjctNGE0OC1iNDJjLTk5YjljZDA1MjRhNg==?action=revert\",\"setlogging\":\"http://12.0.34.99:9501/v1/volumes/cHZjLTk2NTc2Y2RmLWViYjctNGE0OC1iNDJjLTk5YjljZDA1MjRhNg==?action=setlogging\",\"shutdown\":\"http://12.0.34.99:9501/v1/volumes/cHZjLTk2NTc2Y2RmLWViYjctNGE0OC1iNDJjLTk5YjljZDA1MjRhNg==?action=shutdown\",\"snapshot\":\"http://12.0.34.99:9501/v1/volumes/cHZjLTk2NTc2Y2RmLWViYjctNGE0OC1iNDJjLTk5YjljZDA1MjRhNg==?action=snapshot\"},\"id\":\"cHZjLTk2NTc2Y2RmLWViYjctNGE0OC1iNDJjLTk5YjljZDA1MjRhNg==\",\"links\":{\"self\":\"http://12.0.34.99:9501/v1/volumes/cHZjLTk2NTc2Y2RmLWViYjctNGE0OC1iNDJjLTk5YjljZDA1MjRhNg==\"},\"name\":\"pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6\",\"readOnly\":\"false\",\"replicaCount\":3,\"type\":\"volume\"}],\"links\":{\"self\":\"http://12.0.34.99:9501/v1/volumes\"},\"resourceType\":\"volume\",\"type\":\"collection\"}\n", "content_length": "1126", "content_type": "application/json", "cookies": {}, "cookies_string": "", "date": "Tue, 11 Aug 2020 14:00:37 GMT", "json": {"data": [{"actions": {"deleteSnapshot": "http://12.0.34.99:9501/v1/volumes/cHZjLTk2NTc2Y2RmLWViYjctNGE0OC1iNDJjLTk5YjljZDA1MjRhNg==?action=deleteSnapshot", "resize": "http://12.0.34.99:9501/v1/volumes/cHZjLTk2NTc2Y2RmLWViYjctNGE0OC1iNDJjLTk5YjljZDA1MjRhNg==?action=resize", "revert": "http://12.0.34.99:9501/v1/volumes/cHZjLTk2NTc2Y2RmLWViYjctNGE0OC1iNDJjLTk5YjljZDA1MjRhNg==?action=revert", "setlogging": "http://12.0.34.99:9501/v1/volumes/cHZjLTk2NTc2Y2RmLWViYjctNGE0OC1iNDJjLTk5YjljZDA1MjRhNg==?action=setlogging", "shutdown": "http://12.0.34.99:9501/v1/volumes/cHZjLTk2NTc2Y2RmLWViYjctNGE0OC1iNDJjLTk5YjljZDA1MjRhNg==?action=shutdown", "snapshot": "http://12.0.34.99:9501/v1/volumes/cHZjLTk2NTc2Y2RmLWViYjctNGE0OC1iNDJjLTk5YjljZDA1MjRhNg==?action=snapshot"}, "id": "cHZjLTk2NTc2Y2RmLWViYjctNGE0OC1iNDJjLTk5YjljZDA1MjRhNg==", "links": {"self": "http://12.0.34.99:9501/v1/volumes/cHZjLTk2NTc2Y2RmLWViYjctNGE0OC1iNDJjLTk5YjljZDA1MjRhNg=="}, "name": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6", "readOnly": "false", "replicaCount": 3, "type": "volume"}], "links": {"self": "http://12.0.34.99:9501/v1/volumes"}, "resourceType": "volume", "type": "collection"}, "msg": "OK (1126 bytes)", "redirected": false, "status": 200, "url": "http://12.0.34.99:9501/v1/volumes", "x_api_schemas": "http://12.0.34.99:9501/v1/schemas"}
TASK [Get rw replica status from controller] ***********************************
2020-08-11T14:00:33.376597 (delta: 0.459176) elapsed: 1306.669355 *****
ok: [127.0.0.1] => {"attempts": 1, "changed": false, "connection": "close", "content": "{\"createTypes\":{\"replica\":\"http://12.0.34.99:9501/v1/replicas\"},\"data\":[{\"actions\":{\"preparerebuild\":\"http://12.0.34.99:9501/v1/replicas/dGNwOi8vMTcyLjE2OC4yOS40ODo5NTAy?action=preparerebuild\",\"verifyrebuild\":\"http://12.0.34.99:9501/v1/replicas/dGNwOi8vMTcyLjE2OC4yOS40ODo5NTAy?action=verifyrebuild\"},\"address\":\"tcp://172.168.29.48:9502\",\"id\":\"dGNwOi8vMTcyLjE2OC4yOS40ODo5NTAy\",\"links\":{\"self\":\"http://12.0.34.99:9501/v1/replicas/dGNwOi8vMTcyLjE2OC4yOS40ODo5NTAy\"},\"mode\":\"RW\",\"type\":\"replica\"},{\"actions\":{\"preparerebuild\":\"http://12.0.34.99:9501/v1/replicas/dGNwOi8vMTcyLjE2OC4xNjYuOTA6OTUwMg==?action=preparerebuild\",\"verifyrebuild\":\"http://12.0.34.99:9501/v1/replicas/dGNwOi8vMTcyLjE2OC4xNjYuOTA6OTUwMg==?action=verifyrebuild\"},\"address\":\"tcp://172.168.166.90:9502\",\"id\":\"dGNwOi8vMTcyLjE2OC4xNjYuOTA6OTUwMg==\",\"links\":{\"self\":\"http://12.0.34.99:9501/v1/replicas/dGNwOi8vMTcyLjE2OC4xNjYuOTA6OTUwMg==\"},\"mode\":\"RW\",\"type\":\"replica\"},{\"actions\":{\"preparerebuild\":\"http://12.0.34.99:9501/v1/replicas/dGNwOi8vMTcyLjE2OC4xMjUuMjI3Ojk1MDI=?action=preparerebuild\",\"verifyrebuild\":\"http://12.0.34.99:9501/v1/replicas/dGNwOi8vMTcyLjE2OC4xMjUuMjI3Ojk1MDI=?action=verifyrebuild\"},\"address\":\"tcp://172.168.125.227:9502\",\"id\":\"dGNwOi8vMTcyLjE2OC4xMjUuMjI3Ojk1MDI=\",\"links\":{\"self\":\"http://12.0.34.99:9501/v1/replicas/dGNwOi8vMTcyLjE2OC4xMjUuMjI3Ojk1MDI=\"},\"mode\":\"RW\",\"type\":\"replica\"}],\"links\":{\"self\":\"http://12.0.34.99:9501/v1/replicas\"},\"resourceType\":\"replica\",\"type\":\"collection\"}\n", "content_length": "1477", "content_type": "application/json", "cookies": {}, "cookies_string": "", "date": "Tue, 11 Aug 2020 14:00:37 GMT", "json": {"createTypes": {"replica": "http://12.0.34.99:9501/v1/replicas"}, "data": [{"actions": {"preparerebuild": "http://12.0.34.99:9501/v1/replicas/dGNwOi8vMTcyLjE2OC4yOS40ODo5NTAy?action=preparerebuild", "verifyrebuild": "http://12.0.34.99:9501/v1/replicas/dGNwOi8vMTcyLjE2OC4yOS40ODo5NTAy?action=verifyrebuild"}, "address": "tcp://172.168.29.48:9502", "id": "dGNwOi8vMTcyLjE2OC4yOS40ODo5NTAy", "links": {"self": "http://12.0.34.99:9501/v1/replicas/dGNwOi8vMTcyLjE2OC4yOS40ODo5NTAy"}, "mode": "RW", "type": "replica"}, {"actions": {"preparerebuild": "http://12.0.34.99:9501/v1/replicas/dGNwOi8vMTcyLjE2OC4xNjYuOTA6OTUwMg==?action=preparerebuild", "verifyrebuild": "http://12.0.34.99:9501/v1/replicas/dGNwOi8vMTcyLjE2OC4xNjYuOTA6OTUwMg==?action=verifyrebuild"}, "address": "tcp://172.168.166.90:9502", "id": "dGNwOi8vMTcyLjE2OC4xNjYuOTA6OTUwMg==", "links": {"self": "http://12.0.34.99:9501/v1/replicas/dGNwOi8vMTcyLjE2OC4xNjYuOTA6OTUwMg=="}, "mode": "RW", "type": "replica"}, {"actions": {"preparerebuild": "http://12.0.34.99:9501/v1/replicas/dGNwOi8vMTcyLjE2OC4xMjUuMjI3Ojk1MDI=?action=preparerebuild", "verifyrebuild": "http://12.0.34.99:9501/v1/replicas/dGNwOi8vMTcyLjE2OC4xMjUuMjI3Ojk1MDI=?action=verifyrebuild"}, "address": "tcp://172.168.125.227:9502", "id": "dGNwOi8vMTcyLjE2OC4xMjUuMjI3Ojk1MDI=", "links": {"self": "http://12.0.34.99:9501/v1/replicas/dGNwOi8vMTcyLjE2OC4xMjUuMjI3Ojk1MDI="}, "mode": "RW", "type": "replica"}], "links": {"self": "http://12.0.34.99:9501/v1/replicas"}, "resourceType": "replica", "type": "collection"}, "msg": "OK (1477 bytes)", "redirected": false, "status": 200, "url": "http://12.0.34.99:9501/v1/replicas", "x_api_schemas": "http://12.0.34.99:9501/v1/schemas"}
TASK [Check if fio write job is completed] *************************************
2020-08-11T14:00:33.691253 (delta: 0.314599) elapsed: 1306.984011 *****
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "kubectl get pods -n fio-t -o jsonpath='{.items[?(@.metadata.labels.name==\"fio-write\")].status.containerStatuses[*].state.terminated.reason}'", "delta": "0:00:00.864798", "end": "2020-08-11 14:00:34.745060", "rc": 0, "start": "2020-08-11 14:00:33.880262", "stderr": "", "stderr_lines": [], "stdout": "Completed", "stdout_lines": ["Completed"]}
TASK [Verify the fio logs to check if run is complete w/o errors] **************
2020-08-11T14:00:34.789042 (delta: 1.097751) elapsed: 1308.0818 *******
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl logs fio-b8nzw -n fio-t | grep -i error | cut -d \":\" -f 2 | sort | uniq", "delta": "0:00:00.812885", "end": "2020-08-11 14:00:35.765761", "failed_when_result": false, "rc": 0, "start": "2020-08-11 14:00:34.952876", "stderr": "", "stderr_lines": [], "stdout": " 0,", "stdout_lines": [" 0,"]}
TASK [Deploy fio read test job] ************************************************
2020-08-11T14:00:35.810044 (delta: 1.020953) elapsed: 1309.102802 *****
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl apply -f fio-read.yml -n fio-t", "delta": "0:00:00.982254", "end": "2020-08-11 14:00:36.954148", "rc": 0, "start": "2020-08-11 14:00:35.971894", "stderr": "", "stderr_lines": [], "stdout": "configmap/basic-read created\njob.batch/fio-read created", "stdout_lines": ["configmap/basic-read created", "job.batch/fio-read created"]}
TASK [Obtaining the fio read job pod name] *************************************
2020-08-11T14:00:36.991893 (delta: 1.181794) elapsed: 1310.284651 *****
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pods -n fio-t -l name=fio-read -o custom-columns=:metadata.name --no-headers", "delta": "0:00:00.903388", "end": "2020-08-11 14:00:38.058554", "rc": 0, "start": "2020-08-11 14:00:37.155166", "stderr": "", "stderr_lines": [], "stdout": "fio-read-xbjzk", "stdout_lines": ["fio-read-xbjzk"]}
TASK [Check if fio read job is completed] **************************************
2020-08-11T14:00:38.097701 (delta: 1.105768) elapsed: 1311.390459 *****
FAILED - RETRYING: Check if fio read job is completed (20 retries left).
FAILED - RETRYING: Check if fio read job is completed (19 retries left).
changed: [127.0.0.1] => {"attempts": 3, "changed": true, "cmd": "kubectl get pods -n fio-t -o jsonpath='{.items[?(@.metadata.labels.name==\"fio-read\")].status.containerStatuses[*].state.terminated.reason}'", "delta": "0:00:01.023107", "end": "2020-08-11 14:02:41.280837", "rc": 0, "start": "2020-08-11 14:02:40.257730", "stderr": "", "stderr_lines": [], "stdout": "Completed", "stdout_lines": ["Completed"]}
TASK [Verify the data integrity check] *****************************************
2020-08-11T14:02:41.329103 (delta: 123.231363) elapsed: 1434.621861 ***
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl logs fio-read-xbjzk -n fio-t | grep -i '\"error\"' | cut -d \":\" -f 2 | sort | uniq", "delta": "0:00:01.234213", "end": "2020-08-11 14:02:42.725462", "failed_when_result": false, "rc": 0, "start": "2020-08-11 14:02:41.491249", "stderr": "", "stderr_lines": [], "stdout": " 0,", "stdout_lines": [" 0,"]}
TASK [Verify Snapshot deletion] ************************************************
2020-08-11T14:02:42.764630 (delta: 1.435488) elapsed: 1436.057388 *****
included: /experiments/functional/snapshot_deletion/jiva/snapshot_delete_verify_test.yml for 127.0.0.1
TASK [Get the name of controller pod] ******************************************
2020-08-11T14:02:42.858111 (delta: 0.093447) elapsed: 1436.150869 *****
changed: [127.0.0.1] => {"changed": true, "cmd": "kubectl get pods -l openebs.io/controller=jiva-controller,openebs.io/persistent-volume=pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6 -n openebs -o jsonpath='{range .items[*]}{.metadata.name}{\"\\n\"}{end}'", "delta": "0:00:00.781524", "end": "2020-08-11 14:02:43.801335", "rc": 0, "start": "2020-08-11 14:02:43.019811", "stderr": "", "stderr_lines": [], "stdout": "pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-ctrl-567f886c8f-99rt9", "stdout_lines": ["pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-ctrl-567f886c8f-99rt9"]}
TASK [Getting number of snapshots] *********************************************
2020-08-11T14:02:43.838024 (delta: 0.979859) elapsed: 1437.130782 *****
changed: [127.0.0.1] => {"attempts": 1, "changed": true, "cmd": "kubectl exec -it pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-ctrl-567f886c8f-99rt9 -n openebs -- jivactl snapshot ls | grep -v ID | wc -l", "delta": "0:00:00.963510", "end": "2020-08-11 14:02:44.957768", "rc": 0, "start": "2020-08-11 14:02:43.994258", "stderr": "Defaulting container name to pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-ctrl-con.\nUse 'kubectl describe pod/pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-ctrl-567f886c8f-99rt9 -n openebs' to see all of the containers in this pod.\nUnable to use a TTY - input is not a terminal or the right kind of file", "stderr_lines": ["Defaulting container name to pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-ctrl-con.", "Use 'kubectl describe pod/pvc-96576cdf-ebb7-4a48-b42c-99b9cd0524a6-ctrl-567f886c8f-99rt9 -n openebs' to see all of the containers in this pod.", "Unable to use a TTY - input is not a terminal or the right kind of file"], "stdout": "11", "stdout_lines": ["11"]}
TASK [set_fact] ****************************************************************
2020-08-11T14:02:44.998241 (delta: 1.160177) elapsed: 1438.290999 *****
ok: [127.0.0.1] => {"ansible_facts": {"flag": "Pass"}, "changed": false}
TASK [include_tasks] ***********************************************************
2020-08-11T14:02:45.043967 (delta: 0.045689) elapsed: 1438.336725 *****
included: /utils/fcm/update_litmus_result_resource.yml for 127.0.0.1
TASK [Generate the litmus result CR to reflect SOT (Start of Test)] ************
2020-08-11T14:02:45.127943 (delta: 0.083937) elapsed: 1438.420701 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Analyze the cr yaml] *****************************************************
2020-08-11T14:02:45.154657 (delta: 0.026662) elapsed: 1438.447415 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Apply the litmus result CR] **********************************************
2020-08-11T14:02:45.185153 (delta: 0.030459) elapsed: 1438.477911 *****
skipping: [127.0.0.1] => {"changed": false, "skip_reason": "Conditional result was False"}
TASK [Generate the litmus result CR to reflect EOT (End of Test)] **************
2020-08-11T14:02:45.219602 (delta: 0.03441) elapsed: 1438.51236 *******
changed: [127.0.0.1] => {"changed": true, "checksum": "7d051c9ab2830b420ab57e65c1a77919cc01ffae", "dest": "./litmus-result.yaml", "gid": 0, "group": "root", "md5sum": "27037d02b557dafa06fac7875ada3e9a", "mode": "0644", "owner": "root", "size": 427, "src": "/root/.ansible/tmp/ansible-tmp-1597154565.27-90962257648149/source", "state": "file", "uid": 0}
TASK [Analyze the cr yaml] *****************************************************
2020-08-11T14:02:47.008066 (delta: 1.788428) elapsed: 1440.300824 *****
changed: [127.0.0.1] => {"changed": true, "cmd": "cat litmus-result.yaml", "delta": "0:00:00.700614", "end": "2020-08-11 14:02:47.868954", "rc": 0, "start": "2020-08-11 14:02:47.168340", "stderr": "", "stderr_lines": [], "stdout": "---\napiVersion: litmus.io/v1alpha1\nkind: LitmusResult\nmetadata:\n\n # name of the litmus testcase\n name: fio-di-delete-jiva-snapshots \nspec:\n\n # holds information on the testcase\n testMetadata:\n app: \n chaostype: \n\n # holds the state of testcase, manually updated by json merge patch\n # result is the useful value today, but anticipate phase use in future \n testStatus: \n phase: completed \n result: Pass ", "stdout_lines": ["---", "apiVersion: litmus.io/v1alpha1", "kind: LitmusResult", "metadata:", "", " # name of the litmus testcase", " name: fio-di-delete-jiva-snapshots ", "spec:", "", " # holds information on the testcase", " testMetadata:", " app: ", " chaostype: ", "", " # holds the state of testcase, manually updated by json merge patch", " # result is the useful value today, but anticipate phase use in future ", " testStatus: ", " phase: completed ", " result: Pass "]}
TASK [Apply the litmus result CR] **********************************************
2020-08-11T14:02:47.901938 (delta: 0.893833) elapsed: 1441.194696 *****
changed: [127.0.0.1] => {"changed": true, "failed_when_result": false, "method": "patch", "result": {"apiVersion": "litmus.io/v1alpha1", "kind": "LitmusResult", "metadata": {"creationTimestamp": "2020-08-11T13:38:54Z", "generation": 2, "name": "fio-di-delete-jiva-snapshots", "resourceVersion": "3285834", "selfLink": "/apis/litmus.io/v1alpha1/litmusresults/fio-di-delete-jiva-snapshots", "uid": "01b11ac6-9989-4515-8a5f-ab760ebc41b4"}, "spec": {"testMetadata": {}, "testStatus": {"phase": "completed", "result": "Pass"}}}}
PLAY RECAP *********************************************************************
127.0.0.1 : ok=144 changed=72 unreachable=0 failed=0
2020-08-11T14:02:48.617900 (delta: 0.715923) elapsed: 1441.910658 *****
===============================================================================
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment