Skip to content

Instantly share code, notes, and snippets.

@Pothulapati
Created April 24, 2023 11:39
Show Gist options
  • Save Pothulapati/2e3acc35e8915fc5d5d3dd83cb51f87f to your computer and use it in GitHub Desktop.
Save Pothulapati/2e3acc35e8915fc5d5d3dd83cb51f87f to your computer and use it in GitHub Desktop.
2023-04-24T10:39:08Z INFO controller-runtime.metrics Metrics server is starting to listen {"addr": "127.0.0.1:8080"}
2023-04-24T10:39:08Z INFO setup starting manager
I0424 10:39:08.125347 1 leaderelection.go:248] attempting to acquire leader lease dragonfly-operator-system/31079dea.dragonflydb.io...
2023-04-24T10:39:08Z INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"}
2023-04-24T10:39:08Z INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
I0424 10:39:08.129063 1 leaderelection.go:258] successfully acquired lease dragonfly-operator-system/31079dea.dragonflydb.io
2023-04-24T10:39:08Z DEBUG events dragonfly-operator-controller-manager-78d587fbd9-k8zmw_c4c7cc7e-0336-4d01-8a49-44437a1f5cf7 became leader {"type": "Normal", "object": {"kind":"Lease","namespace":"dragonfly-operator-system","name":"31079dea.dragonflydb.io","uid":"52ca94b9-3253-4d3f-8db4-7103ed5a894c","apiVersion":"coordination.k8s.io/v1","resourceVersion":"530"}, "reason": "LeaderElection"}
2023-04-24T10:39:08Z INFO Starting EventSource {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "source": "kind source: *v1alpha1.Dragonfly"}
2023-04-24T10:39:08Z INFO Starting EventSource {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "source": "kind source: *v1.StatefulSet"}
2023-04-24T10:39:08Z INFO Starting EventSource {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "source": "kind source: *v1.Service"}
2023-04-24T10:39:08Z INFO Starting Controller {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly"}
2023-04-24T10:39:08Z INFO Starting EventSource {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2023-04-24T10:39:08Z INFO Starting Controller {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod"}
2023-04-24T10:39:08Z INFO Starting workers {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "worker count": 1}
2023-04-24T10:39:08Z INFO Starting workers {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "worker count": 1}
2023-04-24T10:39:08Z INFO Reconciling Dragonfly object {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "da067102-fa49-445b-b7b6-65c37b2464ee"}
2023-04-24T10:39:08Z INFO Creating resources {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "da067102-fa49-445b-b7b6-65c37b2464ee"}
2023-04-24T10:39:08Z INFO Creating resources for df-test {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "da067102-fa49-445b-b7b6-65c37b2464ee"}
2023-04-24T10:39:08Z INFO Created resources for object {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "da067102-fa49-445b-b7b6-65c37b2464ee"}
2023-04-24T10:39:08Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "807da821-f848-462d-8232-85d132d8e365", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:39:08Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "807da821-f848-462d-8232-85d132d8e365"}
2023-04-24T10:39:08Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "c5cb96ca-c54e-495b-a06b-f32c982e6105", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:39:08Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "c5cb96ca-c54e-495b-a06b-f32c982e6105"}
2023-04-24T10:39:08Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "09c5c689-c5ff-4f4e-9962-141598d88127", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:39:08Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "09c5c689-c5ff-4f4e-9962-141598d88127"}
2023-04-24T10:39:13Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "df77d10a-d077-4200-9447-38dbccad38e4", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:39:13Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "df77d10a-d077-4200-9447-38dbccad38e4"}
2023-04-24T10:39:18Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "7424e680-3122-4b54-af36-5a4769f17123", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:39:18Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "7424e680-3122-4b54-af36-5a4769f17123"}
2023-04-24T10:39:23Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "a39fea85-b72d-4fe0-ba9e-d1be3e267c01", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:39:23Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "a39fea85-b72d-4fe0-ba9e-d1be3e267c01"}
2023-04-24T10:39:23Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "c5ad7a83-aa6a-4640-bee2-cad82b32aeb1", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:39:23Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "c5ad7a83-aa6a-4640-bee2-cad82b32aeb1"}
2023-04-24T10:39:28Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "11d30def-04f2-4a89-aac4-6d9972d2edf4", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:39:28Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "11d30def-04f2-4a89-aac4-6d9972d2edf4"}
2023-04-24T10:39:33Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "80db827b-2b08-4aab-91dd-e0f6da75e003", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:39:33Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "80db827b-2b08-4aab-91dd-e0f6da75e003"}
2023-04-24T10:39:38Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "8ebadda7-3c99-4cbd-bd79-f0ea6456021a", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:39:38Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "8ebadda7-3c99-4cbd-bd79-f0ea6456021a"}
2023-04-24T10:39:39Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "8bc52ffe-52bd-4d27-976c-382ff1cc3ed4", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:39:39Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "8bc52ffe-52bd-4d27-976c-382ff1cc3ed4", "phase": "resources-created"}
2023-04-24T10:39:39Z INFO Dragonfly object is only initialized. Configuring replication for the first time {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "8bc52ffe-52bd-4d27-976c-382ff1cc3ed4"}
2023-04-24T10:39:39Z INFO Configuring replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "8bc52ffe-52bd-4d27-976c-382ff1cc3ed4"}
2023-04-24T10:39:39Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "8bc52ffe-52bd-4d27-976c-382ff1cc3ed4"}
2023-04-24T10:39:39Z INFO Marking pod as master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "8bc52ffe-52bd-4d27-976c-382ff1cc3ed4", "podName": "df-test-0", "ip": "10.244.0.6"}
2023-04-24T10:39:39Z INFO Running SLAVE OF NO ONE command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "8bc52ffe-52bd-4d27-976c-382ff1cc3ed4", "pod": "df-test-0"}
2023-04-24T10:39:39Z INFO Marking pod role as master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "8bc52ffe-52bd-4d27-976c-382ff1cc3ed4", "pod": "df-test-0"}
2023-04-24T10:39:39Z INFO Checking pod {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "8bc52ffe-52bd-4d27-976c-382ff1cc3ed4", "podName": "df-test-0", "ip": "10.244.0.6", "status": "Running", "deletiontimestamp": "<nil>"}
2023-04-24T10:39:39Z INFO Successfully marked 0/0 replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "8bc52ffe-52bd-4d27-976c-382ff1cc3ed4"}
2023-04-24T10:39:39Z INFO Updating status {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "8bc52ffe-52bd-4d27-976c-382ff1cc3ed4", "phase": "ready"}
2023-04-24T10:39:39Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "06cb8d88-5894-4d87-a581-d50c60ec2888", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:39:39Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "06cb8d88-5894-4d87-a581-d50c60ec2888"}
2023-04-24T10:39:39Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "040b4213-cdd7-4878-b614-eff4abd72cd4", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:39:39Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "040b4213-cdd7-4878-b614-eff4abd72cd4", "pod": "default/df-test-0", "role": "master"}
2023-04-24T10:39:39Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "040b4213-cdd7-4878-b614-eff4abd72cd4"}
2023-04-24T10:39:39Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "040b4213-cdd7-4878-b614-eff4abd72cd4"}
2023-04-24T10:39:39Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "040b4213-cdd7-4878-b614-eff4abd72cd4", "replicas": []}
2023-04-24T10:39:39Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "040b4213-cdd7-4878-b614-eff4abd72cd4"}
2023-04-24T10:39:39Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "040b4213-cdd7-4878-b614-eff4abd72cd4"}
2023-04-24T10:39:39Z INFO all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "040b4213-cdd7-4878-b614-eff4abd72cd4", "dfi": "df-test"}
2023-04-24T10:39:39Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "4efac518-e7e8-42fe-803d-ae03984e5abb", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:39:39Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "4efac518-e7e8-42fe-803d-ae03984e5abb"}
2023-04-24T10:39:39Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "3d13e370-d565-4cfb-9e8f-84ff4ee87a20", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:39:39Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "3d13e370-d565-4cfb-9e8f-84ff4ee87a20"}
2023-04-24T10:39:43Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "e3435d0e-6231-4af6-a5db-7f475d5c6cca", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:39:43Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "e3435d0e-6231-4af6-a5db-7f475d5c6cca", "pod": "default/df-test-0", "role": "master"}
2023-04-24T10:39:43Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "e3435d0e-6231-4af6-a5db-7f475d5c6cca"}
2023-04-24T10:39:43Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "e3435d0e-6231-4af6-a5db-7f475d5c6cca"}
2023-04-24T10:39:43Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "e3435d0e-6231-4af6-a5db-7f475d5c6cca", "replicas": []}
2023-04-24T10:39:43Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "e3435d0e-6231-4af6-a5db-7f475d5c6cca"}
2023-04-24T10:39:43Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "e3435d0e-6231-4af6-a5db-7f475d5c6cca"}
2023-04-24T10:39:43Z INFO all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "e3435d0e-6231-4af6-a5db-7f475d5c6cca", "dfi": "df-test"}
2023-04-24T10:39:43Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "593c9a8a-a7de-470f-a3e2-b1c504d08c86", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:39:43Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "593c9a8a-a7de-470f-a3e2-b1c504d08c86"}
2023-04-24T10:39:44Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "3a37d3ec-273e-48f9-a5c0-b56ee4342dc1", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:39:44Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "3a37d3ec-273e-48f9-a5c0-b56ee4342dc1"}
2023-04-24T10:39:49Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "390fd7f1-acaf-4f78-8fdf-75f01fdf9776", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:39:49Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "390fd7f1-acaf-4f78-8fdf-75f01fdf9776"}
2023-04-24T10:39:54Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "3cfd3684-5df7-4317-8462-003216f3c9a0", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:39:54Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "3cfd3684-5df7-4317-8462-003216f3c9a0"}
2023-04-24T10:39:59Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "efe8d804-b528-4139-ad55-2472591e7063", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:39:59Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "efe8d804-b528-4139-ad55-2472591e7063"}
2023-04-24T10:40:01Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "0cbaa06a-5de2-41b7-91c9-6c6017be5421", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:40:01Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "0cbaa06a-5de2-41b7-91c9-6c6017be5421", "phase": "ready"}
2023-04-24T10:40:01Z INFO Pod restart from a ready Dragonfly instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "0cbaa06a-5de2-41b7-91c9-6c6017be5421"}
2023-04-24T10:40:01Z INFO checking if a master exists already {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "0cbaa06a-5de2-41b7-91c9-6c6017be5421"}
2023-04-24T10:40:01Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "0cbaa06a-5de2-41b7-91c9-6c6017be5421"}
2023-04-24T10:40:01Z INFO Master exists. Configuring 10.244.0.7 as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "0cbaa06a-5de2-41b7-91c9-6c6017be5421"}
2023-04-24T10:40:01Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "0cbaa06a-5de2-41b7-91c9-6c6017be5421", "pod": "df-test-1"}
2023-04-24T10:40:01Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "0cbaa06a-5de2-41b7-91c9-6c6017be5421"}
2023-04-24T10:40:01Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "0cbaa06a-5de2-41b7-91c9-6c6017be5421"}
2023-04-24T10:40:01Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "0cbaa06a-5de2-41b7-91c9-6c6017be5421", "pod": "df-test-1", "master": "10.244.0.6"}
2023-04-24T10:40:01Z INFO Marking pod role as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "0cbaa06a-5de2-41b7-91c9-6c6017be5421", "pod": "df-test-1"}
2023-04-24T10:40:01Z INFO Updating status {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "0cbaa06a-5de2-41b7-91c9-6c6017be5421", "phase": "ready"}
2023-04-24T10:40:01Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "9513ae68-f928-445b-b3cb-cb6bbee9cedf", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:40:01Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "9513ae68-f928-445b-b3cb-cb6bbee9cedf"}
2023-04-24T10:40:01Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "efd3c1e8-051c-4b8b-b69b-b5c9259a0c49", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:40:01Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "efd3c1e8-051c-4b8b-b69b-b5c9259a0c49", "pod": "default/df-test-1", "role": "replica"}
2023-04-24T10:40:01Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "efd3c1e8-051c-4b8b-b69b-b5c9259a0c49"}
2023-04-24T10:40:01Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "efd3c1e8-051c-4b8b-b69b-b5c9259a0c49"}
2023-04-24T10:40:01Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "efd3c1e8-051c-4b8b-b69b-b5c9259a0c49", "replicas": ["df-test-1"]}
2023-04-24T10:40:01Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "efd3c1e8-051c-4b8b-b69b-b5c9259a0c49"}
2023-04-24T10:40:01Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "efd3c1e8-051c-4b8b-b69b-b5c9259a0c49"}
2023-04-24T10:40:01Z INFO all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "efd3c1e8-051c-4b8b-b69b-b5c9259a0c49", "dfi": "df-test"}
2023-04-24T10:40:03Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "f47e1b8e-e443-4545-b05c-f8351cd11081", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:40:03Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "f47e1b8e-e443-4545-b05c-f8351cd11081"}
2023-04-24T10:40:04Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "2a04558f-6dfb-4f9c-8a9a-8e9967acaba1", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:40:04Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "2a04558f-6dfb-4f9c-8a9a-8e9967acaba1", "pod": "default/df-test-1", "role": "replica"}
2023-04-24T10:40:04Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "2a04558f-6dfb-4f9c-8a9a-8e9967acaba1"}
2023-04-24T10:40:04Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "2a04558f-6dfb-4f9c-8a9a-8e9967acaba1"}
2023-04-24T10:40:04Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "2a04558f-6dfb-4f9c-8a9a-8e9967acaba1", "replicas": ["df-test-1"]}
2023-04-24T10:40:04Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "2a04558f-6dfb-4f9c-8a9a-8e9967acaba1"}
2023-04-24T10:40:04Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "2a04558f-6dfb-4f9c-8a9a-8e9967acaba1"}
2023-04-24T10:40:04Z INFO all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "2a04558f-6dfb-4f9c-8a9a-8e9967acaba1", "dfi": "df-test"}
2023-04-24T10:40:06Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "3ad0e00c-44a1-42d1-800e-e469a86b040c", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:40:06Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "3ad0e00c-44a1-42d1-800e-e469a86b040c"}
2023-04-24T10:40:11Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "61ffa941-79ed-48cc-bd6b-4dce415ef95d", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:40:11Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "61ffa941-79ed-48cc-bd6b-4dce415ef95d"}
2023-04-24T10:40:16Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "af62a656-1b23-45fe-a137-f643de46f811", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:40:16Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "af62a656-1b23-45fe-a137-f643de46f811"}
2023-04-24T10:40:21Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "441a4867-cfd9-4806-a601-3ba8d9e64448", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:40:21Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "441a4867-cfd9-4806-a601-3ba8d9e64448"}
2023-04-24T10:40:22Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "ed324f55-eb5d-496d-99d8-6ded7b8c8f6a", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:40:22Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "ed324f55-eb5d-496d-99d8-6ded7b8c8f6a", "phase": "ready"}
2023-04-24T10:40:22Z INFO Pod restart from a ready Dragonfly instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "ed324f55-eb5d-496d-99d8-6ded7b8c8f6a"}
2023-04-24T10:40:22Z INFO checking if a master exists already {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "ed324f55-eb5d-496d-99d8-6ded7b8c8f6a"}
2023-04-24T10:40:22Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "ed324f55-eb5d-496d-99d8-6ded7b8c8f6a"}
2023-04-24T10:40:22Z INFO Master exists. Configuring 10.244.0.8 as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "ed324f55-eb5d-496d-99d8-6ded7b8c8f6a"}
2023-04-24T10:40:22Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "ed324f55-eb5d-496d-99d8-6ded7b8c8f6a", "pod": "df-test-2"}
2023-04-24T10:40:22Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "ed324f55-eb5d-496d-99d8-6ded7b8c8f6a"}
2023-04-24T10:40:22Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "ed324f55-eb5d-496d-99d8-6ded7b8c8f6a"}
2023-04-24T10:40:22Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "ed324f55-eb5d-496d-99d8-6ded7b8c8f6a", "pod": "df-test-2", "master": "10.244.0.6"}
2023-04-24T10:40:22Z INFO Reconciling Dragonfly object {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "fb89ac6c-9264-4f99-ac3e-e7b80824f5ec"}
2023-04-24T10:40:22Z INFO updating existing resources {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "fb89ac6c-9264-4f99-ac3e-e7b80824f5ec"}
2023-04-24T10:40:22Z INFO Creating resources for df-test {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "fb89ac6c-9264-4f99-ac3e-e7b80824f5ec"}
2023-04-24T10:40:22Z INFO Updated resources for object {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "fb89ac6c-9264-4f99-ac3e-e7b80824f5ec"}
2023-04-24T10:40:23Z INFO Marking pod role as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "ed324f55-eb5d-496d-99d8-6ded7b8c8f6a", "pod": "df-test-2"}
2023-04-24T10:40:23Z INFO Updating status {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "ed324f55-eb5d-496d-99d8-6ded7b8c8f6a", "phase": "ready"}
2023-04-24T10:40:23Z ERROR could not mark replica from db {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "ed324f55-eb5d-496d-99d8-6ded7b8c8f6a", "error": "Operation cannot be fulfilled on dragonflies.dragonflydb.io \"df-test\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:123
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:40:23Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "ed324f55-eb5d-496d-99d8-6ded7b8c8f6a", "error": "Operation cannot be fulfilled on dragonflies.dragonflydb.io \"df-test\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:40:23Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "a92a9f7f-4593-45e1-bd62-3b95bd279bb1", "pod": {"namespace": "default", "name": "df-test-3"}}
2023-04-24T10:40:23Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "a92a9f7f-4593-45e1-bd62-3b95bd279bb1"}
2023-04-24T10:40:23Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "cca8d3fb-49da-4490-a964-ce9ce76e12ec", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:40:23Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "cca8d3fb-49da-4490-a964-ce9ce76e12ec", "pod": "default/df-test-2", "role": "replica"}
2023-04-24T10:40:23Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "cca8d3fb-49da-4490-a964-ce9ce76e12ec"}
2023-04-24T10:40:23Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "cca8d3fb-49da-4490-a964-ce9ce76e12ec"}
2023-04-24T10:40:23Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "cca8d3fb-49da-4490-a964-ce9ce76e12ec", "replicas": ["df-test-1", "df-test-2"]}
2023-04-24T10:40:23Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "cca8d3fb-49da-4490-a964-ce9ce76e12ec"}
2023-04-24T10:40:23Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "cca8d3fb-49da-4490-a964-ce9ce76e12ec"}
2023-04-24T10:40:23Z INFO all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "cca8d3fb-49da-4490-a964-ce9ce76e12ec", "dfi": "df-test"}
2023-04-24T10:40:23Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "476473ea-292f-41e1-bca5-01371d358325", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:40:23Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "476473ea-292f-41e1-bca5-01371d358325", "pod": "default/df-test-2", "role": "replica"}
2023-04-24T10:40:23Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "476473ea-292f-41e1-bca5-01371d358325"}
2023-04-24T10:40:23Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "476473ea-292f-41e1-bca5-01371d358325"}
2023-04-24T10:40:23Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "476473ea-292f-41e1-bca5-01371d358325", "replicas": ["df-test-1", "df-test-2"]}
2023-04-24T10:40:23Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "476473ea-292f-41e1-bca5-01371d358325"}
2023-04-24T10:40:23Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "476473ea-292f-41e1-bca5-01371d358325"}
2023-04-24T10:40:23Z INFO all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "476473ea-292f-41e1-bca5-01371d358325", "dfi": "df-test"}
2023-04-24T10:40:28Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "94b41153-45fc-4d22-9375-c19b94a96933", "pod": {"namespace": "default", "name": "df-test-3"}}
2023-04-24T10:40:28Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "94b41153-45fc-4d22-9375-c19b94a96933"}
2023-04-24T10:40:30Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "b84e9f0b-cd48-497d-ad4a-50f93c7b4f81", "pod": {"namespace": "default", "name": "df-test-3"}}
2023-04-24T10:40:30Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "b84e9f0b-cd48-497d-ad4a-50f93c7b4f81"}
2023-04-24T10:40:33Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "d248ee38-fc03-47c9-ba84-f09861b346c4", "pod": {"namespace": "default", "name": "df-test-3"}}
2023-04-24T10:40:33Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "d248ee38-fc03-47c9-ba84-f09861b346c4"}
2023-04-24T10:40:38Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "821fbedf-deba-4569-8c15-471ad673a8c2", "pod": {"namespace": "default", "name": "df-test-3"}}
2023-04-24T10:40:38Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "821fbedf-deba-4569-8c15-471ad673a8c2"}
2023-04-24T10:40:43Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "5b748dfa-7e97-4367-9fe0-798b19f5ea78", "pod": {"namespace": "default", "name": "df-test-3"}}
2023-04-24T10:40:43Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "5b748dfa-7e97-4367-9fe0-798b19f5ea78"}
2023-04-24T10:40:44Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "77fc086a-42e9-4e11-b8a5-31551843c7fb", "pod": {"namespace": "default", "name": "df-test-3"}}
2023-04-24T10:40:44Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "77fc086a-42e9-4e11-b8a5-31551843c7fb", "phase": "ready"}
2023-04-24T10:40:44Z INFO Pod restart from a ready Dragonfly instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "77fc086a-42e9-4e11-b8a5-31551843c7fb"}
2023-04-24T10:40:44Z INFO checking if a master exists already {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "77fc086a-42e9-4e11-b8a5-31551843c7fb"}
2023-04-24T10:40:44Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "77fc086a-42e9-4e11-b8a5-31551843c7fb"}
2023-04-24T10:40:44Z INFO Master exists. Configuring 10.244.0.9 as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "77fc086a-42e9-4e11-b8a5-31551843c7fb"}
2023-04-24T10:40:44Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "77fc086a-42e9-4e11-b8a5-31551843c7fb", "pod": "df-test-3"}
2023-04-24T10:40:44Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "77fc086a-42e9-4e11-b8a5-31551843c7fb"}
2023-04-24T10:40:44Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "77fc086a-42e9-4e11-b8a5-31551843c7fb"}
2023-04-24T10:40:44Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "77fc086a-42e9-4e11-b8a5-31551843c7fb", "pod": "df-test-3", "master": "10.244.0.6"}
2023-04-24T10:40:44Z INFO Marking pod role as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "77fc086a-42e9-4e11-b8a5-31551843c7fb", "pod": "df-test-3"}
2023-04-24T10:40:44Z INFO Updating status {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "77fc086a-42e9-4e11-b8a5-31551843c7fb", "phase": "ready"}
2023-04-24T10:40:44Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "28282d80-70e5-4c10-a812-1968ffb41397", "pod": {"namespace": "default", "name": "df-test-4"}}
2023-04-24T10:40:44Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "28282d80-70e5-4c10-a812-1968ffb41397"}
2023-04-24T10:40:44Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "e373b6de-2a11-4fc7-8d2c-1b37b2d8db97", "pod": {"namespace": "default", "name": "df-test-3"}}
2023-04-24T10:40:44Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "e373b6de-2a11-4fc7-8d2c-1b37b2d8db97", "pod": "default/df-test-3", "role": "replica"}
2023-04-24T10:40:44Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "e373b6de-2a11-4fc7-8d2c-1b37b2d8db97"}
2023-04-24T10:40:44Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "e373b6de-2a11-4fc7-8d2c-1b37b2d8db97"}
2023-04-24T10:40:44Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "e373b6de-2a11-4fc7-8d2c-1b37b2d8db97", "replicas": ["df-test-3", "df-test-1", "df-test-2"]}
2023-04-24T10:40:44Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "e373b6de-2a11-4fc7-8d2c-1b37b2d8db97"}
2023-04-24T10:40:44Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "e373b6de-2a11-4fc7-8d2c-1b37b2d8db97"}
2023-04-24T10:40:45Z INFO all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "e373b6de-2a11-4fc7-8d2c-1b37b2d8db97", "dfi": "df-test"}
2023-04-24T10:40:47Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "a6dac5a4-9c11-4089-9760-df18a5279e0b", "pod": {"namespace": "default", "name": "df-test-4"}}
2023-04-24T10:40:47Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "a6dac5a4-9c11-4089-9760-df18a5279e0b"}
2023-04-24T10:40:48Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "0b7712f0-759f-49be-809a-f463c54b31bd", "pod": {"namespace": "default", "name": "df-test-3"}}
2023-04-24T10:40:48Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "0b7712f0-759f-49be-809a-f463c54b31bd", "pod": "default/df-test-3", "role": "replica"}
2023-04-24T10:40:48Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "0b7712f0-759f-49be-809a-f463c54b31bd"}
2023-04-24T10:40:48Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "0b7712f0-759f-49be-809a-f463c54b31bd"}
2023-04-24T10:40:48Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "0b7712f0-759f-49be-809a-f463c54b31bd", "replicas": ["df-test-1", "df-test-2", "df-test-3"]}
2023-04-24T10:40:48Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "0b7712f0-759f-49be-809a-f463c54b31bd"}
2023-04-24T10:40:48Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "0b7712f0-759f-49be-809a-f463c54b31bd"}
2023-04-24T10:40:48Z INFO all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "0b7712f0-759f-49be-809a-f463c54b31bd", "dfi": "df-test"}
2023-04-24T10:40:49Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c1049ca9-65fe-451e-8fa5-0a6eeac084d1", "pod": {"namespace": "default", "name": "df-test-4"}}
2023-04-24T10:40:49Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c1049ca9-65fe-451e-8fa5-0a6eeac084d1"}
2023-04-24T10:40:54Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "2b1b32e0-e52c-4391-b60a-160e767af6a8", "pod": {"namespace": "default", "name": "df-test-4"}}
2023-04-24T10:40:54Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "2b1b32e0-e52c-4391-b60a-160e767af6a8"}
2023-04-24T10:40:59Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "7a9d8804-1815-4abf-a15e-4ceda8d47926", "pod": {"namespace": "default", "name": "df-test-4"}}
2023-04-24T10:40:59Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "7a9d8804-1815-4abf-a15e-4ceda8d47926"}
2023-04-24T10:41:04Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "37e966bb-d31a-46b3-bbf0-ee48e24bf11f", "pod": {"namespace": "default", "name": "df-test-4"}}
2023-04-24T10:41:04Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "37e966bb-d31a-46b3-bbf0-ee48e24bf11f"}
2023-04-24T10:41:05Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "d1ee736b-c732-491a-8d0d-11fa2fe4802e", "pod": {"namespace": "default", "name": "df-test-4"}}
2023-04-24T10:41:05Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "d1ee736b-c732-491a-8d0d-11fa2fe4802e", "phase": "ready"}
2023-04-24T10:41:05Z INFO Pod restart from a ready Dragonfly instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "d1ee736b-c732-491a-8d0d-11fa2fe4802e"}
2023-04-24T10:41:05Z INFO checking if a master exists already {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "d1ee736b-c732-491a-8d0d-11fa2fe4802e"}
2023-04-24T10:41:05Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "d1ee736b-c732-491a-8d0d-11fa2fe4802e"}
2023-04-24T10:41:05Z INFO Master exists. Configuring 10.244.0.10 as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "d1ee736b-c732-491a-8d0d-11fa2fe4802e"}
2023-04-24T10:41:05Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "d1ee736b-c732-491a-8d0d-11fa2fe4802e", "pod": "df-test-4"}
2023-04-24T10:41:05Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "d1ee736b-c732-491a-8d0d-11fa2fe4802e"}
2023-04-24T10:41:05Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "d1ee736b-c732-491a-8d0d-11fa2fe4802e"}
2023-04-24T10:41:05Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "d1ee736b-c732-491a-8d0d-11fa2fe4802e", "pod": "df-test-4", "master": "10.244.0.6"}
2023-04-24T10:41:06Z INFO Reconciling Dragonfly object {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "b3bcb448-b5af-4f79-8544-a0fbf530cb40"}
2023-04-24T10:41:06Z INFO updating existing resources {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "b3bcb448-b5af-4f79-8544-a0fbf530cb40"}
2023-04-24T10:41:06Z INFO Creating resources for df-test {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "b3bcb448-b5af-4f79-8544-a0fbf530cb40"}
2023-04-24T10:41:06Z INFO Updated resources for object {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "b3bcb448-b5af-4f79-8544-a0fbf530cb40"}
2023-04-24T10:41:06Z INFO Marking pod role as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "d1ee736b-c732-491a-8d0d-11fa2fe4802e", "pod": "df-test-4"}
2023-04-24T10:41:06Z ERROR could not mark replica from db {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "d1ee736b-c732-491a-8d0d-11fa2fe4802e", "error": "could not update replica label"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:123
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:06Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "d1ee736b-c732-491a-8d0d-11fa2fe4802e", "error": "could not update replica label"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:06Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "92049d7d-b102-40fb-9797-6f2de2a65d23", "pod": {"namespace": "default", "name": "df-test-4"}}
2023-04-24T10:41:06Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "92049d7d-b102-40fb-9797-6f2de2a65d23", "phase": "ready"}
2023-04-24T10:41:06Z INFO Pod restart from a ready Dragonfly instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "92049d7d-b102-40fb-9797-6f2de2a65d23"}
2023-04-24T10:41:06Z INFO checking if a master exists already {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "92049d7d-b102-40fb-9797-6f2de2a65d23"}
2023-04-24T10:41:06Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "92049d7d-b102-40fb-9797-6f2de2a65d23"}
2023-04-24T10:41:06Z INFO Master exists. Configuring 10.244.0.10 as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "92049d7d-b102-40fb-9797-6f2de2a65d23"}
2023-04-24T10:41:06Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "92049d7d-b102-40fb-9797-6f2de2a65d23", "pod": "df-test-4"}
2023-04-24T10:41:06Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "92049d7d-b102-40fb-9797-6f2de2a65d23"}
2023-04-24T10:41:06Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "92049d7d-b102-40fb-9797-6f2de2a65d23"}
2023-04-24T10:41:06Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "92049d7d-b102-40fb-9797-6f2de2a65d23", "pod": "df-test-4", "master": "10.244.0.6"}
2023-04-24T10:41:06Z ERROR could not mark replica from db {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "92049d7d-b102-40fb-9797-6f2de2a65d23", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43970->10.244.0.10:6379: read: connection reset by peer"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:123
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:06Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "92049d7d-b102-40fb-9797-6f2de2a65d23", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43970->10.244.0.10:6379: read: connection reset by peer"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:06Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "03fedac1-ed8e-489c-a8ee-d352eeaddb93", "pod": {"namespace": "default", "name": "df-test-4"}}
2023-04-24T10:41:06Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "03fedac1-ed8e-489c-a8ee-d352eeaddb93", "phase": "ready"}
2023-04-24T10:41:06Z INFO Pod restart from a ready Dragonfly instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "03fedac1-ed8e-489c-a8ee-d352eeaddb93"}
2023-04-24T10:41:06Z INFO checking if a master exists already {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "03fedac1-ed8e-489c-a8ee-d352eeaddb93"}
2023-04-24T10:41:06Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "03fedac1-ed8e-489c-a8ee-d352eeaddb93"}
2023-04-24T10:41:06Z INFO Master exists. Configuring 10.244.0.10 as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "03fedac1-ed8e-489c-a8ee-d352eeaddb93"}
2023-04-24T10:41:06Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "03fedac1-ed8e-489c-a8ee-d352eeaddb93", "pod": "df-test-4"}
2023-04-24T10:41:06Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "03fedac1-ed8e-489c-a8ee-d352eeaddb93"}
2023-04-24T10:41:06Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "03fedac1-ed8e-489c-a8ee-d352eeaddb93"}
2023-04-24T10:41:06Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "03fedac1-ed8e-489c-a8ee-d352eeaddb93", "pod": "df-test-4", "master": "10.244.0.6"}
2023-04-24T10:41:06Z ERROR could not mark replica from db {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "03fedac1-ed8e-489c-a8ee-d352eeaddb93", "error": "Error running SLAVE OF command: dial tcp 10.244.0.10:6379: connect: connection refused"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:123
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:06Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "03fedac1-ed8e-489c-a8ee-d352eeaddb93", "error": "Error running SLAVE OF command: dial tcp 10.244.0.10:6379: connect: connection refused"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:06Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "0c8701c2-0af6-41ea-bc9e-256c6bbdb201", "pod": {"namespace": "default", "name": "df-test-4"}}
2023-04-24T10:41:06Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "0c8701c2-0af6-41ea-bc9e-256c6bbdb201", "phase": "ready"}
2023-04-24T10:41:06Z INFO Pod restart from a ready Dragonfly instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "0c8701c2-0af6-41ea-bc9e-256c6bbdb201"}
2023-04-24T10:41:06Z INFO checking if a master exists already {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "0c8701c2-0af6-41ea-bc9e-256c6bbdb201"}
2023-04-24T10:41:06Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "0c8701c2-0af6-41ea-bc9e-256c6bbdb201"}
2023-04-24T10:41:06Z INFO Master exists. Configuring 10.244.0.10 as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "0c8701c2-0af6-41ea-bc9e-256c6bbdb201"}
2023-04-24T10:41:06Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "0c8701c2-0af6-41ea-bc9e-256c6bbdb201", "pod": "df-test-4"}
2023-04-24T10:41:06Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "0c8701c2-0af6-41ea-bc9e-256c6bbdb201"}
2023-04-24T10:41:06Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "0c8701c2-0af6-41ea-bc9e-256c6bbdb201"}
2023-04-24T10:41:06Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "0c8701c2-0af6-41ea-bc9e-256c6bbdb201", "pod": "df-test-4", "master": "10.244.0.6"}
2023-04-24T10:41:06Z ERROR could not mark replica from db {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "0c8701c2-0af6-41ea-bc9e-256c6bbdb201", "error": "Error running SLAVE OF command: dial tcp 10.244.0.10:6379: connect: connection refused"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:123
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:06Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "0c8701c2-0af6-41ea-bc9e-256c6bbdb201", "error": "Error running SLAVE OF command: dial tcp 10.244.0.10:6379: connect: connection refused"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:06Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c9d0e6dc-3335-4d02-b4cd-b81f1c50d72e", "pod": {"namespace": "default", "name": "df-test-4"}}
2023-04-24T10:41:06Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c9d0e6dc-3335-4d02-b4cd-b81f1c50d72e", "phase": "ready"}
2023-04-24T10:41:06Z INFO Pod restart from a ready Dragonfly instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c9d0e6dc-3335-4d02-b4cd-b81f1c50d72e"}
2023-04-24T10:41:06Z INFO checking if a master exists already {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c9d0e6dc-3335-4d02-b4cd-b81f1c50d72e"}
2023-04-24T10:41:06Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c9d0e6dc-3335-4d02-b4cd-b81f1c50d72e"}
2023-04-24T10:41:06Z INFO Master exists. Configuring 10.244.0.10 as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c9d0e6dc-3335-4d02-b4cd-b81f1c50d72e"}
2023-04-24T10:41:06Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c9d0e6dc-3335-4d02-b4cd-b81f1c50d72e", "pod": "df-test-4"}
2023-04-24T10:41:06Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c9d0e6dc-3335-4d02-b4cd-b81f1c50d72e"}
2023-04-24T10:41:06Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c9d0e6dc-3335-4d02-b4cd-b81f1c50d72e"}
2023-04-24T10:41:06Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c9d0e6dc-3335-4d02-b4cd-b81f1c50d72e", "pod": "df-test-4", "master": "10.244.0.6"}
2023-04-24T10:41:06Z ERROR could not mark replica from db {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c9d0e6dc-3335-4d02-b4cd-b81f1c50d72e", "error": "Error running SLAVE OF command: dial tcp 10.244.0.10:6379: connect: connection refused"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:123
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:06Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c9d0e6dc-3335-4d02-b4cd-b81f1c50d72e", "error": "Error running SLAVE OF command: dial tcp 10.244.0.10:6379: connect: connection refused"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:07Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "033604b8-0c1a-4930-8546-a65aa8ed65ca", "pod": {"namespace": "default", "name": "df-test-4"}}
2023-04-24T10:41:07Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "033604b8-0c1a-4930-8546-a65aa8ed65ca", "phase": "ready"}
2023-04-24T10:41:07Z INFO Pod restart from a ready Dragonfly instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "033604b8-0c1a-4930-8546-a65aa8ed65ca"}
2023-04-24T10:41:07Z INFO checking if a master exists already {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "033604b8-0c1a-4930-8546-a65aa8ed65ca"}
2023-04-24T10:41:07Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "033604b8-0c1a-4930-8546-a65aa8ed65ca"}
2023-04-24T10:41:07Z INFO Master exists. Configuring 10.244.0.10 as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "033604b8-0c1a-4930-8546-a65aa8ed65ca"}
2023-04-24T10:41:07Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "033604b8-0c1a-4930-8546-a65aa8ed65ca", "pod": "df-test-4"}
2023-04-24T10:41:07Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "033604b8-0c1a-4930-8546-a65aa8ed65ca"}
2023-04-24T10:41:07Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "033604b8-0c1a-4930-8546-a65aa8ed65ca"}
2023-04-24T10:41:07Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "033604b8-0c1a-4930-8546-a65aa8ed65ca", "pod": "df-test-4", "master": "10.244.0.6"}
2023-04-24T10:41:07Z ERROR could not mark replica from db {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "033604b8-0c1a-4930-8546-a65aa8ed65ca", "error": "Error running SLAVE OF command: dial tcp 10.244.0.10:6379: connect: connection refused"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:123
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:07Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "033604b8-0c1a-4930-8546-a65aa8ed65ca", "error": "Error running SLAVE OF command: dial tcp 10.244.0.10:6379: connect: connection refused"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:07Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "3b8f1ddf-8df8-4697-94f4-893414bd9b38", "pod": {"namespace": "default", "name": "df-test-4"}}
2023-04-24T10:41:07Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "3b8f1ddf-8df8-4697-94f4-893414bd9b38", "phase": "ready"}
2023-04-24T10:41:07Z INFO Pod restart from a ready Dragonfly instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "3b8f1ddf-8df8-4697-94f4-893414bd9b38"}
2023-04-24T10:41:07Z INFO checking if a master exists already {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "3b8f1ddf-8df8-4697-94f4-893414bd9b38"}
2023-04-24T10:41:07Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "3b8f1ddf-8df8-4697-94f4-893414bd9b38"}
2023-04-24T10:41:07Z INFO Master exists. Configuring 10.244.0.10 as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "3b8f1ddf-8df8-4697-94f4-893414bd9b38"}
2023-04-24T10:41:07Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "3b8f1ddf-8df8-4697-94f4-893414bd9b38", "pod": "df-test-4"}
2023-04-24T10:41:07Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "3b8f1ddf-8df8-4697-94f4-893414bd9b38"}
2023-04-24T10:41:07Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "3b8f1ddf-8df8-4697-94f4-893414bd9b38"}
2023-04-24T10:41:07Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "3b8f1ddf-8df8-4697-94f4-893414bd9b38", "pod": "df-test-4", "master": "10.244.0.6"}
2023-04-24T10:41:07Z ERROR could not mark replica from db {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "3b8f1ddf-8df8-4697-94f4-893414bd9b38", "error": "Error running SLAVE OF command: dial tcp 10.244.0.10:6379: connect: connection refused"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:123
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:07Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "3b8f1ddf-8df8-4697-94f4-893414bd9b38", "error": "Error running SLAVE OF command: dial tcp 10.244.0.10:6379: connect: connection refused"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:07Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "2e516ef8-9c7b-4edd-a12e-82f6131020eb", "pod": {"namespace": "default", "name": "df-test-4"}}
2023-04-24T10:41:07Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "2e516ef8-9c7b-4edd-a12e-82f6131020eb", "phase": "ready"}
2023-04-24T10:41:07Z INFO Pod restart from a ready Dragonfly instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "2e516ef8-9c7b-4edd-a12e-82f6131020eb"}
2023-04-24T10:41:07Z INFO checking if a master exists already {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "2e516ef8-9c7b-4edd-a12e-82f6131020eb"}
2023-04-24T10:41:07Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "2e516ef8-9c7b-4edd-a12e-82f6131020eb"}
2023-04-24T10:41:07Z INFO Master exists. Configuring 10.244.0.10 as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "2e516ef8-9c7b-4edd-a12e-82f6131020eb"}
2023-04-24T10:41:07Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "2e516ef8-9c7b-4edd-a12e-82f6131020eb", "pod": "df-test-4"}
2023-04-24T10:41:07Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "2e516ef8-9c7b-4edd-a12e-82f6131020eb"}
2023-04-24T10:41:07Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "2e516ef8-9c7b-4edd-a12e-82f6131020eb"}
2023-04-24T10:41:07Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "2e516ef8-9c7b-4edd-a12e-82f6131020eb", "pod": "df-test-4", "master": "10.244.0.6"}
2023-04-24T10:41:07Z ERROR could not mark replica from db {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "2e516ef8-9c7b-4edd-a12e-82f6131020eb", "error": "Error running SLAVE OF command: dial tcp 10.244.0.10:6379: connect: connection refused"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:123
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:07Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "2e516ef8-9c7b-4edd-a12e-82f6131020eb", "error": "Error running SLAVE OF command: dial tcp 10.244.0.10:6379: connect: connection refused"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:08Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c7516d8f-000e-4e29-a753-a06b0a76ead8", "pod": {"namespace": "default", "name": "df-test-4"}}
2023-04-24T10:41:08Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c7516d8f-000e-4e29-a753-a06b0a76ead8", "phase": "ready"}
2023-04-24T10:41:08Z INFO Pod restart from a ready Dragonfly instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c7516d8f-000e-4e29-a753-a06b0a76ead8"}
2023-04-24T10:41:08Z INFO checking if a master exists already {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c7516d8f-000e-4e29-a753-a06b0a76ead8"}
2023-04-24T10:41:08Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c7516d8f-000e-4e29-a753-a06b0a76ead8"}
2023-04-24T10:41:08Z INFO Master exists. Configuring 10.244.0.10 as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c7516d8f-000e-4e29-a753-a06b0a76ead8"}
2023-04-24T10:41:08Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c7516d8f-000e-4e29-a753-a06b0a76ead8", "pod": "df-test-4"}
2023-04-24T10:41:08Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c7516d8f-000e-4e29-a753-a06b0a76ead8"}
2023-04-24T10:41:08Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c7516d8f-000e-4e29-a753-a06b0a76ead8"}
2023-04-24T10:41:08Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c7516d8f-000e-4e29-a753-a06b0a76ead8", "pod": "df-test-4", "master": "10.244.0.6"}
2023-04-24T10:41:09Z INFO Reconciling Dragonfly object {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "2249ae87-0fc0-4d64-8464-a000619c1c71"}
2023-04-24T10:41:09Z INFO updating existing resources {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "2249ae87-0fc0-4d64-8464-a000619c1c71"}
2023-04-24T10:41:09Z INFO Creating resources for df-test {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "2249ae87-0fc0-4d64-8464-a000619c1c71"}
2023-04-24T10:41:09Z INFO Updated resources for object {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "2249ae87-0fc0-4d64-8464-a000619c1c71"}
2023-04-24T10:41:09Z INFO Pod spec has changed, performing a rollout {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "2249ae87-0fc0-4d64-8464-a000619c1c71"}
2023-04-24T10:41:10Z INFO Reconciling Dragonfly object {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "55084505-9b0c-4dbe-a077-894a38fb054f"}
2023-04-24T10:41:10Z INFO Rolling out new version {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "55084505-9b0c-4dbe-a077-894a38fb054f"}
2023-04-24T10:41:10Z INFO deleting replica {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "55084505-9b0c-4dbe-a077-894a38fb054f", "pod": "df-test-1"}
2023-04-24T10:41:13Z ERROR could not mark replica from db {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c7516d8f-000e-4e29-a753-a06b0a76ead8", "error": "Error running SLAVE OF command: dial tcp 10.244.0.10:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:123
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:13Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "c7516d8f-000e-4e29-a753-a06b0a76ead8", "error": "Error running SLAVE OF command: dial tcp 10.244.0.10:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:13Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-3","namespace":"default"}, "namespace": "default", "name": "df-test-3", "reconcileID": "ed499ee2-5e2d-437c-8e72-2634b87bafd1", "pod": {"namespace": "default", "name": "df-test-3"}}
2023-04-24T10:41:13Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "c53df1ad-6e4b-40f6-aa5c-3d9083fad909", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:41:13Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "c53df1ad-6e4b-40f6-aa5c-3d9083fad909"}
2023-04-24T10:41:13Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "76fd8b2a-4f6a-4e20-a512-72312bcc7d28", "pod": {"namespace": "default", "name": "df-test-4"}}
2023-04-24T10:41:14Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-4","namespace":"default"}, "namespace": "default", "name": "df-test-4", "reconcileID": "11dc83be-781b-4ea6-b314-6c76e6ad6dcd", "pod": {"namespace": "default", "name": "df-test-4"}}
2023-04-24T10:41:15Z INFO Reconciling Dragonfly object {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "c2df7c9c-7ab2-4ad6-87e4-d847d1a32a45"}
2023-04-24T10:41:15Z INFO Rolling out new version {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "c2df7c9c-7ab2-4ad6-87e4-d847d1a32a45"}
2023-04-24T10:41:15Z INFO deleting replica {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "c2df7c9c-7ab2-4ad6-87e4-d847d1a32a45", "pod": "df-test-2"}
2023-04-24T10:41:15Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "94542d67-3d0c-4ac5-a5eb-f60e667c3c8d", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:41:15Z INFO Pod is being deleted {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "94542d67-3d0c-4ac5-a5eb-f60e667c3c8d", "pod": "default/df-test-2"}
2023-04-24T10:41:15Z INFO replica is being deleted. nothing to do {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "94542d67-3d0c-4ac5-a5eb-f60e667c3c8d"}
2023-04-24T10:41:16Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "d77427b5-a261-4059-a9af-90563a18b8ee", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:41:16Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "d77427b5-a261-4059-a9af-90563a18b8ee"}
2023-04-24T10:41:16Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "d11f759e-df8f-4e43-951e-321327c25b49", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:41:16Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "d11f759e-df8f-4e43-951e-321327c25b49"}
2023-04-24T10:41:16Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "cd6e6d5e-6974-4d53-ae4f-79475545b5fa", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:41:18Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "5af23d45-6da2-4065-9f3f-952939ca179d", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:41:18Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "5af23d45-6da2-4065-9f3f-952939ca179d"}
2023-04-24T10:41:20Z INFO Reconciling Dragonfly object {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "0203714b-06d8-4948-9d19-180ed8d27509"}
2023-04-24T10:41:20Z INFO Rolling out new version {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"df-test","namespace":"default"}, "namespace": "default", "name": "df-test", "reconcileID": "0203714b-06d8-4948-9d19-180ed8d27509"}
2023-04-24T10:41:20Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "4de1f41e-8be1-4a90-84c7-dd3f28cc068c", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:41:20Z INFO Pod is being deleted {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "4de1f41e-8be1-4a90-84c7-dd3f28cc068c", "pod": "default/df-test-0"}
2023-04-24T10:41:20Z INFO master is being removed. re-configuring replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "4de1f41e-8be1-4a90-84c7-dd3f28cc068c"}
2023-04-24T10:41:20Z INFO Configuring replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "4de1f41e-8be1-4a90-84c7-dd3f28cc068c"}
2023-04-24T10:41:20Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "4de1f41e-8be1-4a90-84c7-dd3f28cc068c"}
2023-04-24T10:41:20Z INFO Couldn't find a healthy pod to configure as master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "4de1f41e-8be1-4a90-84c7-dd3f28cc068c"}
2023-04-24T10:41:20Z ERROR couldn't find healthy and mark active {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "4de1f41e-8be1-4a90-84c7-dd3f28cc068c", "error": "couldn't find a healthy pod to configure as master"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:139
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:20Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "4de1f41e-8be1-4a90-84c7-dd3f28cc068c", "error": "couldn't find a healthy pod to configure as master"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:41:20Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "4b75d3f4-0bac-4d94-9708-f9ab6e254398", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:41:20Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "4b75d3f4-0bac-4d94-9708-f9ab6e254398", "phase": "rolling-update"}
2023-04-24T10:41:20Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "f586b7bf-0191-45da-bab2-4a0adaeb276a", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:41:20Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "f586b7bf-0191-45da-bab2-4a0adaeb276a", "phase": "rolling-update"}
2023-04-24T10:41:21Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "206a721b-ea0f-4434-b68e-1272db17ac5e", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:41:21Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "f53e3c0e-4bea-404b-badd-363a8a847d30", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:41:21Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "f53e3c0e-4bea-404b-badd-363a8a847d30"}
2023-04-24T10:41:21Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "de76a79c-bd25-4428-83fe-b7bc87b55ea0", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:41:21Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "de76a79c-bd25-4428-83fe-b7bc87b55ea0"}
2023-04-24T10:41:21Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "eb3aa635-20fa-4a3a-9574-983e25024bd6", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:41:21Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "2eb4a70b-fe6b-414c-b695-fce865a8071c", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:41:21Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "2eb4a70b-fe6b-414c-b695-fce865a8071c"}
2023-04-24T10:41:21Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "03b50141-e510-4e3d-a70d-4f7b75c339df", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:41:21Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "03b50141-e510-4e3d-a70d-4f7b75c339df"}
2023-04-24T10:41:21Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "642ad8cd-4c06-466a-9119-aea6351f3601", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:41:21Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "642ad8cd-4c06-466a-9119-aea6351f3601"}
2023-04-24T10:41:23Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "50a0327e-a5af-46b6-b2d7-6097871209d9", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:41:23Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "50a0327e-a5af-46b6-b2d7-6097871209d9"}
2023-04-24T10:41:26Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "45d35b7d-deed-4fe9-a270-f450c17fdea8", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:41:26Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "45d35b7d-deed-4fe9-a270-f450c17fdea8"}
2023-04-24T10:41:28Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "4e58d343-322f-414e-9119-535699bc13a4", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:41:28Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "4e58d343-322f-414e-9119-535699bc13a4"}
2023-04-24T10:41:31Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "707aff04-fc60-4d04-ab5f-d11ef592bbc3", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:41:31Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "707aff04-fc60-4d04-ab5f-d11ef592bbc3"}
2023-04-24T10:41:33Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "f143ada1-1876-4305-aca5-badaf535fdad", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:41:33Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "f143ada1-1876-4305-aca5-badaf535fdad"}
2023-04-24T10:41:36Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "f841139d-b7b6-4b81-812f-9742a9ecf03d", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:41:36Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "f841139d-b7b6-4b81-812f-9742a9ecf03d"}
2023-04-24T10:41:38Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "9ef52128-2647-4fae-8438-e890911438e0", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:41:38Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "9ef52128-2647-4fae-8438-e890911438e0"}
2023-04-24T10:41:41Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "9f179dce-5d90-4bc8-abcb-76839dcf390a", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:41:41Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "9f179dce-5d90-4bc8-abcb-76839dcf390a"}
2023-04-24T10:41:43Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "54ebf9e6-05b5-4fef-93e4-a270c6d42f33", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:41:43Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "54ebf9e6-05b5-4fef-93e4-a270c6d42f33"}
2023-04-24T10:41:46Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "c1ad8f6b-5f24-4837-b697-ac2afcf82385", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:41:46Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "c1ad8f6b-5f24-4837-b697-ac2afcf82385"}
2023-04-24T10:41:48Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "bb805410-ac9d-4e67-b66f-d1f960b01d9a", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:41:48Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "bb805410-ac9d-4e67-b66f-d1f960b01d9a"}
2023-04-24T10:41:51Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "917c81c8-0b84-46cc-b31d-c330d16f5c27", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:41:51Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "917c81c8-0b84-46cc-b31d-c330d16f5c27"}
2023-04-24T10:41:53Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "e136b840-3d9c-4282-b0aa-e6c630a3922b", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:41:53Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "e136b840-3d9c-4282-b0aa-e6c630a3922b"}
2023-04-24T10:41:56Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "3fad1e63-8804-47c0-a564-67e201270961", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:41:56Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "3fad1e63-8804-47c0-a564-67e201270961"}
2023-04-24T10:41:56Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "b23a035f-34fa-4598-bff6-349d1c0707c9", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:41:56Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "b23a035f-34fa-4598-bff6-349d1c0707c9"}
2023-04-24T10:41:58Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "8d8d4b68-bd97-4e39-8099-3f62a5a2bc63", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:41:58Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "8d8d4b68-bd97-4e39-8099-3f62a5a2bc63"}
2023-04-24T10:41:58Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "e143e69e-30a7-4d3a-b8e4-19899ed4889f", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:41:58Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "e143e69e-30a7-4d3a-b8e4-19899ed4889f"}
2023-04-24T10:42:01Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "efcf61a6-e55f-45c1-bd5d-6385766d712c", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:42:01Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "efcf61a6-e55f-45c1-bd5d-6385766d712c"}
2023-04-24T10:42:03Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "47b788a5-8a75-4e5e-91bb-6a820a0c2047", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:42:03Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "47b788a5-8a75-4e5e-91bb-6a820a0c2047"}
2023-04-24T10:42:06Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "1c6315bb-0213-48dc-93c9-6dc8805bc292", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:42:06Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "1c6315bb-0213-48dc-93c9-6dc8805bc292"}
2023-04-24T10:42:08Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "d16f43eb-aa2e-4fce-9aea-6bf45a46155f", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:42:08Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "d16f43eb-aa2e-4fce-9aea-6bf45a46155f"}
2023-04-24T10:42:11Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "e280e72c-c153-4857-87c1-d37fa79bb477", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:42:11Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "e280e72c-c153-4857-87c1-d37fa79bb477"}
2023-04-24T10:42:12Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "40c30ba4-8d38-4871-97d9-7ad7c74da0b4", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:42:12Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "40c30ba4-8d38-4871-97d9-7ad7c74da0b4", "phase": "rolling-update"}
2023-04-24T10:42:13Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "ac416d8a-99e2-48c4-8ddd-eafc721295e0", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:42:13Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "ac416d8a-99e2-48c4-8ddd-eafc721295e0", "phase": "rolling-update"}
2023-04-24T10:42:13Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "169b4bf7-09ba-475a-ad41-e5617b531c01", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:42:13Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "169b4bf7-09ba-475a-ad41-e5617b531c01"}
2023-04-24T10:42:13Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "d2610b73-cd59-46c8-a6b7-087fa2405b84", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:42:13Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "d2610b73-cd59-46c8-a6b7-087fa2405b84"}
2023-04-24T10:42:13Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "7ac83823-d004-4663-bd8f-5d162a9afb9b", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:42:13Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "7ac83823-d004-4663-bd8f-5d162a9afb9b"}
2023-04-24T10:42:13Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "c66b9a93-67d8-4960-bde6-1ad43972372a", "pod": {"namespace": "default", "name": "df-test-1"}}
2023-04-24T10:42:13Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-1","namespace":"default"}, "namespace": "default", "name": "df-test-1", "reconcileID": "c66b9a93-67d8-4960-bde6-1ad43972372a", "phase": "rolling-update"}
2023-04-24T10:42:16Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "70970c78-1a79-4be2-8d84-dddb2316ec5d", "pod": {"namespace": "default", "name": "df-test-0"}}
2023-04-24T10:42:16Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-0","namespace":"default"}, "namespace": "default", "name": "df-test-0", "reconcileID": "70970c78-1a79-4be2-8d84-dddb2316ec5d", "phase": "rolling-update"}
2023-04-24T10:42:17Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "9e784f72-e133-4c0a-85f2-9269000bedea", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:42:17Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "9e784f72-e133-4c0a-85f2-9269000bedea"}
2023-04-24T10:42:18Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "4a556500-458a-4bfa-82e6-1762988754a2", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:42:18Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "4a556500-458a-4bfa-82e6-1762988754a2"}
2023-04-24T10:42:23Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "5d35f37a-72b1-464a-ae68-19a3d4a4a731", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:42:23Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "5d35f37a-72b1-464a-ae68-19a3d4a4a731"}
2023-04-24T10:42:28Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "7494ea14-7d24-40c3-b700-cf73b8ed0af2", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:42:28Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "7494ea14-7d24-40c3-b700-cf73b8ed0af2"}
2023-04-24T10:42:33Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "9e47539d-1eb1-4d77-89e4-5edb90854505", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:42:33Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "9e47539d-1eb1-4d77-89e4-5edb90854505"}
2023-04-24T10:42:34Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "900976f7-1fa7-4b15-aad0-29a9bd2a3d9b", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:42:34Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "900976f7-1fa7-4b15-aad0-29a9bd2a3d9b", "phase": "rolling-update"}
2023-04-24T10:42:38Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "79744cfc-5fcb-4daa-aec1-5b277c4bc1f1", "pod": {"namespace": "default", "name": "df-test-2"}}
2023-04-24T10:42:38Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"df-test-2","namespace":"default"}, "namespace": "default", "name": "df-test-2", "reconcileID": "79744cfc-5fcb-4daa-aec1-5b277c4bc1f1", "phase": "rolling-update"}
2023-04-24T10:44:10Z INFO Reconciling Dragonfly object {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"health-test","namespace":"default"}, "namespace": "default", "name": "health-test", "reconcileID": "7e5ae5d8-f863-4fe7-a761-a2cb8370e462"}
2023-04-24T10:44:10Z INFO Creating resources {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"health-test","namespace":"default"}, "namespace": "default", "name": "health-test", "reconcileID": "7e5ae5d8-f863-4fe7-a761-a2cb8370e462"}
2023-04-24T10:44:10Z INFO Creating resources for health-test {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"health-test","namespace":"default"}, "namespace": "default", "name": "health-test", "reconcileID": "7e5ae5d8-f863-4fe7-a761-a2cb8370e462"}
2023-04-24T10:44:10Z INFO Created resources for object {"controller": "dragonfly", "controllerGroup": "dragonflydb.io", "controllerKind": "Dragonfly", "Dragonfly": {"name":"health-test","namespace":"default"}, "namespace": "default", "name": "health-test", "reconcileID": "7e5ae5d8-f863-4fe7-a761-a2cb8370e462"}
2023-04-24T10:44:10Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "0126f5d0-b76c-4534-81dd-876ccd6d655c", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:44:10Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "0126f5d0-b76c-4534-81dd-876ccd6d655c"}
2023-04-24T10:44:10Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "21b5923d-582a-4f0b-ac0f-85d418db5a09", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:44:10Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "21b5923d-582a-4f0b-ac0f-85d418db5a09"}
2023-04-24T10:44:10Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a64c6ae9-32b3-44ea-b389-e1e00be20b80", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:44:10Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a64c6ae9-32b3-44ea-b389-e1e00be20b80"}
2023-04-24T10:44:14Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8da75477-8d65-4521-9c0d-7a1c24f82760", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:44:14Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8da75477-8d65-4521-9c0d-7a1c24f82760"}
2023-04-24T10:44:15Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "f2beebc9-16aa-4270-90f7-3ad164650ff6", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:44:15Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "f2beebc9-16aa-4270-90f7-3ad164650ff6"}
2023-04-24T10:44:20Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "73635966-5eb5-4599-9012-fea7a160881d", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:44:20Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "73635966-5eb5-4599-9012-fea7a160881d"}
2023-04-24T10:44:25Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "54a7e7c3-7f4b-48bd-b425-2f8f16cd743c", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:44:25Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "54a7e7c3-7f4b-48bd-b425-2f8f16cd743c"}
2023-04-24T10:44:30Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "371d4c33-29fa-486d-b149-7600aabb5451", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:44:30Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "371d4c33-29fa-486d-b149-7600aabb5451"}
2023-04-24T10:44:31Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6b76a411-7583-4002-813a-353f6e4153c1", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:44:31Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6b76a411-7583-4002-813a-353f6e4153c1", "phase": "resources-created"}
2023-04-24T10:44:31Z INFO Dragonfly object is only initialized. Configuring replication for the first time {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6b76a411-7583-4002-813a-353f6e4153c1"}
2023-04-24T10:44:31Z INFO Configuring replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6b76a411-7583-4002-813a-353f6e4153c1"}
2023-04-24T10:44:31Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6b76a411-7583-4002-813a-353f6e4153c1"}
2023-04-24T10:44:31Z INFO Marking pod as master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6b76a411-7583-4002-813a-353f6e4153c1", "podName": "health-test-0", "ip": "10.244.0.14"}
2023-04-24T10:44:31Z INFO Running SLAVE OF NO ONE command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6b76a411-7583-4002-813a-353f6e4153c1", "pod": "health-test-0"}
2023-04-24T10:44:31Z INFO Marking pod role as master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6b76a411-7583-4002-813a-353f6e4153c1", "pod": "health-test-0"}
2023-04-24T10:44:31Z INFO Checking pod {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6b76a411-7583-4002-813a-353f6e4153c1", "podName": "health-test-0", "ip": "10.244.0.14", "status": "Running", "deletiontimestamp": "<nil>"}
2023-04-24T10:44:31Z INFO Successfully marked 0/0 replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6b76a411-7583-4002-813a-353f6e4153c1"}
2023-04-24T10:44:31Z INFO Updating status {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6b76a411-7583-4002-813a-353f6e4153c1", "phase": "ready"}
2023-04-24T10:44:31Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "264e0168-67fc-41a6-8ce5-79c4f7212a69", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:44:31Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "264e0168-67fc-41a6-8ce5-79c4f7212a69"}
2023-04-24T10:44:31Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a57d143f-5af2-46a2-8f82-3911e83f1a59", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:44:31Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a57d143f-5af2-46a2-8f82-3911e83f1a59", "pod": "default/health-test-0", "role": "master"}
2023-04-24T10:44:31Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a57d143f-5af2-46a2-8f82-3911e83f1a59"}
2023-04-24T10:44:31Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a57d143f-5af2-46a2-8f82-3911e83f1a59"}
2023-04-24T10:44:31Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a57d143f-5af2-46a2-8f82-3911e83f1a59", "replicas": []}
2023-04-24T10:44:31Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a57d143f-5af2-46a2-8f82-3911e83f1a59"}
2023-04-24T10:44:31Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a57d143f-5af2-46a2-8f82-3911e83f1a59"}
2023-04-24T10:44:31Z INFO all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a57d143f-5af2-46a2-8f82-3911e83f1a59", "dfi": "health-test"}
2023-04-24T10:44:31Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "69f29ff3-5c92-40ca-b774-bf65659bbaa9", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:44:31Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "69f29ff3-5c92-40ca-b774-bf65659bbaa9"}
2023-04-24T10:44:35Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "c7cafdb7-bd59-482d-9bda-5ea5f7411267", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:44:35Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "c7cafdb7-bd59-482d-9bda-5ea5f7411267", "pod": "default/health-test-0", "role": "master"}
2023-04-24T10:44:35Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "c7cafdb7-bd59-482d-9bda-5ea5f7411267"}
2023-04-24T10:44:35Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "c7cafdb7-bd59-482d-9bda-5ea5f7411267"}
2023-04-24T10:44:35Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "c7cafdb7-bd59-482d-9bda-5ea5f7411267", "replicas": []}
2023-04-24T10:44:35Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "c7cafdb7-bd59-482d-9bda-5ea5f7411267"}
2023-04-24T10:44:35Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "c7cafdb7-bd59-482d-9bda-5ea5f7411267"}
2023-04-24T10:44:35Z INFO all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "c7cafdb7-bd59-482d-9bda-5ea5f7411267", "dfi": "health-test"}
2023-04-24T10:44:36Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "a4f4f111-8606-45f3-9577-075c13ee3ad3", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:44:36Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "a4f4f111-8606-45f3-9577-075c13ee3ad3"}
2023-04-24T10:44:37Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "27a24ca3-c0e7-4e5b-91f2-e8d081c0f746", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:44:37Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "27a24ca3-c0e7-4e5b-91f2-e8d081c0f746"}
2023-04-24T10:44:41Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "5a572b2b-a9c9-4c44-a1c9-585a2a0b8c08", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:44:41Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "5a572b2b-a9c9-4c44-a1c9-585a2a0b8c08"}
2023-04-24T10:44:46Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "75965d6a-48cb-4236-93f7-b547c0461368", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:44:46Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "75965d6a-48cb-4236-93f7-b547c0461368"}
2023-04-24T10:44:51Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b9f88192-9c1b-48fe-a2a7-9edd9d054152", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:44:51Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b9f88192-9c1b-48fe-a2a7-9edd9d054152"}
2023-04-24T10:44:53Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9cc688a3-dac0-4151-8651-00b8ae434136", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:44:53Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9cc688a3-dac0-4151-8651-00b8ae434136", "phase": "ready"}
2023-04-24T10:44:53Z INFO Pod restart from a ready Dragonfly instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9cc688a3-dac0-4151-8651-00b8ae434136"}
2023-04-24T10:44:53Z INFO checking if a master exists already {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9cc688a3-dac0-4151-8651-00b8ae434136"}
2023-04-24T10:44:53Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9cc688a3-dac0-4151-8651-00b8ae434136"}
2023-04-24T10:44:53Z INFO Master exists. Configuring 10.244.0.15 as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9cc688a3-dac0-4151-8651-00b8ae434136"}
2023-04-24T10:44:53Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9cc688a3-dac0-4151-8651-00b8ae434136", "pod": "health-test-1"}
2023-04-24T10:44:53Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9cc688a3-dac0-4151-8651-00b8ae434136"}
2023-04-24T10:44:53Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9cc688a3-dac0-4151-8651-00b8ae434136"}
2023-04-24T10:44:53Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9cc688a3-dac0-4151-8651-00b8ae434136", "pod": "health-test-1", "master": "10.244.0.14"}
2023-04-24T10:44:53Z INFO Marking pod role as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9cc688a3-dac0-4151-8651-00b8ae434136", "pod": "health-test-1"}
2023-04-24T10:44:53Z INFO Updating status {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9cc688a3-dac0-4151-8651-00b8ae434136", "phase": "ready"}
2023-04-24T10:44:53Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "cb45311f-8a5f-498e-aae1-e2c7f5ec29be", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:44:53Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "cb45311f-8a5f-498e-aae1-e2c7f5ec29be"}
2023-04-24T10:44:53Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "01234d99-4b4c-45bd-b8c2-d4dcaf3be673", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:44:53Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "01234d99-4b4c-45bd-b8c2-d4dcaf3be673", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:44:53Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "01234d99-4b4c-45bd-b8c2-d4dcaf3be673"}
2023-04-24T10:44:53Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "01234d99-4b4c-45bd-b8c2-d4dcaf3be673"}
2023-04-24T10:44:53Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "01234d99-4b4c-45bd-b8c2-d4dcaf3be673", "replicas": ["health-test-1"]}
2023-04-24T10:44:53Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "01234d99-4b4c-45bd-b8c2-d4dcaf3be673"}
2023-04-24T10:44:53Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "01234d99-4b4c-45bd-b8c2-d4dcaf3be673"}
2023-04-24T10:44:53Z INFO all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "01234d99-4b4c-45bd-b8c2-d4dcaf3be673", "dfi": "health-test"}
2023-04-24T10:44:56Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "5154a203-61ff-47c1-a33c-4c95eaaa1148", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:44:56Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "5154a203-61ff-47c1-a33c-4c95eaaa1148"}
2023-04-24T10:44:56Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "e1517a4a-3327-4d5d-8863-0e21ffbd39f1", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:44:56Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "e1517a4a-3327-4d5d-8863-0e21ffbd39f1", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:44:56Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "e1517a4a-3327-4d5d-8863-0e21ffbd39f1"}
2023-04-24T10:44:56Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "e1517a4a-3327-4d5d-8863-0e21ffbd39f1"}
2023-04-24T10:44:56Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "e1517a4a-3327-4d5d-8863-0e21ffbd39f1", "replicas": ["health-test-1"]}
2023-04-24T10:44:56Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "e1517a4a-3327-4d5d-8863-0e21ffbd39f1"}
2023-04-24T10:44:56Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "e1517a4a-3327-4d5d-8863-0e21ffbd39f1"}
2023-04-24T10:44:56Z INFO all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "e1517a4a-3327-4d5d-8863-0e21ffbd39f1", "dfi": "health-test"}
2023-04-24T10:44:58Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "cede3ef9-3097-4d98-92b0-5811b4940496", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:44:58Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "cede3ef9-3097-4d98-92b0-5811b4940496"}
2023-04-24T10:45:03Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "c99d4603-9b20-46a6-8227-ec5c48bda671", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:45:03Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "c99d4603-9b20-46a6-8227-ec5c48bda671"}
2023-04-24T10:45:08Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "902344a0-bde0-483e-93a9-d11b976c5781", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:45:08Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "902344a0-bde0-483e-93a9-d11b976c5781"}
2023-04-24T10:45:13Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "8fe0d0c3-7b85-4b01-ad13-8c1f0a1c7d8e", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:45:13Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "8fe0d0c3-7b85-4b01-ad13-8c1f0a1c7d8e"}
2023-04-24T10:45:14Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "a4dbe738-d89b-43f9-baa3-fa3ad9573a5d", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:45:14Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "a4dbe738-d89b-43f9-baa3-fa3ad9573a5d", "phase": "ready"}
2023-04-24T10:45:14Z INFO Pod restart from a ready Dragonfly instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "a4dbe738-d89b-43f9-baa3-fa3ad9573a5d"}
2023-04-24T10:45:14Z INFO checking if a master exists already {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "a4dbe738-d89b-43f9-baa3-fa3ad9573a5d"}
2023-04-24T10:45:14Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "a4dbe738-d89b-43f9-baa3-fa3ad9573a5d"}
2023-04-24T10:45:14Z INFO Master exists. Configuring 10.244.0.16 as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "a4dbe738-d89b-43f9-baa3-fa3ad9573a5d"}
2023-04-24T10:45:14Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "a4dbe738-d89b-43f9-baa3-fa3ad9573a5d", "pod": "health-test-2"}
2023-04-24T10:45:14Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "a4dbe738-d89b-43f9-baa3-fa3ad9573a5d"}
2023-04-24T10:45:14Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "a4dbe738-d89b-43f9-baa3-fa3ad9573a5d"}
2023-04-24T10:45:14Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "a4dbe738-d89b-43f9-baa3-fa3ad9573a5d", "pod": "health-test-2", "master": "10.244.0.14"}
2023-04-24T10:45:14Z INFO Marking pod role as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "a4dbe738-d89b-43f9-baa3-fa3ad9573a5d", "pod": "health-test-2"}
2023-04-24T10:45:14Z INFO Updating status {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "a4dbe738-d89b-43f9-baa3-fa3ad9573a5d", "phase": "ready"}
2023-04-24T10:45:14Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "0054b656-cf8f-4f57-bfe4-9b8aca2c5e0a", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:45:14Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "0054b656-cf8f-4f57-bfe4-9b8aca2c5e0a"}
2023-04-24T10:45:14Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "7c2d7eef-51ea-49cc-9867-d96007cd150b", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:45:14Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "7c2d7eef-51ea-49cc-9867-d96007cd150b", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:45:14Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "7c2d7eef-51ea-49cc-9867-d96007cd150b"}
2023-04-24T10:45:14Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "7c2d7eef-51ea-49cc-9867-d96007cd150b"}
2023-04-24T10:45:14Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "7c2d7eef-51ea-49cc-9867-d96007cd150b", "replicas": ["health-test-1", "health-test-2"]}
2023-04-24T10:45:14Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "7c2d7eef-51ea-49cc-9867-d96007cd150b"}
2023-04-24T10:45:14Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "7c2d7eef-51ea-49cc-9867-d96007cd150b"}
2023-04-24T10:45:14Z INFO all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "7c2d7eef-51ea-49cc-9867-d96007cd150b", "dfi": "health-test"}
2023-04-24T10:45:18Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae697278-0b0a-4502-983a-56658c1d8b57", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:45:18Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae697278-0b0a-4502-983a-56658c1d8b57", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:45:18Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae697278-0b0a-4502-983a-56658c1d8b57"}
2023-04-24T10:45:18Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae697278-0b0a-4502-983a-56658c1d8b57"}
2023-04-24T10:45:18Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae697278-0b0a-4502-983a-56658c1d8b57", "replicas": ["health-test-1", "health-test-2"]}
2023-04-24T10:45:18Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae697278-0b0a-4502-983a-56658c1d8b57"}
2023-04-24T10:45:18Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae697278-0b0a-4502-983a-56658c1d8b57"}
2023-04-24T10:45:18Z INFO all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae697278-0b0a-4502-983a-56658c1d8b57", "dfi": "health-test"}
2023-04-24T10:45:19Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "b293aebb-77fb-4518-a8d7-7ce839682790", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:45:19Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "b293aebb-77fb-4518-a8d7-7ce839682790"}
2023-04-24T10:45:19Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "5e2824e9-cf81-483e-b34c-6d49abaf1b4d", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:45:19Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "5e2824e9-cf81-483e-b34c-6d49abaf1b4d"}
2023-04-24T10:45:24Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "7d0ca11f-138c-4395-a97d-565deb5d3f5c", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:45:24Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "7d0ca11f-138c-4395-a97d-565deb5d3f5c"}
2023-04-24T10:45:29Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "65a09142-4eb6-42a5-a63a-36b92f34fd7a", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:45:29Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "65a09142-4eb6-42a5-a63a-36b92f34fd7a"}
2023-04-24T10:45:34Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "368c2bbc-224e-4863-9712-7473d08c95c2", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:45:34Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "368c2bbc-224e-4863-9712-7473d08c95c2"}
2023-04-24T10:45:36Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "fae58d5c-426f-4b04-b3fc-e17d313c2534", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:45:36Z INFO No replication role was set yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "fae58d5c-426f-4b04-b3fc-e17d313c2534", "phase": "ready"}
2023-04-24T10:45:36Z INFO Pod restart from a ready Dragonfly instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "fae58d5c-426f-4b04-b3fc-e17d313c2534"}
2023-04-24T10:45:36Z INFO checking if a master exists already {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "fae58d5c-426f-4b04-b3fc-e17d313c2534"}
2023-04-24T10:45:36Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "fae58d5c-426f-4b04-b3fc-e17d313c2534"}
2023-04-24T10:45:36Z INFO Master exists. Configuring 10.244.0.17 as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "fae58d5c-426f-4b04-b3fc-e17d313c2534"}
2023-04-24T10:45:36Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "fae58d5c-426f-4b04-b3fc-e17d313c2534", "pod": "health-test-3"}
2023-04-24T10:45:36Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "fae58d5c-426f-4b04-b3fc-e17d313c2534"}
2023-04-24T10:45:36Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "fae58d5c-426f-4b04-b3fc-e17d313c2534"}
2023-04-24T10:45:36Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "fae58d5c-426f-4b04-b3fc-e17d313c2534", "pod": "health-test-3", "master": "10.244.0.14"}
2023-04-24T10:45:36Z INFO Marking pod role as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "fae58d5c-426f-4b04-b3fc-e17d313c2534", "pod": "health-test-3"}
2023-04-24T10:45:36Z INFO Updating status {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "fae58d5c-426f-4b04-b3fc-e17d313c2534", "phase": "ready"}
2023-04-24T10:45:36Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3dc09243-a4df-427d-835e-3362ba957630", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:45:36Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3dc09243-a4df-427d-835e-3362ba957630", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:45:36Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3dc09243-a4df-427d-835e-3362ba957630"}
2023-04-24T10:45:36Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3dc09243-a4df-427d-835e-3362ba957630"}
2023-04-24T10:45:36Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3dc09243-a4df-427d-835e-3362ba957630"}
2023-04-24T10:45:36Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3dc09243-a4df-427d-835e-3362ba957630"}
2023-04-24T10:45:36Z INFO all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3dc09243-a4df-427d-835e-3362ba957630", "dfi": "health-test"}
2023-04-24T10:45:36Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:45:36Z INFO Pod is being deleted {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e", "pod": "default/health-test-0"}
2023-04-24T10:45:36Z INFO master is being removed. re-configuring replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e"}
2023-04-24T10:45:36Z INFO Configuring replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e"}
2023-04-24T10:45:36Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e"}
2023-04-24T10:45:36Z INFO Marking pod as master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e", "podName": "health-test-1", "ip": "10.244.0.15"}
2023-04-24T10:45:36Z INFO Running SLAVE OF NO ONE command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e", "pod": "health-test-1"}
2023-04-24T10:45:36Z INFO Marking pod role as master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e", "pod": "health-test-1"}
2023-04-24T10:45:36Z INFO Checking pod {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e", "podName": "health-test-0", "ip": "10.244.0.14", "status": "Running", "deletiontimestamp": "2023-04-24 10:46:06 +0000 UTC"}
2023-04-24T10:45:36Z INFO Checking pod {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e", "podName": "health-test-1", "ip": "10.244.0.15", "status": "Running", "deletiontimestamp": "<nil>"}
2023-04-24T10:45:36Z INFO Checking pod {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e", "podName": "health-test-2", "ip": "10.244.0.16", "status": "Running", "deletiontimestamp": "<nil>"}
2023-04-24T10:45:36Z INFO Marking pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e", "podName": "health-test-2", "ip": "10.244.0.16", "status": "Running"}
2023-04-24T10:45:36Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e", "pod": "health-test-2", "master": "10.244.0.15"}
2023-04-24T10:45:36Z INFO Marking pod role as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e", "pod": "health-test-2"}
2023-04-24T10:45:36Z INFO Checking pod {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e", "podName": "health-test-3", "ip": "10.244.0.17", "status": "Running", "deletiontimestamp": "<nil>"}
2023-04-24T10:45:36Z INFO Marking pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e", "podName": "health-test-3", "ip": "10.244.0.17", "status": "Running"}
2023-04-24T10:45:36Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:45:39Z ERROR Failed to mark pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e", "podName": "health-test-3", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43902->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DragonflyInstance).configureReplication
/workspace/internal/controller/dragonfly_instance.go:130
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:138
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:45:39Z ERROR couldn't find healthy and mark active {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43902->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:139
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:45:39Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "51117256-912a-4ee7-abd4-a9e6a5f07a5e", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43902->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:45:39Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "88108961-0225-4198-af61-bb9ae1247afd", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:45:39Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "88108961-0225-4198-af61-bb9ae1247afd", "pod": "default/health-test-1", "role": "master"}
2023-04-24T10:45:39Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "88108961-0225-4198-af61-bb9ae1247afd"}
2023-04-24T10:45:39Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "88108961-0225-4198-af61-bb9ae1247afd"}
2023-04-24T10:45:39Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "88108961-0225-4198-af61-bb9ae1247afd", "replicas": ["health-test-2", "health-test-3"]}
2023-04-24T10:45:39Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "88108961-0225-4198-af61-bb9ae1247afd"}
2023-04-24T10:45:39Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "88108961-0225-4198-af61-bb9ae1247afd"}
2023-04-24T10:45:39Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "88108961-0225-4198-af61-bb9ae1247afd", "pod": "health-test-3"}
2023-04-24T10:45:39Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "88108961-0225-4198-af61-bb9ae1247afd", "pod": "health-test-3"}
2023-04-24T10:45:39Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "88108961-0225-4198-af61-bb9ae1247afd"}
2023-04-24T10:45:39Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "88108961-0225-4198-af61-bb9ae1247afd"}
2023-04-24T10:45:39Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "88108961-0225-4198-af61-bb9ae1247afd", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:45:42Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "88108961-0225-4198-af61-bb9ae1247afd", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43918->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:45:42Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "88108961-0225-4198-af61-bb9ae1247afd", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43918->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:45:42Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "26069798-0d04-4b9b-8ecb-80c18a8f4acd", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:45:42Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "26069798-0d04-4b9b-8ecb-80c18a8f4acd", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:45:42Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "26069798-0d04-4b9b-8ecb-80c18a8f4acd"}
2023-04-24T10:45:42Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "26069798-0d04-4b9b-8ecb-80c18a8f4acd"}
2023-04-24T10:45:42Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "26069798-0d04-4b9b-8ecb-80c18a8f4acd", "replicas": ["health-test-2", "health-test-3"]}
2023-04-24T10:45:42Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "26069798-0d04-4b9b-8ecb-80c18a8f4acd"}
2023-04-24T10:45:42Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "26069798-0d04-4b9b-8ecb-80c18a8f4acd"}
2023-04-24T10:45:42Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "26069798-0d04-4b9b-8ecb-80c18a8f4acd", "pod": "health-test-3"}
2023-04-24T10:45:42Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "26069798-0d04-4b9b-8ecb-80c18a8f4acd", "pod": "health-test-3"}
2023-04-24T10:45:42Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "26069798-0d04-4b9b-8ecb-80c18a8f4acd"}
2023-04-24T10:45:42Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "26069798-0d04-4b9b-8ecb-80c18a8f4acd"}
2023-04-24T10:45:42Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "26069798-0d04-4b9b-8ecb-80c18a8f4acd", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:45:45Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "26069798-0d04-4b9b-8ecb-80c18a8f4acd", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43936->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:45:45Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "26069798-0d04-4b9b-8ecb-80c18a8f4acd", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43936->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:45:45Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "13884173-bc2f-4049-9005-d65d6405fa15", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:45:45Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "13884173-bc2f-4049-9005-d65d6405fa15"}
2023-04-24T10:45:45Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "23b7f43c-aac9-4d2e-9870-db1d95edd3ef", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:45:45Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "23b7f43c-aac9-4d2e-9870-db1d95edd3ef", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:45:45Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "23b7f43c-aac9-4d2e-9870-db1d95edd3ef"}
2023-04-24T10:45:45Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "23b7f43c-aac9-4d2e-9870-db1d95edd3ef"}
2023-04-24T10:45:45Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "23b7f43c-aac9-4d2e-9870-db1d95edd3ef", "replicas": ["health-test-2", "health-test-3"]}
2023-04-24T10:45:45Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "23b7f43c-aac9-4d2e-9870-db1d95edd3ef"}
2023-04-24T10:45:45Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "23b7f43c-aac9-4d2e-9870-db1d95edd3ef"}
2023-04-24T10:45:45Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "23b7f43c-aac9-4d2e-9870-db1d95edd3ef", "pod": "health-test-3"}
2023-04-24T10:45:45Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "23b7f43c-aac9-4d2e-9870-db1d95edd3ef", "pod": "health-test-3"}
2023-04-24T10:45:45Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "23b7f43c-aac9-4d2e-9870-db1d95edd3ef"}
2023-04-24T10:45:45Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "23b7f43c-aac9-4d2e-9870-db1d95edd3ef"}
2023-04-24T10:45:45Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "23b7f43c-aac9-4d2e-9870-db1d95edd3ef", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:45:48Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "23b7f43c-aac9-4d2e-9870-db1d95edd3ef", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43344->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:45:48Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "23b7f43c-aac9-4d2e-9870-db1d95edd3ef", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43344->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:45:48Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d0c34f0-271e-44aa-9418-86d54046d7d6", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:45:48Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d0c34f0-271e-44aa-9418-86d54046d7d6", "pod": "default/health-test-1", "role": "master"}
2023-04-24T10:45:48Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d0c34f0-271e-44aa-9418-86d54046d7d6"}
2023-04-24T10:45:48Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d0c34f0-271e-44aa-9418-86d54046d7d6"}
2023-04-24T10:45:48Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d0c34f0-271e-44aa-9418-86d54046d7d6", "replicas": ["health-test-2", "health-test-3"]}
2023-04-24T10:45:48Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d0c34f0-271e-44aa-9418-86d54046d7d6"}
2023-04-24T10:45:48Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d0c34f0-271e-44aa-9418-86d54046d7d6"}
2023-04-24T10:45:48Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d0c34f0-271e-44aa-9418-86d54046d7d6", "pod": "health-test-3"}
2023-04-24T10:45:48Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d0c34f0-271e-44aa-9418-86d54046d7d6", "pod": "health-test-3"}
2023-04-24T10:45:48Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d0c34f0-271e-44aa-9418-86d54046d7d6"}
2023-04-24T10:45:48Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d0c34f0-271e-44aa-9418-86d54046d7d6"}
2023-04-24T10:45:48Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d0c34f0-271e-44aa-9418-86d54046d7d6", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:45:51Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d0c34f0-271e-44aa-9418-86d54046d7d6", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43358->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:45:51Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d0c34f0-271e-44aa-9418-86d54046d7d6", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43358->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:45:51Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ac5ff704-0429-4145-851b-1dc842642a85", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:45:51Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ac5ff704-0429-4145-851b-1dc842642a85", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:45:51Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ac5ff704-0429-4145-851b-1dc842642a85"}
2023-04-24T10:45:51Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ac5ff704-0429-4145-851b-1dc842642a85"}
2023-04-24T10:45:51Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ac5ff704-0429-4145-851b-1dc842642a85", "replicas": ["health-test-2", "health-test-3"]}
2023-04-24T10:45:51Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ac5ff704-0429-4145-851b-1dc842642a85"}
2023-04-24T10:45:51Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ac5ff704-0429-4145-851b-1dc842642a85"}
2023-04-24T10:45:51Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ac5ff704-0429-4145-851b-1dc842642a85", "pod": "health-test-3"}
2023-04-24T10:45:51Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ac5ff704-0429-4145-851b-1dc842642a85", "pod": "health-test-3"}
2023-04-24T10:45:51Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ac5ff704-0429-4145-851b-1dc842642a85"}
2023-04-24T10:45:51Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ac5ff704-0429-4145-851b-1dc842642a85"}
2023-04-24T10:45:51Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ac5ff704-0429-4145-851b-1dc842642a85", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:45:54Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ac5ff704-0429-4145-851b-1dc842642a85", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43382->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:45:54Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ac5ff704-0429-4145-851b-1dc842642a85", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43382->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:45:54Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5098af7-df9a-44e5-95f9-a56c8c7aaefc", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:45:54Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5098af7-df9a-44e5-95f9-a56c8c7aaefc", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:45:54Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5098af7-df9a-44e5-95f9-a56c8c7aaefc"}
2023-04-24T10:45:54Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5098af7-df9a-44e5-95f9-a56c8c7aaefc"}
2023-04-24T10:45:54Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5098af7-df9a-44e5-95f9-a56c8c7aaefc", "replicas": ["health-test-3", "health-test-2"]}
2023-04-24T10:45:54Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5098af7-df9a-44e5-95f9-a56c8c7aaefc"}
2023-04-24T10:45:54Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5098af7-df9a-44e5-95f9-a56c8c7aaefc"}
2023-04-24T10:45:54Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5098af7-df9a-44e5-95f9-a56c8c7aaefc", "pod": "health-test-3"}
2023-04-24T10:45:54Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5098af7-df9a-44e5-95f9-a56c8c7aaefc", "pod": "health-test-3"}
2023-04-24T10:45:54Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5098af7-df9a-44e5-95f9-a56c8c7aaefc"}
2023-04-24T10:45:54Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5098af7-df9a-44e5-95f9-a56c8c7aaefc"}
2023-04-24T10:45:54Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5098af7-df9a-44e5-95f9-a56c8c7aaefc", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:45:57Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5098af7-df9a-44e5-95f9-a56c8c7aaefc", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:42492->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:45:57Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5098af7-df9a-44e5-95f9-a56c8c7aaefc", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:42492->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:45:57Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a4d1415a-4500-425f-b461-c9b2cee7f5d4", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:45:57Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a4d1415a-4500-425f-b461-c9b2cee7f5d4"}
2023-04-24T10:45:57Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "89da1aa2-7610-49da-8e5c-3e0658af855c", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:45:57Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "89da1aa2-7610-49da-8e5c-3e0658af855c", "pod": "default/health-test-1", "role": "master"}
2023-04-24T10:45:57Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "89da1aa2-7610-49da-8e5c-3e0658af855c"}
2023-04-24T10:45:57Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "89da1aa2-7610-49da-8e5c-3e0658af855c"}
2023-04-24T10:45:57Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "89da1aa2-7610-49da-8e5c-3e0658af855c", "replicas": ["health-test-2", "health-test-3"]}
2023-04-24T10:45:57Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "89da1aa2-7610-49da-8e5c-3e0658af855c"}
2023-04-24T10:45:57Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "89da1aa2-7610-49da-8e5c-3e0658af855c"}
2023-04-24T10:45:57Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "89da1aa2-7610-49da-8e5c-3e0658af855c", "pod": "health-test-3"}
2023-04-24T10:45:57Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "89da1aa2-7610-49da-8e5c-3e0658af855c", "pod": "health-test-3"}
2023-04-24T10:45:57Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "89da1aa2-7610-49da-8e5c-3e0658af855c"}
2023-04-24T10:45:57Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "89da1aa2-7610-49da-8e5c-3e0658af855c"}
2023-04-24T10:45:57Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "89da1aa2-7610-49da-8e5c-3e0658af855c", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:46:00Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "89da1aa2-7610-49da-8e5c-3e0658af855c", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:42500->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:00Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "89da1aa2-7610-49da-8e5c-3e0658af855c", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:42500->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:00Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:46:00Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:46:00Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5"}
2023-04-24T10:46:00Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5"}
2023-04-24T10:46:00Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5", "replicas": ["health-test-2", "health-test-3"]}
2023-04-24T10:46:00Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5", "pod": "health-test-0"}
2023-04-24T10:46:00Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5"}
2023-04-24T10:46:00Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5"}
2023-04-24T10:46:00Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5", "pod": "health-test-0", "master": "10.244.0.15"}
2023-04-24T10:46:00Z INFO Marking pod role as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5", "pod": "health-test-0"}
2023-04-24T10:46:00Z INFO Updating status {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5", "phase": "ready"}
2023-04-24T10:46:00Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5"}
2023-04-24T10:46:00Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5"}
2023-04-24T10:46:00Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5", "pod": "health-test-3"}
2023-04-24T10:46:00Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5", "pod": "health-test-3"}
2023-04-24T10:46:00Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5"}
2023-04-24T10:46:00Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5"}
2023-04-24T10:46:00Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:46:03Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:42516->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:03Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b2e7a892-bf27-4693-a71e-28c5e66ccbb5", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:42516->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:03Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e029932-f5fa-453a-b3bd-d2bd872c843d", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:46:03Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e029932-f5fa-453a-b3bd-d2bd872c843d", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:46:03Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e029932-f5fa-453a-b3bd-d2bd872c843d"}
2023-04-24T10:46:03Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e029932-f5fa-453a-b3bd-d2bd872c843d"}
2023-04-24T10:46:03Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e029932-f5fa-453a-b3bd-d2bd872c843d"}
2023-04-24T10:46:03Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e029932-f5fa-453a-b3bd-d2bd872c843d"}
2023-04-24T10:46:03Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e029932-f5fa-453a-b3bd-d2bd872c843d", "pod": "health-test-3"}
2023-04-24T10:46:03Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e029932-f5fa-453a-b3bd-d2bd872c843d", "pod": "health-test-3"}
2023-04-24T10:46:03Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e029932-f5fa-453a-b3bd-d2bd872c843d"}
2023-04-24T10:46:03Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e029932-f5fa-453a-b3bd-d2bd872c843d"}
2023-04-24T10:46:03Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e029932-f5fa-453a-b3bd-d2bd872c843d", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:46:06Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e029932-f5fa-453a-b3bd-d2bd872c843d", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43584->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:06Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e029932-f5fa-453a-b3bd-d2bd872c843d", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43584->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:06Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d0b4a6c1-7c46-4859-af78-0ace35fc61f1", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:46:06Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d0b4a6c1-7c46-4859-af78-0ace35fc61f1", "pod": "default/health-test-0", "role": "replica"}
2023-04-24T10:46:06Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d0b4a6c1-7c46-4859-af78-0ace35fc61f1"}
2023-04-24T10:46:06Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d0b4a6c1-7c46-4859-af78-0ace35fc61f1"}
2023-04-24T10:46:06Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d0b4a6c1-7c46-4859-af78-0ace35fc61f1"}
2023-04-24T10:46:06Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d0b4a6c1-7c46-4859-af78-0ace35fc61f1"}
2023-04-24T10:46:06Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d0b4a6c1-7c46-4859-af78-0ace35fc61f1", "pod": "health-test-3"}
2023-04-24T10:46:06Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d0b4a6c1-7c46-4859-af78-0ace35fc61f1", "pod": "health-test-3"}
2023-04-24T10:46:06Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d0b4a6c1-7c46-4859-af78-0ace35fc61f1"}
2023-04-24T10:46:06Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d0b4a6c1-7c46-4859-af78-0ace35fc61f1"}
2023-04-24T10:46:06Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d0b4a6c1-7c46-4859-af78-0ace35fc61f1", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:46:09Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d0b4a6c1-7c46-4859-af78-0ace35fc61f1", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43608->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:09Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d0b4a6c1-7c46-4859-af78-0ace35fc61f1", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43608->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:09Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6811b4a2-6ce8-46eb-8452-6e06ce2f5421", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:46:09Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6811b4a2-6ce8-46eb-8452-6e06ce2f5421", "pod": "default/health-test-1", "role": "master"}
2023-04-24T10:46:09Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6811b4a2-6ce8-46eb-8452-6e06ce2f5421"}
2023-04-24T10:46:09Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6811b4a2-6ce8-46eb-8452-6e06ce2f5421"}
2023-04-24T10:46:09Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6811b4a2-6ce8-46eb-8452-6e06ce2f5421"}
2023-04-24T10:46:09Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6811b4a2-6ce8-46eb-8452-6e06ce2f5421"}
2023-04-24T10:46:09Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6811b4a2-6ce8-46eb-8452-6e06ce2f5421", "pod": "health-test-3"}
2023-04-24T10:46:09Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6811b4a2-6ce8-46eb-8452-6e06ce2f5421", "pod": "health-test-3"}
2023-04-24T10:46:09Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6811b4a2-6ce8-46eb-8452-6e06ce2f5421"}
2023-04-24T10:46:09Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6811b4a2-6ce8-46eb-8452-6e06ce2f5421"}
2023-04-24T10:46:09Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6811b4a2-6ce8-46eb-8452-6e06ce2f5421", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:46:12Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6811b4a2-6ce8-46eb-8452-6e06ce2f5421", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43628->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:12Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6811b4a2-6ce8-46eb-8452-6e06ce2f5421", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43628->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:12Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae100495-1824-4171-aebb-df2a3e16eaea", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:46:12Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae100495-1824-4171-aebb-df2a3e16eaea", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:46:12Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae100495-1824-4171-aebb-df2a3e16eaea"}
2023-04-24T10:46:12Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae100495-1824-4171-aebb-df2a3e16eaea"}
2023-04-24T10:46:12Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae100495-1824-4171-aebb-df2a3e16eaea"}
2023-04-24T10:46:12Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae100495-1824-4171-aebb-df2a3e16eaea"}
2023-04-24T10:46:12Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae100495-1824-4171-aebb-df2a3e16eaea", "pod": "health-test-3"}
2023-04-24T10:46:12Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae100495-1824-4171-aebb-df2a3e16eaea", "pod": "health-test-3"}
2023-04-24T10:46:12Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae100495-1824-4171-aebb-df2a3e16eaea"}
2023-04-24T10:46:12Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae100495-1824-4171-aebb-df2a3e16eaea"}
2023-04-24T10:46:12Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae100495-1824-4171-aebb-df2a3e16eaea", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:46:15Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae100495-1824-4171-aebb-df2a3e16eaea", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43658->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:15Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ae100495-1824-4171-aebb-df2a3e16eaea", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:43658->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:15Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2f5b60b7-e88d-4d6e-b4a2-7aec8dc00a57", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:46:15Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2f5b60b7-e88d-4d6e-b4a2-7aec8dc00a57", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:46:15Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2f5b60b7-e88d-4d6e-b4a2-7aec8dc00a57"}
2023-04-24T10:46:15Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2f5b60b7-e88d-4d6e-b4a2-7aec8dc00a57"}
2023-04-24T10:46:15Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2f5b60b7-e88d-4d6e-b4a2-7aec8dc00a57"}
2023-04-24T10:46:15Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2f5b60b7-e88d-4d6e-b4a2-7aec8dc00a57"}
2023-04-24T10:46:15Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2f5b60b7-e88d-4d6e-b4a2-7aec8dc00a57", "pod": "health-test-3"}
2023-04-24T10:46:15Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2f5b60b7-e88d-4d6e-b4a2-7aec8dc00a57", "pod": "health-test-3"}
2023-04-24T10:46:15Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2f5b60b7-e88d-4d6e-b4a2-7aec8dc00a57"}
2023-04-24T10:46:15Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2f5b60b7-e88d-4d6e-b4a2-7aec8dc00a57"}
2023-04-24T10:46:15Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2f5b60b7-e88d-4d6e-b4a2-7aec8dc00a57", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:46:18Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2f5b60b7-e88d-4d6e-b4a2-7aec8dc00a57", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:37354->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:18Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2f5b60b7-e88d-4d6e-b4a2-7aec8dc00a57", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:37354->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:18Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "49657551-e3fa-4ecb-b860-2b17faa9b02f", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:46:18Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "49657551-e3fa-4ecb-b860-2b17faa9b02f", "pod": "default/health-test-0", "role": "replica"}
2023-04-24T10:46:18Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "49657551-e3fa-4ecb-b860-2b17faa9b02f"}
2023-04-24T10:46:18Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "49657551-e3fa-4ecb-b860-2b17faa9b02f"}
2023-04-24T10:46:18Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "49657551-e3fa-4ecb-b860-2b17faa9b02f"}
2023-04-24T10:46:18Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "49657551-e3fa-4ecb-b860-2b17faa9b02f"}
2023-04-24T10:46:18Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "49657551-e3fa-4ecb-b860-2b17faa9b02f", "pod": "health-test-3"}
2023-04-24T10:46:18Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "49657551-e3fa-4ecb-b860-2b17faa9b02f", "pod": "health-test-3"}
2023-04-24T10:46:18Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "49657551-e3fa-4ecb-b860-2b17faa9b02f"}
2023-04-24T10:46:18Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "49657551-e3fa-4ecb-b860-2b17faa9b02f"}
2023-04-24T10:46:18Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "49657551-e3fa-4ecb-b860-2b17faa9b02f", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:46:21Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "49657551-e3fa-4ecb-b860-2b17faa9b02f", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:37384->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:21Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "49657551-e3fa-4ecb-b860-2b17faa9b02f", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:37384->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:21Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b7e7b401-5094-4ef9-ae0b-e2596029d830", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:46:21Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b7e7b401-5094-4ef9-ae0b-e2596029d830", "pod": "default/health-test-1", "role": "master"}
2023-04-24T10:46:21Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b7e7b401-5094-4ef9-ae0b-e2596029d830"}
2023-04-24T10:46:21Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b7e7b401-5094-4ef9-ae0b-e2596029d830"}
2023-04-24T10:46:21Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b7e7b401-5094-4ef9-ae0b-e2596029d830"}
2023-04-24T10:46:21Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b7e7b401-5094-4ef9-ae0b-e2596029d830"}
2023-04-24T10:46:21Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b7e7b401-5094-4ef9-ae0b-e2596029d830", "pod": "health-test-3"}
2023-04-24T10:46:21Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b7e7b401-5094-4ef9-ae0b-e2596029d830", "pod": "health-test-3"}
2023-04-24T10:46:21Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b7e7b401-5094-4ef9-ae0b-e2596029d830"}
2023-04-24T10:46:21Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b7e7b401-5094-4ef9-ae0b-e2596029d830"}
2023-04-24T10:46:21Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b7e7b401-5094-4ef9-ae0b-e2596029d830", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:46:24Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b7e7b401-5094-4ef9-ae0b-e2596029d830", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:37412->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:24Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b7e7b401-5094-4ef9-ae0b-e2596029d830", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:37412->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:24Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "620c29b2-4c7b-415b-894e-3f4d687374d0", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:46:24Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "620c29b2-4c7b-415b-894e-3f4d687374d0", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:46:24Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "620c29b2-4c7b-415b-894e-3f4d687374d0"}
2023-04-24T10:46:24Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "620c29b2-4c7b-415b-894e-3f4d687374d0"}
2023-04-24T10:46:24Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "620c29b2-4c7b-415b-894e-3f4d687374d0"}
2023-04-24T10:46:24Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "620c29b2-4c7b-415b-894e-3f4d687374d0"}
2023-04-24T10:46:24Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "620c29b2-4c7b-415b-894e-3f4d687374d0", "pod": "health-test-3"}
2023-04-24T10:46:24Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "620c29b2-4c7b-415b-894e-3f4d687374d0", "pod": "health-test-3"}
2023-04-24T10:46:24Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "620c29b2-4c7b-415b-894e-3f4d687374d0"}
2023-04-24T10:46:24Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "620c29b2-4c7b-415b-894e-3f4d687374d0"}
2023-04-24T10:46:24Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "620c29b2-4c7b-415b-894e-3f4d687374d0", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:46:27Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "620c29b2-4c7b-415b-894e-3f4d687374d0", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:36074->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:27Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "620c29b2-4c7b-415b-894e-3f4d687374d0", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:36074->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:27Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ca92bf26-511c-4135-a85b-2ddb22da3e93", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:46:27Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ca92bf26-511c-4135-a85b-2ddb22da3e93", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:46:27Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ca92bf26-511c-4135-a85b-2ddb22da3e93"}
2023-04-24T10:46:27Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ca92bf26-511c-4135-a85b-2ddb22da3e93"}
2023-04-24T10:46:27Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ca92bf26-511c-4135-a85b-2ddb22da3e93"}
2023-04-24T10:46:27Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ca92bf26-511c-4135-a85b-2ddb22da3e93"}
2023-04-24T10:46:27Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ca92bf26-511c-4135-a85b-2ddb22da3e93", "pod": "health-test-3"}
2023-04-24T10:46:27Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ca92bf26-511c-4135-a85b-2ddb22da3e93", "pod": "health-test-3"}
2023-04-24T10:46:27Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ca92bf26-511c-4135-a85b-2ddb22da3e93"}
2023-04-24T10:46:27Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ca92bf26-511c-4135-a85b-2ddb22da3e93"}
2023-04-24T10:46:27Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ca92bf26-511c-4135-a85b-2ddb22da3e93", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:46:30Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ca92bf26-511c-4135-a85b-2ddb22da3e93", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:36096->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:30Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ca92bf26-511c-4135-a85b-2ddb22da3e93", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:36096->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:30Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "14665841-e4b9-4ca1-bee2-b1c9abc323e2", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:46:30Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "14665841-e4b9-4ca1-bee2-b1c9abc323e2", "pod": "default/health-test-0", "role": "replica"}
2023-04-24T10:46:30Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "14665841-e4b9-4ca1-bee2-b1c9abc323e2"}
2023-04-24T10:46:30Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "14665841-e4b9-4ca1-bee2-b1c9abc323e2"}
2023-04-24T10:46:30Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "14665841-e4b9-4ca1-bee2-b1c9abc323e2"}
2023-04-24T10:46:30Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "14665841-e4b9-4ca1-bee2-b1c9abc323e2"}
2023-04-24T10:46:30Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "14665841-e4b9-4ca1-bee2-b1c9abc323e2", "pod": "health-test-3"}
2023-04-24T10:46:30Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "14665841-e4b9-4ca1-bee2-b1c9abc323e2", "pod": "health-test-3"}
2023-04-24T10:46:30Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "14665841-e4b9-4ca1-bee2-b1c9abc323e2"}
2023-04-24T10:46:30Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "14665841-e4b9-4ca1-bee2-b1c9abc323e2"}
2023-04-24T10:46:30Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "14665841-e4b9-4ca1-bee2-b1c9abc323e2", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:46:33Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "14665841-e4b9-4ca1-bee2-b1c9abc323e2", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:36112->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:33Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "14665841-e4b9-4ca1-bee2-b1c9abc323e2", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:36112->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:33Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c1e18fba-a0de-48d3-aff8-a811489145b3", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:46:33Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c1e18fba-a0de-48d3-aff8-a811489145b3", "pod": "default/health-test-1", "role": "master"}
2023-04-24T10:46:33Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c1e18fba-a0de-48d3-aff8-a811489145b3"}
2023-04-24T10:46:33Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c1e18fba-a0de-48d3-aff8-a811489145b3"}
2023-04-24T10:46:33Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c1e18fba-a0de-48d3-aff8-a811489145b3"}
2023-04-24T10:46:33Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c1e18fba-a0de-48d3-aff8-a811489145b3"}
2023-04-24T10:46:33Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c1e18fba-a0de-48d3-aff8-a811489145b3", "pod": "health-test-3"}
2023-04-24T10:46:33Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c1e18fba-a0de-48d3-aff8-a811489145b3", "pod": "health-test-3"}
2023-04-24T10:46:33Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c1e18fba-a0de-48d3-aff8-a811489145b3"}
2023-04-24T10:46:33Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c1e18fba-a0de-48d3-aff8-a811489145b3"}
2023-04-24T10:46:33Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c1e18fba-a0de-48d3-aff8-a811489145b3", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:46:36Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c1e18fba-a0de-48d3-aff8-a811489145b3", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:58846->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:36Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c1e18fba-a0de-48d3-aff8-a811489145b3", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:58846->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:36Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "06cc0e18-8a8f-4518-a2ad-0237b3f709d6", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:46:36Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "06cc0e18-8a8f-4518-a2ad-0237b3f709d6", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:46:36Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "06cc0e18-8a8f-4518-a2ad-0237b3f709d6"}
2023-04-24T10:46:36Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "06cc0e18-8a8f-4518-a2ad-0237b3f709d6"}
2023-04-24T10:46:36Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "06cc0e18-8a8f-4518-a2ad-0237b3f709d6"}
2023-04-24T10:46:36Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "06cc0e18-8a8f-4518-a2ad-0237b3f709d6"}
2023-04-24T10:46:36Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "06cc0e18-8a8f-4518-a2ad-0237b3f709d6", "pod": "health-test-3"}
2023-04-24T10:46:36Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "06cc0e18-8a8f-4518-a2ad-0237b3f709d6", "pod": "health-test-3"}
2023-04-24T10:46:36Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "06cc0e18-8a8f-4518-a2ad-0237b3f709d6"}
2023-04-24T10:46:36Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "06cc0e18-8a8f-4518-a2ad-0237b3f709d6"}
2023-04-24T10:46:36Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "06cc0e18-8a8f-4518-a2ad-0237b3f709d6", "pod": "health-test-3", "master": "10.244.0.15"}
2023-04-24T10:46:39Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "06cc0e18-8a8f-4518-a2ad-0237b3f709d6", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:58866->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:39Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "06cc0e18-8a8f-4518-a2ad-0237b3f709d6", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:58866->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:39Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:46:39Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:46:39Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318"}
2023-04-24T10:46:39Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318"}
2023-04-24T10:46:39Z INFO incorrect number of masters. reconfiguring replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318", "masters": []}
2023-04-24T10:46:39Z INFO Configuring replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318"}
2023-04-24T10:46:39Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318"}
2023-04-24T10:46:39Z INFO Marking pod as master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318", "podName": "health-test-0", "ip": "10.244.0.18"}
2023-04-24T10:46:39Z INFO Running SLAVE OF NO ONE command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318", "pod": "health-test-0"}
2023-04-24T10:46:39Z INFO Marking pod role as master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318", "pod": "health-test-0"}
2023-04-24T10:46:39Z INFO Checking pod {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318", "podName": "health-test-0", "ip": "10.244.0.18", "status": "Running", "deletiontimestamp": "<nil>"}
2023-04-24T10:46:39Z INFO Checking pod {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318", "podName": "health-test-1", "ip": "", "status": "Pending", "deletiontimestamp": "<nil>"}
2023-04-24T10:46:39Z INFO Checking pod {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318", "podName": "health-test-2", "ip": "10.244.0.16", "status": "Running", "deletiontimestamp": "<nil>"}
2023-04-24T10:46:39Z INFO Marking pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318", "podName": "health-test-2", "ip": "10.244.0.16", "status": "Running"}
2023-04-24T10:46:39Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318", "pod": "health-test-2", "master": "10.244.0.18"}
2023-04-24T10:46:39Z INFO Marking pod role as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318", "pod": "health-test-2"}
2023-04-24T10:46:39Z INFO Checking pod {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318", "podName": "health-test-3", "ip": "10.244.0.17", "status": "Running", "deletiontimestamp": "<nil>"}
2023-04-24T10:46:39Z INFO Marking pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318", "podName": "health-test-3", "ip": "10.244.0.17", "status": "Running"}
2023-04-24T10:46:39Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:46:42Z ERROR Failed to mark pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318", "podName": "health-test-3", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:58874->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DragonflyInstance).configureReplication
/workspace/internal/controller/dragonfly_instance.go:130
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DragonflyInstance).checkAndConfigureReplication
/workspace/internal/controller/dragonfly_instance.go:263
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:150
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:42Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:58874->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:42Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "42e4b4b3-b3d6-49c8-b1f7-bbf8bd63a318", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:58874->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:42Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8e75dfbd-3e66-4ea9-854e-a8a5bc97a422", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:46:42Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8e75dfbd-3e66-4ea9-854e-a8a5bc97a422", "pod": "default/health-test-0", "role": "master"}
2023-04-24T10:46:42Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8e75dfbd-3e66-4ea9-854e-a8a5bc97a422"}
2023-04-24T10:46:42Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8e75dfbd-3e66-4ea9-854e-a8a5bc97a422"}
2023-04-24T10:46:42Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8e75dfbd-3e66-4ea9-854e-a8a5bc97a422", "replicas": ["health-test-2", "health-test-3"]}
2023-04-24T10:46:42Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8e75dfbd-3e66-4ea9-854e-a8a5bc97a422"}
2023-04-24T10:46:42Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8e75dfbd-3e66-4ea9-854e-a8a5bc97a422"}
2023-04-24T10:46:42Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8e75dfbd-3e66-4ea9-854e-a8a5bc97a422", "pod": "health-test-3"}
2023-04-24T10:46:42Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8e75dfbd-3e66-4ea9-854e-a8a5bc97a422", "pod": "health-test-3"}
2023-04-24T10:46:42Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8e75dfbd-3e66-4ea9-854e-a8a5bc97a422"}
2023-04-24T10:46:42Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8e75dfbd-3e66-4ea9-854e-a8a5bc97a422"}
2023-04-24T10:46:42Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8e75dfbd-3e66-4ea9-854e-a8a5bc97a422", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:46:45Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8e75dfbd-3e66-4ea9-854e-a8a5bc97a422", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:58888->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:45Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8e75dfbd-3e66-4ea9-854e-a8a5bc97a422", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:58888->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:45Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "8fd3f417-6cc0-412b-b9e6-159546b30e43", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:46:45Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "8fd3f417-6cc0-412b-b9e6-159546b30e43"}
2023-04-24T10:46:45Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65acdd37-ca11-41b5-b82c-f4496335e78b", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:46:45Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65acdd37-ca11-41b5-b82c-f4496335e78b", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:46:45Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65acdd37-ca11-41b5-b82c-f4496335e78b"}
2023-04-24T10:46:45Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65acdd37-ca11-41b5-b82c-f4496335e78b"}
2023-04-24T10:46:45Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65acdd37-ca11-41b5-b82c-f4496335e78b", "replicas": ["health-test-2", "health-test-3"]}
2023-04-24T10:46:45Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65acdd37-ca11-41b5-b82c-f4496335e78b"}
2023-04-24T10:46:45Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65acdd37-ca11-41b5-b82c-f4496335e78b"}
2023-04-24T10:46:45Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65acdd37-ca11-41b5-b82c-f4496335e78b", "pod": "health-test-3"}
2023-04-24T10:46:45Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65acdd37-ca11-41b5-b82c-f4496335e78b", "pod": "health-test-3"}
2023-04-24T10:46:45Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65acdd37-ca11-41b5-b82c-f4496335e78b"}
2023-04-24T10:46:45Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65acdd37-ca11-41b5-b82c-f4496335e78b"}
2023-04-24T10:46:45Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65acdd37-ca11-41b5-b82c-f4496335e78b", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:46:48Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65acdd37-ca11-41b5-b82c-f4496335e78b", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:40548->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:48Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65acdd37-ca11-41b5-b82c-f4496335e78b", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:40548->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:48Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "13e059f4-d798-4d2d-a9ab-1024a1f2043d", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:46:48Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "13e059f4-d798-4d2d-a9ab-1024a1f2043d", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:46:48Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "13e059f4-d798-4d2d-a9ab-1024a1f2043d"}
2023-04-24T10:46:48Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "13e059f4-d798-4d2d-a9ab-1024a1f2043d"}
2023-04-24T10:46:48Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "13e059f4-d798-4d2d-a9ab-1024a1f2043d", "replicas": ["health-test-2", "health-test-3"]}
2023-04-24T10:46:48Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "13e059f4-d798-4d2d-a9ab-1024a1f2043d"}
2023-04-24T10:46:48Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "13e059f4-d798-4d2d-a9ab-1024a1f2043d"}
2023-04-24T10:46:48Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "13e059f4-d798-4d2d-a9ab-1024a1f2043d", "pod": "health-test-3"}
2023-04-24T10:46:48Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "13e059f4-d798-4d2d-a9ab-1024a1f2043d", "pod": "health-test-3"}
2023-04-24T10:46:48Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "13e059f4-d798-4d2d-a9ab-1024a1f2043d"}
2023-04-24T10:46:48Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "13e059f4-d798-4d2d-a9ab-1024a1f2043d"}
2023-04-24T10:46:48Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "13e059f4-d798-4d2d-a9ab-1024a1f2043d", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:46:51Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "13e059f4-d798-4d2d-a9ab-1024a1f2043d", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:40558->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:51Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "13e059f4-d798-4d2d-a9ab-1024a1f2043d", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:40558->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:51Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8a33162e-516e-4d33-9817-0ea8aade7939", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:46:51Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8a33162e-516e-4d33-9817-0ea8aade7939", "pod": "default/health-test-0", "role": "master"}
2023-04-24T10:46:51Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8a33162e-516e-4d33-9817-0ea8aade7939"}
2023-04-24T10:46:51Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8a33162e-516e-4d33-9817-0ea8aade7939"}
2023-04-24T10:46:51Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8a33162e-516e-4d33-9817-0ea8aade7939", "replicas": ["health-test-2", "health-test-3"]}
2023-04-24T10:46:51Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8a33162e-516e-4d33-9817-0ea8aade7939"}
2023-04-24T10:46:51Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8a33162e-516e-4d33-9817-0ea8aade7939"}
2023-04-24T10:46:51Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8a33162e-516e-4d33-9817-0ea8aade7939", "pod": "health-test-3"}
2023-04-24T10:46:51Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8a33162e-516e-4d33-9817-0ea8aade7939", "pod": "health-test-3"}
2023-04-24T10:46:51Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8a33162e-516e-4d33-9817-0ea8aade7939"}
2023-04-24T10:46:51Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8a33162e-516e-4d33-9817-0ea8aade7939"}
2023-04-24T10:46:51Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8a33162e-516e-4d33-9817-0ea8aade7939", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:46:54Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8a33162e-516e-4d33-9817-0ea8aade7939", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:40586->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:54Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "8a33162e-516e-4d33-9817-0ea8aade7939", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:40586->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:54Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "617115f1-0a3a-411c-9b69-551f5540fa78", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:46:54Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "617115f1-0a3a-411c-9b69-551f5540fa78", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:46:54Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "617115f1-0a3a-411c-9b69-551f5540fa78"}
2023-04-24T10:46:54Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "617115f1-0a3a-411c-9b69-551f5540fa78"}
2023-04-24T10:46:54Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "617115f1-0a3a-411c-9b69-551f5540fa78", "replicas": ["health-test-2", "health-test-3"]}
2023-04-24T10:46:54Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "617115f1-0a3a-411c-9b69-551f5540fa78"}
2023-04-24T10:46:54Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "617115f1-0a3a-411c-9b69-551f5540fa78"}
2023-04-24T10:46:54Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "617115f1-0a3a-411c-9b69-551f5540fa78", "pod": "health-test-3"}
2023-04-24T10:46:54Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "617115f1-0a3a-411c-9b69-551f5540fa78", "pod": "health-test-3"}
2023-04-24T10:46:54Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "617115f1-0a3a-411c-9b69-551f5540fa78"}
2023-04-24T10:46:54Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "617115f1-0a3a-411c-9b69-551f5540fa78"}
2023-04-24T10:46:54Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "617115f1-0a3a-411c-9b69-551f5540fa78", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:46:57Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "617115f1-0a3a-411c-9b69-551f5540fa78", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:54180->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:57Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "617115f1-0a3a-411c-9b69-551f5540fa78", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:54180->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:46:57Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "cfd04adf-de7b-4420-8773-2888f106e714", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:46:57Z INFO Pod is not ready yet {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "cfd04adf-de7b-4420-8773-2888f106e714"}
2023-04-24T10:46:57Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ac3e22d8-cf45-4085-94dd-3154301d7478", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:46:57Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ac3e22d8-cf45-4085-94dd-3154301d7478", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:46:57Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ac3e22d8-cf45-4085-94dd-3154301d7478"}
2023-04-24T10:46:57Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ac3e22d8-cf45-4085-94dd-3154301d7478"}
2023-04-24T10:46:57Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ac3e22d8-cf45-4085-94dd-3154301d7478", "replicas": ["health-test-2", "health-test-3"]}
2023-04-24T10:46:57Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ac3e22d8-cf45-4085-94dd-3154301d7478"}
2023-04-24T10:46:57Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ac3e22d8-cf45-4085-94dd-3154301d7478"}
2023-04-24T10:46:57Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ac3e22d8-cf45-4085-94dd-3154301d7478", "pod": "health-test-3"}
2023-04-24T10:46:57Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ac3e22d8-cf45-4085-94dd-3154301d7478", "pod": "health-test-3"}
2023-04-24T10:46:57Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ac3e22d8-cf45-4085-94dd-3154301d7478"}
2023-04-24T10:46:57Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ac3e22d8-cf45-4085-94dd-3154301d7478"}
2023-04-24T10:46:57Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ac3e22d8-cf45-4085-94dd-3154301d7478", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:00Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ac3e22d8-cf45-4085-94dd-3154301d7478", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:54194->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:00Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ac3e22d8-cf45-4085-94dd-3154301d7478", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:54194->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:00Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:47:00Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06", "pod": "default/health-test-0", "role": "master"}
2023-04-24T10:47:00Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06"}
2023-04-24T10:47:00Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06"}
2023-04-24T10:47:00Z INFO incorrect number of replicas {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06", "replicas": ["health-test-3", "health-test-2"]}
2023-04-24T10:47:00Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06", "pod": "health-test-1"}
2023-04-24T10:47:00Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06"}
2023-04-24T10:47:00Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06"}
2023-04-24T10:47:00Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06", "pod": "health-test-1", "master": "10.244.0.18"}
2023-04-24T10:47:01Z INFO Marking pod role as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06", "pod": "health-test-1"}
2023-04-24T10:47:01Z INFO Updating status {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06", "phase": "ready"}
2023-04-24T10:47:01Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06"}
2023-04-24T10:47:01Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06"}
2023-04-24T10:47:01Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06", "pod": "health-test-3"}
2023-04-24T10:47:01Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06", "pod": "health-test-3"}
2023-04-24T10:47:01Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06"}
2023-04-24T10:47:01Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06"}
2023-04-24T10:47:01Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:04Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:54220->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:04Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "24068ff4-8b1d-4e5e-b53f-86ae695e2e06", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:54220->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:04Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "73c04014-4887-40a7-9cd3-1df26e51a4b2", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:47:04Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "73c04014-4887-40a7-9cd3-1df26e51a4b2", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:47:04Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "73c04014-4887-40a7-9cd3-1df26e51a4b2"}
2023-04-24T10:47:04Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "73c04014-4887-40a7-9cd3-1df26e51a4b2"}
2023-04-24T10:47:04Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "73c04014-4887-40a7-9cd3-1df26e51a4b2"}
2023-04-24T10:47:04Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "73c04014-4887-40a7-9cd3-1df26e51a4b2"}
2023-04-24T10:47:04Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "73c04014-4887-40a7-9cd3-1df26e51a4b2", "pod": "health-test-3"}
2023-04-24T10:47:04Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "73c04014-4887-40a7-9cd3-1df26e51a4b2", "pod": "health-test-3"}
2023-04-24T10:47:04Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "73c04014-4887-40a7-9cd3-1df26e51a4b2"}
2023-04-24T10:47:04Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "73c04014-4887-40a7-9cd3-1df26e51a4b2"}
2023-04-24T10:47:04Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "73c04014-4887-40a7-9cd3-1df26e51a4b2", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:07Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "73c04014-4887-40a7-9cd3-1df26e51a4b2", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:55916->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:07Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "73c04014-4887-40a7-9cd3-1df26e51a4b2", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:55916->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:07Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c90018ec-bb3f-4906-83c8-94071b8d340b", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:47:07Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c90018ec-bb3f-4906-83c8-94071b8d340b", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:47:07Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c90018ec-bb3f-4906-83c8-94071b8d340b"}
2023-04-24T10:47:07Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c90018ec-bb3f-4906-83c8-94071b8d340b"}
2023-04-24T10:47:07Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c90018ec-bb3f-4906-83c8-94071b8d340b"}
2023-04-24T10:47:07Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c90018ec-bb3f-4906-83c8-94071b8d340b"}
2023-04-24T10:47:07Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c90018ec-bb3f-4906-83c8-94071b8d340b", "pod": "health-test-3"}
2023-04-24T10:47:07Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c90018ec-bb3f-4906-83c8-94071b8d340b", "pod": "health-test-3"}
2023-04-24T10:47:07Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c90018ec-bb3f-4906-83c8-94071b8d340b"}
2023-04-24T10:47:07Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c90018ec-bb3f-4906-83c8-94071b8d340b"}
2023-04-24T10:47:07Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c90018ec-bb3f-4906-83c8-94071b8d340b", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:10Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c90018ec-bb3f-4906-83c8-94071b8d340b", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:55930->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:10Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c90018ec-bb3f-4906-83c8-94071b8d340b", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:55930->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:10Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ee4151dd-9381-4754-9c35-39cb7537880a", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:47:10Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ee4151dd-9381-4754-9c35-39cb7537880a", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:47:10Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ee4151dd-9381-4754-9c35-39cb7537880a"}
2023-04-24T10:47:10Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ee4151dd-9381-4754-9c35-39cb7537880a"}
2023-04-24T10:47:10Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ee4151dd-9381-4754-9c35-39cb7537880a"}
2023-04-24T10:47:10Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ee4151dd-9381-4754-9c35-39cb7537880a"}
2023-04-24T10:47:10Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ee4151dd-9381-4754-9c35-39cb7537880a", "pod": "health-test-3"}
2023-04-24T10:47:10Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ee4151dd-9381-4754-9c35-39cb7537880a", "pod": "health-test-3"}
2023-04-24T10:47:10Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ee4151dd-9381-4754-9c35-39cb7537880a"}
2023-04-24T10:47:10Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ee4151dd-9381-4754-9c35-39cb7537880a"}
2023-04-24T10:47:10Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ee4151dd-9381-4754-9c35-39cb7537880a", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:13Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ee4151dd-9381-4754-9c35-39cb7537880a", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:55938->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:13Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ee4151dd-9381-4754-9c35-39cb7537880a", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:55938->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:13Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ffb4e26c-e38d-4256-af3c-a14770cfa820", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:47:13Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ffb4e26c-e38d-4256-af3c-a14770cfa820", "pod": "default/health-test-0", "role": "master"}
2023-04-24T10:47:13Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ffb4e26c-e38d-4256-af3c-a14770cfa820"}
2023-04-24T10:47:13Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ffb4e26c-e38d-4256-af3c-a14770cfa820"}
2023-04-24T10:47:13Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ffb4e26c-e38d-4256-af3c-a14770cfa820"}
2023-04-24T10:47:13Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ffb4e26c-e38d-4256-af3c-a14770cfa820"}
2023-04-24T10:47:13Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ffb4e26c-e38d-4256-af3c-a14770cfa820", "pod": "health-test-3"}
2023-04-24T10:47:13Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ffb4e26c-e38d-4256-af3c-a14770cfa820", "pod": "health-test-3"}
2023-04-24T10:47:13Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ffb4e26c-e38d-4256-af3c-a14770cfa820"}
2023-04-24T10:47:13Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ffb4e26c-e38d-4256-af3c-a14770cfa820"}
2023-04-24T10:47:13Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ffb4e26c-e38d-4256-af3c-a14770cfa820", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:16Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ffb4e26c-e38d-4256-af3c-a14770cfa820", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:48638->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:16Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ffb4e26c-e38d-4256-af3c-a14770cfa820", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:48638->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:16Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "662dddce-93c4-4d59-81e1-152b10f623c1", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:47:16Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "662dddce-93c4-4d59-81e1-152b10f623c1", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:47:16Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "662dddce-93c4-4d59-81e1-152b10f623c1"}
2023-04-24T10:47:16Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "662dddce-93c4-4d59-81e1-152b10f623c1"}
2023-04-24T10:47:16Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "662dddce-93c4-4d59-81e1-152b10f623c1"}
2023-04-24T10:47:16Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "662dddce-93c4-4d59-81e1-152b10f623c1"}
2023-04-24T10:47:16Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "662dddce-93c4-4d59-81e1-152b10f623c1", "pod": "health-test-3"}
2023-04-24T10:47:16Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "662dddce-93c4-4d59-81e1-152b10f623c1", "pod": "health-test-3"}
2023-04-24T10:47:16Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "662dddce-93c4-4d59-81e1-152b10f623c1"}
2023-04-24T10:47:16Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "662dddce-93c4-4d59-81e1-152b10f623c1"}
2023-04-24T10:47:16Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "662dddce-93c4-4d59-81e1-152b10f623c1", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:19Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "662dddce-93c4-4d59-81e1-152b10f623c1", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:48666->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:19Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "662dddce-93c4-4d59-81e1-152b10f623c1", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:48666->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:19Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "73ba267a-dc92-4273-b9fc-9dfa37bedce5", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:47:19Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "73ba267a-dc92-4273-b9fc-9dfa37bedce5", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:47:19Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "73ba267a-dc92-4273-b9fc-9dfa37bedce5"}
2023-04-24T10:47:19Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "73ba267a-dc92-4273-b9fc-9dfa37bedce5"}
2023-04-24T10:47:19Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "73ba267a-dc92-4273-b9fc-9dfa37bedce5"}
2023-04-24T10:47:19Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "73ba267a-dc92-4273-b9fc-9dfa37bedce5"}
2023-04-24T10:47:19Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "73ba267a-dc92-4273-b9fc-9dfa37bedce5", "pod": "health-test-3"}
2023-04-24T10:47:19Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "73ba267a-dc92-4273-b9fc-9dfa37bedce5", "pod": "health-test-3"}
2023-04-24T10:47:19Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "73ba267a-dc92-4273-b9fc-9dfa37bedce5"}
2023-04-24T10:47:19Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "73ba267a-dc92-4273-b9fc-9dfa37bedce5"}
2023-04-24T10:47:19Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "73ba267a-dc92-4273-b9fc-9dfa37bedce5", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:22Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "73ba267a-dc92-4273-b9fc-9dfa37bedce5", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:48676->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:22Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "73ba267a-dc92-4273-b9fc-9dfa37bedce5", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:48676->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:22Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "8d290d93-980e-413a-b59b-511a6c1b8494", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:47:22Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "8d290d93-980e-413a-b59b-511a6c1b8494", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:47:22Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "8d290d93-980e-413a-b59b-511a6c1b8494"}
2023-04-24T10:47:22Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "8d290d93-980e-413a-b59b-511a6c1b8494"}
2023-04-24T10:47:22Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "8d290d93-980e-413a-b59b-511a6c1b8494"}
2023-04-24T10:47:22Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "8d290d93-980e-413a-b59b-511a6c1b8494"}
2023-04-24T10:47:22Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "8d290d93-980e-413a-b59b-511a6c1b8494", "pod": "health-test-3"}
2023-04-24T10:47:22Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "8d290d93-980e-413a-b59b-511a6c1b8494", "pod": "health-test-3"}
2023-04-24T10:47:22Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "8d290d93-980e-413a-b59b-511a6c1b8494"}
2023-04-24T10:47:22Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "8d290d93-980e-413a-b59b-511a6c1b8494"}
2023-04-24T10:47:22Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "8d290d93-980e-413a-b59b-511a6c1b8494", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:25Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "8d290d93-980e-413a-b59b-511a6c1b8494", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:48692->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:25Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "8d290d93-980e-413a-b59b-511a6c1b8494", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:48692->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:25Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6024b3e0-3a4c-4812-8125-c010049e7f35", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:47:25Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6024b3e0-3a4c-4812-8125-c010049e7f35", "pod": "default/health-test-0", "role": "master"}
2023-04-24T10:47:25Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6024b3e0-3a4c-4812-8125-c010049e7f35"}
2023-04-24T10:47:25Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6024b3e0-3a4c-4812-8125-c010049e7f35"}
2023-04-24T10:47:25Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6024b3e0-3a4c-4812-8125-c010049e7f35"}
2023-04-24T10:47:25Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6024b3e0-3a4c-4812-8125-c010049e7f35"}
2023-04-24T10:47:25Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6024b3e0-3a4c-4812-8125-c010049e7f35", "pod": "health-test-3"}
2023-04-24T10:47:25Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6024b3e0-3a4c-4812-8125-c010049e7f35", "pod": "health-test-3"}
2023-04-24T10:47:25Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6024b3e0-3a4c-4812-8125-c010049e7f35"}
2023-04-24T10:47:25Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6024b3e0-3a4c-4812-8125-c010049e7f35"}
2023-04-24T10:47:25Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6024b3e0-3a4c-4812-8125-c010049e7f35", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:28Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6024b3e0-3a4c-4812-8125-c010049e7f35", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:57066->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:28Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6024b3e0-3a4c-4812-8125-c010049e7f35", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:57066->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:28Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65f4b81e-1326-4991-811e-7292d6dc366d", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:47:28Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65f4b81e-1326-4991-811e-7292d6dc366d", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:47:28Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65f4b81e-1326-4991-811e-7292d6dc366d"}
2023-04-24T10:47:28Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65f4b81e-1326-4991-811e-7292d6dc366d"}
2023-04-24T10:47:28Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65f4b81e-1326-4991-811e-7292d6dc366d"}
2023-04-24T10:47:28Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65f4b81e-1326-4991-811e-7292d6dc366d"}
2023-04-24T10:47:28Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65f4b81e-1326-4991-811e-7292d6dc366d", "pod": "health-test-3"}
2023-04-24T10:47:28Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65f4b81e-1326-4991-811e-7292d6dc366d", "pod": "health-test-3"}
2023-04-24T10:47:28Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65f4b81e-1326-4991-811e-7292d6dc366d"}
2023-04-24T10:47:28Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65f4b81e-1326-4991-811e-7292d6dc366d"}
2023-04-24T10:47:28Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65f4b81e-1326-4991-811e-7292d6dc366d", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:31Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65f4b81e-1326-4991-811e-7292d6dc366d", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:57088->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:31Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65f4b81e-1326-4991-811e-7292d6dc366d", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:57088->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:31Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f9a16431-71cb-48f0-8126-5f20239278ff", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:47:31Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f9a16431-71cb-48f0-8126-5f20239278ff", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:47:31Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f9a16431-71cb-48f0-8126-5f20239278ff"}
2023-04-24T10:47:31Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f9a16431-71cb-48f0-8126-5f20239278ff"}
2023-04-24T10:47:31Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f9a16431-71cb-48f0-8126-5f20239278ff"}
2023-04-24T10:47:31Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f9a16431-71cb-48f0-8126-5f20239278ff"}
2023-04-24T10:47:31Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f9a16431-71cb-48f0-8126-5f20239278ff", "pod": "health-test-3"}
2023-04-24T10:47:31Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f9a16431-71cb-48f0-8126-5f20239278ff", "pod": "health-test-3"}
2023-04-24T10:47:31Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f9a16431-71cb-48f0-8126-5f20239278ff"}
2023-04-24T10:47:31Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f9a16431-71cb-48f0-8126-5f20239278ff"}
2023-04-24T10:47:31Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f9a16431-71cb-48f0-8126-5f20239278ff", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:34Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f9a16431-71cb-48f0-8126-5f20239278ff", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:57096->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:34Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f9a16431-71cb-48f0-8126-5f20239278ff", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:57096->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:34Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "adc4f240-5d4a-45c4-8172-d5467d8c4530", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:47:34Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "adc4f240-5d4a-45c4-8172-d5467d8c4530", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:47:34Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "adc4f240-5d4a-45c4-8172-d5467d8c4530"}
2023-04-24T10:47:34Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "adc4f240-5d4a-45c4-8172-d5467d8c4530"}
2023-04-24T10:47:34Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "adc4f240-5d4a-45c4-8172-d5467d8c4530"}
2023-04-24T10:47:34Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "adc4f240-5d4a-45c4-8172-d5467d8c4530"}
2023-04-24T10:47:34Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "adc4f240-5d4a-45c4-8172-d5467d8c4530", "pod": "health-test-3"}
2023-04-24T10:47:34Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "adc4f240-5d4a-45c4-8172-d5467d8c4530", "pod": "health-test-3"}
2023-04-24T10:47:34Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "adc4f240-5d4a-45c4-8172-d5467d8c4530"}
2023-04-24T10:47:34Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "adc4f240-5d4a-45c4-8172-d5467d8c4530"}
2023-04-24T10:47:34Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "adc4f240-5d4a-45c4-8172-d5467d8c4530", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:37Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "adc4f240-5d4a-45c4-8172-d5467d8c4530", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:59026->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:37Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "adc4f240-5d4a-45c4-8172-d5467d8c4530", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:59026->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:37Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d3983a3d-e0d7-4d69-849b-6bd367e9fa28", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:47:37Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d3983a3d-e0d7-4d69-849b-6bd367e9fa28", "pod": "default/health-test-0", "role": "master"}
2023-04-24T10:47:37Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d3983a3d-e0d7-4d69-849b-6bd367e9fa28"}
2023-04-24T10:47:37Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d3983a3d-e0d7-4d69-849b-6bd367e9fa28"}
2023-04-24T10:47:37Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d3983a3d-e0d7-4d69-849b-6bd367e9fa28"}
2023-04-24T10:47:37Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d3983a3d-e0d7-4d69-849b-6bd367e9fa28"}
2023-04-24T10:47:37Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d3983a3d-e0d7-4d69-849b-6bd367e9fa28", "pod": "health-test-3"}
2023-04-24T10:47:37Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d3983a3d-e0d7-4d69-849b-6bd367e9fa28", "pod": "health-test-3"}
2023-04-24T10:47:37Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d3983a3d-e0d7-4d69-849b-6bd367e9fa28"}
2023-04-24T10:47:37Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d3983a3d-e0d7-4d69-849b-6bd367e9fa28"}
2023-04-24T10:47:37Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d3983a3d-e0d7-4d69-849b-6bd367e9fa28", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:40Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d3983a3d-e0d7-4d69-849b-6bd367e9fa28", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:59046->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:40Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "d3983a3d-e0d7-4d69-849b-6bd367e9fa28", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:59046->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:40Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "aa2bcf3a-e730-4a9d-9e80-fc2cfa167b50", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:47:40Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "aa2bcf3a-e730-4a9d-9e80-fc2cfa167b50", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:47:40Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "aa2bcf3a-e730-4a9d-9e80-fc2cfa167b50"}
2023-04-24T10:47:40Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "aa2bcf3a-e730-4a9d-9e80-fc2cfa167b50"}
2023-04-24T10:47:40Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "aa2bcf3a-e730-4a9d-9e80-fc2cfa167b50"}
2023-04-24T10:47:40Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "aa2bcf3a-e730-4a9d-9e80-fc2cfa167b50"}
2023-04-24T10:47:40Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "aa2bcf3a-e730-4a9d-9e80-fc2cfa167b50", "pod": "health-test-3"}
2023-04-24T10:47:40Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "aa2bcf3a-e730-4a9d-9e80-fc2cfa167b50", "pod": "health-test-3"}
2023-04-24T10:47:40Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "aa2bcf3a-e730-4a9d-9e80-fc2cfa167b50"}
2023-04-24T10:47:40Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "aa2bcf3a-e730-4a9d-9e80-fc2cfa167b50"}
2023-04-24T10:47:40Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "aa2bcf3a-e730-4a9d-9e80-fc2cfa167b50", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:43Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "aa2bcf3a-e730-4a9d-9e80-fc2cfa167b50", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:59060->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:43Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "aa2bcf3a-e730-4a9d-9e80-fc2cfa167b50", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:59060->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:43Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "453e7373-b6b9-45a1-914d-fdd488c91c20", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:47:43Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "453e7373-b6b9-45a1-914d-fdd488c91c20", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:47:43Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "453e7373-b6b9-45a1-914d-fdd488c91c20"}
2023-04-24T10:47:43Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "453e7373-b6b9-45a1-914d-fdd488c91c20"}
2023-04-24T10:47:43Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "453e7373-b6b9-45a1-914d-fdd488c91c20"}
2023-04-24T10:47:43Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "453e7373-b6b9-45a1-914d-fdd488c91c20"}
2023-04-24T10:47:43Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "453e7373-b6b9-45a1-914d-fdd488c91c20", "pod": "health-test-3"}
2023-04-24T10:47:43Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "453e7373-b6b9-45a1-914d-fdd488c91c20", "pod": "health-test-3"}
2023-04-24T10:47:43Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "453e7373-b6b9-45a1-914d-fdd488c91c20"}
2023-04-24T10:47:43Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "453e7373-b6b9-45a1-914d-fdd488c91c20"}
2023-04-24T10:47:43Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "453e7373-b6b9-45a1-914d-fdd488c91c20", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:46Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "453e7373-b6b9-45a1-914d-fdd488c91c20", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:47498->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:46Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "453e7373-b6b9-45a1-914d-fdd488c91c20", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:47498->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:46Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "eeb63663-8024-430c-849f-b4d29fe13987", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:47:46Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "eeb63663-8024-430c-849f-b4d29fe13987", "pod": "default/health-test-0", "role": "master"}
2023-04-24T10:47:46Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "eeb63663-8024-430c-849f-b4d29fe13987"}
2023-04-24T10:47:46Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "eeb63663-8024-430c-849f-b4d29fe13987"}
2023-04-24T10:47:46Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "eeb63663-8024-430c-849f-b4d29fe13987"}
2023-04-24T10:47:46Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "eeb63663-8024-430c-849f-b4d29fe13987"}
2023-04-24T10:47:46Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "eeb63663-8024-430c-849f-b4d29fe13987", "pod": "health-test-3"}
2023-04-24T10:47:46Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "eeb63663-8024-430c-849f-b4d29fe13987", "pod": "health-test-3"}
2023-04-24T10:47:46Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "eeb63663-8024-430c-849f-b4d29fe13987"}
2023-04-24T10:47:46Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "eeb63663-8024-430c-849f-b4d29fe13987"}
2023-04-24T10:47:46Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "eeb63663-8024-430c-849f-b4d29fe13987", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:49Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "eeb63663-8024-430c-849f-b4d29fe13987", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:47508->10.244.0.17:6379: i/o timeout"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:49Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "eeb63663-8024-430c-849f-b4d29fe13987", "error": "Error running SLAVE OF command: read tcp 10.244.0.5:47508->10.244.0.17:6379: i/o timeout"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:49Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ce6d5d0a-44f9-4537-96a6-8dff5878e5a9", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:47:49Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ce6d5d0a-44f9-4537-96a6-8dff5878e5a9", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:47:49Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ce6d5d0a-44f9-4537-96a6-8dff5878e5a9"}
2023-04-24T10:47:49Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ce6d5d0a-44f9-4537-96a6-8dff5878e5a9"}
2023-04-24T10:47:49Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ce6d5d0a-44f9-4537-96a6-8dff5878e5a9"}
2023-04-24T10:47:49Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ce6d5d0a-44f9-4537-96a6-8dff5878e5a9"}
2023-04-24T10:47:49Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ce6d5d0a-44f9-4537-96a6-8dff5878e5a9", "pod": "health-test-3"}
2023-04-24T10:47:49Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ce6d5d0a-44f9-4537-96a6-8dff5878e5a9", "pod": "health-test-3"}
2023-04-24T10:47:49Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ce6d5d0a-44f9-4537-96a6-8dff5878e5a9"}
2023-04-24T10:47:49Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ce6d5d0a-44f9-4537-96a6-8dff5878e5a9"}
2023-04-24T10:47:49Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ce6d5d0a-44f9-4537-96a6-8dff5878e5a9", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:49Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ce6d5d0a-44f9-4537-96a6-8dff5878e5a9", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:49Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "ce6d5d0a-44f9-4537-96a6-8dff5878e5a9", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:49Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b8c8de6a-c988-4b91-97db-eca37b626ffb", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:47:49Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b8c8de6a-c988-4b91-97db-eca37b626ffb", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:47:49Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b8c8de6a-c988-4b91-97db-eca37b626ffb"}
2023-04-24T10:47:49Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b8c8de6a-c988-4b91-97db-eca37b626ffb"}
2023-04-24T10:47:49Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b8c8de6a-c988-4b91-97db-eca37b626ffb"}
2023-04-24T10:47:49Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b8c8de6a-c988-4b91-97db-eca37b626ffb"}
2023-04-24T10:47:49Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b8c8de6a-c988-4b91-97db-eca37b626ffb", "pod": "health-test-3"}
2023-04-24T10:47:49Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b8c8de6a-c988-4b91-97db-eca37b626ffb", "pod": "health-test-3"}
2023-04-24T10:47:49Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b8c8de6a-c988-4b91-97db-eca37b626ffb"}
2023-04-24T10:47:49Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b8c8de6a-c988-4b91-97db-eca37b626ffb"}
2023-04-24T10:47:49Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b8c8de6a-c988-4b91-97db-eca37b626ffb", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:49Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b8c8de6a-c988-4b91-97db-eca37b626ffb", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:49Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "b8c8de6a-c988-4b91-97db-eca37b626ffb", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:49Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "2c35dceb-20b4-4278-977e-4804b2f900b6", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:47:49Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "2c35dceb-20b4-4278-977e-4804b2f900b6", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:47:49Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "2c35dceb-20b4-4278-977e-4804b2f900b6"}
2023-04-24T10:47:49Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "2c35dceb-20b4-4278-977e-4804b2f900b6"}
2023-04-24T10:47:49Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "2c35dceb-20b4-4278-977e-4804b2f900b6"}
2023-04-24T10:47:49Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "2c35dceb-20b4-4278-977e-4804b2f900b6"}
2023-04-24T10:47:49Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "2c35dceb-20b4-4278-977e-4804b2f900b6", "pod": "health-test-3"}
2023-04-24T10:47:49Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "2c35dceb-20b4-4278-977e-4804b2f900b6", "pod": "health-test-3"}
2023-04-24T10:47:49Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "2c35dceb-20b4-4278-977e-4804b2f900b6"}
2023-04-24T10:47:49Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "2c35dceb-20b4-4278-977e-4804b2f900b6"}
2023-04-24T10:47:49Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "2c35dceb-20b4-4278-977e-4804b2f900b6", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:49Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "2c35dceb-20b4-4278-977e-4804b2f900b6", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:49Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "2c35dceb-20b4-4278-977e-4804b2f900b6", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:49Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "3cb821be-d133-46b5-abbd-68c830dfad98", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:47:49Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "3cb821be-d133-46b5-abbd-68c830dfad98", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:47:49Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "3cb821be-d133-46b5-abbd-68c830dfad98"}
2023-04-24T10:47:49Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "3cb821be-d133-46b5-abbd-68c830dfad98"}
2023-04-24T10:47:49Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "3cb821be-d133-46b5-abbd-68c830dfad98"}
2023-04-24T10:47:49Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "3cb821be-d133-46b5-abbd-68c830dfad98"}
2023-04-24T10:47:49Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "3cb821be-d133-46b5-abbd-68c830dfad98", "pod": "health-test-3"}
2023-04-24T10:47:49Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "3cb821be-d133-46b5-abbd-68c830dfad98", "pod": "health-test-3"}
2023-04-24T10:47:49Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "3cb821be-d133-46b5-abbd-68c830dfad98"}
2023-04-24T10:47:49Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "3cb821be-d133-46b5-abbd-68c830dfad98"}
2023-04-24T10:47:49Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "3cb821be-d133-46b5-abbd-68c830dfad98", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:49Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "3cb821be-d133-46b5-abbd-68c830dfad98", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:49Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "3cb821be-d133-46b5-abbd-68c830dfad98", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:49Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "84986489-5271-40f5-a430-30a9a3920ae1", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:47:49Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "84986489-5271-40f5-a430-30a9a3920ae1", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:47:49Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "84986489-5271-40f5-a430-30a9a3920ae1"}
2023-04-24T10:47:49Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "84986489-5271-40f5-a430-30a9a3920ae1"}
2023-04-24T10:47:49Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "84986489-5271-40f5-a430-30a9a3920ae1"}
2023-04-24T10:47:49Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "84986489-5271-40f5-a430-30a9a3920ae1"}
2023-04-24T10:47:49Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "84986489-5271-40f5-a430-30a9a3920ae1", "pod": "health-test-3"}
2023-04-24T10:47:49Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "84986489-5271-40f5-a430-30a9a3920ae1", "pod": "health-test-3"}
2023-04-24T10:47:49Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "84986489-5271-40f5-a430-30a9a3920ae1"}
2023-04-24T10:47:49Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "84986489-5271-40f5-a430-30a9a3920ae1"}
2023-04-24T10:47:49Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "84986489-5271-40f5-a430-30a9a3920ae1", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:49Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "84986489-5271-40f5-a430-30a9a3920ae1", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:49Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "84986489-5271-40f5-a430-30a9a3920ae1", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:50Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "530b3a26-01be-423f-8754-6d7a4fcac6e3", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:47:50Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "530b3a26-01be-423f-8754-6d7a4fcac6e3", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:47:50Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "530b3a26-01be-423f-8754-6d7a4fcac6e3"}
2023-04-24T10:47:50Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "530b3a26-01be-423f-8754-6d7a4fcac6e3"}
2023-04-24T10:47:50Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "530b3a26-01be-423f-8754-6d7a4fcac6e3"}
2023-04-24T10:47:50Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "530b3a26-01be-423f-8754-6d7a4fcac6e3"}
2023-04-24T10:47:50Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "530b3a26-01be-423f-8754-6d7a4fcac6e3", "pod": "health-test-3"}
2023-04-24T10:47:50Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "530b3a26-01be-423f-8754-6d7a4fcac6e3", "pod": "health-test-3"}
2023-04-24T10:47:50Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "530b3a26-01be-423f-8754-6d7a4fcac6e3"}
2023-04-24T10:47:50Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "530b3a26-01be-423f-8754-6d7a4fcac6e3"}
2023-04-24T10:47:50Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "530b3a26-01be-423f-8754-6d7a4fcac6e3", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:50Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "530b3a26-01be-423f-8754-6d7a4fcac6e3", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:50Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "530b3a26-01be-423f-8754-6d7a4fcac6e3", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:51Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "dc31e836-09ae-440f-8272-68c25f62bf5b", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:47:51Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "dc31e836-09ae-440f-8272-68c25f62bf5b", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:47:51Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "dc31e836-09ae-440f-8272-68c25f62bf5b"}
2023-04-24T10:47:51Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "dc31e836-09ae-440f-8272-68c25f62bf5b"}
2023-04-24T10:47:51Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "dc31e836-09ae-440f-8272-68c25f62bf5b"}
2023-04-24T10:47:51Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "dc31e836-09ae-440f-8272-68c25f62bf5b"}
2023-04-24T10:47:51Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "dc31e836-09ae-440f-8272-68c25f62bf5b", "pod": "health-test-3"}
2023-04-24T10:47:51Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "dc31e836-09ae-440f-8272-68c25f62bf5b", "pod": "health-test-3"}
2023-04-24T10:47:51Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "dc31e836-09ae-440f-8272-68c25f62bf5b"}
2023-04-24T10:47:51Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "dc31e836-09ae-440f-8272-68c25f62bf5b"}
2023-04-24T10:47:51Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "dc31e836-09ae-440f-8272-68c25f62bf5b", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:51Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "dc31e836-09ae-440f-8272-68c25f62bf5b", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:51Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "dc31e836-09ae-440f-8272-68c25f62bf5b", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:51Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "22802120-e248-4c18-9e32-70b94d2ff160", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:47:51Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "22802120-e248-4c18-9e32-70b94d2ff160", "pod": "default/health-test-0", "role": "master"}
2023-04-24T10:47:51Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "22802120-e248-4c18-9e32-70b94d2ff160"}
2023-04-24T10:47:51Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "22802120-e248-4c18-9e32-70b94d2ff160"}
2023-04-24T10:47:51Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "22802120-e248-4c18-9e32-70b94d2ff160"}
2023-04-24T10:47:51Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "22802120-e248-4c18-9e32-70b94d2ff160"}
2023-04-24T10:47:51Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "22802120-e248-4c18-9e32-70b94d2ff160", "pod": "health-test-3"}
2023-04-24T10:47:51Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "22802120-e248-4c18-9e32-70b94d2ff160", "pod": "health-test-3"}
2023-04-24T10:47:51Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "22802120-e248-4c18-9e32-70b94d2ff160"}
2023-04-24T10:47:51Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "22802120-e248-4c18-9e32-70b94d2ff160"}
2023-04-24T10:47:51Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "22802120-e248-4c18-9e32-70b94d2ff160", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:51Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "22802120-e248-4c18-9e32-70b94d2ff160", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:51Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "22802120-e248-4c18-9e32-70b94d2ff160", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:54Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c95c4d35-3cfd-402c-8d24-8c0a3ce058c8", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:47:54Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c95c4d35-3cfd-402c-8d24-8c0a3ce058c8", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:47:54Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c95c4d35-3cfd-402c-8d24-8c0a3ce058c8"}
2023-04-24T10:47:54Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c95c4d35-3cfd-402c-8d24-8c0a3ce058c8"}
2023-04-24T10:47:54Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c95c4d35-3cfd-402c-8d24-8c0a3ce058c8"}
2023-04-24T10:47:54Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c95c4d35-3cfd-402c-8d24-8c0a3ce058c8"}
2023-04-24T10:47:54Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c95c4d35-3cfd-402c-8d24-8c0a3ce058c8", "pod": "health-test-3"}
2023-04-24T10:47:54Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c95c4d35-3cfd-402c-8d24-8c0a3ce058c8", "pod": "health-test-3"}
2023-04-24T10:47:54Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c95c4d35-3cfd-402c-8d24-8c0a3ce058c8"}
2023-04-24T10:47:54Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c95c4d35-3cfd-402c-8d24-8c0a3ce058c8"}
2023-04-24T10:47:54Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c95c4d35-3cfd-402c-8d24-8c0a3ce058c8", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:54Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c95c4d35-3cfd-402c-8d24-8c0a3ce058c8", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:54Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "c95c4d35-3cfd-402c-8d24-8c0a3ce058c8", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:56Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ce39ca8f-3e6d-4f67-bba5-b802248634fb", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:47:56Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ce39ca8f-3e6d-4f67-bba5-b802248634fb", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:47:56Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ce39ca8f-3e6d-4f67-bba5-b802248634fb"}
2023-04-24T10:47:56Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ce39ca8f-3e6d-4f67-bba5-b802248634fb"}
2023-04-24T10:47:56Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ce39ca8f-3e6d-4f67-bba5-b802248634fb"}
2023-04-24T10:47:56Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ce39ca8f-3e6d-4f67-bba5-b802248634fb"}
2023-04-24T10:47:56Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ce39ca8f-3e6d-4f67-bba5-b802248634fb", "pod": "health-test-3"}
2023-04-24T10:47:56Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ce39ca8f-3e6d-4f67-bba5-b802248634fb", "pod": "health-test-3"}
2023-04-24T10:47:56Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ce39ca8f-3e6d-4f67-bba5-b802248634fb"}
2023-04-24T10:47:56Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ce39ca8f-3e6d-4f67-bba5-b802248634fb"}
2023-04-24T10:47:56Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ce39ca8f-3e6d-4f67-bba5-b802248634fb", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:56Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ce39ca8f-3e6d-4f67-bba5-b802248634fb", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:56Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ce39ca8f-3e6d-4f67-bba5-b802248634fb", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:56Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "59c591d5-7c32-4a13-aaec-dd71ce78e2e7", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:47:56Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "59c591d5-7c32-4a13-aaec-dd71ce78e2e7", "pod": "default/health-test-0", "role": "master"}
2023-04-24T10:47:56Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "59c591d5-7c32-4a13-aaec-dd71ce78e2e7"}
2023-04-24T10:47:56Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "59c591d5-7c32-4a13-aaec-dd71ce78e2e7"}
2023-04-24T10:47:56Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "59c591d5-7c32-4a13-aaec-dd71ce78e2e7"}
2023-04-24T10:47:56Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "59c591d5-7c32-4a13-aaec-dd71ce78e2e7"}
2023-04-24T10:47:56Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "59c591d5-7c32-4a13-aaec-dd71ce78e2e7", "pod": "health-test-3"}
2023-04-24T10:47:56Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "59c591d5-7c32-4a13-aaec-dd71ce78e2e7", "pod": "health-test-3"}
2023-04-24T10:47:56Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "59c591d5-7c32-4a13-aaec-dd71ce78e2e7"}
2023-04-24T10:47:56Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "59c591d5-7c32-4a13-aaec-dd71ce78e2e7"}
2023-04-24T10:47:56Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "59c591d5-7c32-4a13-aaec-dd71ce78e2e7", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:56Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "59c591d5-7c32-4a13-aaec-dd71ce78e2e7", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:56Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "59c591d5-7c32-4a13-aaec-dd71ce78e2e7", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:59Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6522b61c-3bb2-497e-911a-aa2315c186d9", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:47:59Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6522b61c-3bb2-497e-911a-aa2315c186d9", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:47:59Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6522b61c-3bb2-497e-911a-aa2315c186d9"}
2023-04-24T10:47:59Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6522b61c-3bb2-497e-911a-aa2315c186d9"}
2023-04-24T10:47:59Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6522b61c-3bb2-497e-911a-aa2315c186d9"}
2023-04-24T10:47:59Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6522b61c-3bb2-497e-911a-aa2315c186d9"}
2023-04-24T10:47:59Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6522b61c-3bb2-497e-911a-aa2315c186d9", "pod": "health-test-3"}
2023-04-24T10:47:59Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6522b61c-3bb2-497e-911a-aa2315c186d9", "pod": "health-test-3"}
2023-04-24T10:47:59Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6522b61c-3bb2-497e-911a-aa2315c186d9"}
2023-04-24T10:47:59Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6522b61c-3bb2-497e-911a-aa2315c186d9"}
2023-04-24T10:47:59Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6522b61c-3bb2-497e-911a-aa2315c186d9", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:59Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6522b61c-3bb2-497e-911a-aa2315c186d9", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:59Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6522b61c-3bb2-497e-911a-aa2315c186d9", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:59Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "56bb203e-2853-4687-9bc5-5c5e05d429e6", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:47:59Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "56bb203e-2853-4687-9bc5-5c5e05d429e6", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:47:59Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "56bb203e-2853-4687-9bc5-5c5e05d429e6"}
2023-04-24T10:47:59Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "56bb203e-2853-4687-9bc5-5c5e05d429e6"}
2023-04-24T10:47:59Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "56bb203e-2853-4687-9bc5-5c5e05d429e6"}
2023-04-24T10:47:59Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "56bb203e-2853-4687-9bc5-5c5e05d429e6"}
2023-04-24T10:47:59Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "56bb203e-2853-4687-9bc5-5c5e05d429e6", "pod": "health-test-3"}
2023-04-24T10:47:59Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "56bb203e-2853-4687-9bc5-5c5e05d429e6", "pod": "health-test-3"}
2023-04-24T10:47:59Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "56bb203e-2853-4687-9bc5-5c5e05d429e6"}
2023-04-24T10:47:59Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "56bb203e-2853-4687-9bc5-5c5e05d429e6"}
2023-04-24T10:47:59Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "56bb203e-2853-4687-9bc5-5c5e05d429e6", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:47:59Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "56bb203e-2853-4687-9bc5-5c5e05d429e6", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:47:59Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "56bb203e-2853-4687-9bc5-5c5e05d429e6", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:48:07Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "4a99a0e3-4b2e-48a4-b0c9-08bded64fc36", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:48:07Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "4a99a0e3-4b2e-48a4-b0c9-08bded64fc36", "pod": "default/health-test-0", "role": "master"}
2023-04-24T10:48:07Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "4a99a0e3-4b2e-48a4-b0c9-08bded64fc36"}
2023-04-24T10:48:07Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "4a99a0e3-4b2e-48a4-b0c9-08bded64fc36"}
2023-04-24T10:48:07Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "4a99a0e3-4b2e-48a4-b0c9-08bded64fc36"}
2023-04-24T10:48:07Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "4a99a0e3-4b2e-48a4-b0c9-08bded64fc36"}
2023-04-24T10:48:07Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "4a99a0e3-4b2e-48a4-b0c9-08bded64fc36", "pod": "health-test-3"}
2023-04-24T10:48:07Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "4a99a0e3-4b2e-48a4-b0c9-08bded64fc36", "pod": "health-test-3"}
2023-04-24T10:48:07Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "4a99a0e3-4b2e-48a4-b0c9-08bded64fc36"}
2023-04-24T10:48:07Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "4a99a0e3-4b2e-48a4-b0c9-08bded64fc36"}
2023-04-24T10:48:07Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "4a99a0e3-4b2e-48a4-b0c9-08bded64fc36", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:48:07Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "4a99a0e3-4b2e-48a4-b0c9-08bded64fc36", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:48:07Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "4a99a0e3-4b2e-48a4-b0c9-08bded64fc36", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:48:09Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f24250bb-418b-4c91-8880-3de7b2d2ab00", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:48:09Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f24250bb-418b-4c91-8880-3de7b2d2ab00", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:48:09Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f24250bb-418b-4c91-8880-3de7b2d2ab00"}
2023-04-24T10:48:09Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f24250bb-418b-4c91-8880-3de7b2d2ab00"}
2023-04-24T10:48:09Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f24250bb-418b-4c91-8880-3de7b2d2ab00"}
2023-04-24T10:48:09Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f24250bb-418b-4c91-8880-3de7b2d2ab00"}
2023-04-24T10:48:09Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f24250bb-418b-4c91-8880-3de7b2d2ab00", "pod": "health-test-3"}
2023-04-24T10:48:09Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f24250bb-418b-4c91-8880-3de7b2d2ab00", "pod": "health-test-3"}
2023-04-24T10:48:09Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f24250bb-418b-4c91-8880-3de7b2d2ab00"}
2023-04-24T10:48:09Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f24250bb-418b-4c91-8880-3de7b2d2ab00"}
2023-04-24T10:48:09Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f24250bb-418b-4c91-8880-3de7b2d2ab00", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:48:09Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f24250bb-418b-4c91-8880-3de7b2d2ab00", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:48:09Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "f24250bb-418b-4c91-8880-3de7b2d2ab00", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:48:16Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "4a38ce3e-e841-4788-ac95-50eaf5f47b72", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:48:16Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "4a38ce3e-e841-4788-ac95-50eaf5f47b72", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:48:16Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "4a38ce3e-e841-4788-ac95-50eaf5f47b72"}
2023-04-24T10:48:16Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "4a38ce3e-e841-4788-ac95-50eaf5f47b72"}
2023-04-24T10:48:16Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "4a38ce3e-e841-4788-ac95-50eaf5f47b72"}
2023-04-24T10:48:16Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "4a38ce3e-e841-4788-ac95-50eaf5f47b72"}
2023-04-24T10:48:16Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "4a38ce3e-e841-4788-ac95-50eaf5f47b72", "pod": "health-test-3"}
2023-04-24T10:48:16Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "4a38ce3e-e841-4788-ac95-50eaf5f47b72", "pod": "health-test-3"}
2023-04-24T10:48:16Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "4a38ce3e-e841-4788-ac95-50eaf5f47b72"}
2023-04-24T10:48:16Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "4a38ce3e-e841-4788-ac95-50eaf5f47b72"}
2023-04-24T10:48:16Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "4a38ce3e-e841-4788-ac95-50eaf5f47b72", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:48:16Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "4a38ce3e-e841-4788-ac95-50eaf5f47b72", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:48:16Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "4a38ce3e-e841-4788-ac95-50eaf5f47b72", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:48:19Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "36e3a79b-492e-4202-84a1-d2ef124b35a1", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:48:19Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "36e3a79b-492e-4202-84a1-d2ef124b35a1", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:48:19Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "36e3a79b-492e-4202-84a1-d2ef124b35a1"}
2023-04-24T10:48:19Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "36e3a79b-492e-4202-84a1-d2ef124b35a1"}
2023-04-24T10:48:19Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "36e3a79b-492e-4202-84a1-d2ef124b35a1"}
2023-04-24T10:48:19Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "36e3a79b-492e-4202-84a1-d2ef124b35a1"}
2023-04-24T10:48:19Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "36e3a79b-492e-4202-84a1-d2ef124b35a1", "pod": "health-test-3"}
2023-04-24T10:48:19Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "36e3a79b-492e-4202-84a1-d2ef124b35a1", "pod": "health-test-3"}
2023-04-24T10:48:19Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "36e3a79b-492e-4202-84a1-d2ef124b35a1"}
2023-04-24T10:48:19Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "36e3a79b-492e-4202-84a1-d2ef124b35a1"}
2023-04-24T10:48:19Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "36e3a79b-492e-4202-84a1-d2ef124b35a1", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:48:19Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "36e3a79b-492e-4202-84a1-d2ef124b35a1", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:48:19Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "36e3a79b-492e-4202-84a1-d2ef124b35a1", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:48:27Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6575e83d-52b3-4884-93b2-c19dd7599c8b", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:48:27Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6575e83d-52b3-4884-93b2-c19dd7599c8b", "pod": "default/health-test-0", "role": "master"}
2023-04-24T10:48:27Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6575e83d-52b3-4884-93b2-c19dd7599c8b"}
2023-04-24T10:48:27Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6575e83d-52b3-4884-93b2-c19dd7599c8b"}
2023-04-24T10:48:27Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6575e83d-52b3-4884-93b2-c19dd7599c8b"}
2023-04-24T10:48:27Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6575e83d-52b3-4884-93b2-c19dd7599c8b"}
2023-04-24T10:48:27Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6575e83d-52b3-4884-93b2-c19dd7599c8b", "pod": "health-test-3"}
2023-04-24T10:48:27Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6575e83d-52b3-4884-93b2-c19dd7599c8b", "pod": "health-test-3"}
2023-04-24T10:48:27Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6575e83d-52b3-4884-93b2-c19dd7599c8b"}
2023-04-24T10:48:27Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6575e83d-52b3-4884-93b2-c19dd7599c8b"}
2023-04-24T10:48:27Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6575e83d-52b3-4884-93b2-c19dd7599c8b", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:48:27Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6575e83d-52b3-4884-93b2-c19dd7599c8b", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:48:27Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "6575e83d-52b3-4884-93b2-c19dd7599c8b", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:48:30Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "59a3c50f-a73e-466d-8808-9f47f5e280e7", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:48:30Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "59a3c50f-a73e-466d-8808-9f47f5e280e7", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:48:30Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "59a3c50f-a73e-466d-8808-9f47f5e280e7"}
2023-04-24T10:48:30Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "59a3c50f-a73e-466d-8808-9f47f5e280e7"}
2023-04-24T10:48:30Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "59a3c50f-a73e-466d-8808-9f47f5e280e7"}
2023-04-24T10:48:30Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "59a3c50f-a73e-466d-8808-9f47f5e280e7"}
2023-04-24T10:48:30Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "59a3c50f-a73e-466d-8808-9f47f5e280e7", "pod": "health-test-3"}
2023-04-24T10:48:30Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "59a3c50f-a73e-466d-8808-9f47f5e280e7", "pod": "health-test-3"}
2023-04-24T10:48:30Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "59a3c50f-a73e-466d-8808-9f47f5e280e7"}
2023-04-24T10:48:30Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "59a3c50f-a73e-466d-8808-9f47f5e280e7"}
2023-04-24T10:48:30Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "59a3c50f-a73e-466d-8808-9f47f5e280e7", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:48:30Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "59a3c50f-a73e-466d-8808-9f47f5e280e7", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:48:30Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "59a3c50f-a73e-466d-8808-9f47f5e280e7", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:48:57Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ab7b7c5e-8b6c-4f04-a00c-ef9eeeb95b59", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:48:57Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ab7b7c5e-8b6c-4f04-a00c-ef9eeeb95b59", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:48:57Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ab7b7c5e-8b6c-4f04-a00c-ef9eeeb95b59"}
2023-04-24T10:48:57Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ab7b7c5e-8b6c-4f04-a00c-ef9eeeb95b59"}
2023-04-24T10:48:57Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ab7b7c5e-8b6c-4f04-a00c-ef9eeeb95b59"}
2023-04-24T10:48:57Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ab7b7c5e-8b6c-4f04-a00c-ef9eeeb95b59"}
2023-04-24T10:48:57Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ab7b7c5e-8b6c-4f04-a00c-ef9eeeb95b59", "pod": "health-test-3"}
2023-04-24T10:48:57Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ab7b7c5e-8b6c-4f04-a00c-ef9eeeb95b59", "pod": "health-test-3"}
2023-04-24T10:48:57Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ab7b7c5e-8b6c-4f04-a00c-ef9eeeb95b59"}
2023-04-24T10:48:57Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ab7b7c5e-8b6c-4f04-a00c-ef9eeeb95b59"}
2023-04-24T10:48:57Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ab7b7c5e-8b6c-4f04-a00c-ef9eeeb95b59", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:48:57Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ab7b7c5e-8b6c-4f04-a00c-ef9eeeb95b59", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:48:57Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "ab7b7c5e-8b6c-4f04-a00c-ef9eeeb95b59", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:49:00Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e04a4ee-0893-4ece-b432-2d38e6e0a654", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:49:00Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e04a4ee-0893-4ece-b432-2d38e6e0a654", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:49:00Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e04a4ee-0893-4ece-b432-2d38e6e0a654"}
2023-04-24T10:49:00Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e04a4ee-0893-4ece-b432-2d38e6e0a654"}
2023-04-24T10:49:00Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e04a4ee-0893-4ece-b432-2d38e6e0a654"}
2023-04-24T10:49:00Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e04a4ee-0893-4ece-b432-2d38e6e0a654"}
2023-04-24T10:49:00Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e04a4ee-0893-4ece-b432-2d38e6e0a654", "pod": "health-test-3"}
2023-04-24T10:49:00Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e04a4ee-0893-4ece-b432-2d38e6e0a654", "pod": "health-test-3"}
2023-04-24T10:49:00Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e04a4ee-0893-4ece-b432-2d38e6e0a654"}
2023-04-24T10:49:00Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e04a4ee-0893-4ece-b432-2d38e6e0a654"}
2023-04-24T10:49:00Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e04a4ee-0893-4ece-b432-2d38e6e0a654", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:49:00Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e04a4ee-0893-4ece-b432-2d38e6e0a654", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:49:00Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "2e04a4ee-0893-4ece-b432-2d38e6e0a654", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:49:08Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ae923ed5-6e2e-4678-9318-c0b45851a0fb", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:49:08Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ae923ed5-6e2e-4678-9318-c0b45851a0fb", "pod": "default/health-test-0", "role": "master"}
2023-04-24T10:49:08Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ae923ed5-6e2e-4678-9318-c0b45851a0fb"}
2023-04-24T10:49:08Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ae923ed5-6e2e-4678-9318-c0b45851a0fb"}
2023-04-24T10:49:08Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ae923ed5-6e2e-4678-9318-c0b45851a0fb"}
2023-04-24T10:49:08Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ae923ed5-6e2e-4678-9318-c0b45851a0fb"}
2023-04-24T10:49:08Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ae923ed5-6e2e-4678-9318-c0b45851a0fb", "pod": "health-test-3"}
2023-04-24T10:49:08Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ae923ed5-6e2e-4678-9318-c0b45851a0fb", "pod": "health-test-3"}
2023-04-24T10:49:08Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ae923ed5-6e2e-4678-9318-c0b45851a0fb"}
2023-04-24T10:49:08Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ae923ed5-6e2e-4678-9318-c0b45851a0fb"}
2023-04-24T10:49:08Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ae923ed5-6e2e-4678-9318-c0b45851a0fb", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:49:08Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ae923ed5-6e2e-4678-9318-c0b45851a0fb", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:49:08Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "ae923ed5-6e2e-4678-9318-c0b45851a0fb", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:49:11Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d5498c4-ec18-4724-9307-739b7070a6b1", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:49:11Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d5498c4-ec18-4724-9307-739b7070a6b1", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:49:11Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d5498c4-ec18-4724-9307-739b7070a6b1"}
2023-04-24T10:49:11Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d5498c4-ec18-4724-9307-739b7070a6b1"}
2023-04-24T10:49:11Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d5498c4-ec18-4724-9307-739b7070a6b1"}
2023-04-24T10:49:11Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d5498c4-ec18-4724-9307-739b7070a6b1"}
2023-04-24T10:49:11Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d5498c4-ec18-4724-9307-739b7070a6b1", "pod": "health-test-3"}
2023-04-24T10:49:11Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d5498c4-ec18-4724-9307-739b7070a6b1", "pod": "health-test-3"}
2023-04-24T10:49:11Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d5498c4-ec18-4724-9307-739b7070a6b1"}
2023-04-24T10:49:11Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d5498c4-ec18-4724-9307-739b7070a6b1"}
2023-04-24T10:49:11Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d5498c4-ec18-4724-9307-739b7070a6b1", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:49:11Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d5498c4-ec18-4724-9307-739b7070a6b1", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:49:11Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "6d5498c4-ec18-4724-9307-739b7070a6b1", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:50:19Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "2f49ad9c-bf9c-490e-9736-519cb8aa3f3b", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:50:19Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "2f49ad9c-bf9c-490e-9736-519cb8aa3f3b", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:50:19Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "2f49ad9c-bf9c-490e-9736-519cb8aa3f3b"}
2023-04-24T10:50:19Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "2f49ad9c-bf9c-490e-9736-519cb8aa3f3b"}
2023-04-24T10:50:19Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "2f49ad9c-bf9c-490e-9736-519cb8aa3f3b"}
2023-04-24T10:50:19Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "2f49ad9c-bf9c-490e-9736-519cb8aa3f3b"}
2023-04-24T10:50:19Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "2f49ad9c-bf9c-490e-9736-519cb8aa3f3b", "pod": "health-test-3"}
2023-04-24T10:50:19Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "2f49ad9c-bf9c-490e-9736-519cb8aa3f3b", "pod": "health-test-3"}
2023-04-24T10:50:19Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "2f49ad9c-bf9c-490e-9736-519cb8aa3f3b"}
2023-04-24T10:50:19Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "2f49ad9c-bf9c-490e-9736-519cb8aa3f3b"}
2023-04-24T10:50:19Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "2f49ad9c-bf9c-490e-9736-519cb8aa3f3b", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:50:19Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "2f49ad9c-bf9c-490e-9736-519cb8aa3f3b", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:50:19Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "2f49ad9c-bf9c-490e-9736-519cb8aa3f3b", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:50:22Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "1e916337-e66e-4c95-b29f-abf5958d5d28", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:50:22Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "1e916337-e66e-4c95-b29f-abf5958d5d28", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:50:22Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "1e916337-e66e-4c95-b29f-abf5958d5d28"}
2023-04-24T10:50:22Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "1e916337-e66e-4c95-b29f-abf5958d5d28"}
2023-04-24T10:50:22Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "1e916337-e66e-4c95-b29f-abf5958d5d28"}
2023-04-24T10:50:22Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "1e916337-e66e-4c95-b29f-abf5958d5d28"}
2023-04-24T10:50:22Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "1e916337-e66e-4c95-b29f-abf5958d5d28", "pod": "health-test-3"}
2023-04-24T10:50:22Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "1e916337-e66e-4c95-b29f-abf5958d5d28", "pod": "health-test-3"}
2023-04-24T10:50:22Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "1e916337-e66e-4c95-b29f-abf5958d5d28"}
2023-04-24T10:50:22Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "1e916337-e66e-4c95-b29f-abf5958d5d28"}
2023-04-24T10:50:22Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "1e916337-e66e-4c95-b29f-abf5958d5d28", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:50:22Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "1e916337-e66e-4c95-b29f-abf5958d5d28", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:50:22Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "1e916337-e66e-4c95-b29f-abf5958d5d28", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:50:30Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "70f5d5a8-3bf5-49fc-865c-5dcd70a6dd5e", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:50:30Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "70f5d5a8-3bf5-49fc-865c-5dcd70a6dd5e", "pod": "default/health-test-0", "role": "master"}
2023-04-24T10:50:30Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "70f5d5a8-3bf5-49fc-865c-5dcd70a6dd5e"}
2023-04-24T10:50:30Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "70f5d5a8-3bf5-49fc-865c-5dcd70a6dd5e"}
2023-04-24T10:50:30Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "70f5d5a8-3bf5-49fc-865c-5dcd70a6dd5e"}
2023-04-24T10:50:30Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "70f5d5a8-3bf5-49fc-865c-5dcd70a6dd5e"}
2023-04-24T10:50:30Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "70f5d5a8-3bf5-49fc-865c-5dcd70a6dd5e", "pod": "health-test-3"}
2023-04-24T10:50:30Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "70f5d5a8-3bf5-49fc-865c-5dcd70a6dd5e", "pod": "health-test-3"}
2023-04-24T10:50:30Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "70f5d5a8-3bf5-49fc-865c-5dcd70a6dd5e"}
2023-04-24T10:50:30Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "70f5d5a8-3bf5-49fc-865c-5dcd70a6dd5e"}
2023-04-24T10:50:30Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "70f5d5a8-3bf5-49fc-865c-5dcd70a6dd5e", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:50:30Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "70f5d5a8-3bf5-49fc-865c-5dcd70a6dd5e", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:50:30Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "70f5d5a8-3bf5-49fc-865c-5dcd70a6dd5e", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:50:32Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "eac50e3c-0bd5-48b5-bb4f-a87f53c3f4fb", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:50:32Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "eac50e3c-0bd5-48b5-bb4f-a87f53c3f4fb", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:50:32Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "eac50e3c-0bd5-48b5-bb4f-a87f53c3f4fb"}
2023-04-24T10:50:32Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "eac50e3c-0bd5-48b5-bb4f-a87f53c3f4fb"}
2023-04-24T10:50:32Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "eac50e3c-0bd5-48b5-bb4f-a87f53c3f4fb"}
2023-04-24T10:50:32Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "eac50e3c-0bd5-48b5-bb4f-a87f53c3f4fb"}
2023-04-24T10:50:32Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "eac50e3c-0bd5-48b5-bb4f-a87f53c3f4fb", "pod": "health-test-3"}
2023-04-24T10:50:32Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "eac50e3c-0bd5-48b5-bb4f-a87f53c3f4fb", "pod": "health-test-3"}
2023-04-24T10:50:32Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "eac50e3c-0bd5-48b5-bb4f-a87f53c3f4fb"}
2023-04-24T10:50:32Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "eac50e3c-0bd5-48b5-bb4f-a87f53c3f4fb"}
2023-04-24T10:50:32Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "eac50e3c-0bd5-48b5-bb4f-a87f53c3f4fb", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:50:32Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "eac50e3c-0bd5-48b5-bb4f-a87f53c3f4fb", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:50:32Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "eac50e3c-0bd5-48b5-bb4f-a87f53c3f4fb", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:53:03Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65721b34-e335-4f7c-85ca-b72f696e32ea", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:53:03Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65721b34-e335-4f7c-85ca-b72f696e32ea", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:53:03Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65721b34-e335-4f7c-85ca-b72f696e32ea"}
2023-04-24T10:53:03Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65721b34-e335-4f7c-85ca-b72f696e32ea"}
2023-04-24T10:53:03Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65721b34-e335-4f7c-85ca-b72f696e32ea"}
2023-04-24T10:53:03Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65721b34-e335-4f7c-85ca-b72f696e32ea"}
2023-04-24T10:53:03Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65721b34-e335-4f7c-85ca-b72f696e32ea", "pod": "health-test-3"}
2023-04-24T10:53:03Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65721b34-e335-4f7c-85ca-b72f696e32ea", "pod": "health-test-3"}
2023-04-24T10:53:03Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65721b34-e335-4f7c-85ca-b72f696e32ea"}
2023-04-24T10:53:03Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65721b34-e335-4f7c-85ca-b72f696e32ea"}
2023-04-24T10:53:03Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65721b34-e335-4f7c-85ca-b72f696e32ea", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:53:03Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65721b34-e335-4f7c-85ca-b72f696e32ea", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:53:03Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "65721b34-e335-4f7c-85ca-b72f696e32ea", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:53:06Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5f28e5f-5e80-4849-b303-aac7c171be0d", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:53:06Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5f28e5f-5e80-4849-b303-aac7c171be0d", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:53:06Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5f28e5f-5e80-4849-b303-aac7c171be0d"}
2023-04-24T10:53:06Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5f28e5f-5e80-4849-b303-aac7c171be0d"}
2023-04-24T10:53:06Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5f28e5f-5e80-4849-b303-aac7c171be0d"}
2023-04-24T10:53:06Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5f28e5f-5e80-4849-b303-aac7c171be0d"}
2023-04-24T10:53:06Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5f28e5f-5e80-4849-b303-aac7c171be0d", "pod": "health-test-3"}
2023-04-24T10:53:06Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5f28e5f-5e80-4849-b303-aac7c171be0d", "pod": "health-test-3"}
2023-04-24T10:53:06Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5f28e5f-5e80-4849-b303-aac7c171be0d"}
2023-04-24T10:53:06Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5f28e5f-5e80-4849-b303-aac7c171be0d"}
2023-04-24T10:53:06Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5f28e5f-5e80-4849-b303-aac7c171be0d", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:53:06Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5f28e5f-5e80-4849-b303-aac7c171be0d", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:53:06Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "f5f28e5f-5e80-4849-b303-aac7c171be0d", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:53:14Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "832fa272-6564-49f2-ae0b-02424bd32052", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:53:14Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "832fa272-6564-49f2-ae0b-02424bd32052", "pod": "default/health-test-0", "role": "master"}
2023-04-24T10:53:14Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "832fa272-6564-49f2-ae0b-02424bd32052"}
2023-04-24T10:53:14Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "832fa272-6564-49f2-ae0b-02424bd32052"}
2023-04-24T10:53:14Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "832fa272-6564-49f2-ae0b-02424bd32052"}
2023-04-24T10:53:14Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "832fa272-6564-49f2-ae0b-02424bd32052"}
2023-04-24T10:53:14Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "832fa272-6564-49f2-ae0b-02424bd32052", "pod": "health-test-3"}
2023-04-24T10:53:14Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "832fa272-6564-49f2-ae0b-02424bd32052", "pod": "health-test-3"}
2023-04-24T10:53:14Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "832fa272-6564-49f2-ae0b-02424bd32052"}
2023-04-24T10:53:14Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "832fa272-6564-49f2-ae0b-02424bd32052"}
2023-04-24T10:53:14Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "832fa272-6564-49f2-ae0b-02424bd32052", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:53:14Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "832fa272-6564-49f2-ae0b-02424bd32052", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:53:14Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "832fa272-6564-49f2-ae0b-02424bd32052", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:53:16Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "d8a49c82-e944-4de8-9d19-837148e0b8e7", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:53:16Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "d8a49c82-e944-4de8-9d19-837148e0b8e7", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:53:16Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "d8a49c82-e944-4de8-9d19-837148e0b8e7"}
2023-04-24T10:53:16Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "d8a49c82-e944-4de8-9d19-837148e0b8e7"}
2023-04-24T10:53:16Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "d8a49c82-e944-4de8-9d19-837148e0b8e7"}
2023-04-24T10:53:16Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "d8a49c82-e944-4de8-9d19-837148e0b8e7"}
2023-04-24T10:53:16Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "d8a49c82-e944-4de8-9d19-837148e0b8e7", "pod": "health-test-3"}
2023-04-24T10:53:16Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "d8a49c82-e944-4de8-9d19-837148e0b8e7", "pod": "health-test-3"}
2023-04-24T10:53:16Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "d8a49c82-e944-4de8-9d19-837148e0b8e7"}
2023-04-24T10:53:16Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "d8a49c82-e944-4de8-9d19-837148e0b8e7"}
2023-04-24T10:53:16Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "d8a49c82-e944-4de8-9d19-837148e0b8e7", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:53:16Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "d8a49c82-e944-4de8-9d19-837148e0b8e7", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:53:16Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "d8a49c82-e944-4de8-9d19-837148e0b8e7", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:58:31Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "906090a6-0799-4b04-9f45-3467dd1c9cee", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T10:58:31Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "906090a6-0799-4b04-9f45-3467dd1c9cee", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T10:58:31Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "906090a6-0799-4b04-9f45-3467dd1c9cee"}
2023-04-24T10:58:31Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "906090a6-0799-4b04-9f45-3467dd1c9cee"}
2023-04-24T10:58:31Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "906090a6-0799-4b04-9f45-3467dd1c9cee"}
2023-04-24T10:58:31Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "906090a6-0799-4b04-9f45-3467dd1c9cee"}
2023-04-24T10:58:31Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "906090a6-0799-4b04-9f45-3467dd1c9cee", "pod": "health-test-3"}
2023-04-24T10:58:31Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "906090a6-0799-4b04-9f45-3467dd1c9cee", "pod": "health-test-3"}
2023-04-24T10:58:31Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "906090a6-0799-4b04-9f45-3467dd1c9cee"}
2023-04-24T10:58:31Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "906090a6-0799-4b04-9f45-3467dd1c9cee"}
2023-04-24T10:58:31Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "906090a6-0799-4b04-9f45-3467dd1c9cee", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:58:31Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "906090a6-0799-4b04-9f45-3467dd1c9cee", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:58:31Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "906090a6-0799-4b04-9f45-3467dd1c9cee", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:58:34Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3741aed5-dde7-451e-a361-3ca8fb458276", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T10:58:34Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3741aed5-dde7-451e-a361-3ca8fb458276", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T10:58:34Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3741aed5-dde7-451e-a361-3ca8fb458276"}
2023-04-24T10:58:34Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3741aed5-dde7-451e-a361-3ca8fb458276"}
2023-04-24T10:58:34Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3741aed5-dde7-451e-a361-3ca8fb458276"}
2023-04-24T10:58:34Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3741aed5-dde7-451e-a361-3ca8fb458276"}
2023-04-24T10:58:34Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3741aed5-dde7-451e-a361-3ca8fb458276", "pod": "health-test-3"}
2023-04-24T10:58:34Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3741aed5-dde7-451e-a361-3ca8fb458276", "pod": "health-test-3"}
2023-04-24T10:58:34Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3741aed5-dde7-451e-a361-3ca8fb458276"}
2023-04-24T10:58:34Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3741aed5-dde7-451e-a361-3ca8fb458276"}
2023-04-24T10:58:34Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3741aed5-dde7-451e-a361-3ca8fb458276", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:58:34Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3741aed5-dde7-451e-a361-3ca8fb458276", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:58:34Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "3741aed5-dde7-451e-a361-3ca8fb458276", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:58:41Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a00e9486-40bd-4c25-a3c4-371f76081c92", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T10:58:41Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a00e9486-40bd-4c25-a3c4-371f76081c92", "pod": "default/health-test-0", "role": "master"}
2023-04-24T10:58:41Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a00e9486-40bd-4c25-a3c4-371f76081c92"}
2023-04-24T10:58:41Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a00e9486-40bd-4c25-a3c4-371f76081c92"}
2023-04-24T10:58:41Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a00e9486-40bd-4c25-a3c4-371f76081c92"}
2023-04-24T10:58:41Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a00e9486-40bd-4c25-a3c4-371f76081c92"}
2023-04-24T10:58:41Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a00e9486-40bd-4c25-a3c4-371f76081c92", "pod": "health-test-3"}
2023-04-24T10:58:41Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a00e9486-40bd-4c25-a3c4-371f76081c92", "pod": "health-test-3"}
2023-04-24T10:58:41Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a00e9486-40bd-4c25-a3c4-371f76081c92"}
2023-04-24T10:58:41Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a00e9486-40bd-4c25-a3c4-371f76081c92"}
2023-04-24T10:58:41Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a00e9486-40bd-4c25-a3c4-371f76081c92", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:58:41Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a00e9486-40bd-4c25-a3c4-371f76081c92", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:58:41Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a00e9486-40bd-4c25-a3c4-371f76081c92", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:58:44Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "573c8608-aa10-4ae6-9cc6-9e35bebf9770", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T10:58:44Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "573c8608-aa10-4ae6-9cc6-9e35bebf9770", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T10:58:44Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "573c8608-aa10-4ae6-9cc6-9e35bebf9770"}
2023-04-24T10:58:44Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "573c8608-aa10-4ae6-9cc6-9e35bebf9770"}
2023-04-24T10:58:44Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "573c8608-aa10-4ae6-9cc6-9e35bebf9770"}
2023-04-24T10:58:44Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "573c8608-aa10-4ae6-9cc6-9e35bebf9770"}
2023-04-24T10:58:44Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "573c8608-aa10-4ae6-9cc6-9e35bebf9770", "pod": "health-test-3"}
2023-04-24T10:58:44Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "573c8608-aa10-4ae6-9cc6-9e35bebf9770", "pod": "health-test-3"}
2023-04-24T10:58:44Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "573c8608-aa10-4ae6-9cc6-9e35bebf9770"}
2023-04-24T10:58:44Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "573c8608-aa10-4ae6-9cc6-9e35bebf9770"}
2023-04-24T10:58:44Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "573c8608-aa10-4ae6-9cc6-9e35bebf9770", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T10:58:44Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "573c8608-aa10-4ae6-9cc6-9e35bebf9770", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T10:58:44Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "573c8608-aa10-4ae6-9cc6-9e35bebf9770", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T11:09:26Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b0853fd6-91d2-494a-b41a-224cd87b8a65", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T11:09:26Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b0853fd6-91d2-494a-b41a-224cd87b8a65", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T11:09:26Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b0853fd6-91d2-494a-b41a-224cd87b8a65"}
2023-04-24T11:09:26Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b0853fd6-91d2-494a-b41a-224cd87b8a65"}
2023-04-24T11:09:26Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b0853fd6-91d2-494a-b41a-224cd87b8a65"}
2023-04-24T11:09:26Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b0853fd6-91d2-494a-b41a-224cd87b8a65"}
2023-04-24T11:09:26Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b0853fd6-91d2-494a-b41a-224cd87b8a65", "pod": "health-test-3"}
2023-04-24T11:09:26Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b0853fd6-91d2-494a-b41a-224cd87b8a65", "pod": "health-test-3"}
2023-04-24T11:09:26Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b0853fd6-91d2-494a-b41a-224cd87b8a65"}
2023-04-24T11:09:26Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b0853fd6-91d2-494a-b41a-224cd87b8a65"}
2023-04-24T11:09:26Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b0853fd6-91d2-494a-b41a-224cd87b8a65", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T11:09:26Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b0853fd6-91d2-494a-b41a-224cd87b8a65", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T11:09:26Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "b0853fd6-91d2-494a-b41a-224cd87b8a65", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T11:09:29Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "b58f31bc-404a-47a4-98d0-cbdc7a696bdb", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T11:09:29Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "b58f31bc-404a-47a4-98d0-cbdc7a696bdb", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T11:09:29Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "b58f31bc-404a-47a4-98d0-cbdc7a696bdb"}
2023-04-24T11:09:29Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "b58f31bc-404a-47a4-98d0-cbdc7a696bdb"}
2023-04-24T11:09:29Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "b58f31bc-404a-47a4-98d0-cbdc7a696bdb"}
2023-04-24T11:09:29Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "b58f31bc-404a-47a4-98d0-cbdc7a696bdb"}
2023-04-24T11:09:29Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "b58f31bc-404a-47a4-98d0-cbdc7a696bdb", "pod": "health-test-3"}
2023-04-24T11:09:29Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "b58f31bc-404a-47a4-98d0-cbdc7a696bdb", "pod": "health-test-3"}
2023-04-24T11:09:29Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "b58f31bc-404a-47a4-98d0-cbdc7a696bdb"}
2023-04-24T11:09:29Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "b58f31bc-404a-47a4-98d0-cbdc7a696bdb"}
2023-04-24T11:09:29Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "b58f31bc-404a-47a4-98d0-cbdc7a696bdb", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T11:09:29Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "b58f31bc-404a-47a4-98d0-cbdc7a696bdb", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T11:09:29Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "b58f31bc-404a-47a4-98d0-cbdc7a696bdb", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T11:09:37Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "3f2c464b-0473-483a-be2b-bb9f0dcc5610", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T11:09:37Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "3f2c464b-0473-483a-be2b-bb9f0dcc5610", "pod": "default/health-test-0", "role": "master"}
2023-04-24T11:09:37Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "3f2c464b-0473-483a-be2b-bb9f0dcc5610"}
2023-04-24T11:09:37Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "3f2c464b-0473-483a-be2b-bb9f0dcc5610"}
2023-04-24T11:09:37Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "3f2c464b-0473-483a-be2b-bb9f0dcc5610"}
2023-04-24T11:09:37Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "3f2c464b-0473-483a-be2b-bb9f0dcc5610"}
2023-04-24T11:09:37Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "3f2c464b-0473-483a-be2b-bb9f0dcc5610", "pod": "health-test-3"}
2023-04-24T11:09:37Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "3f2c464b-0473-483a-be2b-bb9f0dcc5610", "pod": "health-test-3"}
2023-04-24T11:09:37Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "3f2c464b-0473-483a-be2b-bb9f0dcc5610"}
2023-04-24T11:09:37Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "3f2c464b-0473-483a-be2b-bb9f0dcc5610"}
2023-04-24T11:09:37Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "3f2c464b-0473-483a-be2b-bb9f0dcc5610", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T11:09:37Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "3f2c464b-0473-483a-be2b-bb9f0dcc5610", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T11:09:37Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "3f2c464b-0473-483a-be2b-bb9f0dcc5610", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T11:09:39Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9fbfb8c1-9680-4d42-8737-70baac8e58d4", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T11:09:39Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9fbfb8c1-9680-4d42-8737-70baac8e58d4", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T11:09:39Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9fbfb8c1-9680-4d42-8737-70baac8e58d4"}
2023-04-24T11:09:39Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9fbfb8c1-9680-4d42-8737-70baac8e58d4"}
2023-04-24T11:09:39Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9fbfb8c1-9680-4d42-8737-70baac8e58d4"}
2023-04-24T11:09:39Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9fbfb8c1-9680-4d42-8737-70baac8e58d4"}
2023-04-24T11:09:39Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9fbfb8c1-9680-4d42-8737-70baac8e58d4", "pod": "health-test-3"}
2023-04-24T11:09:39Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9fbfb8c1-9680-4d42-8737-70baac8e58d4", "pod": "health-test-3"}
2023-04-24T11:09:39Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9fbfb8c1-9680-4d42-8737-70baac8e58d4"}
2023-04-24T11:09:39Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9fbfb8c1-9680-4d42-8737-70baac8e58d4"}
2023-04-24T11:09:39Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9fbfb8c1-9680-4d42-8737-70baac8e58d4", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T11:09:39Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9fbfb8c1-9680-4d42-8737-70baac8e58d4", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T11:09:39Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "9fbfb8c1-9680-4d42-8737-70baac8e58d4", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T11:26:06Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "956d45f0-6eb3-4b18-bf7d-b27a5ffe5572", "pod": {"namespace": "default", "name": "health-test-2"}}
2023-04-24T11:26:06Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "956d45f0-6eb3-4b18-bf7d-b27a5ffe5572", "pod": "default/health-test-2", "role": "replica"}
2023-04-24T11:26:06Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "956d45f0-6eb3-4b18-bf7d-b27a5ffe5572"}
2023-04-24T11:26:06Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "956d45f0-6eb3-4b18-bf7d-b27a5ffe5572"}
2023-04-24T11:26:06Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "956d45f0-6eb3-4b18-bf7d-b27a5ffe5572"}
2023-04-24T11:26:06Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "956d45f0-6eb3-4b18-bf7d-b27a5ffe5572"}
2023-04-24T11:26:06Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "956d45f0-6eb3-4b18-bf7d-b27a5ffe5572", "pod": "health-test-3"}
2023-04-24T11:26:06Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "956d45f0-6eb3-4b18-bf7d-b27a5ffe5572", "pod": "health-test-3"}
2023-04-24T11:26:06Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "956d45f0-6eb3-4b18-bf7d-b27a5ffe5572"}
2023-04-24T11:26:06Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "956d45f0-6eb3-4b18-bf7d-b27a5ffe5572"}
2023-04-24T11:26:06Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "956d45f0-6eb3-4b18-bf7d-b27a5ffe5572", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T11:26:06Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "956d45f0-6eb3-4b18-bf7d-b27a5ffe5572", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T11:26:06Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-2","namespace":"default"}, "namespace": "default", "name": "health-test-2", "reconcileID": "956d45f0-6eb3-4b18-bf7d-b27a5ffe5572", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T11:26:09Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "31dbb8ec-f416-4f7f-9298-d67a103daac8", "pod": {"namespace": "default", "name": "health-test-3"}}
2023-04-24T11:26:09Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "31dbb8ec-f416-4f7f-9298-d67a103daac8", "pod": "default/health-test-3", "role": "replica"}
2023-04-24T11:26:09Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "31dbb8ec-f416-4f7f-9298-d67a103daac8"}
2023-04-24T11:26:09Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "31dbb8ec-f416-4f7f-9298-d67a103daac8"}
2023-04-24T11:26:09Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "31dbb8ec-f416-4f7f-9298-d67a103daac8"}
2023-04-24T11:26:09Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "31dbb8ec-f416-4f7f-9298-d67a103daac8"}
2023-04-24T11:26:09Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "31dbb8ec-f416-4f7f-9298-d67a103daac8", "pod": "health-test-3"}
2023-04-24T11:26:09Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "31dbb8ec-f416-4f7f-9298-d67a103daac8", "pod": "health-test-3"}
2023-04-24T11:26:09Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "31dbb8ec-f416-4f7f-9298-d67a103daac8"}
2023-04-24T11:26:09Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "31dbb8ec-f416-4f7f-9298-d67a103daac8"}
2023-04-24T11:26:09Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "31dbb8ec-f416-4f7f-9298-d67a103daac8", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T11:26:09Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "31dbb8ec-f416-4f7f-9298-d67a103daac8", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T11:26:09Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-3","namespace":"default"}, "namespace": "default", "name": "health-test-3", "reconcileID": "31dbb8ec-f416-4f7f-9298-d67a103daac8", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T11:26:17Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a4818a29-9fe2-400a-b664-991056e4f597", "pod": {"namespace": "default", "name": "health-test-0"}}
2023-04-24T11:26:17Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a4818a29-9fe2-400a-b664-991056e4f597", "pod": "default/health-test-0", "role": "master"}
2023-04-24T11:26:17Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a4818a29-9fe2-400a-b664-991056e4f597"}
2023-04-24T11:26:17Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a4818a29-9fe2-400a-b664-991056e4f597"}
2023-04-24T11:26:17Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a4818a29-9fe2-400a-b664-991056e4f597"}
2023-04-24T11:26:17Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a4818a29-9fe2-400a-b664-991056e4f597"}
2023-04-24T11:26:17Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a4818a29-9fe2-400a-b664-991056e4f597", "pod": "health-test-3"}
2023-04-24T11:26:17Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a4818a29-9fe2-400a-b664-991056e4f597", "pod": "health-test-3"}
2023-04-24T11:26:17Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a4818a29-9fe2-400a-b664-991056e4f597"}
2023-04-24T11:26:17Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a4818a29-9fe2-400a-b664-991056e4f597"}
2023-04-24T11:26:17Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a4818a29-9fe2-400a-b664-991056e4f597", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T11:26:17Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a4818a29-9fe2-400a-b664-991056e4f597", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T11:26:17Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-0","namespace":"default"}, "namespace": "default", "name": "health-test-0", "reconcileID": "a4818a29-9fe2-400a-b664-991056e4f597", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T11:26:19Z INFO Received {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "ba46d478-4dab-44a7-ba7a-859b4b087291", "pod": {"namespace": "default", "name": "health-test-1"}}
2023-04-24T11:26:19Z INFO Non-deletion event for a pod with an existing role. checking if something is wrong {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "ba46d478-4dab-44a7-ba7a-859b4b087291", "pod": "default/health-test-1", "role": "replica"}
2023-04-24T11:26:19Z INFO checking if all pods are configured correctly {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "ba46d478-4dab-44a7-ba7a-859b4b087291"}
2023-04-24T11:26:19Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "ba46d478-4dab-44a7-ba7a-859b4b087291"}
2023-04-24T11:26:19Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "ba46d478-4dab-44a7-ba7a-859b4b087291"}
2023-04-24T11:26:19Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "ba46d478-4dab-44a7-ba7a-859b4b087291"}
2023-04-24T11:26:19Z INFO configuring pod as replica to the right master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "ba46d478-4dab-44a7-ba7a-859b4b087291", "pod": "health-test-3"}
2023-04-24T11:26:19Z INFO configuring pod as replica {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "ba46d478-4dab-44a7-ba7a-859b4b087291", "pod": "health-test-3"}
2023-04-24T11:26:19Z INFO retrieving ip of the master {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "ba46d478-4dab-44a7-ba7a-859b4b087291"}
2023-04-24T11:26:19Z INFO getting all pods relevant to the instance {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "ba46d478-4dab-44a7-ba7a-859b4b087291"}
2023-04-24T11:26:19Z INFO Trying to invoke SLAVE OF command {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "ba46d478-4dab-44a7-ba7a-859b4b087291", "pod": "health-test-3", "master": "10.244.0.18"}
2023-04-24T11:26:19Z ERROR could not check and configure replication {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "ba46d478-4dab-44a7-ba7a-859b4b087291", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
github.com/dragonflydb/dragonfly-operator/internal/controller.(*DfPodLifeCycleReconciler).Reconcile
/workspace/internal/controller/dragonfly_pod_lifecycle_controller.go:151
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
2023-04-24T11:26:19Z ERROR Reconciler error {"controller": "pod", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"health-test-1","namespace":"default"}, "namespace": "default", "name": "health-test-1", "reconcileID": "ba46d478-4dab-44a7-ba7a-859b4b087291", "error": "Error running SLAVE OF command: ERR Can not execute during LOADING"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment