Overview.
-
If stateful app(/Assuing Storage NAS) running on one node.
- In case of network failure with Node, Pod will process the data. But K8s will rechedule the pod to new instance.
- In such cases data corrucption can happens!.
In case of failure on Node.
- Notice that we can’t unmount the volume from the failed node
- Use the Storage api to force detach the volume
- Now the volume is available – we are able to attach and mount it to a healthy node https://portworx.com/understand-resolve-warning-failed-attach-volume-warning-failed-mount-errors-kubernetes-azure/
Or - replicate the data on both disk (DRDB) - simply fail the failed node. - failover is smooth in terms of data switch.
Or - Use a Controller (Operator), Which decide the logic of Master and standby. - The switch should be done by operator, rather than the nodes with it.