Environment details:
DC -> region-1
Host1 (10.186.10.56) -> zone-a. VMs: k8s-node-018, k8s-node-0888
Host2 (10.186.7.239) -> zone-b
Host3 (10.186.9.77) -> zone-c. VMs: k8s-master-964, k8s-node-0316
Datastore info:
vSAN DS:
ds:///vmfs/volumes/vsan:523aa97b16f3f102-4b52e46770c1d990/
Storage Policy: non-shared-ds-policy
Local VMFS DS:
10.186.10.56:
ds:///vmfs/volumes/60da3096-535e431d-f452-0200990825a9/
ds:///vmfs/volumes/60da3097-d38efce1-054f-0200990825a9/
10.186.9.77:
ds:///vmfs/volumes/60da309a-8c65c3e7-716b-02009999d220/
ds:///vmfs/volumes/60da309c-2ef62605-8d75-02009999d220/
10.186.7.239:
ds:///vmfs/volumes/60da3097-05196e9b-5457-020099fa6513/
ds:///vmfs/volumes/60da3098-95d9fddb-9005-020099fa6513/
Placement of pods in k8s cluster:
root@k8s-master-964:~# kc get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
vsphere-csi-controller-86dcc99d98-9qc69 6/6 Running 0 151m 10.244.0.6 k8s-master-964 <none> <none>
vsphere-csi-node-brnv9 3/3 Running 0 142m 10.244.2.5 k8s-node-018 <none> <none>
vsphere-csi-node-gjvtz 3/3 Running 0 143m 10.244.3.4 k8s-node-0888 <none> <none>
vsphere-csi-node-hmsjv 3/3 Running 0 142m 10.244.0.7 k8s-master-964 <none> <none>
vsphere-csi-node-skkz9 3/3 Running 0 143m 10.244.1.5 k8s-node-0316 <none> <none>
Topology labels applied on nodes:
root@k8s-master-964:~# kubectl get nodes -o wide -L topology.kubernetes.io/zone -L topology.kubernetes.io/region
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ZONE REGION
k8s-master-964 Ready control-plane,master 7d23h v1.21.1 10.186.4.100 10.186.4.100 Ubuntu 20.04.2 LTS 5.4.0-66-generic docker://20.10.5 zone-c region-1
k8s-node-018 Ready <none> 7d23h v1.21.1 10.186.9.247 10.186.9.247 Ubuntu 20.04.2 LTS 5.4.0-66-generic docker://20.10.5 zone-a region-1
k8s-node-0316 Ready <none> 7d23h v1.21.1 10.186.0.244 10.186.0.244 Ubuntu 20.04.2 LTS 5.4.0-66-generic docker://20.10.5 zone-c region-1
k8s-node-0888 Ready <none> 7d23h v1.21.1 10.186.2.189 10.186.2.189 Ubuntu 20.04.2 LTS 5.4.0-66-generic docker://20.10.5 zone-a region-1
Without strict-topology:
- No accessibility requirements and no WFFC:
Storage class definition:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: block-sc
provisioner: csi.vsphere.vmware.com
parameters:
storagepolicyname: "non-shared-ds-policy"
Logs:
AccessibilityRequirements:
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-c" > >
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > >
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-c" > >
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > >
2021-07-07T16:43:25.370Z INFO volume/util.go:324 Volume created successfully. VolumeName: "pvc-3a4f4ad8-15d9-402d-a758-1f124b2db248", volumeID: "98d952da-308b-487d-904d-2a49a4e3c9c3" {"TraceId": "2fe7d940-7576-45d5-adda-cd22a5b7eecb"}
2021-07-07T16:43:25.375Z DEBUG vanilla/controller.go:556 Volume: 98d952da-308b-487d-904d-2a49a4e3c9c3 is provisioned on the datastore: ds:///vmfs/volumes/60da309c-2ef62605-8d75-02009999d220/ {"TraceId": "2fe7d940-7576-45d5-adda-cd22a5b7eecb"}
2021-07-07T16:43:25.380Z DEBUG vanilla/controller.go:561 volumeAccessibleTopology: [map[topology.kubernetes.io/region:region-1 topology.kubernetes.io/zone:zone-c]] is selected for datastore: ds:///vmfs/volumes/60da309c-2ef62605-8d75-02009999d220/ {"TraceId": "2fe7d940-7576-45d5-adda-cd22a5b7eecb"}
PV describe output:
Name: pvc-3a4f4ad8-15d9-402d-a758-1f124b2db248
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers: [kubernetes.io/pv-protection]
StorageClass: block-sc-1
Status: Bound
Claim: default/block-pvc-1
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 100Mi
Node Affinity:
Required Terms:
Term 0: topology.kubernetes.io/region in [region-1]
topology.kubernetes.io/zone in [zone-c]
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: csi.vsphere.vmware.com
FSType: ext4
VolumeHandle: 98d952da-308b-487d-904d-2a49a4e3c9c3
ReadOnly: false
VolumeAttributes: storage.kubernetes.io/csiProvisionerIdentity=1625623664749-8081-csi.vsphere.vmware.com
type=vSphere CNS Block Volume
Events: <none>
- Single zone accessibility requirements and no WFFC:
Storage class definition:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: single-zone-sc
provisioner: csi.vsphere.vmware.com
parameters:
storagepolicyname: "non-shared-ds-policy"
allowedTopologies:
- matchLabelExpressions:
- key: topology.kubernetes.io/zone
values:
- zone-c
- key: topology.kubernetes.io/region
values:
- region-1
Logs:
AccessibilityRequirements:
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-c" > >
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-c" > >
2021-07-07T16:47:48.528Z INFO volume/util.go:324 Volume created successfully. VolumeName: "pvc-9eb6e528-84ce-4a0f-bf75-e1303c399eaf", volumeID: "a66cfc2c-9af6-4c8d-8e2c-a77018c8ec49" {"TraceId": "0a0ad652-2180-48b6-94c5-cd88786cfc64"}
2021-07-07T16:47:48.530Z DEBUG vanilla/controller.go:556 Volume: a66cfc2c-9af6-4c8d-8e2c-a77018c8ec49 is provisioned on the datastore: ds:///vmfs/volumes/60da309a-8c65c3e7-716b-02009999d220/ {"TraceId": "0a0ad652-2180-48b6-94c5-cd88786cfc64"}
2021-07-07T16:47:48.533Z DEBUG vanilla/controller.go:561 volumeAccessibleTopology: [map[topology.kubernetes.io/region:region-1 topology.kubernetes.io/zone:zone-c]] is selected for datastore: ds:///vmfs/volumes/60da309a-8c65c3e7-716b-02009999d220/ {"TraceId": "0a0ad652-2180-48b6-94c5-cd88786cfc64"}
PV describe output:
Name: pvc-9eb6e528-84ce-4a0f-bf75-e1303c399eaf
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers: [kubernetes.io/pv-protection]
StorageClass: single-zone-sc-1
Status: Bound
Claim: default/single-zone-pvc-1
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 100Mi
Node Affinity:
Required Terms:
Term 0: topology.kubernetes.io/zone in [zone-c]
topology.kubernetes.io/region in [region-1]
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: csi.vsphere.vmware.com
FSType: ext4
VolumeHandle: a66cfc2c-9af6-4c8d-8e2c-a77018c8ec49
ReadOnly: false
VolumeAttributes: storage.kubernetes.io/csiProvisionerIdentity=1625623664749-8081-csi.vsphere.vmware.com
type=vSphere CNS Block Volume
Events: <none>
- Multi-zone accessibility requirements and no WFFC:
Storage class definition:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: multi-zone-sc
provisioner: csi.vsphere.vmware.com
parameters:
storagepolicyname: "non-shared-ds-policy"
allowedTopologies:
- matchLabelExpressions:
- key: topology.kubernetes.io/zone
values:
- zone-a
- zone-b
- key: topology.kubernetes.io/region
values:
- region-1
Logs:
AccessibilityRequirements:
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-b" > >
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > >
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-b" > >
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > >
2021-07-07T16:48:48.047Z INFO volume/util.go:324 Volume created successfully. VolumeName: "pvc-721dc962-ad9f-47de-8a4f-f01fe60a8f5f", volumeID: "693e136e-7e45-41c4-bcce-bc6ec6b1c9de" {"TraceId": "2cf2edeb-7165-4c70-a167-2e2467a8087d"}
2021-07-07T16:48:48.049Z DEBUG vanilla/controller.go:556 Volume: 693e136e-7e45-41c4-bcce-bc6ec6b1c9de is provisioned on the datastore: ds:///vmfs/volumes/60da3097-d38efce1-054f-0200990825a9/ {"TraceId": "2cf2edeb-7165-4c70-a167-2e2467a8087d"}
2021-07-07T16:48:48.050Z DEBUG vanilla/controller.go:561 volumeAccessibleTopology: [map[topology.kubernetes.io/region:region-1 topology.kubernetes.io/zone:zone-a]] is selected for datastore: ds:///vmfs/volumes/60da3097-d38efce1-054f-0200990825a9/ {"TraceId": "2cf2edeb-7165-4c70-a167-2e2467a8087d"}
PV describe output:
Name: pvc-721dc962-ad9f-47de-8a4f-f01fe60a8f5f
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers: [kubernetes.io/pv-protection]
StorageClass: multi-zone-sc-1
Status: Bound
Claim: default/multi-zone-pvc-1
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 100Mi
Node Affinity:
Required Terms:
Term 0: topology.kubernetes.io/region in [region-1]
topology.kubernetes.io/zone in [zone-a]
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: csi.vsphere.vmware.com
FSType: ext4
VolumeHandle: 693e136e-7e45-41c4-bcce-bc6ec6b1c9de
ReadOnly: false
VolumeAttributes: storage.kubernetes.io/csiProvisionerIdentity=1625623664749-8081-csi.vsphere.vmware.com
type=vSphere CNS Block Volume
Events: <none>
- No accessibility requirements with WFFC:
Storage class definition:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: wffc-sc
provisioner: csi.vsphere.vmware.com
volumeBindingMode: WaitForFirstConsumer
parameters:
storagepolicyname: "non-shared-ds-policy"
Logs:
AccessibilityRequirements:
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-c" > >
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > >
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > >
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-c" > >
2021-07-07T17:00:14.768Z INFO volume/util.go:324 Volume created successfully. VolumeName: "pvc-c18838bd-eec0-4bb3-9dcf-a8888a051a3e", volumeID: "0a87d173-1847-4a22-a8f9-92c32bb2b30e" {"TraceId": "416c2f81-0df2-4f0e-b28a-22e1bc0b0f7f"}
2021-07-07T17:00:14.769Z DEBUG volume/util.go:326 CreateVolume volumeId {{} "0a87d173-1847-4a22-a8f9-92c32bb2b30e"} is placed on datastore "ds:///vmfs/volumes/60da309c-2ef62605-8d75-02009999d220/" {"TraceId": "416c2f81-0df2-4f0e-b28a-22e1bc0b0f7f"}
2021-07-07T17:00:14.769Z DEBUG vanilla/controller.go:556 Volume: 0a87d173-1847-4a22-a8f9-92c32bb2b30e is provisioned on the datastore: ds:///vmfs/volumes/60da309c-2ef62605-8d75-02009999d220/ {"TraceId": "416c2f81-0df2-4f0e-b28a-22e1bc0b0f7f"}
2021-07-07T17:00:14.770Z DEBUG vanilla/controller.go:561 volumeAccessibleTopology: [map[topology.kubernetes.io/region:region-1 topology.kubernetes.io/zone:zone-c]] is selected for datastore: ds:///vmfs/volumes/60da309c-2ef62605-8d75-02009999d220/ {"TraceId": "416c2f81-0df2-4f0e-b28a-22e1bc0b0f7f"}
2021-07-07T17:00:18.343Z INFO vanilla/controller.go:771 ControllerPublishVolume: called with args {VolumeId:0a87d173-1847-4a22-a8f9-92c32bb2b30e NodeId:k8s-node-0316 VolumeCapability:mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > Readonly:false Secrets:map[] VolumeContext:map[storage.kubernetes.io/csiProvisionerIdentity:1625623664749-8081-csi.vsphere.vmware.com type:vSphere CNS Block Volume] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {"TraceId": "0b8e4d73-f26d-4797-ad7a-e985072ecc3e"}
2021-07-07T17:00:22.598Z DEBUG common/vsphereutil.go:440 Successfully attached disk 0a87d173-1847-4a22-a8f9-92c32bb2b30e to VM VirtualMachine:vm-49 [VirtualCenterHost: 10.186.2.101, UUID: 423fe073-afc4-fb51-3dee-4423550e2211, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-3, VirtualCenterHost: 10.186.2.101]]. Disk UUID is 6000C29c-77f9-4212-db2a-82dee1faaf66 {"TraceId": "9174b9a0-b04a-4a32-8f33-f7d3c5cc5e9c"}
2021-07-07T17:00:22.598Z INFO vanilla/controller.go:858 ControllerPublishVolume successful with publish context: map[diskUUID:6000c29c77f94212db2a82dee1faaf66 type:vSphere CNS Block Volume] {"TraceId": "9174b9a0-b04a-4a32-8f33-f7d3c5cc5e9c"}
PV describe output:
Name: pvc-c18838bd-eec0-4bb3-9dcf-a8888a051a3e
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers: [kubernetes.io/pv-protection external-attacher/csi-vsphere-vmware-com]
StorageClass: wffc-sc-1
Status: Bound
Claim: default/wffc-pvc-1
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 100Mi
Node Affinity:
Required Terms:
Term 0: topology.kubernetes.io/zone in [zone-c]
topology.kubernetes.io/region in [region-1]
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: csi.vsphere.vmware.com
FSType: ext4
VolumeHandle: 0a87d173-1847-4a22-a8f9-92c32bb2b30e
ReadOnly: false
VolumeAttributes: storage.kubernetes.io/csiProvisionerIdentity=1625623664749-8081-csi.vsphere.vmware.com
type=vSphere CNS Block Volume
Events: <none>
Pod placement:
root@k8s-master-964:~# kp wffc-pod-1 -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
wffc-pod-1 1/1 Running 0 19m 10.244.1.14 k8s-node-0316 <none> <none>
2nd attempt:
Logs:
AccessibilityRequirements:
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-c" > >
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > >
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > >
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-c" > >
2021-07-07T17:52:58.182Z INFO volume/util.go:324 Volume created successfully. VolumeName: "pvc-34366c6e-b84f-46d9-bdd9-95d87b0efc3c", volumeID: "268f78ef-2f33-41fb-b3ff-62f4be05a367" {"TraceId": "0b0944cc-49cd-4794-bf0f-d13a60421728"}
2021-07-07T17:52:58.184Z DEBUG vanilla/controller.go:556 Volume: 268f78ef-2f33-41fb-b3ff-62f4be05a367 is provisioned on the datastore: ds:///vmfs/volumes/60da3096-535e431d-f452-0200990825a9/ {"TraceId": "0b0944cc-49cd-4794-bf0f-d13a60421728"}
2021-07-07T17:52:58.184Z DEBUG vanilla/controller.go:561 volumeAccessibleTopology: [map[topology.kubernetes.io/region:region-1 topology.kubernetes.io/zone:zone-a]] is selected for datastore: ds:///vmfs/volumes/60da3096-535e431d-f452-0200990825a9/ {"TraceId": "0b0944cc-49cd-4794-bf0f-d13a60421728"}
2021-07-07T17:52:59.008Z INFO vanilla/controller.go:771 ControllerPublishVolume: called with args {VolumeId:268f78ef-2f33-41fb-b3ff-62f4be05a367 NodeId:k8s-node-018 VolumeCapability:mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > Readonly:false Secrets:map[] VolumeContext:map[storage.kubernetes.io/csiProvisionerIdentity:1625623664749-8081-csi.vsphere.vmware.com type:vSphere CNS Block Volume] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {"TraceId": "76e7137f-8bf4-4c9f-89db-03dfb98aa237"}
2021-07-07T17:53:01.149Z DEBUG common/vsphereutil.go:440 Successfully attached disk 268f78ef-2f33-41fb-b3ff-62f4be05a367 to VM VirtualMachine:vm-50 [VirtualCenterHost: 10.186.2.101, UUID: 423fe8b9-c453-ef3b-df25-79218df9ca24, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-3, VirtualCenterHost: 10.186.2.101]]. Disk UUID is 6000C294-4eae-8bad-9519-ba0b537c4609 {"TraceId": "76e7137f-8bf4-4c9f-89db-03dfb98aa237"}
2021-07-07T17:53:01.152Z INFO vanilla/controller.go:858 ControllerPublishVolume successful with publish context: map[diskUUID:6000c2944eae8bad9519ba0b537c4609 type:vSphere CNS Block Volume] {"TraceId": "76e7137f-8bf4-4c9f-89db-03dfb98aa237"}
PV describe output:
Name: pvc-34366c6e-b84f-46d9-bdd9-95d87b0efc3c
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers: [kubernetes.io/pv-protection external-attacher/csi-vsphere-vmware-com]
StorageClass: wffc-sc-1
Status: Bound
Claim: default/wffc-pvc-1
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 100Mi
Node Affinity:
Required Terms:
Term 0: topology.kubernetes.io/zone in [zone-a]
topology.kubernetes.io/region in [region-1]
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: csi.vsphere.vmware.com
FSType: ext4
VolumeHandle: 268f78ef-2f33-41fb-b3ff-62f4be05a367
ReadOnly: false
VolumeAttributes: storage.kubernetes.io/csiProvisionerIdentity=1625623664749-8081-csi.vsphere.vmware.com
type=vSphere CNS Block Volume
Events: <none>
Pod placement:
root@k8s-master-964:~# kp wffc-pod-1 -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
wffc-pod-1 1/1 Running 0 68s 10.244.2.9 k8s-node-018 <none> <none>
On further analysis, I found that the pod gets rescheduled as the selected node (which is the first segment in the preferred accessibility requirements) does not match the PV node affinity required rules. The pod describe output in such cases looks like: (k8s code ref: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/scheduling/scheduler_binder.go#L678 )
root@k8s-master-964:~# kdsc pod wffc-pod-4
Name: wffc-pod-4
Namespace: default
Priority: 0
Node: k8s-node-0316/10.186.0.244
Start Time: Mon, 12 Jul 2021 19:01:04 +0000
Labels: <none>
Annotations: <none>
Status: Running
IP: 10.244.1.16
IPs:
IP: 10.244.1.16
Containers:
test-container:
Container ID: docker://5909f88a2ea0b0d978bffdb0caa27fe793358bd16d9e7b420dd8bbf4545cfbbc
Image: gcr.io/google_containers/busybox:1.24
Image ID: docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff
Port: <none>
Host Port: <none>
Command:
/bin/sh
-c
echo 'hello' > /mnt/volume1/index.html && chmod o+rX /mnt /mnt/volume1/index.html && while true ; do sleep 2 ; done
State: Running
Started: Mon, 12 Jul 2021 19:01:35 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/mnt/volume1 from test-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-54pgb (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
test-volume:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: wffc-pvc-3
ReadOnly: false
kube-api-access-54pgb:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 15m default-scheduler running PreBind plugin "VolumeBinding": binding volumes: pv "pvc-b2c43dce-2502-45e8-b371-fefe3d7d3c71" node affinity doesn't match node "k8s-node-0888": no matching NodeSelectorTerms
Normal Scheduled 15m default-scheduler Successfully assigned default/wffc-pod-4 to k8s-node-0316
Normal SuccessfulAttachVolume 15m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-b2c43dce-2502-45e8-b371-fefe3d7d3c71"
Normal Pulled 15m kubelet Container image "gcr.io/google_containers/busybox:1.24" already present on machine
Normal Created 15m kubelet Created container test-container
Normal Started 15m kubelet Started container test-container
- Single zone accessibility requirements with WFFC:
Storage class definition:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: single-zone-wffc-sc
provisioner: csi.vsphere.vmware.com
volumeBindingMode: WaitForFirstConsumer
parameters:
storagepolicyname: "non-shared-ds-policy"
allowedTopologies:
- matchLabelExpressions:
- key: topology.kubernetes.io/zone
values:
- zone-a
- key: topology.kubernetes.io/region
values:
- region-1
Logs:
AccessibilityRequirements:
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > >
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > >
2021-07-07T16:58:26.497Z INFO volume/util.go:324 Volume created successfully. VolumeName: "pvc-11235cc3-ec32-4099-842b-3189274b3873", volumeID: "2994bfc4-b841-4b39-8d36-00ee9ed04931" {"TraceId": "8e43706a-8ca0-43f4-8b06-8adf5bde4408"}
2021-07-07T16:58:26.497Z DEBUG volume/util.go:326 CreateVolume volumeId {{} "2994bfc4-b841-4b39-8d36-00ee9ed04931"} is placed on datastore "ds:///vmfs/volumes/60da3097-d38efce1-054f-0200990825a9/" {"TraceId": "8e43706a-8ca0-43f4-8b06-8adf5bde4408"}
2021-07-07T16:58:26.499Z DEBUG vanilla/controller.go:556 Volume: 2994bfc4-b841-4b39-8d36-00ee9ed04931 is provisioned on the datastore: ds:///vmfs/volumes/60da3097-d38efce1-054f-0200990825a9/ {"TraceId": "8e43706a-8ca0-43f4-8b06-8adf5bde4408"}
2021-07-07T16:58:26.499Z DEBUG vanilla/controller.go:561 volumeAccessibleTopology: [map[topology.kubernetes.io/region:region-1 topology.kubernetes.io/zone:zone-a]] is selected for datastore: ds:///vmfs/volumes/60da3097-d38efce1-054f-0200990825a9/ {"TraceId": "8e43706a-8ca0-43f4-8b06-8adf5bde4408"}
2021-07-07T16:58:27.603Z INFO vanilla/controller.go:771 ControllerPublishVolume: called with args {VolumeId:2994bfc4-b841-4b39-8d36-00ee9ed04931 NodeId:k8s-node-0888 VolumeCapability:mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > Readonly:false Secrets:map[] VolumeContext:map[storage.kubernetes.io/csiProvisionerIdentity:1625623664749-8081-csi.vsphere.vmware.com type:vSphere CNS Block Volume] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {"TraceId": "ce108514-e759-469b-b897-8aa9245dd0db"}
2021-07-07T16:58:29.005Z DEBUG common/vsphereutil.go:440 Successfully attached disk 2994bfc4-b841-4b39-8d36-00ee9ed04931 to VM VirtualMachine:vm-51 [VirtualCenterHost: 10.186.2.101, UUID: 423f474d-bf84-6d1a-a6b1-f3b45e505592, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-3, VirtualCenterHost: 10.186.2.101]]. Disk UUID is 6000C298-7f84-4c50-80b7-fab336f60f42 {"TraceId": "ce108514-e759-469b-b897-8aa9245dd0db"}
2021-07-07T16:58:29.005Z INFO vanilla/controller.go:858 ControllerPublishVolume successful with publish context: map[diskUUID:6000c2987f844c5080b7fab336f60f42 type:vSphere CNS Block Volume] {"TraceId": "ce108514-e759-469b-b897-8aa9245dd0db"}
PV describe output:
Name: pvc-11235cc3-ec32-4099-842b-3189274b3873
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers: [kubernetes.io/pv-protection external-attacher/csi-vsphere-vmware-com]
StorageClass: single-zone-wffc-sc-1
Status: Bound
Claim: default/single-zone-wffc-pvc-1
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 100Mi
Node Affinity:
Required Terms:
Term 0: topology.kubernetes.io/region in [region-1]
topology.kubernetes.io/zone in [zone-a]
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: csi.vsphere.vmware.com
FSType: ext4
VolumeHandle: 2994bfc4-b841-4b39-8d36-00ee9ed04931
ReadOnly: false
VolumeAttributes: storage.kubernetes.io/csiProvisionerIdentity=1625623664749-8081-csi.vsphere.vmware.com
type=vSphere CNS Block Volume
Events: <none>
- Multi-zone accessibility requirements with WFFC:
Storage class definition:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: multi-zone-wffc-sc
provisioner: csi.vsphere.vmware.com
volumeBindingMode: WaitForFirstConsumer
parameters:
storagepolicyname: "non-shared-ds-policy"
allowedTopologies:
- matchLabelExpressions:
- key: topology.kubernetes.io/zone
values:
- zone-a
- zone-b
- key: topology.kubernetes.io/region
values:
- region-1
Logs:
AccessibilityRequirements:
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > >
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-b" > >
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > >
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-b" > >
2021-07-07T16:56:42.245Z INFO volume/util.go:324 Volume created successfully. VolumeName: "pvc-10a938a6-fb51-4446-8350-5210e753e874", volumeID: "d9b30614-7149-4c10-bc1a-bdd5618cce17" {"TraceId": "f2928d89-edbc-48d3-b834-5bd520ab0a67"}
2021-07-07T16:56:42.246Z DEBUG vanilla/controller.go:556 Volume: d9b30614-7149-4c10-bc1a-bdd5618cce17 is provisioned on the datastore: ds:///vmfs/volumes/60da3096-535e431d-f452-0200990825a9/ {"TraceId": "f2928d89-edbc-48d3-b834-5bd520ab0a67"}
2021-07-07T16:56:42.247Z DEBUG vanilla/controller.go:561 volumeAccessibleTopology: [map[topology.kubernetes.io/region:region-1 topology.kubernetes.io/zone:zone-a]] is selected for datastore: ds:///vmfs/volumes/60da3096-535e431d-f452-0200990825a9/ {"TraceId": "f2928d89-edbc-48d3-b834-5bd520ab0a67"}
2021-07-07T16:56:43.456Z INFO vanilla/controller.go:771 ControllerPublishVolume: called with args {VolumeId:d9b30614-7149-4c10-bc1a-bdd5618cce17 NodeId:k8s-node-018 VolumeCapability:mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > Readonly:false Secrets:map[] VolumeContext:map[storage.kubernetes.io/csiProvisionerIdentity:1625623664749-8081-csi.vsphere.vmware.com type:vSphere CNS Block Volume] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0} {"TraceId": "8c6894cd-b031-4ffc-a56e-2f11dafd7d29"}
2021-07-07T16:56:45.129Z DEBUG common/vsphereutil.go:440 Successfully attached disk d9b30614-7149-4c10-bc1a-bdd5618cce17 to VM VirtualMachine:vm-50 [VirtualCenterHost: 10.186.2.101, UUID: 423fe8b9-c453-ef3b-df25-79218df9ca24, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-3, VirtualCenterHost: 10.186.2.101]]. Disk UUID is 6000C29a-7baa-c28f-9379-33e532671b95 {"TraceId": "8c6894cd-b031-4ffc-a56e-2f11dafd7d29"}
2021-07-07T16:56:45.130Z INFO vanilla/controller.go:858 ControllerPublishVolume successful with publish context: map[diskUUID:6000c29a7baac28f937933e532671b95 type:vSphere CNS Block Volume] {"TraceId": "8c6894cd-b031-4ffc-a56e-2f11dafd7d29"}
PV describe output:
Name: pvc-10a938a6-fb51-4446-8350-5210e753e874
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers: [kubernetes.io/pv-protection external-attacher/csi-vsphere-vmware-com]
StorageClass: multi-zone-wffc-sc-1
Status: Bound
Claim: default/multi-zone-wffc-pvc-1
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 100Mi
Node Affinity:
Required Terms:
Term 0: topology.kubernetes.io/zone in [zone-a]
topology.kubernetes.io/region in [region-1]
Message:
Source:
Type: CSI (a Container Storage Interface (CSI) volume source)
Driver: csi.vsphere.vmware.com
FSType: ext4
VolumeHandle: d9b30614-7149-4c10-bc1a-bdd5618cce17
ReadOnly: false
VolumeAttributes: storage.kubernetes.io/csiProvisionerIdentity=1625623664749-8081-csi.vsphere.vmware.com
type=vSphere CNS Block Volume
Events: <none>
Final State:
root@k8s-master-964:~# kp -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
block-pod 1/1 Running 0 16h 10.244.1.12 k8s-node-0316 <none> <none>
multi-zone-pod 1/1 Running 0 16h 10.244.3.7 k8s-node-0888 <none> <none>
multi-zone-wffc-pod 1/1 Running 0 21h 10.244.3.5 k8s-node-0888 <none> <none>
multi-zone-wffc-pod-1 1/1 Running 0 23m 10.244.2.8 k8s-node-018 <none> <none>
single-zone-pod 1/1 Running 0 16h 10.244.1.13 k8s-node-0316 <none> <none>
single-zone-wffc-pod 1/1 Running 0 21h 10.244.2.6 k8s-node-018 <none> <none>
single-zone-wffc-pod-1 1/1 Running 0 21m 10.244.3.8 k8s-node-0888 <none> <none>
wffc-pod 1/1 Running 0 21h 10.244.1.6 k8s-node-0316 <none> <none>
wffc-pod-1 1/1 Running 0 19m 10.244.1.14 k8s-node-0316 <none> <none>
Case 4: kube-scheduler reschedules pod due to https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/scheduling/scheduler_binder.go#L678 (failure in matching with PV node affinity)