Skip to content

Instantly share code, notes, and snippets.

@shalini-b

shalini-b/output-1.md

Last active July 23, 2021 23:11
Show Gist options
  • Save shalini-b/46dec5f1eb215a57885416246be76134 to your computer and use it in GitHub Desktop.
Save shalini-b/46dec5f1eb215a57885416246be76134 to your computer and use it in GitHub Desktop.
Download ZIP
CreateVolume REQ/REP with topology, WFFC and not strict-topology
Raw
output-1.md

Environment details:

DC -> region-1
Host1 (10.186.10.56) -> zone-a. VMs: k8s-node-018, k8s-node-0888
Host2 (10.186.7.239) -> zone-b
Host3 (10.186.9.77) -> zone-c. VMs: k8s-master-964, k8s-node-0316

Datastore info:

vSAN DS:
ds:///vmfs/volumes/vsan:523aa97b16f3f102-4b52e46770c1d990/

Storage Policy: non-shared-ds-policy
Local VMFS DS:
10.186.10.56:
ds:///vmfs/volumes/60da3096-535e431d-f452-0200990825a9/
ds:///vmfs/volumes/60da3097-d38efce1-054f-0200990825a9/
10.186.9.77:
ds:///vmfs/volumes/60da309a-8c65c3e7-716b-02009999d220/
ds:///vmfs/volumes/60da309c-2ef62605-8d75-02009999d220/
10.186.7.239:
ds:///vmfs/volumes/60da3097-05196e9b-5457-020099fa6513/
ds:///vmfs/volumes/60da3098-95d9fddb-9005-020099fa6513/

Placement of pods in k8s cluster:

root@k8s-master-964:~# kc get pods -o wide
NAME                                      READY   STATUS    RESTARTS   AGE    IP           NODE             NOMINATED NODE   READINESS GATES
vsphere-csi-controller-86dcc99d98-9qc69   6/6     Running   0          151m   10.244.0.6   k8s-master-964   <none>           <none>
vsphere-csi-node-brnv9                    3/3     Running   0          142m   10.244.2.5   k8s-node-018     <none>           <none>
vsphere-csi-node-gjvtz                    3/3     Running   0          143m   10.244.3.4   k8s-node-0888    <none>           <none>
vsphere-csi-node-hmsjv                    3/3     Running   0          142m   10.244.0.7   k8s-master-964   <none>           <none>
vsphere-csi-node-skkz9                    3/3     Running   0          143m   10.244.1.5   k8s-node-0316    <none>           <none>

Topology labels applied on nodes:

root@k8s-master-964:~# kubectl get nodes -o wide -L topology.kubernetes.io/zone -L topology.kubernetes.io/region
NAME             STATUS   ROLES                  AGE     VERSION   INTERNAL-IP    EXTERNAL-IP    OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME   ZONE     REGION
k8s-master-964   Ready    control-plane,master   7d23h   v1.21.1   10.186.4.100   10.186.4.100   Ubuntu 20.04.2 LTS   5.4.0-66-generic   docker://20.10.5    zone-c   region-1
k8s-node-018     Ready    <none>                 7d23h   v1.21.1   10.186.9.247   10.186.9.247   Ubuntu 20.04.2 LTS   5.4.0-66-generic   docker://20.10.5    zone-a   region-1
k8s-node-0316    Ready    <none>                 7d23h   v1.21.1   10.186.0.244   10.186.0.244   Ubuntu 20.04.2 LTS   5.4.0-66-generic   docker://20.10.5    zone-c   region-1
k8s-node-0888    Ready    <none>                 7d23h   v1.21.1   10.186.2.189   10.186.2.189   Ubuntu 20.04.2 LTS   5.4.0-66-generic   docker://20.10.5    zone-a   region-1

Without strict-topology:

  1. No accessibility requirements and no WFFC:

Storage class definition:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: block-sc
provisioner: csi.vsphere.vmware.com
parameters:
  storagepolicyname: "non-shared-ds-policy"

Logs:

AccessibilityRequirements:
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-c" > > 
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > > 
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-c" > > 
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > > 

2021-07-07T16:43:25.370Z	INFO	volume/util.go:324	Volume created successfully. VolumeName: "pvc-3a4f4ad8-15d9-402d-a758-1f124b2db248", volumeID: "98d952da-308b-487d-904d-2a49a4e3c9c3"	{"TraceId": "2fe7d940-7576-45d5-adda-cd22a5b7eecb"}
2021-07-07T16:43:25.375Z	DEBUG	vanilla/controller.go:556	Volume: 98d952da-308b-487d-904d-2a49a4e3c9c3 is provisioned on the datastore: ds:///vmfs/volumes/60da309c-2ef62605-8d75-02009999d220/ 	{"TraceId": "2fe7d940-7576-45d5-adda-cd22a5b7eecb"}
2021-07-07T16:43:25.380Z	DEBUG	vanilla/controller.go:561	volumeAccessibleTopology: [map[topology.kubernetes.io/region:region-1 topology.kubernetes.io/zone:zone-c]] is selected for datastore: ds:///vmfs/volumes/60da309c-2ef62605-8d75-02009999d220/ 	{"TraceId": "2fe7d940-7576-45d5-adda-cd22a5b7eecb"}

PV describe output:

Name:              pvc-3a4f4ad8-15d9-402d-a758-1f124b2db248
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      block-sc-1
Status:            Bound
Claim:             default/block-pvc-1
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          100Mi
Node Affinity:     
  Required Terms:  
    Term 0:        topology.kubernetes.io/region in [region-1]
                   topology.kubernetes.io/zone in [zone-c]
Message:           
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            csi.vsphere.vmware.com
    FSType:            ext4
    VolumeHandle:      98d952da-308b-487d-904d-2a49a4e3c9c3
    ReadOnly:          false
    VolumeAttributes:      storage.kubernetes.io/csiProvisionerIdentity=1625623664749-8081-csi.vsphere.vmware.com
                           type=vSphere CNS Block Volume
Events:                <none>
  1. Single zone accessibility requirements and no WFFC:

Storage class definition:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: single-zone-sc
provisioner: csi.vsphere.vmware.com
parameters:
  storagepolicyname: "non-shared-ds-policy"
allowedTopologies:
  - matchLabelExpressions:
      - key: topology.kubernetes.io/zone
        values:
          - zone-c
      - key: topology.kubernetes.io/region
        values:
          - region-1

Logs:

AccessibilityRequirements:
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-c" > > 
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-c" > >

2021-07-07T16:47:48.528Z	INFO	volume/util.go:324	Volume created successfully. VolumeName: "pvc-9eb6e528-84ce-4a0f-bf75-e1303c399eaf", volumeID: "a66cfc2c-9af6-4c8d-8e2c-a77018c8ec49"	{"TraceId": "0a0ad652-2180-48b6-94c5-cd88786cfc64"}
2021-07-07T16:47:48.530Z	DEBUG	vanilla/controller.go:556	Volume: a66cfc2c-9af6-4c8d-8e2c-a77018c8ec49 is provisioned on the datastore: ds:///vmfs/volumes/60da309a-8c65c3e7-716b-02009999d220/ 	{"TraceId": "0a0ad652-2180-48b6-94c5-cd88786cfc64"}
2021-07-07T16:47:48.533Z	DEBUG	vanilla/controller.go:561	volumeAccessibleTopology: [map[topology.kubernetes.io/region:region-1 topology.kubernetes.io/zone:zone-c]] is selected for datastore: ds:///vmfs/volumes/60da309a-8c65c3e7-716b-02009999d220/ 	{"TraceId": "0a0ad652-2180-48b6-94c5-cd88786cfc64"}

PV describe output:

Name:              pvc-9eb6e528-84ce-4a0f-bf75-e1303c399eaf
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      single-zone-sc-1
Status:            Bound
Claim:             default/single-zone-pvc-1
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          100Mi
Node Affinity:     
  Required Terms:  
    Term 0:        topology.kubernetes.io/zone in [zone-c]
                   topology.kubernetes.io/region in [region-1]
Message:           
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            csi.vsphere.vmware.com
    FSType:            ext4
    VolumeHandle:      a66cfc2c-9af6-4c8d-8e2c-a77018c8ec49
    ReadOnly:          false
    VolumeAttributes:      storage.kubernetes.io/csiProvisionerIdentity=1625623664749-8081-csi.vsphere.vmware.com
                           type=vSphere CNS Block Volume
Events:                <none>
  1. Multi-zone accessibility requirements and no WFFC:

Storage class definition:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: multi-zone-sc
provisioner: csi.vsphere.vmware.com
parameters:
  storagepolicyname: "non-shared-ds-policy"
allowedTopologies:
  - matchLabelExpressions:
      - key: topology.kubernetes.io/zone
        values:
          - zone-a
          - zone-b
      - key: topology.kubernetes.io/region
        values:
          - region-1

Logs:

AccessibilityRequirements:
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-b" > > 
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > > 
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-b" > > 
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > >

2021-07-07T16:48:48.047Z	INFO	volume/util.go:324	Volume created successfully. VolumeName: "pvc-721dc962-ad9f-47de-8a4f-f01fe60a8f5f", volumeID: "693e136e-7e45-41c4-bcce-bc6ec6b1c9de"	{"TraceId": "2cf2edeb-7165-4c70-a167-2e2467a8087d"}
2021-07-07T16:48:48.049Z	DEBUG	vanilla/controller.go:556	Volume: 693e136e-7e45-41c4-bcce-bc6ec6b1c9de is provisioned on the datastore: ds:///vmfs/volumes/60da3097-d38efce1-054f-0200990825a9/ 	{"TraceId": "2cf2edeb-7165-4c70-a167-2e2467a8087d"}
2021-07-07T16:48:48.050Z	DEBUG	vanilla/controller.go:561	volumeAccessibleTopology: [map[topology.kubernetes.io/region:region-1 topology.kubernetes.io/zone:zone-a]] is selected for datastore: ds:///vmfs/volumes/60da3097-d38efce1-054f-0200990825a9/ 	{"TraceId": "2cf2edeb-7165-4c70-a167-2e2467a8087d"}

PV describe output:

Name:              pvc-721dc962-ad9f-47de-8a4f-f01fe60a8f5f
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers:        [kubernetes.io/pv-protection]
StorageClass:      multi-zone-sc-1
Status:            Bound
Claim:             default/multi-zone-pvc-1
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          100Mi
Node Affinity:     
  Required Terms:  
    Term 0:        topology.kubernetes.io/region in [region-1]
                   topology.kubernetes.io/zone in [zone-a]
Message:           
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            csi.vsphere.vmware.com
    FSType:            ext4
    VolumeHandle:      693e136e-7e45-41c4-bcce-bc6ec6b1c9de
    ReadOnly:          false
    VolumeAttributes:      storage.kubernetes.io/csiProvisionerIdentity=1625623664749-8081-csi.vsphere.vmware.com
                           type=vSphere CNS Block Volume
Events:                <none>
  1. No accessibility requirements with WFFC:

Storage class definition:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: wffc-sc
provisioner: csi.vsphere.vmware.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  storagepolicyname: "non-shared-ds-policy"

Logs:

AccessibilityRequirements:
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-c" > > 
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > > 
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > > 
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-c" > > 

2021-07-07T17:00:14.768Z	INFO	volume/util.go:324	Volume created successfully. VolumeName: "pvc-c18838bd-eec0-4bb3-9dcf-a8888a051a3e", volumeID: "0a87d173-1847-4a22-a8f9-92c32bb2b30e"	{"TraceId": "416c2f81-0df2-4f0e-b28a-22e1bc0b0f7f"}
2021-07-07T17:00:14.769Z	DEBUG	volume/util.go:326	CreateVolume volumeId {{} "0a87d173-1847-4a22-a8f9-92c32bb2b30e"} is placed on datastore "ds:///vmfs/volumes/60da309c-2ef62605-8d75-02009999d220/"	{"TraceId": "416c2f81-0df2-4f0e-b28a-22e1bc0b0f7f"}
2021-07-07T17:00:14.769Z	DEBUG	vanilla/controller.go:556	Volume: 0a87d173-1847-4a22-a8f9-92c32bb2b30e is provisioned on the datastore: ds:///vmfs/volumes/60da309c-2ef62605-8d75-02009999d220/ 	{"TraceId": "416c2f81-0df2-4f0e-b28a-22e1bc0b0f7f"}
2021-07-07T17:00:14.770Z	DEBUG	vanilla/controller.go:561	volumeAccessibleTopology: [map[topology.kubernetes.io/region:region-1 topology.kubernetes.io/zone:zone-c]] is selected for datastore: ds:///vmfs/volumes/60da309c-2ef62605-8d75-02009999d220/ 	{"TraceId": "416c2f81-0df2-4f0e-b28a-22e1bc0b0f7f"}
2021-07-07T17:00:18.343Z	INFO	vanilla/controller.go:771	ControllerPublishVolume: called with args {VolumeId:0a87d173-1847-4a22-a8f9-92c32bb2b30e NodeId:k8s-node-0316 VolumeCapability:mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER >  Readonly:false Secrets:map[] VolumeContext:map[storage.kubernetes.io/csiProvisionerIdentity:1625623664749-8081-csi.vsphere.vmware.com type:vSphere CNS Block Volume] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}	{"TraceId": "0b8e4d73-f26d-4797-ad7a-e985072ecc3e"}
2021-07-07T17:00:22.598Z	DEBUG	common/vsphereutil.go:440	Successfully attached disk 0a87d173-1847-4a22-a8f9-92c32bb2b30e to VM VirtualMachine:vm-49 [VirtualCenterHost: 10.186.2.101, UUID: 423fe073-afc4-fb51-3dee-4423550e2211, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-3, VirtualCenterHost: 10.186.2.101]]. Disk UUID is 6000C29c-77f9-4212-db2a-82dee1faaf66	{"TraceId": "9174b9a0-b04a-4a32-8f33-f7d3c5cc5e9c"}
2021-07-07T17:00:22.598Z	INFO	vanilla/controller.go:858	ControllerPublishVolume successful with publish context: map[diskUUID:6000c29c77f94212db2a82dee1faaf66 type:vSphere CNS Block Volume]	{"TraceId": "9174b9a0-b04a-4a32-8f33-f7d3c5cc5e9c"}

PV describe output:

Name:              pvc-c18838bd-eec0-4bb3-9dcf-a8888a051a3e
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers:        [kubernetes.io/pv-protection external-attacher/csi-vsphere-vmware-com]
StorageClass:      wffc-sc-1
Status:            Bound
Claim:             default/wffc-pvc-1
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          100Mi
Node Affinity:     
  Required Terms:  
    Term 0:        topology.kubernetes.io/zone in [zone-c]
                   topology.kubernetes.io/region in [region-1]
Message:           
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            csi.vsphere.vmware.com
    FSType:            ext4
    VolumeHandle:      0a87d173-1847-4a22-a8f9-92c32bb2b30e
    ReadOnly:          false
    VolumeAttributes:      storage.kubernetes.io/csiProvisionerIdentity=1625623664749-8081-csi.vsphere.vmware.com
                           type=vSphere CNS Block Volume
Events:                <none>

Pod placement:

root@k8s-master-964:~# kp wffc-pod-1 -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE            NOMINATED NODE   READINESS GATES
wffc-pod-1               1/1     Running   0          19m   10.244.1.14   k8s-node-0316   <none>           <none>

2nd attempt:

Logs:

AccessibilityRequirements:
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-c" > > 
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > > 
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > > 
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-c" > >

2021-07-07T17:52:58.182Z	INFO	volume/util.go:324	Volume created successfully. VolumeName: "pvc-34366c6e-b84f-46d9-bdd9-95d87b0efc3c", volumeID: "268f78ef-2f33-41fb-b3ff-62f4be05a367"	{"TraceId": "0b0944cc-49cd-4794-bf0f-d13a60421728"}
2021-07-07T17:52:58.184Z	DEBUG	vanilla/controller.go:556	Volume: 268f78ef-2f33-41fb-b3ff-62f4be05a367 is provisioned on the datastore: ds:///vmfs/volumes/60da3096-535e431d-f452-0200990825a9/ 	{"TraceId": "0b0944cc-49cd-4794-bf0f-d13a60421728"}
2021-07-07T17:52:58.184Z	DEBUG	vanilla/controller.go:561	volumeAccessibleTopology: [map[topology.kubernetes.io/region:region-1 topology.kubernetes.io/zone:zone-a]] is selected for datastore: ds:///vmfs/volumes/60da3096-535e431d-f452-0200990825a9/ 	{"TraceId": "0b0944cc-49cd-4794-bf0f-d13a60421728"}
2021-07-07T17:52:59.008Z	INFO	vanilla/controller.go:771	ControllerPublishVolume: called with args {VolumeId:268f78ef-2f33-41fb-b3ff-62f4be05a367 NodeId:k8s-node-018 VolumeCapability:mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER >  Readonly:false Secrets:map[] VolumeContext:map[storage.kubernetes.io/csiProvisionerIdentity:1625623664749-8081-csi.vsphere.vmware.com type:vSphere CNS Block Volume] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}	{"TraceId": "76e7137f-8bf4-4c9f-89db-03dfb98aa237"}
2021-07-07T17:53:01.149Z	DEBUG	common/vsphereutil.go:440	Successfully attached disk 268f78ef-2f33-41fb-b3ff-62f4be05a367 to VM VirtualMachine:vm-50 [VirtualCenterHost: 10.186.2.101, UUID: 423fe8b9-c453-ef3b-df25-79218df9ca24, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-3, VirtualCenterHost: 10.186.2.101]]. Disk UUID is 6000C294-4eae-8bad-9519-ba0b537c4609	{"TraceId": "76e7137f-8bf4-4c9f-89db-03dfb98aa237"}
2021-07-07T17:53:01.152Z	INFO	vanilla/controller.go:858	ControllerPublishVolume successful with publish context: map[diskUUID:6000c2944eae8bad9519ba0b537c4609 type:vSphere CNS Block Volume]	{"TraceId": "76e7137f-8bf4-4c9f-89db-03dfb98aa237"}

PV describe output:

Name:              pvc-34366c6e-b84f-46d9-bdd9-95d87b0efc3c
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers:        [kubernetes.io/pv-protection external-attacher/csi-vsphere-vmware-com]
StorageClass:      wffc-sc-1
Status:            Bound
Claim:             default/wffc-pvc-1
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          100Mi
Node Affinity:     
  Required Terms:  
    Term 0:        topology.kubernetes.io/zone in [zone-a]
                   topology.kubernetes.io/region in [region-1]
Message:           
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            csi.vsphere.vmware.com
    FSType:            ext4
    VolumeHandle:      268f78ef-2f33-41fb-b3ff-62f4be05a367
    ReadOnly:          false
    VolumeAttributes:      storage.kubernetes.io/csiProvisionerIdentity=1625623664749-8081-csi.vsphere.vmware.com
                           type=vSphere CNS Block Volume
Events:                <none>

Pod placement:

root@k8s-master-964:~# kp wffc-pod-1 -o wide
NAME         READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
wffc-pod-1   1/1     Running   0          68s   10.244.2.9   k8s-node-018   <none>           <none>

On further analysis, I found that the pod gets rescheduled as the selected node (which is the first segment in the preferred accessibility requirements) does not match the PV node affinity required rules. The pod describe output in such cases looks like: (k8s code ref: https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/volume/scheduling/scheduler_binder.go#L678 )

root@k8s-master-964:~# kdsc pod wffc-pod-4
Name:         wffc-pod-4
Namespace:    default
Priority:     0
Node:         k8s-node-0316/10.186.0.244
Start Time:   Mon, 12 Jul 2021 19:01:04 +0000
Labels:       <none>
Annotations:  <none>
Status:       Running
IP:           10.244.1.16
IPs:
  IP:  10.244.1.16
Containers:
  test-container:
    Container ID:  docker://5909f88a2ea0b0d978bffdb0caa27fe793358bd16d9e7b420dd8bbf4545cfbbc
    Image:         gcr.io/google_containers/busybox:1.24
    Image ID:      docker-pullable://gcr.io/google_containers/busybox@sha256:4bdd623e848417d96127e16037743f0cd8b528c026e9175e22a84f639eca58ff
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      echo 'hello' > /mnt/volume1/index.html  && chmod o+rX /mnt /mnt/volume1/index.html && while true ; do sleep 2 ; done
    State:          Running
      Started:      Mon, 12 Jul 2021 19:01:35 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /mnt/volume1 from test-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-54pgb (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  test-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  wffc-pvc-3
    ReadOnly:   false
  kube-api-access-54pgb:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age   From                     Message
  ----     ------                  ----  ----                     -------
  Warning  FailedScheduling        15m   default-scheduler        running PreBind plugin "VolumeBinding": binding volumes: pv "pvc-b2c43dce-2502-45e8-b371-fefe3d7d3c71" node affinity doesn't match node "k8s-node-0888": no matching NodeSelectorTerms
  Normal   Scheduled               15m   default-scheduler        Successfully assigned default/wffc-pod-4 to k8s-node-0316
  Normal   SuccessfulAttachVolume  15m   attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-b2c43dce-2502-45e8-b371-fefe3d7d3c71"
  Normal   Pulled                  15m   kubelet                  Container image "gcr.io/google_containers/busybox:1.24" already present on machine
  Normal   Created                 15m   kubelet                  Created container test-container
  Normal   Started                 15m   kubelet                  Started container test-container
  1. Single zone accessibility requirements with WFFC:

Storage class definition:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: single-zone-wffc-sc
provisioner: csi.vsphere.vmware.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  storagepolicyname: "non-shared-ds-policy"
allowedTopologies:
  - matchLabelExpressions:
      - key: topology.kubernetes.io/zone
        values:
          - zone-a
      - key: topology.kubernetes.io/region
        values:
          - region-1

Logs:

AccessibilityRequirements:
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > > 
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > >

2021-07-07T16:58:26.497Z	INFO	volume/util.go:324	Volume created successfully. VolumeName: "pvc-11235cc3-ec32-4099-842b-3189274b3873", volumeID: "2994bfc4-b841-4b39-8d36-00ee9ed04931"	{"TraceId": "8e43706a-8ca0-43f4-8b06-8adf5bde4408"}
2021-07-07T16:58:26.497Z	DEBUG	volume/util.go:326	CreateVolume volumeId {{} "2994bfc4-b841-4b39-8d36-00ee9ed04931"} is placed on datastore "ds:///vmfs/volumes/60da3097-d38efce1-054f-0200990825a9/"	{"TraceId": "8e43706a-8ca0-43f4-8b06-8adf5bde4408"}
2021-07-07T16:58:26.499Z	DEBUG	vanilla/controller.go:556	Volume: 2994bfc4-b841-4b39-8d36-00ee9ed04931 is provisioned on the datastore: ds:///vmfs/volumes/60da3097-d38efce1-054f-0200990825a9/ 	{"TraceId": "8e43706a-8ca0-43f4-8b06-8adf5bde4408"}
2021-07-07T16:58:26.499Z	DEBUG	vanilla/controller.go:561	volumeAccessibleTopology: [map[topology.kubernetes.io/region:region-1 topology.kubernetes.io/zone:zone-a]] is selected for datastore: ds:///vmfs/volumes/60da3097-d38efce1-054f-0200990825a9/ 	{"TraceId": "8e43706a-8ca0-43f4-8b06-8adf5bde4408"}
2021-07-07T16:58:27.603Z	INFO	vanilla/controller.go:771	ControllerPublishVolume: called with args {VolumeId:2994bfc4-b841-4b39-8d36-00ee9ed04931 NodeId:k8s-node-0888 VolumeCapability:mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER >  Readonly:false Secrets:map[] VolumeContext:map[storage.kubernetes.io/csiProvisionerIdentity:1625623664749-8081-csi.vsphere.vmware.com type:vSphere CNS Block Volume] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}	{"TraceId": "ce108514-e759-469b-b897-8aa9245dd0db"}
2021-07-07T16:58:29.005Z	DEBUG	common/vsphereutil.go:440	Successfully attached disk 2994bfc4-b841-4b39-8d36-00ee9ed04931 to VM VirtualMachine:vm-51 [VirtualCenterHost: 10.186.2.101, UUID: 423f474d-bf84-6d1a-a6b1-f3b45e505592, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-3, VirtualCenterHost: 10.186.2.101]]. Disk UUID is 6000C298-7f84-4c50-80b7-fab336f60f42	{"TraceId": "ce108514-e759-469b-b897-8aa9245dd0db"}
2021-07-07T16:58:29.005Z	INFO	vanilla/controller.go:858	ControllerPublishVolume successful with publish context: map[diskUUID:6000c2987f844c5080b7fab336f60f42 type:vSphere CNS Block Volume]	{"TraceId": "ce108514-e759-469b-b897-8aa9245dd0db"}

PV describe output:

Name:              pvc-11235cc3-ec32-4099-842b-3189274b3873
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers:        [kubernetes.io/pv-protection external-attacher/csi-vsphere-vmware-com]
StorageClass:      single-zone-wffc-sc-1
Status:            Bound
Claim:             default/single-zone-wffc-pvc-1
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          100Mi
Node Affinity:     
  Required Terms:  
    Term 0:        topology.kubernetes.io/region in [region-1]
                   topology.kubernetes.io/zone in [zone-a]
Message:           
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            csi.vsphere.vmware.com
    FSType:            ext4
    VolumeHandle:      2994bfc4-b841-4b39-8d36-00ee9ed04931
    ReadOnly:          false
    VolumeAttributes:      storage.kubernetes.io/csiProvisionerIdentity=1625623664749-8081-csi.vsphere.vmware.com
                           type=vSphere CNS Block Volume
Events:                <none>
  1. Multi-zone accessibility requirements with WFFC:

Storage class definition:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: multi-zone-wffc-sc
provisioner: csi.vsphere.vmware.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  storagepolicyname: "non-shared-ds-policy"
allowedTopologies:
  - matchLabelExpressions:
      - key: topology.kubernetes.io/zone
        values:
          - zone-a
          - zone-b
      - key: topology.kubernetes.io/region
        values:
          - region-1

Logs:

AccessibilityRequirements:
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > > 
requisite:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-b" > > 
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-a" > > 
preferred:<segments:<key:"topology.kubernetes.io/region" value:"region-1" > segments:<key:"topology.kubernetes.io/zone" value:"zone-b" > > 

2021-07-07T16:56:42.245Z	INFO	volume/util.go:324	Volume created successfully. VolumeName: "pvc-10a938a6-fb51-4446-8350-5210e753e874", volumeID: "d9b30614-7149-4c10-bc1a-bdd5618cce17"	{"TraceId": "f2928d89-edbc-48d3-b834-5bd520ab0a67"}
2021-07-07T16:56:42.246Z	DEBUG	vanilla/controller.go:556	Volume: d9b30614-7149-4c10-bc1a-bdd5618cce17 is provisioned on the datastore: ds:///vmfs/volumes/60da3096-535e431d-f452-0200990825a9/ 	{"TraceId": "f2928d89-edbc-48d3-b834-5bd520ab0a67"}
2021-07-07T16:56:42.247Z	DEBUG	vanilla/controller.go:561	volumeAccessibleTopology: [map[topology.kubernetes.io/region:region-1 topology.kubernetes.io/zone:zone-a]] is selected for datastore: ds:///vmfs/volumes/60da3096-535e431d-f452-0200990825a9/ 	{"TraceId": "f2928d89-edbc-48d3-b834-5bd520ab0a67"}
2021-07-07T16:56:43.456Z	INFO	vanilla/controller.go:771	ControllerPublishVolume: called with args {VolumeId:d9b30614-7149-4c10-bc1a-bdd5618cce17 NodeId:k8s-node-018 VolumeCapability:mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER >  Readonly:false Secrets:map[] VolumeContext:map[storage.kubernetes.io/csiProvisionerIdentity:1625623664749-8081-csi.vsphere.vmware.com type:vSphere CNS Block Volume] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}	{"TraceId": "8c6894cd-b031-4ffc-a56e-2f11dafd7d29"}
2021-07-07T16:56:45.129Z	DEBUG	common/vsphereutil.go:440	Successfully attached disk d9b30614-7149-4c10-bc1a-bdd5618cce17 to VM VirtualMachine:vm-50 [VirtualCenterHost: 10.186.2.101, UUID: 423fe8b9-c453-ef3b-df25-79218df9ca24, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-3, VirtualCenterHost: 10.186.2.101]]. Disk UUID is 6000C29a-7baa-c28f-9379-33e532671b95	{"TraceId": "8c6894cd-b031-4ffc-a56e-2f11dafd7d29"}
2021-07-07T16:56:45.130Z	INFO	vanilla/controller.go:858	ControllerPublishVolume successful with publish context: map[diskUUID:6000c29a7baac28f937933e532671b95 type:vSphere CNS Block Volume]	{"TraceId": "8c6894cd-b031-4ffc-a56e-2f11dafd7d29"}

PV describe output:

Name:              pvc-10a938a6-fb51-4446-8350-5210e753e874
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers:        [kubernetes.io/pv-protection external-attacher/csi-vsphere-vmware-com]
StorageClass:      multi-zone-wffc-sc-1
Status:            Bound
Claim:             default/multi-zone-wffc-pvc-1
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          100Mi
Node Affinity:     
  Required Terms:  
    Term 0:        topology.kubernetes.io/zone in [zone-a]
                   topology.kubernetes.io/region in [region-1]
Message:           
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            csi.vsphere.vmware.com
    FSType:            ext4
    VolumeHandle:      d9b30614-7149-4c10-bc1a-bdd5618cce17
    ReadOnly:          false
    VolumeAttributes:      storage.kubernetes.io/csiProvisionerIdentity=1625623664749-8081-csi.vsphere.vmware.com
                           type=vSphere CNS Block Volume
Events:                <none>

Final State:

root@k8s-master-964:~# kp -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE            NOMINATED NODE   READINESS GATES
block-pod                1/1     Running   0          16h   10.244.1.12   k8s-node-0316   <none>           <none>
multi-zone-pod           1/1     Running   0          16h   10.244.3.7    k8s-node-0888   <none>           <none>
multi-zone-wffc-pod      1/1     Running   0          21h   10.244.3.5    k8s-node-0888   <none>           <none>
multi-zone-wffc-pod-1    1/1     Running   0          23m   10.244.2.8    k8s-node-018    <none>           <none>
single-zone-pod          1/1     Running   0          16h   10.244.1.13   k8s-node-0316   <none>           <none>
single-zone-wffc-pod     1/1     Running   0          21h   10.244.2.6    k8s-node-018    <none>           <none>
single-zone-wffc-pod-1   1/1     Running   0          21m   10.244.3.8    k8s-node-0888   <none>           <none>
wffc-pod                 1/1     Running   0          21h   10.244.1.6    k8s-node-0316   <none>           <none>
wffc-pod-1               1/1     Running   0          19m   10.244.1.14   k8s-node-0316   <none>           <none>
@shalini-b
Copy link
Author

Generate accessibility requirements code in kubernetes - https://github.com/kubernetes-csi/external-provisioner/blob/61317e204eb271d8f47ae7983f97abbb60cbf3a4/pkg/controller/topology.go#L136

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment