- Created config-map for feature states.**
# cat csi-feature-states.conf
[FeatureStates]
csi-migration= "true"
# kubectl create configmap csi-feature-states --from-file=csi-feature-states.conf --namespace=kube-system
configmap/csi-feature-states created
- Deployed controller using YAMLs - https://gist.github.com/divyenpatel/a9d0f45d4ace041e385814cd4cb0a376
Note: Will create another PR to check-in yamls changes once all dependent external sidecar containers are available with migration shim changes.
- Log of Initialization of volume migration service
2020-06-03T06:11:22.945997612Z {"level":"info","time":"2020-06-03T06:11:22.940106092Z","caller":"vanilla/controller.go:197","msg":"CSI Migration Feature is Enabled. Loading Volume Migration Service","TraceId":"13383e3b-88fa-4efb-9a2c-4976f17c9ea0"}
2020-06-03T06:11:22.946000933Z {"level":"info","time":"2020-06-03T06:11:22.94012114Z","caller":"migration/migration.go:81","msg":"Initializing volume migration service...","TraceId":"13383e3b-88fa-4efb-9a2c-4976f17c9ea0"}
2020-06-03T06:11:23.100490326Z {"level":"info","time":"2020-06-03T06:11:23.059547127Z","caller":"migration/migration.go:146","msg":"\"cnsvspherevolumemigrations.cns.vmware.com\" CRD created successfully","TraceId":"13383e3b-88fa-4efb-9a2c-4976f17c9ea0"}
2020-06-03T06:11:28.067009825Z {"level":"info","time":"2020-06-03T06:11:28.066095938Z","caller":"kubernetes/kubernetes.go:66","msg":"k8s client using in-cluster config","TraceId":"13383e3b-88fa-4efb-9a2c-4976f17c9ea0"}
2020-06-03T06:11:28.123213719Z {"level":"info","time":"2020-06-03T06:11:28.122860458Z","caller":"migration/migration.go:100","msg":"volume migration service initialized","TraceId":"13383e3b-88fa-4efb-9a2c-4976f17c9ea0"}
2020-06-03T06:11:28.240086678Z {"level":"info","time":"2020-06-03T06:11:28.239653032Z","caller":"migration/migration.go:267","msg":"successfully loaded all CR for CnsvSphereVolumeMigration in the cache","TraceId":"13383e3b-88fa-4efb-9a2c-4976f17c9ea0"}
- Log of migration of pre-provisioned volumes
% kubectl describe pvc
Name: vcppvc
Namespace: default
StorageClass: vcpsc
Status: Bound
Volume: pvc-b9c0b278-67db-4cb5-b01e-3f9a36f93876
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
pv.kubernetes.io/migrated-to: csi.vsphere.vmware.com
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 2Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: <none>
Events: <none>
% kubectl describe pv
Name: pvc-b9c0b278-67db-4cb5-b01e-3f9a36f93876
Labels: <none>
Annotations: kubernetes.io/createdby: vsphere-volume-dynamic-provisioner
pv.kubernetes.io/bound-by-controller: yes
pv.kubernetes.io/migrated-to: csi.vsphere.vmware.com
pv.kubernetes.io/provisioned-by: kubernetes.io/vsphere-volume
Finalizers: [kubernetes.io/pv-protection]
StorageClass: vcpsc
Status: Bound
Claim: default/vcppvc
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 2Gi
Node Affinity: <none>
Message:
Source:
Type: vSphereVolume (a Persistent Disk resource in vSphere)
VolumePath: [vsanDatastore] 0c6ad55e-88c2-e431-bccb-02000359156d/kubernetes-dynamic-pvc-b9c0b278-67db-4cb5-b01e-3f9a36f93876.vmdk
FSType: ext4
StoragePolicyName: vSAN Default Storage Policy
Events: <none>
# cat pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: vcppod
spec:
containers:
- name: test-container
image: gcr.io/google_containers/busybox:1.24
command: ["/bin/sh", "-c", "echo 'hello' > /mnt/volume1/index.html && chmod o+rX /mnt /mnt/volume1/index.html && while true ; do sleep 2 ; done"]
volumeMounts:
- name: test-volume
mountPath: /mnt/volume1
restartPolicy: Never
volumes:
- name: test-volume
persistentVolumeClaim:
claimName: vcppvc
# kubectl create -f pod.yaml
pod/vcppod created
ControllerPublishVolume Logs
2020-06-02T00:35:59.197587482Z {"level":"info","time":"2020-06-02T00:35:59.197256331Z","caller":"vanilla/controller.go:484","msg":"ControllerPublishVolume: called with args {VolumeId:[vsanDatastore] 0c6ad55e-88c2-e431-bccb-02000359156d/kubernetes-dynamic-pvc-b9c0b278-67db-4cb5-b01e-3f9a36f93876.vmdk NodeId:k8s-node1 VolumeCapability:mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > Readonly:false Secrets:map[] VolumeContext:map[storagepolicyname:vSAN Default Storage Policy] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"7a8e6562-4022-4735-bfe6-91ecf2a8f776"}
2020-06-02T00:35:59.205420469Z {"level":"info","time":"2020-06-02T00:35:59.205075417Z","caller":"migration/migration.go:213","msg":"Could not retrieve CnsvSphereVolumeMigrationSpec from cache and K8s CR for Volume Path: \"[vsanDatastore] 0c6ad55e-88c2-e431-bccb-02000359156d/kubernetes-dynamic-pvc-b9c0b278-67db-4cb5-b01e-3f9a36f93876.vmdk\". volume may not be registered","TraceId":"7a8e6562-4022-4735-bfe6-91ecf2a8f776"}
2020-06-02T00:35:59.205441654Z {"level":"info","time":"2020-06-02T00:35:59.205156546Z","caller":"vanilla/controller_helper.go:80","msg":"Registering in-tree volume: \"[vsanDatastore] 0c6ad55e-88c2-e431-bccb-02000359156d/kubernetes-dynamic-pvc-b9c0b278-67db-4cb5-b01e-3f9a36f93876.vmdk\"","TraceId":"7a8e6562-4022-4735-bfe6-91ecf2a8f776"}
2020-06-02T00:35:59.209949509Z {"level":"info","time":"2020-06-02T00:35:59.209580061Z","caller":"migration/migration.go:303","msg":"Registering volume: \"[vsanDatastore] 0c6ad55e-88c2-e431-bccb-02000359156d/kubernetes-dynamic-pvc-b9c0b278-67db-4cb5-b01e-3f9a36f93876.vmdk\" using backingDiskURLPath :\"https://10.160.41.232/folder/0c6ad55e-88c2-e431-bccb-02000359156d/kubernetes-dynamic-pvc-b9c0b278-67db-4cb5-b01e-3f9a36f93876.vmdk?dcPath=cns-datacenter&dsName=vsanDatastore\"","TraceId":"7a8e6562-4022-4735-bfe6-91ecf2a8f776"}
2020-06-02T00:36:01.162566223Z {"level":"info","time":"2020-06-02T00:36:01.162351316Z","caller":"volume/manager.go:195","msg":"CreateVolume: VolumeName: \"0778133a-a469-11ea-a69a-12f99881383e\", opId: \"4c7d1a23\"","TraceId":"7a8e6562-4022-4735-bfe6-91ecf2a8f776"}
2020-06-02T00:36:01.162989486Z {"level":"info","time":"2020-06-02T00:36:01.162810592Z","caller":"volume/manager.go:213","msg":"Observed CnsAlreadyRegisteredFault fault","TraceId":"7a8e6562-4022-4735-bfe6-91ecf2a8f776"}
2020-06-02T00:36:01.163176684Z {"level":"info","time":"2020-06-02T00:36:01.163052738Z","caller":"volume/manager.go:214","msg":"CreateVolume: Volume created successfully. VolumeName: \"0778133a-a469-11ea-a69a-12f99881383e\", opId: \"4c7d1a23\", volumeID: \"064f8b98-4e0d-40b9-a819-2269fe39862a\"","TraceId":"7a8e6562-4022-4735-bfe6-91ecf2a8f776"}
2020-06-02T00:36:01.163465988Z {"level":"info","time":"2020-06-02T00:36:01.163309067Z","caller":"migration/migration.go:324","msg":"successfully registered volume \"[vsanDatastore] 0c6ad55e-88c2-e431-bccb-02000359156d/kubernetes-dynamic-pvc-b9c0b278-67db-4cb5-b01e-3f9a36f93876.vmdk\" as container volume with ID: \"064f8b98-4e0d-40b9-a819-2269fe39862a\"","TraceId":"7a8e6562-4022-4735-bfe6-91ecf2a8f776"}
2020-06-02T00:36:01.164115487Z {"level":"info","time":"2020-06-02T00:36:01.163804322Z","caller":"migration/migration.go:221","msg":"creating CR for cnsvSphereVolumeMigration: {TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:0778133a-a469-11ea-a69a-12f99881383e GenerateName: Namespace: SelfLink: UID: ResourceVersion: Generation:0 CreationTimestamp:0001-01-01 00:00:00 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[] Initializers:nil Finalizers:[] ClusterName: ManagedFields:[]} Spec:{VolumePath:[vsanDatastore] 0c6ad55e-88c2-e431-bccb-02000359156d/kubernetes-dynamic-pvc-b9c0b278-67db-4cb5-b01e-3f9a36f93876.vmdk VolumeName:0778133a-a469-11ea-a69a-12f99881383e VolumeID:064f8b98-4e0d-40b9-a819-2269fe39862a}}","TraceId":"7a8e6562-4022-4735-bfe6-91ecf2a8f776"}
2020-06-02T00:36:01.219137009Z {"level":"info","time":"2020-06-02T00:36:01.218951887Z","caller":"migration/migration.go:227","msg":"successfully created CR for cnsvSphereVolumeMigration: {{ } {0778133a-a469-11ea-a69a-12f99881383e /apis/cns.vmware.com/v1alpha1/cnsvspherevolumemigrations/0778133a-a469-11ea-a69a-12f99881383e 751d7d04-adc0-400f-ab65-2a55b014693f 48718 1 2020-06-02 00:36:01 +0000 UTC <nil> <nil> map[] map[] [] nil [] [{vsphere-csi Update cns.vmware.com/v1alpha1 2020-06-02 00:36:01 +0000 UTC nil}]} {[vsanDatastore] 0c6ad55e-88c2-e431-bccb-02000359156d/kubernetes-dynamic-pvc-b9c0b278-67db-4cb5-b01e-3f9a36f93876.vmdk 0778133a-a469-11ea-a69a-12f99881383e 064f8b98-4e0d-40b9-a819-2269fe39862a}}","TraceId":"7a8e6562-4022-4735-bfe6-91ecf2a8f776"}
2020-06-02T00:36:01.219170186Z {"level":"info","time":"2020-06-02T00:36:01.219007997Z","caller":"migration/migration.go:229","msg":"successfully updated cache for cnsvSphereVolumeMigrationSpec: {{ } {0778133a-a469-11ea-a69a-12f99881383e /apis/cns.vmware.com/v1alpha1/cnsvspherevolumemigrations/0778133a-a469-11ea-a69a-12f99881383e 751d7d04-adc0-400f-ab65-2a55b014693f 48718 1 2020-06-02 00:36:01 +0000 UTC <nil> <nil> map[] map[] [] nil [] [{vsphere-csi Update cns.vmware.com/v1alpha1 2020-06-02 00:36:01 +0000 UTC nil}]} {[vsanDatastore] 0c6ad55e-88c2-e431-bccb-02000359156d/kubernetes-dynamic-pvc-b9c0b278-67db-4cb5-b01e-3f9a36f93876.vmdk 0778133a-a469-11ea-a69a-12f99881383e 064f8b98-4e0d-40b9-a819-2269fe39862a}}","TraceId":"7a8e6562-4022-4735-bfe6-91ecf2a8f776"}
2020-06-02T00:36:06.353955902Z {"level":"info","time":"2020-06-02T00:36:06.353428661Z","caller":"volume/manager.go:266","msg":"AttachVolume: volumeID: \"064f8b98-4e0d-40b9-a819-2269fe39862a\", vm: \"VirtualMachine:vm-86 [VirtualCenterHost: 10.160.41.232, UUID: 42246c52-c2bd-49cf-1111-fbcbd29087da, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-3, VirtualCenterHost: 10.160.41.232]]\", opId: \"4c7d1a24\"","TraceId":"7a8e6562-4022-4735-bfe6-91ecf2a8f776"}
2020-06-02T00:36:06.353997599Z {"level":"info","time":"2020-06-02T00:36:06.353534968Z","caller":"volume/manager.go:299","msg":"AttachVolume: Volume attached successfully. volumeID: \"064f8b98-4e0d-40b9-a819-2269fe39862a\", opId: \"4c7d1a24\", vm: \"VirtualMachine:vm-86 [VirtualCenterHost: 10.160.41.232, UUID: 42246c52-c2bd-49cf-1111-fbcbd29087da, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-3, VirtualCenterHost: 10.160.41.232]]\", diskUUID: \"6000C29a-3706-eabe-091c-44c5d7d325bf\"","TraceId":"7a8e6562-4022-4735-bfe6-91ecf2a8f776"}
Verified cnsvspherevolumemigration CR is created.
# kubectl get cnsvspherevolumemigrations.cns.vmware.com
NAME AGE
0778133a-a469-11ea-a69a-12f99881383e 70s
# kubectl describe cnsvspherevolumemigrations.cns.vmware.com
Name: 0778133a-a469-11ea-a69a-12f99881383e
Namespace:
Labels: <none>
Annotations: <none>
API Version: cns.vmware.com/v1alpha1
Kind: CnsvSphereVolumeMigration
Metadata:
Creation Timestamp: 2020-06-02T00:36:01Z
Generation: 1
Managed Fields:
API Version: cns.vmware.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:spec:
.:
f:volumeid:
f:volumename:
f:volumepath:
Manager: vsphere-csi
Operation: Update
Time: 2020-06-02T00:36:01Z
Resource Version: 48718
Self Link: /apis/cns.vmware.com/v1alpha1/cnsvspherevolumemigrations/0778133a-a469-11ea-a69a-12f99881383e
UID: 751d7d04-adc0-400f-ab65-2a55b014693f
Spec:
Volumeid: 064f8b98-4e0d-40b9-a819-2269fe39862a
Volumename: 0778133a-a469-11ea-a69a-12f99881383e
Volumepath: [vsanDatastore] 0c6ad55e-88c2-e431-bccb-02000359156d/kubernetes-dynamic-pvc-b9c0b278-67db-4cb5-b01e-3f9a36f93876.vmdk
Events: <none>
# kubectl get pods
NAME READY STATUS RESTARTS AGE
vcppod 1/1 Running 0 2m4s
Deletion of Pod using migrated volume.
# kubectl delete pod vcppod
pod "vcppod" deleted
ControllerUnpublishVolume Logs
2020-06-02T00:39:18.317683280Z {"level":"info","time":"2020-06-02T00:39:18.316801394Z","caller":"vanilla/controller.go:564","msg":"ControllerUnpublishVolume: called with args {VolumeId:[vsanDatastore] 0c6ad55e-88c2-e431-bccb-02000359156d/kubernetes-dynamic-pvc-b9c0b278-67db-4cb5-b01e-3f9a36f93876.vmdk NodeId:k8s-node1 Secrets:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"d153de05-a124-46b3-8842-6d2ef35400aa"}
2020-06-02T00:39:18.317703293Z {"level":"info","time":"2020-06-02T00:39:18.316962225Z","caller":"migration/migration.go:196","msg":"CnsvSphereVolumeMigrationSpec found from the cache for for VolumePath: \"[vsanDatastore] 0c6ad55e-88c2-e431-bccb-02000359156d/kubernetes-dynamic-pvc-b9c0b278-67db-4cb5-b01e-3f9a36f93876.vmdk\"","TraceId":"d153de05-a124-46b3-8842-6d2ef35400aa"}
2020-06-02T00:39:21.178987210Z {"level":"info","time":"2020-06-02T00:39:21.178603973Z","caller":"volume/manager.go:357","msg":"DetachVolume: volumeID: \"064f8b98-4e0d-40b9-a819-2269fe39862a\", vm: \"VirtualMachine:vm-86 [VirtualCenterHost: 10.160.41.232, UUID: 42246c52-c2bd-49cf-1111-fbcbd29087da, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-3, VirtualCenterHost: 10.160.41.232]]\", opId: \"4c7d1a45\"","TraceId":"d153de05-a124-46b3-8842-6d2ef35400aa"}
2020-06-02T00:39:21.179250070Z {"level":"info","time":"2020-06-02T00:39:21.179079569Z","caller":"volume/manager.go:387","msg":"DetachVolume: Volume detached successfully. volumeID: \"064f8b98-4e0d-40b9-a819-2269fe39862a\", vm: \"4c7d1a45\", opId: \"VirtualMachine:vm-86 [VirtualCenterHost: 10.160.41.232, UUID: 42246c52-c2bd-49cf-1111-fbcbd29087da, Datacenter: Datacenter [Datacenter: Datacenter:datacenter-3, VirtualCenterHost: 10.160.41.232]]\"","TraceId":"d153de05-a124-46b3-8842-6d2ef35400aa"}
Delete of migrated PVC
# kubectl delete pvc vcppvc
persistentvolumeclaim "vcppvc" deleted
DeleteVolume Logs
2020-06-02T00:41:00.691209908Z {"level":"info","time":"2020-06-02T00:41:00.690766808Z","caller":"vanilla/controller.go:456","msg":"DeleteVolume: called with args: {VolumeId:[vsanDatastore] 0c6ad55e-88c2-e431-bccb-02000359156d/kubernetes-dynamic-pvc-b9c0b278-67db-4cb5-b01e-3f9a36f93876.vmdk Secrets:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"d0c04914-dcf0-478f-a5d7-19eef16e989e"}
2020-06-02T00:41:00.691238450Z {"level":"info","time":"2020-06-02T00:41:00.690857187Z","caller":"migration/migration.go:196","msg":"CnsvSphereVolumeMigrationSpec found from the cache for for VolumePath: \"[vsanDatastore] 0c6ad55e-88c2-e431-bccb-02000359156d/kubernetes-dynamic-pvc-b9c0b278-67db-4cb5-b01e-3f9a36f93876.vmdk\"","TraceId":"d0c04914-dcf0-478f-a5d7-19eef16e989e"}
2020-06-02T00:41:05.138110266Z {"level":"info","time":"2020-06-02T00:41:05.137826649Z","caller":"volume/manager.go:426","msg":"DeleteVolume: volumeID: \"064f8b98-4e0d-40b9-a819-2269fe39862a\", opId: \"4c7d1a54\"","TraceId":"d0c04914-dcf0-478f-a5d7-19eef16e989e"}
2020-06-02T00:41:05.138151841Z {"level":"info","time":"2020-06-02T00:41:05.137916332Z","caller":"volume/manager.go:444","msg":"DeleteVolume: Volume deleted successfully. volumeID: \"064f8b98-4e0d-40b9-a819-2269fe39862a\", opId: \"4c7d1a54\"","TraceId":"d0c04914-dcf0-478f-a5d7-19eef16e989e"}
Creation of the new volume after the migration is enabled.
# cat sc-with-datastore.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: vcpsc-with-datastore
provisioner: kubernetes.io/vsphere-volume
parameters:
storagepolicyname: "vSAN Default Storage Policy"
datastore: "vsanDatastore"
# kubectl create -f sc-with-datastore.yaml
storageclass.storage.k8s.io/vcpsc-with-datastore created
# cat pvc-using-sc-with-datastore.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: vcppvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
storageClassName: vcpsc-with-datastore
# kubectl create -f pvc-using-sc-with-datastore.yaml
persistentvolumeclaim/vcppvc created
Volume Creation Log
2020-06-03T07:09:54.262103485Z {"level":"info","time":"2020-06-03T07:09:54.261264871Z","caller":"vanilla/controller.go:508","msg":"CreateVolume: called with args {Name:pvc-a0420ae4-1371-4500-b170-20b63dd5e78b CapacityRange:required_bytes:2147483648 VolumeCapabilities:[mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[csimigration:true datastore:vsanDatastore storagepolicyname:vSAN Default Storage Policy] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"7c7e1bfb-1a12-44fb-98f8-f37fc4de70a3"}
2020-06-03T07:09:54.262172475Z {"level":"info","time":"2020-06-03T07:09:54.261703246Z","caller":"common/util.go:228","msg":"vSphere CSI Driver supports StoragePolicyName, Datastore and fsType parameters supplied from legacy in-tree provisioner. All other parameters supplied in csimigrationparams will be dropped.","TraceId":"7c7e1bfb-1a12-44fb-98f8-f37fc4de70a3"}
2020-06-03T07:09:54.262178115Z {"level":"info","time":"2020-06-03T07:09:54.261772754Z","caller":"vanilla/controller.go:282","msg":"Converting datastore name: \"vsanDatastore\" to Datastore URL","TraceId":"7c7e1bfb-1a12-44fb-98f8-f37fc4de70a3"}
2020-06-03T07:09:54.304164818Z {"level":"info","time":"2020-06-03T07:09:54.303507027Z","caller":"vanilla/controller.go:301","msg":"Found datastoreURL: \"ds:///vmfs/volumes/vsan:52b7b0d022ddee29-e62dca3400ef5bb7/\" for datastore name: \"vsanDatastore\"","TraceId":"7c7e1bfb-1a12-44fb-98f8-f37fc4de70a3"}
2020-06-03T07:10:11.938887898Z {"level":"info","time":"2020-06-03T07:10:11.937771131Z","caller":"volume/manager.go:197","msg":"CreateVolume: VolumeName: \"pvc-a0420ae4-1371-4500-b170-20b63dd5e78b\", opId: \"4c7d572d\"","TraceId":"7c7e1bfb-1a12-44fb-98f8-f37fc4de70a3"}
2020-06-03T07:10:11.938991806Z {"level":"info","time":"2020-06-03T07:10:11.938450781Z","caller":"volume/manager.go:228","msg":"CreateVolume: Volume created successfully. VolumeName: \"pvc-a0420ae4-1371-4500-b170-20b63dd5e78b\", opId: \"4c7d572d\", volumeID: \"e5120e81-41c8-40f8-b8f7-3d89a9e1a1d8\"","TraceId":"7c7e1bfb-1a12-44fb-98f8-f37fc4de70a3"}
2020-06-03T07:10:12.693127749Z {"level":"info","time":"2020-06-03T07:10:12.692384004Z","caller":"volume/manager.go:648","msg":"QueryVolumeInfo: volumeIDList: [{{} e5120e81-41c8-40f8-b8f7-3d89a9e1a1d8}], opId: \"4c7d5732\"","TraceId":"7c7e1bfb-1a12-44fb-98f8-f37fc4de70a3"}
2020-06-03T07:10:12.693733315Z {"level":"info","time":"2020-06-03T07:10:12.693163833Z","caller":"volume/manager.go:667","msg":"QueryVolumeInfo successfully returned volumeInfo (*types.CnsQueryVolumeInfoResult)(0xc0004ca930)({\n CnsVolumeOperationResult: (types.CnsVolumeOperationResult) {\n DynamicData: (types.DynamicData) {\n },\n VolumeId: (types.CnsVolumeId) {\n DynamicData: (types.DynamicData) {\n },\n Id: (string) \"\"\n },\n Fault: (*types.LocalizedMethodFault)(<nil>)\n },\n VolumeInfo: (*types.CnsBlockVolumeInfo)(0xc00065acc0)({\n CnsVolumeInfo: (types.CnsVolumeInfo) {\n DynamicData: (types.DynamicData) {\n }\n },\n VStorageObject: (types.VStorageObject) {\n DynamicData: (types.DynamicData) {\n },\n Config: (types.VStorageObjectConfigInfo) {\n BaseConfigInfo: (types.BaseConfigInfo) {\n DynamicData: (types.DynamicData) {\n },\n Id: (types.ID) {\n DynamicData: (types.DynamicData) {\n },\n Id: (string) (len=36) \"e5120e81-41c8-40f8-b8f7-3d89a9e1a1d8\"\n },\n Name: (string) (len=40) \"pvc-a0420ae4-1371-4500-b170-20b63dd5e78b\",\n CreateTime: (time.Time) 2020-06-03 07:09:56.043 +0000 UTC,\n KeepAfterDeleteVm: (*bool)(0xc000677298)(true),\n RelocationDisabled: (*bool)(0xc00067736c)(false),\n NativeSnapshotSupported: (*bool)(0xc00067738d)(false),\n ChangedBlockTrackingEnabled: (*bool)(0xc0006773cd)(false),\n Backing: (*types.BaseConfigInfoDiskFileBackingInfo)(0xc000222460)({\n BaseConfigInfoFileBackingInfo: (types.BaseConfigInfoFileBackingInfo) {\n BaseConfigInfoBackingInfo: (types.BaseConfigInfoBackingInfo) {\n DynamicData: (types.DynamicData) {\n },\n Datastore: (types.ManagedObjectReference) Datastore:datastore-68\n },\n FilePath: (string) (len=90) \"[vsanDatastore] 879cd65e-6165-91a3-d047-02000359156d/8a65c35bb78c4e06b47f80e1bc23ce71.vmdk\",\n BackingObjectId: (string) (len=36) \"c34cd75e-c6bd-de8b-6cc7-020003bb3e3c\",\n Parent: (types.BaseBaseConfigInfoFileBackingInfo) <nil>,\n DeltaSizeInMB: (int64) 0,\n KeyId: (*types.CryptoKeyId)(<nil>)\n },\n ProvisioningType: (string) (len=4) \"thin\"\n }),\n Iofilter: ([]string) <nil>\n },\n CapacityInMB: (int64) 2048,\n ConsumptionType: ([]string) (len=1 cap=1) {\n (string) (len=4) \"disk\"\n },\n ConsumerId: ([]types.ID) <nil>\n }\n }\n })\n})\n for volumeIDList [{{} e5120e81-41c8-40f8-b8f7-3d89a9e1a1d8}]:, opId: \"4c7d5732\"","TraceId":"7c7e1bfb-1a12-44fb-98f8-f37fc4de70a3"}
2020-06-03T07:10:12.693846818Z {"level":"info","time":"2020-06-03T07:10:12.693429521Z","caller":"vanilla/controller.go:388","msg":"successfully retrieved volume path: \"[vsanDatastore] 879cd65e-6165-91a3-d047-02000359156d/8a65c35bb78c4e06b47f80e1bc23ce71.vmdk\" for volume id: \"e5120e81-41c8-40f8-b8f7-3d89a9e1a1d8\"","TraceId":"7c7e1bfb-1a12-44fb-98f8-f37fc4de70a3"}
2020-06-03T07:10:12.694994786Z {"level":"info","time":"2020-06-03T07:10:12.694644253Z","caller":"vanilla/controller.go:404","msg":"Saving cnsvSphereVolumeMigration CR: {{ } {4475d347-a569-11ea-981c-2e6e8d29fcf2 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] []} {[vsanDatastore] 879cd65e-6165-91a3-d047-02000359156d/8a65c35bb78c4e06b47f80e1bc23ce71.vmdk pvc-a0420ae4-1371-4500-b170-20b63dd5e78b e5120e81-41c8-40f8-b8f7-3d89a9e1a1d8}}","TraceId":"7c7e1bfb-1a12-44fb-98f8-f37fc4de70a3"}
2020-06-03T07:10:12.695009433Z {"level":"info","time":"2020-06-03T07:10:12.69478611Z","caller":"migration/migration.go:221","msg":"creating CR for cnsvSphereVolumeMigration: {TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:4475d347-a569-11ea-981c-2e6e8d29fcf2 GenerateName: Namespace: SelfLink: UID: ResourceVersion: Generation:0 CreationTimestamp:0001-01-01 00:00:00 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[] Initializers:nil Finalizers:[] ClusterName: ManagedFields:[]} Spec:{VolumePath:[vsanDatastore] 879cd65e-6165-91a3-d047-02000359156d/8a65c35bb78c4e06b47f80e1bc23ce71.vmdk VolumeName:pvc-a0420ae4-1371-4500-b170-20b63dd5e78b VolumeID:e5120e81-41c8-40f8-b8f7-3d89a9e1a1d8}}","TraceId":"7c7e1bfb-1a12-44fb-98f8-f37fc4de70a3"}
2020-06-03T07:10:12.707087746Z {"level":"info","time":"2020-06-03T07:10:12.706282996Z","caller":"migration/migration.go:227","msg":"successfully created CR for cnsvSphereVolumeMigration: {{ } {4475d347-a569-11ea-981c-2e6e8d29fcf2 /apis/cns.vmware.com/v1alpha1/cnsvspherevolumemigrations/4475d347-a569-11ea-981c-2e6e8d29fcf2 1f3ce98f-2e88-4fd9-bf93-5a2b665ebaf8 408846 1 2020-06-03 07:10:12 +0000 UTC <nil> <nil> map[] map[] [] nil [] [{vsphere-csi Update cns.vmware.com/v1alpha1 2020-06-03 07:10:12 +0000 UTC nil}]} {[vsanDatastore] 879cd65e-6165-91a3-d047-02000359156d/8a65c35bb78c4e06b47f80e1bc23ce71.vmdk pvc-a0420ae4-1371-4500-b170-20b63dd5e78b e5120e81-41c8-40f8-b8f7-3d89a9e1a1d8}}","TraceId":"7c7e1bfb-1a12-44fb-98f8-f37fc4de70a3"}
2020-06-03T07:10:12.712619651Z {"level":"info","time":"2020-06-03T07:10:12.707004421Z","caller":"migration/migration.go:229","msg":"successfully updated cache for cnsvSphereVolumeMigrationSpec: {{ } {4475d347-a569-11ea-981c-2e6e8d29fcf2 /apis/cns.vmware.com/v1alpha1/cnsvspherevolumemigrations/4475d347-a569-11ea-981c-2e6e8d29fcf2 1f3ce98f-2e88-4fd9-bf93-5a2b665ebaf8 408846 1 2020-06-03 07:10:12 +0000 UTC <nil> <nil> map[] map[] [] nil [] [{vsphere-csi Update cns.vmware.com/v1alpha1 2020-06-03 07:10:12 +0000 UTC nil}]} {[vsanDatastore] 879cd65e-6165-91a3-d047-02000359156d/8a65c35bb78c4e06b47f80e1bc23ce71.vmdk pvc-a0420ae4-1371-4500-b170-20b63dd5e78b e5120e81-41c8-40f8-b8f7-3d89a9e1a1d8}}","TraceId":"7c7e1bfb-1a12-44fb-98f8-f37fc4de70a3"}
2020-06-03T07:10:12.712637842Z {"level":"info","time":"2020-06-03T07:10:12.707217634Z","caller":"vanilla/controller.go:410","msg":"successfully saved cnsvSphereVolumeMigration CR: {{ } {4475d347-a569-11ea-981c-2e6e8d29fcf2 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] nil [] []} {[vsanDatastore] 879cd65e-6165-91a3-d047-02000359156d/8a65c35bb78c4e06b47f80e1bc23ce71.vmdk pvc-a0420ae4-1371-4500-b170-20b63dd5e78b e5120e81-41c8-40f8-b8f7-3d89a9e1a1d8}}","TraceId":"7c7e1bfb-1a12-44fb-98f8-f37fc4de70a3"}
Describe PVC/PV
# kubectl describe pvc
Name: vcppvc
Namespace: default
StorageClass: vcpsc-with-datastore
Status: Bound
Volume: pvc-a0420ae4-1371-4500-b170-20b63dd5e78b
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: csi.vsphere.vmware.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 2Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Provisioning 47s csi.vsphere.vmware.com_vsphere-csi-controller-6d8ff9688d-7rn2q_6e8eab84-509e-4a30-9e56-90935ff08871 External provisioner is provisioning volume for claim "default/vcppvc"
Normal ExternalProvisioning 29s (x3 over 47s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "csi.vsphere.vmware.com" or manually created by system administrator
Normal ProvisioningSucceeded 29s csi.vsphere.vmware.com_vsphere-csi-controller-6d8ff9688d-7rn2q_6e8eab84-509e-4a30-9e56-90935ff08871 Successfully provisioned volume pvc-a0420ae4-1371-4500-b170-20b63dd5e78b
# kubectl describe pv
Name: pvc-a0420ae4-1371-4500-b170-20b63dd5e78b
Labels: <none>
Annotations: pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
Finalizers: [kubernetes.io/pv-protection]
StorageClass: vcpsc-with-datastore
Status: Bound
Claim: default/vcppvc
Reclaim Policy: Delete
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 2Gi
Node Affinity: <none>
Message:
Source:
Type: vSphereVolume (a Persistent Disk resource in vSphere)
VolumePath: [vsanDatastore] 879cd65e-6165-91a3-d047-02000359156d/8a65c35bb78c4e06b47f80e1bc23ce71.vmdk
FSType: ext4
StoragePolicyName:
Events: <none>
# kubectl describe cnsvspherevolumemigrations.cns.vmware.com/4475d347-a569-11ea-981c-2e6e8d29fcf2
Name: 4475d347-a569-11ea-981c-2e6e8d29fcf2
Namespace:
Labels: <none>
Annotations: <none>
API Version: cns.vmware.com/v1alpha1
Kind: CnsvSphereVolumeMigration
Metadata:
Creation Timestamp: 2020-06-03T07:10:12Z
Generation: 1
Managed Fields:
API Version: cns.vmware.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:spec:
.:
f:volumeid:
f:volumename:
f:volumepath:
Manager: vsphere-csi
Operation: Update
Time: 2020-06-03T07:10:12Z
Resource Version: 408846
Self Link: /apis/cns.vmware.com/v1alpha1/cnsvspherevolumemigrations/4475d347-a569-11ea-981c-2e6e8d29fcf2
UID: 1f3ce98f-2e88-4fd9-bf93-5a2b665ebaf8
Spec:
Volumeid: e5120e81-41c8-40f8-b8f7-3d89a9e1a1d8
Volumename: pvc-a0420ae4-1371-4500-b170-20b63dd5e78b
Volumepath: [vsanDatastore] 879cd65e-6165-91a3-d047-02000359156d/8a65c35bb78c4e06b47f80e1bc23ce71.vmdk
Events: <none>
Deletion of PVC created using in-tree provisioner after the migration is enabled.
# kubectl delete pvc vcppvc
persistentvolumeclaim "vcppvc" deleted
# kubectl describe cnsvspherevolumemigrations.cns.vmware.com/4475d347-a569-11ea-981c-2e6e8d29fcf2
Error from server (NotFound): cnsvspherevolumemigrations.cns.vmware.com "4475d347-a569-11ea-981c-2e6e8d29fcf2" not found
Logs
2020-06-03T07:12:13.556208809Z {"level":"info","time":"2020-06-03T07:12:13.555653756Z","caller":"volume/manager.go:428","msg":"DeleteVolume: volumeID: \"e5120e81-41c8-40f8-b8f7-3d89a9e1a1d8\", opId: \"4c7d5741\"","TraceId":"9043f77f-6106-431f-9a2a-f7420eed7648"}
2020-06-03T07:12:13.556246405Z {"level":"info","time":"2020-06-03T07:12:13.555795613Z","caller":"volume/manager.go:446","msg":"DeleteVolume: Volume deleted successfully. volumeID: \"e5120e81-41c8-40f8-b8f7-3d89a9e1a1d8\", opId: \"4c7d5741\"","TraceId":"9043f77f-6106-431f-9a2a-f7420eed7648"}
2020-06-03T07:12:13.612976742Z {"level":"info","time":"2020-06-03T07:12:13.608716417Z","caller":"migration/migration.go:250","msg":"successfully deleted CR for volumeMigration having volumePath: \"[vsanDatastore] 879cd65e-6165-91a3-d047-02000359156d/8a65c35bb78c4e06b47f80e1bc23ce71.vmdk\"","TraceId":"9043f77f-6106-431f-9a2a-f7420eed7648"}
2020-06-03T07:12:13.613176490Z {"level":"info","time":"2020-06-03T07:12:13.608775204Z","caller":"migration/migration.go:255","msg":"successfully deleted volumeInfo from cache for volumePath: \"[vsanDatastore] 879cd65e-6165-91a3-d047-02000359156d/8a65c35bb78c4e06b47f80e1bc23ce71.vmdk\"","TraceId":"9043f77f-6106-431f-9a2a-f7420eed7648"}