-
-
Save ShyamsundarR/f16d32e3edd5b38df50e90106674a943 to your computer and use it in GitHub Desktop.
--- | |
apiVersion: v1 | |
kind: PersistentVolume | |
metadata: | |
# Can be anything, but has to be matched at line 47 | |
# Also should avoid conflicts with existing PV names in the namespace | |
name: preprov-pv-cephfs-01 | |
spec: | |
accessModes: | |
- ReadWriteMany | |
capacity: | |
storage: 5Gi | |
csi: | |
driver: rook-ceph.cephfs.csi.ceph.com | |
nodeStageSecretRef: | |
name: rook-ceph-csi | |
namespace: rook-ceph | |
volumeAttributes: | |
clusterID: rook-ceph | |
fsName: myfs | |
# The key "staticVolume" states this is pre-provisioned | |
# NOTE: This was "preProvisionedVolume: "true"" in Ceph-CSI versions 1.0 and below | |
staticVolume: "true" | |
# Path of the PV on the CephFS filesystem | |
rootPath: /staticpvs/pv-1 | |
# Can be anything, need not match PV name, or volumeName in PVC | |
# Retained as the same for simplicity and uniquness | |
volumeHandle: preprov-pv-cephfs-01 | |
# Reclaim policy must be "retain" as, | |
# deletion of pre-provisioned volumes is not supported | |
persistentVolumeReclaimPolicy: Retain | |
volumeMode: Filesystem | |
claimRef: | |
# Name should match "claimName" in PVC claim section | |
name: csi-cephfs-pvc-preprov | |
namespace: default | |
--- | |
apiVersion: v1 | |
kind: PersistentVolumeClaim | |
metadata: | |
name: csi-cephfs-pvc-preprov | |
spec: | |
accessModes: | |
- ReadWriteMany | |
resources: | |
requests: | |
storage: 5Gi | |
volumeName: preprov-pv-cephfs-01 | |
--- | |
apiVersion: v1 | |
kind: Pod | |
metadata: | |
name: csicephfs-preprov-demo-pod | |
spec: | |
containers: | |
- image: busybox | |
name: busybox | |
command: | |
- sleep | |
- "3600" | |
imagePullPolicy: IfNotPresent | |
volumeMounts: | |
- name: mypvc | |
mountPath: /mnt | |
volumes: | |
- name: mypvc | |
persistentVolumeClaim: | |
claimName: csi-cephfs-pvc-preprov |
I just realized that I was using wrong secret (something crewed up when I switched between different deployments). I just put correct one and everything seems working fine for me.
The intention of a static PV is that, the path mentioned as rootPath
(here) already exists, and the secret created to access this path is a ceph userID/key with enough credentials to mount and perform required IO on the mount.
So, if using a static PV, the rootPath
should be already created against the backing CephFS volume.
Based on the discussions here so far by @elinesterov and @ztl8702 the above needs to be clarified as having been done, and if so please pass on the csi-cephfsplugin logs from the ceph-csi nodeplugin daemonset logs, on the node where the PV is attempted to mounted.
@ShyamsundarR Thanks for the clarification. Manually creating a secret that matches the form using the client.admin
key from ceph auth ls
worked for me. Do you still want my logs?
I am just wondering if there is a way to more declaratively deploy a CephFS shared filesystem + mount a volume to a Pod that gives access to at a fixed path within cephfs, with ceph-csi. The extra step to get / create a client key in Ceph and then create the correct secret in Kubernetes API bugs me a little bit.
I believe this was previously possible with flexVolume or with provisionVolume: false
?
@ShyamsundarR Thanks for the clarification. Manually creating a secret that matches the form using the
client.admin
key fromceph auth ls
worked for me. Do you still want my logs?
No, thank you.
I am just wondering if there is a way to more declaratively deploy a CephFS shared filesystem + mount a volume to a Pod that gives access to at a fixed path within cephfs, with ceph-csi. The extra step to get / create a client key in Ceph and then create the correct secret in Kubernetes API bugs me a little bit.
The step to create a client key and a secret is a choice for security conscious setups. You could use a single secret across all static PVs that are created. This secret can be in any namespace, but the namespace needs to be reflected in here.
The above reduces it to a single secret creation step.
I believe this was previously possible with flexVolume or with
provisionVolume: false
?
I am not aware of how flexVolume worked, so not commenting on the same.
With provisionVolume: false
the method of using static PVs was to request for one dynamically, and the provisioner detecting it as a pre-provisioned PV and hence using the same secret as that used by the CSI plugins. The entire provisioning step (and hence de-provisioning) was superfluous, and the intention was to go with kubernetes based static PV definitions instead.
The secrets being discussed here need to be of the form: https://gist.github.com/ShyamsundarR/1009b79966992470f588a277429ae2e1#file-pod-with-static-pv-yaml-L1-L10
IOW, a user key and value needs to be specified, in exactly the same format as detailed here: https://github.com/ceph/ceph-csi/blob/3e656769b71a3c43d95f6875ed4934c82a8046e7/examples/cephfs/secret.yaml#L7-L10
If using stringData then base-64 encoding the
userKey
is not required. (if usingdata
instead, then base-64 encoding the same is required)