Ceph-CSI-cephfs-StaticPV-v1.2.0+

---
apiVersion: v1
kind: PersistentVolume
metadata:
  # Can be anything, but has to be matched at line 47
  # Also should avoid conflicts with existing PV names in the namespace
  name: preprov-pv-cephfs-01
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 5Gi
  csi:
    driver: rook-ceph.cephfs.csi.ceph.com
    nodeStageSecretRef:
      name: rook-ceph-csi
      namespace: rook-ceph
    volumeAttributes:
      clusterID: rook-ceph
      fsName: myfs
      # The key "staticVolume" states this is pre-provisioned
      # NOTE: This was "preProvisionedVolume: "true"" in Ceph-CSI versions 1.0 and below
      staticVolume: "true"
      # Path of the PV on the CephFS filesystem
      rootPath: /staticpvs/pv-1
    # Can be anything, need not match PV name, or volumeName in PVC
    # Retained as the same for simplicity and uniquness
    volumeHandle: preprov-pv-cephfs-01
  # Reclaim policy must be "retain" as,
  # deletion of pre-provisioned volumes is not supported
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem
  claimRef:
    # Name should match "claimName" in PVC claim section
    name: csi-cephfs-pvc-preprov
    namespace: default
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-cephfs-pvc-preprov
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  volumeName: preprov-pv-cephfs-01
---
apiVersion: v1
kind: Pod
metadata:
  name: csicephfs-preprov-demo-pod
spec:
  containers:
    - image: busybox
      name: busybox
      command:
        - sleep
        - "3600"
      imagePullPolicy: IfNotPresent
      volumeMounts:
        - name: mypvc
          mountPath: /mnt
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: csi-cephfs-pvc-preprov

I'm trying to use this on an external cluster but can't make it work.
Must be a secret issue...
What exactlly is the bellow refering to?

nodeStageSecretRef:
name: rook-ceph-csi
namespace: rook-ceph

@KipourosV

I'm trying to use this on an external cluster but can't make it work.
Must be a secret issue...
What exactlly is the bellow refering to?

nodeStageSecretRef:
name: rook-ceph-csi
namespace: rook-ceph

You can create secret and reference it in the same file:

apiVersion: v1
kind: Secret
metadata:
  name: csi-cephfs-preprov-secret
  namespace: default
data:
  userID: YWRtaW4=
  userKey: QVFDNTBOQmN1c0hZR0JBQUptU0xtOUF3OThtQlJUNUZ5ZVMxL3c9PQ==

This is a default admin user and password used in the configs. You'll need to change reference to it in the PV section

@elinesterov

This is a default admin user and password used in the configs.

Can you explain in more details how this secret is used? Is it only scoped to the PV? When creating a separate secret file (csi-cephfs-preprov-secret as you suggested) and referencing it from the PV, is the secret then used by Kubernetes to authenticate against Ceph? If so, how does Ceph know that this admin password is valid? Does the userKey need to match the admin password inside the rook-ceph-admin-keyring secret?

When I attempted to mount cephfs inside Rook Toolbox pod (following these steps). It effectively used the username admin and the secret from rook-ceph-admin-keyring to perform the direct mount. And it worked.

However, when I attempted to mount the PV in a pod. The pod could not mount the PV:

MountVolume.MountDevice failed for volume "preprov-pv-cephfs-01" : rpc error: code = Internal desc = an error occurred while running (101) mount [-t ceph 10.43.33.196:6789,10.43.187.206:6789,10.43.149.215:6789:/staticpvs/pv-1 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/preprov-pv-cephfs-01/globalmount -o name=admin,secretfile=/tmp/csi/keys/keyfile-617805680,mds_namespace=testfs]: exit status 1

I am assuming this might have something to do with secretfile=/tmp/csi/keys/keyfile-617805680 not containing the right secret.

Indeed, manually changing the userKey inside csi-cephfs-preprov-secret to match the base64 value of the admin keyring fixed the PV mounting issue. However, this seems cumbersome: having to manually inspect one secret and updating another secret to match is very fragile. And for the record the decoded rook-ceph-admin-keyring looks like:

[client.admin]
        key = <the secret used to mount the cephfs volume>
        caps mds = "allow *"
        caps mon = "allow *"
        caps osd = "allow *"
        caps mgr = "allow *"

which means it cannot be directly referenced from nodeStageSecretRef.

I am definitely missing something -- how does this nodeStageSecretRef relate to the existing secrets that Rook generated when provisioning Ceph?

@ztl8702
I don't know in details about users and passwords. There is a section in doc related to this, but if you get to this point it works, otherwise you'll see the error related to credentials.
I see the same error and when I dig into logs (something similar to kubectl -n rook-ceph logs csi-cephfsplugin-kzp4v csi-cephfsplugin you need to change csi-cephfsplugin-kzp4v with the right pod for you.)

mount [-t ceph 10.96.215.114:6789:/volumes/csi/csi-vol-811e84a6-2b74-11ea-90fb-5a7b60d4d7c3/ /var/lib/kubelet/plugins/kubernetes.io/csi/pv/preprov-pv-cephfs-01/globalmount -o name=admin,secretfile=/tmp/csi/keys/keyfile-446845825,mds_namespace=myfs]

From what I see in log files - there is no directory /var/lib/kubelet/plugins/kubernetes.io/csi/pv/preprov-pv-cephfs-01/globalmount so there is no place to mount cephfs on the node.
I looked into the node when PV created dynamically - and there is some component which creates the directory for PV mount automatically.
So, I tries to create /var/lib/kubelet/plugins/kubernetes.io/csi/pv/preprov-pv-cephfs-01/globalmount dirs on the node manually, but it didn't help because there is something removing these directories.
I guess I need to create an issue for this and dig into corresponding components code, but this is my first few days looking into ceph, rook and cephfs-csi, so I don't know many details about architecture.
Feel free to create an issue, otherwise I'll create one tomorrow.

Indeed, manually changing the userKey inside csi-cephfs-preprov-secret to match the base64 value of the admin keyring fixed the PV mounting issue. However, this seems cumbersome: having to manually inspect one secret and updating another secret to match is very fragile. And for the record the decoded rook-ceph-admin-keyring looks like:

[client.admin]
        key = <the secret used to mount the cephfs volume>
        caps mds = "allow *"
        caps mon = "allow *"
        caps osd = "allow *"
        caps mgr = "allow *"

which means it cannot be directly referenced from nodeStageSecretRef.

I am definitely missing something -- how does this nodeStageSecretRef relate to the existing secrets that Rook generated when provisioning Ceph?

hm. Interesting! It still didn't work for me even with a right creds. I might have a different issue in my case.

There is another step that I did (which might or might not be necessary):

After creating the PV using this gist, I exec'ed into the rook-toolbox Pod, and mounted the CephFS filesystem root directly. And then I made a folder staticpvs/pv-1 inside the CephFS root to match rootPath in the PV manifest.

It still didn't work for me even with a right creds

Did you base64 encode the admin key when you modify the secret?

---

BTW, I just realised, since I am using Rook, that I should (conveniently) be using rook-csi-cephfs-provisioner and rook-csi-cephfs-node (both are created by Rook in the rook-ceph namespace) as provisionerSecretRef and nodeStageSecretRef respectively.

BTW, I just realised, since I am using Rook, that I should (conveniently) be using rook-csi-cephfs-provisioner and rook-csi-cephfs-node (both are created by Rook in the rook-ceph namespace) as provisionerSecretRef and nodeStageSecretRef respectively.

The rook created secrets will not have the keys userID and userKey. So reusing the rook created secret as is for static PVs is not feasible.

The secrets being discussed here need to be of the form: https://gist.github.com/ShyamsundarR/1009b79966992470f588a277429ae2e1#file-pod-with-static-pv-yaml-L1-L10

IOW, a user key and value needs to be specified, in exactly the same format as detailed here: https://github.com/ceph/ceph-csi/blob/3e656769b71a3c43d95f6875ed4934c82a8046e7/examples/cephfs/secret.yaml#L7-L10

If using stringData then base-64 encoding the userKey is not required. (if using data instead, then base-64 encoding the same is required)

I just realized that I was using wrong secret (something crewed up when I switched between different deployments). I just put correct one and everything seems working fine for me.

The intention of a static PV is that, the path mentioned as rootPath (here) already exists, and the secret created to access this path is a ceph userID/key with enough credentials to mount and perform required IO on the mount.

So, if using a static PV, the rootPath should be already created against the backing CephFS volume.

Based on the discussions here so far by @elinesterov and @ztl8702 the above needs to be clarified as having been done, and if so please pass on the csi-cephfsplugin logs from the ceph-csi nodeplugin daemonset logs, on the node where the PV is attempted to mounted.

@ShyamsundarR Thanks for the clarification. Manually creating a secret that matches the form using the client.admin key from ceph auth ls worked for me. Do you still want my logs?

I am just wondering if there is a way to more declaratively deploy a CephFS shared filesystem + mount a volume to a Pod that gives access to at a fixed path within cephfs, with ceph-csi. The extra step to get / create a client key in Ceph and then create the correct secret in Kubernetes API bugs me a little bit.
I believe this was previously possible with flexVolume or with provisionVolume: false?

@ShyamsundarR Thanks for the clarification. Manually creating a secret that matches the form using the client.admin key from ceph auth ls worked for me. Do you still want my logs?

No, thank you.

I am just wondering if there is a way to more declaratively deploy a CephFS shared filesystem + mount a volume to a Pod that gives access to at a fixed path within cephfs, with ceph-csi. The extra step to get / create a client key in Ceph and then create the correct secret in Kubernetes API bugs me a little bit.

The step to create a client key and a secret is a choice for security conscious setups. You could use a single secret across all static PVs that are created. This secret can be in any namespace, but the namespace needs to be reflected in here.

The above reduces it to a single secret creation step.

I believe this was previously possible with flexVolume or with provisionVolume: false?

I am not aware of how flexVolume worked, so not commenting on the same.

With provisionVolume: false the method of using static PVs was to request for one dynamically, and the provisioner detecting it as a pre-provisioned PV and hence using the same secret as that used by the CSI plugins. The entire provisioning step (and hence de-provisioning) was superfluous, and the intention was to go with kubernetes based static PV definitions instead.