Skip to content

Instantly share code, notes, and snippets.

@singlecheeze

singlecheeze/OCP Storage democratic-csi.md

Last active October 17, 2022 15:05
Show Gist options
  • Save singlecheeze/001edf6c2e0ae7ed2b34c64749ecb3c9 to your computer and use it in GitHub Desktop.
Save singlecheeze/001edf6c2e0ae7ed2b34c64749ecb3c9 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
OCP Storage democratic-csi.md

TrueNAS Setup
Helpful Links:
https://www.truenas.com/docs/solutions/integrations/containers/
https://www.lisenet.com/2021/moving-to-truenas-and-democratic-csi-for-kubernetes-persistent-storage/
https://jonathangazeley.com/2021/01/05/using-truenas-to-provide-persistent-storage-for-kubernetes/

NFS
image image image

iSCSI
image image image image image

Helm Command
NOTE: Must do this first oc login https://api.ocp4.lab.localdomain:6443 -u dave

$ helm repo add democratic-csi https://democratic-csi.github.io/charts/
$ helm repo update
$ helm search repo democratic-csi/

NOTE: Required in helm command

--set node.rbac.openshift.privileged=true
--set node.driver.localtimeHostPath=false

NOTE: Unlikely, but in special circumstances may be required in helm command

--set controller.rbac.openshift.privileged=true

NFS

helm upgrade \
  --install \
  --create-namespace \
  --values freenas-nfs.yaml \
  --namespace democratic-csi \
  --set node.rbac.openshift.privileged=true \
  --set node.driver.localtimeHostPath=false \
  --set controller.rbac.openshift.privileged=true \
  zfs-nfs democratic-csi/democratic-csi

freenas-nfs.yaml

csiDriver:
  name: "org.democratic-csi.nfs"

storageClasses:
- name: freenas-nfs-csi
  defaultClass: false
  reclaimPolicy: Retain
  volumeBindingMode: Immediate
  allowVolumeExpansion: true
  parameters:
    fsType: nfs
  mountOptions:
  - noatime
  - nfsvers=4
  secrets:
    provisioner-secret:
    controller-publish-secret:
    node-stage-secret:
    node-publish-secret:
    controller-expand-secret:

driver:
  config:
    driver: freenas-nfs
    instance_id:
    httpConnection:
      protocol: http
      host: 172.16.1.119
      port: 80
      username: root
      password: **********
      allowInsecure: true
      apiVersion: 2
    sshConnection:
      host: 172.16.1.119
      port: 22
      username: root
      password: **********
    zfs:
      # Make sure to use the storage pool that was created previously
      datasetParentName: tank/k8s/nfs/vols
      detachedSnapshotsDatasetParentName: tank/k8s/nfs/snaps
      datasetEnableQuotas: true
      datasetEnableReservation: false
      datasetPermissionsMode: "0777"
      datasetPermissionsUser: 0
      datasetPermissionsGroup: 0
    nfs:
      shareHost: 172.16.1.119
      shareAlldirs: false
      shareAllowedHosts: []
      shareAllowedNetworks: []
      shareMaprootUser: root
      shareMaprootGroup: wheel
      shareMapallUser: ""
      shareMapallGroup: ""

Confirm Pods Are Running

[dave@lenovo ~]$ oc get pods -n democratic-csi
NAME                                                READY   STATUS    RESTARTS   AGE
zfs-nfs-democratic-csi-controller-dfd6d6756-j8m2v   5/5     Running   0          9m59s
zfs-nfs-democratic-csi-node-6c4v2                   4/4     Running   0          9m41s
zfs-nfs-democratic-csi-node-g9wqs                   4/4     Running   0          7m59s
zfs-nfs-democratic-csi-node-l5qbl                   4/4     Running   0          9m7s
zfs-nfs-democratic-csi-node-lx9nx                   4/4     Running   0          7m25s
zfs-nfs-democratic-csi-node-xfffl                   4/4     Running   0          8m33s

Confirm StorageClass is Present

[dave@lenovo ~]$ oc get sc
NAME                                    PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
freenas-nfs-csi                         org.democratic-csi.nfs                  Retain          Immediate              true                   17h

NOTE: If OpenShift Virtualization is in use edit StorageProfile spec under the openshift-cnv namespace

spec:
  claimPropertySets:
  - accessModes:
    - ReadWriteMany
    volumeMode: Filesystem

NFS PersistentVolumeClaim
image
image
image
image

iSCSI

helm upgrade \
  --install \
  --create-namespace \
  --values freenas-iscsi.yaml \
  --namespace democratic-csi \
  --set node.rbac.openshift.privileged=true \
  --set node.driver.localtimeHostPath=false \
  --set controller.rbac.openshift.privileged=true \
  zfs-iscsi democratic-csi/democratic-csi

freenas-iscsi.yaml

csiDriver:
  name: "org.democratic-csi.iscsi"

storageClasses:
- name: freenas-iscsi-csi
  defaultClass: false
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
  allowVolumeExpansion: true
  parameters:
    # for block-based storage can be ext3, ext4, xfs
    fsType: xfs
  mountOptions: []
  secrets:
    provisioner-secret:
    controller-publish-secret:
    node-stage-secret:
    node-publish-secret:
    controller-expand-secret:


driver:
  config:
    driver: freenas-iscsi
    instance_id:
    httpConnection:
      protocol: http
      host: 172.16.1.119
      port: 80
      username: root
      password: **********
      allowInsecure: true
      apiVersion: 2
    sshConnection:
      host: 172.16.1.119
      port: 22
      username: root
      password: **********
    zfs:
      cli:
        paths:
          zfs: /usr/local/sbin/zfs
          zpool: /usr/local/sbin/zpool
          sudo: /usr/local/bin/sudo
          chroot: /usr/sbin/chroot
      # total volume name (zvol/<datasetParentName>/<pvc name>) length cannot exceed 63 chars
      # https://www.ixsystems.com/documentation/freenas/11.2-U5/storage.html#zfs-zvol-config-opts-tab
      # standard volume naming overhead is 46 chars
      # datasetParentName should therefore be 17 chars or less
      datasetParentName: tank/k8s/iscsi/v
      detachedSnapshotsDatasetParentName: tank/k8s/iscsi/s
      # "" (inherit), lz4, gzip-9, etc
      zvolCompression:
      # "" (inherit), on, off, verify
      zvolDedup:
      zvolEnableReservation: false
      # 512, 1K, 2K, 4K, 8K, 16K, 64K, 128K default is 16K
      zvolBlocksize:
    iscsi:
      targetPortal: "172.16.1.119:3260"
      targetPortals: []
      # leave empty to omit usage of -I with iscsiadm
      interface:
      namePrefix: csi-
      nameSuffix: "-cluster"
      # add as many as needed
      targetGroups:
        # get the correct ID from the "portal" section in the UI
        - targetGroupPortalGroup: 1
          # get the correct ID from the "initiators" section in the UI
          targetGroupInitiatorGroup: 1
          # None, CHAP, or CHAP Mutual
          targetGroupAuthType: None
          # get the correct ID from the "Authorized Access" section of the UI
          # only required if using Chap
          targetGroupAuthGroup:
      extentInsecureTpc: true
      extentXenCompat: false
      extentDisablePhysicalBlocksize: true
      # 512, 1024, 2048, or 4096,
      extentBlocksize: 4096
      # "" (let FreeNAS decide, currently defaults to SSD), Unknown, SSD, 5400, 7200, 10000, 15000
      extentRpm: "SSD"
      # 0-100 (0 == ignore)
      extentAvailThreshold: 0

Confirm Pods Are Running

[dave@lenovo ~]$ oc get pods -n democratic-csi
NAME                                                   READY   STATUS    RESTARTS   AGE
zfs-iscsi-democratic-csi-controller-5c78d6f867-5l8nc   5/5     Running   0          2m15s
zfs-iscsi-democratic-csi-node-4htsd                    4/4     Running   0          2m15s
zfs-iscsi-democratic-csi-node-7m2dh                    4/4     Running   0          2m15s
zfs-iscsi-democratic-csi-node-gbv4g                    4/4     Running   0          2m15s
zfs-iscsi-democratic-csi-node-w4mz2                    4/4     Running   0          2m15s
zfs-iscsi-democratic-csi-node-xb7n5                    4/4     Running   0          2m15s

Confirm StorageClass is Present

[dave@lenovo ~]$ oc get sc
NAME                                    PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
freenas-iscsi-csi                       org.democratic-csi.iscsi                Delete          Immediate              true                   3m4s

NOTE: If OpenShift Virtualization is in use edit StorageProfile spec under the openshift-cnv namespace

spec:
  claimPropertySets:
  - accessModes:
    - ReadWriteOnce
    volumeMode: Block

iSCSI PersistentVolumeClaim
image
image
image
image
image

Clean Up Test PVCs
image

Benchmarks
OpenShift Data Foundation

  • 3x R730xd with
    • 1x Optane 900P 480GB
    • 48x Intel Xeon CPU E5-2680 v3 @ 2.50GHz

image

TrueNAS

  • Initiator: 1x R730xd with
    • 48x Intel Xeon CPU E5-2680 v3 @ 2.50GHz
  • Target: 1x R730xd with
    • 24x SAMSUNG MZILS1T9HEJH0D3 2TB in RAID10 over iSCSI
      • compression: lz4
      • sync: disabled
      • pool record size: 128K
      • zvolBlocksize: 16K
      • extentBlocksize: 4096
      • ntfs on xfs on zfs
    • 88x Intel Xeon CPU E5-2699 v4 @ 2.20GHz
      image image
    • 16x Intel Xeon CPU E5-2637 v4 @ 3.50GHz image image

Troubleshooting:
iSCSI Portal Login:
https://www.reddit.com/r/truenas/comments/td5pu4/worker_nodes_not_getting_iscsi_targets_from/
image

SSH into worker nodes (With ssh key file, add -i):

[dave@lenovo ~]$ ssh core@172.16.1.111

Manually run iSCSI discovery (172.16.1.119 is the TrueNAS IP):

[core@trt2ocp1 ~]$ sudo iscsiadm -m discovery -t sendtargets -p 172.16.1.119
172.16.1.119:3260,1 iqn.2005-10.org.freenas.ctl:csi-pvc-5757bd91-d44c-40bb-8e7f-8b2ff4a862f0-cluster

UEFI Windows VMs:
NOTE: While importing, make sure Secure Boot is disabled on boot in OpenShift, even if it was disabled in VMware VM configuration, and Migration Toolkit saw it as present:
image image image image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment