In this guide I will be showing you how to use QuantaStor as a persistent volume provider for Kubernetes on IBM Cloud.
The IBM Cloud Kubernetes service (IKS) has several options for persistent storage but they can vary greatly depending on the type of cluster that you've provisioned. Within IBM Cloud you can deploy Kubernetes Singlezone clusters or Multizone clusters.
For single zone clusters (a cluster whose worker nodes all reside in the same Datacenter) the following options are supported out of the box:
For multizone clusters (a cluster whose worker nodes span multiple Datacenters in the same region) your options are a little more limited. Both the IBM Cloud File and Block offerings are restricted to the Datacenters in which they are ordered. So if you have a File volume provisioned in Dallas 13, your servers and IKS worker nodes in say Dallas 10 cannot actually interact with that storage. For Multizone clusters the following options are supported out of the box:
- IBM Cloud Object Storage
- Portworx
- IBM Cloud Databases - These are fully managed IBM Cloud hosted databases:
Portworx is a great option for supporing Multizone persistent storage but you have to also run an instance of Etcd and it is best used with Bare Metal servers which would increase the cost of your cluster and would likely be overkill for smaller dev/test environment.
As an alternative we'll set up a QuantaStor server to act as an NFS Persistent Volume storage provider. Since the storage is from the local Quantastor disks and not the IBM Cloud the volumes can span multiple Datacenters.
QuantaStor Software Defined Storage supports all major file, block, and object protocols including iSCSI/FC, NFS/SMB, and S3. It features the ability to enable encryption at rest as well as in flight.
- Kubernetes cluster
- QuantaStor server. You can download the Community edition for free. QuantaStor Community Edition keys are capacity limited to 40TB of raw capacity and 4x servers per storage grid.
- kubectl installed. Guide here.
- slcli installed. Guide here.
- helm installed on your cluster. Guide here
Here are the steps we will be going through in this guide:
- Use the slcli to get the IKS Worker Node IPs
- Create Storage Pool in QuantaStor
- Create QuantaStor Network Share
- Enable NFS access from IKS Worker node IPs
- Deploy NFS provisioner Helm chart
- Deploy example pod and write to mounted persistent volume
We can gather our worker node IPs using kubectl
and then use the slcli
to get the subnet range that will be whitelisted for our NFS share.
$ kubectl get nodes -o json | jq -r '.items[].metadata.name'
10.184.249.15
10.184.249.2
10.184.249.34
10.221.28.10
10.221.28.37
10.221.28.5
$ slcli --format json subnet lookup 10.184.249.15 | jq -r '.subnet.identifier'
10.184.249.0/26
$ slcli --format json subnet lookup 10.221.28.37 | jq -r '.subnet.identifier'
10.221.28.0/25
Enter the nfs-client-provisioner helm chart
$ $ helm install stable/nfs-client-provisioner --set nfs.server=10.171.145.12 --set nfs.path=/export/PVShare --name nfs-client
NAME: nfs-client
LAST DEPLOYED: Fri Jan 10 09:09:10 2020
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/ClusterRole
NAME AGE
nfs-client-nfs-client-provisioner-runner 1s
==> v1/ClusterRoleBinding
NAME AGE
run-nfs-client-nfs-client-provisioner 1s
==> v1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
nfs-client-nfs-client-provisioner 0/1 0 0 1s
==> v1/Role
NAME AGE
leader-locking-nfs-client-nfs-client-provisioner 1s
==> v1/RoleBinding
NAME AGE
leader-locking-nfs-client-nfs-client-provisioner 1s
==> v1/ServiceAccount
NAME SECRETS AGE
nfs-client-nfs-client-provisioner 1 2s
==> v1/StorageClass
NAME PROVISIONER AGE
nfs-client cluster.local/nfs-client-nfs-client-provisioner 2s
Check that NFS is available as a storageclass.
$ kubectl get storageclasses | grep nfs-client
nfs-client cluster.local/nfs-client-nfs-client-provisioner 104s
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: qspvc
spec:
storageClassName: nfs-client
accessModes:
- ReadWriteMany
resources:
requests:
storage: 2Gi
$ kubectl create -f qs-test-pvc.yml
persistentvolumeclaim/qspvc created
$ kubectl get persistentvolumeclaim/qspvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
qspvc Bound pvc-3602d021-da6d-466e-8440-c79653339ee5 2Gi RWX nfs-client 14s
kind: Pod
apiVersion: v1
metadata:
name: pod-using-nfs
spec:
volumes:
- name: nfs-volume
persistentVolumeClaim:
claimName: qspvc
containers:
- name: apptest
image: alpine
volumeMounts:
- name: nfs-volume
mountPath: /var/nfs
# Write to a file inside our NFS
command: ["/bin/sh"]
args: ["-c", "while true; do date >> /var/nfs/dates.txt; sleep 5; done"]
$ kubectl create -f nfs-test-pod.yml
pod/pod-using-nfs created
$ kubectl exec -it pod/pod-using-nfs -- /bin/sh
/ # ls -l /var/nfs/dates.txt
-rw-rw-rw- 1 nobody nobody 261 Jan 10 15:24 /var/nfs/dates.txt
/ # cat /var/nfs/dates.txt
Fri Jan 10 15:23:49 UTC 2020
Fri Jan 10 15:23:54 UTC 2020
Fri Jan 10 15:23:59 UTC 2020
Fri Jan 10 15:24:04 UTC 2020
Fri Jan 10 15:24:09 UTC 2020
Fri Jan 10 15:24:14 UTC 2020
Fri Jan 10 15:24:19 UTC 2020
Fri Jan 10 15:24:24 UTC 2020
Fri Jan 10 15:24:29 UTC 2020
Fri Jan 10 15:24:34 UTC 2020