Testing K8s persistent local volumes

README-persistent-local-volume-test.md

Testing local persistent volumes in kind

Reference:

• https://kubernetes.io/docs/concepts/storage/persistent-volumes
• https://kubernetes.io/docs/concepts/storage/volumes/#local
• https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner/tree/master/examples

Create Kind cluster

kind create cluster --name localvoltest --kubeconfig ./localvoltest.conf

If you open a new shell set KUBECONFIG, eg. export KUBECONFIG=$(pwd)/localvoltest.conf

Create persistent volume config then load it. Configure the path you want to use for the volume. I used /var/tmp. See manual-pv.yaml

kubectl apply -f manual-pv.yaml

Note: the hostname for affinity is matching what's in kind or it did not work for me

Create persistent volume claim config then apply it to the cluster. See simple-pvc.yaml

kubectl apply -f simple-pvc.yml

Start an pod which writes to a file in the persistent volume. To show more of how not to do things I'm starting a multiple replicas in my example which all write to the same file w/o properly dealing with lock files :) See #local-writer.yaml. Load with

kubectl apply -f local-writer.yaml

Start another pod to read from the file in the same volume. See local-reader.yaml

kubectl apply -f local-reader.yaml

Monitor the data being written to the local file via the local-reader pod logs:

kubectl logs local-test-reader-TEMP_INSTANCE_NAME --tail=10 --follow

Cleanup the volumes and pods

kubectl delete deploy local-writer
kubectl delete deploy local-reader
kubectl delete pvc example-local-claim
kubectl delete pvc example-local-pv

Delete the kind cluster: kind delete cluster --name localvoltest

local-reader.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: local-test-reader
spec:
  replicas: 1
  selector:
    matchLabels:
      app: local-test-reader
  template:
    metadata:
      labels:
        app: local-test-reader
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: reader
        image: k8s.gcr.io/busybox
        command:
        - "/bin/sh"
        args:
        - "-c"
        - "tail -f /usr/test-pod/test_file"
        volumeMounts:
        - name: local-vol
          mountPath: /usr/test-pod
      volumes:
      - name: local-vol
        persistentVolumeClaim:
          claimName: "example-local-claim"
          #          claimName: "local-vol-local-test-0"

local-writer.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: local-test-writer
spec:
  replicas: 3
  selector:
    matchLabels:
      app: local-test-writer
  template:
    metadata:
      labels:
        app: local-test-writer
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: writer
        image: k8s.gcr.io/busybox
        command:
        - "/bin/sh"
        args:
        - "-c"
        - "date +$(hostname)\\ %c >> /usr/test-pod/test_file"
        volumeMounts:
        - name: local-vol
          mountPath: /usr/test-pod
      volumes:
      - name: local-vol
        persistentVolumeClaim:
          claimName: "example-local-claim"
          #          claimName: "local-vol-local-test-0"

manual-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-local-pv
  labels:
    type: local
spec:
  capacity:
    storage: 5Gi 
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /var/tmp
    #path: /var/tmp/localvoltest
    #    path: /mnt/disks/ssd1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - localvoltest-control-plane
            #          - example-node
            #          - my-node

simple-pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: example-local-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi 
  storageClassName: local-storage

TODO:

• Testing with multiple nodes https://kind.sigs.k8s.io/docs/user/quick-start/#multinode-clusters
• Use something different for affinity?