Migration of Data from one PVC to another

destination-template.yaml

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: NAME
  namespace: NS
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: SIZE
  storageClassName: csi-ceph-dc1-standard 

migration-template.yaml

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: migration-NAME
  namespace: NS
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: SIZE
  storageClassName: csi-ceph-dc1-standard 

pod-backup-template.yaml

apiVersion: v1
kind: Pod
metadata:
  name: migration-backup
  namespace: NS
spec:
  containers:
  - name: migration-backup
    image: busybox
    args:
    - sleep
    - "1000000"
    volumeMounts:
      - name: source
        mountPath: /data-source
      - name: destination
        mountPath: /data-destination
  volumes:
    - name: source
      persistentVolumeClaim:
        claimName: SOURCEPVC
        readOnly: false
    - name: destination  
      persistentVolumeClaim:
        claimName: migration-SOURCEPVC
        readOnly: false

pod-restore-template.yaml

apiVersion: v1
kind: Pod
metadata:
  name: migration-restore
  namespace: NS
spec:
  containers:
  - name: migration-restore
    image: busybox
    args:
    - sleep
    - "1000000"
    volumeMounts:
      - name: source
        mountPath: /data-source
      - name: destination
        mountPath: /data-destination
  volumes:
    - name: source
      persistentVolumeClaim:
        claimName: migration-SOURCEPVC
        readOnly: false
    - name: destination  
      persistentVolumeClaim:
        claimName: SOURCEPVC
        readOnly: false

What does it do ?

Helper manifests and snippets for copying data from one storage-class to another but "keep" the PVC definition (except storageclass)

How does it work

All the follwing steps are done manually. It should be possible to do it scripted, and have the pod copy the stuff automatically. Pods could be run as kubernetes jobs. Currently "works" with only one PVC per application.

• Scale down Application
• create temporary PVC
• create backup pod, that mounts source-pvc & temp-pvc
• copy data inside pod from source to destination
• stop & delete backup pod
• set persistentVolumeReclaimPolicy for source-olume to Retain
• delete source PVC
• recreate "source" PVC with new storage-class and old labels
• create restore-pod that mounts temp-pvc as source, new-pvc as destination
• copy data inside pod from source to destination
• stop & delete restore pod
• Scale up Application
• TEST
• Delete migration PVC and old PV

Requirements

• Access to K8S-cluster with kubectl.
• JQ installed
• Permissions to alter/delete PVs

Manual ;-)

Heavy copy & paste action ;-)

Init

NS=NS-NAME
APPTYPE=deployment|statefulset
APP=APPNAME
SOURCEPVC=Name of source-pvc

Start

PVC_LABEL=
SIZE=$(kubectl -n $NS get pvc $SOURCEPVC -o json | jq -r .spec.resources.requests.storage)
The following line gives an error if no labels are present on PVC. No problem
PVC_LABEL=$(kubectl -n $NS get pvc $SOURCEPVC -o json | jq -r '.metadata.labels| del(.chart)|to_entries|map("(.key)=(.value|tostring)")|join(" ")')

replicas=$(kubectl get $APPTYPE $APP -n $NS -o json | jq '.spec.replicas')
kubectl label $APPTYPE $APP -n $NS replicabackup=$replicas --overwrite
kubectl scale $APPTYPE $APP -n $NS --replicas=0

** WAIT until Pods are stopped **

sed "s/NAME/${SOURCEPVC}/g;s/NS/${NS}/g;s/SIZE/${SIZE}/g" migration-template.yaml | kubectl apply -f -
sed "s/NS/${NS}/g;s/SOURCEPVC/${SOURCEPVC}/g" pod-backup-template.yaml | kubectl apply -f -

** WAIT until migration POD has been created **

kubectl -n $NS exec -ti migration-backup /bin/sh
cp -rfpT /data-source/ /data-destination/
exit

kubectl -n $NS delete pod migration-backup

SOURCEPV=$(kubectl get pvc -n $NS $SOURCEPVC -o json | jq -r '.spec.volumeName')
kubectl patch pv $SOURCEPV -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
kubectl label pv $SOURCEPV csimigration=yes $PVC_LABEL --overwrite

kubectl -n $NS delete pvc $SOURCEPVC
kubectl patch pv $SOURCEPV -p '{"spec":{"claimRef":null}}'
sed "s/NAME/${SOURCEPVC}/g;s/NS/${NS}/g;s/SIZE/${SIZE}/g" destination-template.yaml | kubectl apply -f -

The following line gives an error if no label are present. no problem

kubectl -n $NS label pvc $SOURCEPVC $PVC_LABEL --overwrite

sed "s/NS/${NS}/g;s/SOURCEPVC/${SOURCEPVC}/g" pod-restore-template.yaml | kubectl apply -f -

WAIT until migration POD has been created

kubectl -n $NS exec -ti migration-restore /bin/sh
cp -rfpT /data-source/ /data-destination/
exit

kubectl -n $NS delete pod migration-restore

WAIT until migration POD has been deleted

replicas_new=$(kubectl get $APPTYPE -n $NS $APP -o json | jq -r '.metadata.labels.replicabackup')
kubectl scale $APPTYPE $APP -n $NS --replicas=$replicas_new

TESTS

kubectl -n $NS delete pvc migration-$SOURCEPVC
kubectl patch pv $SOURCEPV -p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'
kubectl delete pv

Very cool. I've used it for 4 PVCs and it worked great.
But, with jq-1.6,

# echo $PVC_LABEL
(.key)=(.value|tostring) (.key)=(.value|tostring) (.key)=(.value|tostring)

I had to set PVC_LABEL by hand.

Good to hear that it still works ;-) I created an ansible role for those migration jobs a couple of months ago that automates pvc-migration, so we don't need those scripts any more.

@iceman91176 Why do an extra step with a temp pvc instead of directly copy all the data to target pvc?

@FedeMarchiniHotovo The target-PVC is essentially the same as the source PVC - same name, same labels, etc. So you cannot create it as long as the source-pvc exists.

I see, thanks !