OpenEBS w/ Velero wrapper scripts

openebs-label-volumes.sh

#!/usr/bin/env ruby

require 'json'

pvcs = JSON.parse(`kubectl get pvc --all-namespaces -o json`)["items"]

pvcs.each do |pvc|
  namespace = pvc["metadata"]["namespace"]
  pv = pvc["spec"]["volumeName"]

  `kubectl label pv #{pv} openebs.io/namespace=#{namespace} --overwrite`
end

openebs-set-target-ip-all.sh

# !/bin/bash

pod_name=`kubectl -n openebs get pod -l app=cstor-pool -o jsonpath='{.items[0].metadata.name}'`
pool_name=`kubectl -n openebs exec -it $pod_name -c cstor-pool -- zpool list -Ho name`
pool_name=${pool_name//[$'\t\r\n']}

for pv in `kubectl get pv -o name`; do
  pv_name=`echo $pv | cut -d '/' -f 2`
  target_ip=`kubectl get $pv -o jsonpath='{.spec.iscsi.targetPortal}' | cut -d ':' -f 1`
  target_name="$pool_name/$pv_name"

  kubectl -n openebs exec -it $pod_name -c cstor-pool -- zfs set io.openebs:targetip=$target_ip $target_name
done

velero-backup.sh

# !/bin/bash

backup=$1
namespaces=$2

clear

openebs-label-volumes

time velero backup create $backup --include-namespaces $namespaces --exclude-resources=orders.certmanager.k8s.io,challenges.certmanager.k8s.io --snapshot-volumes --volume-snapshot-locations=default --wait

velero-restore-scheduled.sh

# !/bin/bash

schedule_name=$1
include_namespaces=$2
up_to_backup=$3

backups_array=( $(kubectl -n velero get backup --sort-by=.status.completionTimestamp | grep "$schedule_name-" | cut -d ' ' -f 1) )
first_backup=${backups_array[0]}
backup_namespaces_array=( $(kubectl -n velero get backup $first_backup -o json | jq -r '.spec.includedNamespaces | join(" ")') )
include_namespaces_array=( $(echo $include_namespaces | sed "s/,/ /g") )
exclude_namespaces_array=()

for i in ${backup_namespaces_array[@]}; do
  skip=

  for j in ${include_namespaces_array[@]}; do
    [[ $i == $j ]] && { skip=1; break; }
  done

  [[ -n $skip ]] || exclude_namespaces_array+=("$i")
done

exclude_namespaces=$(printf ",%s" "${exclude_namespaces_array[@]}")
exclude_namespaces=${exclude_namespaces:1}

for namespace in ${include_namespaces_array[@]}; do
  kubectl create namespace $namespace
done

clear

for backup in $backups_array; do
	time velero restore create --from-backup $backup --include-namespaces $include_namespaces --selector "openebs.io/namespace notin ($exclude_namespaces)" --restore-volumes=true --wait

  if [ "$backup" == "$up_to_backup" ]; then
    break
  fi
done

openebs-set-target-ip-all

velero-restore.sh

# !/bin/bash

backup=$1
include_namespaces=$2

backup_namespaces_array=( $(kubectl -n velero get backup all4 -o json | jq -r '.spec.includedNamespaces | join(" ")') )
include_namespaces_array=( $(echo $include_namespaces | sed "s/,/ /g") )
exclude_namespaces_array=()

for i in ${backup_namespaces_array[@]}; do
  skip=

  for j in ${include_namespaces_array[@]}; do
    [[ $i == $j ]] && { skip=1; break; }
  done

  [[ -n $skip ]] || exclude_namespaces_array+=("$i")
done

exclude_namespaces=$(printf ",%s" "${exclude_namespaces_array[@]}")
exclude_namespaces=${exclude_namespaces:1}

for namespace in ${include_namespaces_array[@]}; do
  kubectl create namespace $namespace
done

clear

time velero restore create --from-backup $backup --include-namespaces $include_namespaces --selector "openebs.io/namespace notin ($exclude_namespaces)" --restore-volumes=true --wait $command

openebs-set-target-ip-all

velero-schedule.sh

# !/bin/bash

name=$1
schedule=$2
namespaces=$3

openebs-label-volumes

velero schedule create $name --schedule="$schedule" --include-namespaces $namespaces --exclude-resources=orders.certmanager.k8s.io,challenges.certmanager.k8s.io --snapshot-volumes --volume-snapshot-locations=default

These are just a bunch of scripts to more easily use OpenEBS's Velero plugin to include cStor volume snapshots in Velero backups. See https://github.com/openebs/velero-plugin for reference.

I use these scripts so I don't have to remember all the parameters and additional steps to correctly back up and restore namespaces that include OpenEBS cStor volumes.

Notes

• Apparently when restoring the --include-namespaces parameter is kinda ignored for volumes, while it works for everything else in the namespace. I was told by the OpenEBS devs that in order to be able restore only volumes for the selected namespace(s) I have to apply some label to the PVs before backing up and then use a selector during the restore. Therefore the backup and schedule scripts call another script that adds a label to PVs in the format openebs.io/namespace=#{namespace} before backing up/scheduling. The restore script then figures out which namespaces to ignore (that is, all the namespaces included in the backup minus the namespaces to include in the restore) and uses a notin selector with the relevant labels to prevent restoring volumes in those namespaces;

• The backup and schedule scripts exclude a couple cert-manager related resources that otherwise will cause restores to fail. This doesn't affect the restore of certificates etc which will work just fine after restoring;

• The restore scripts set the target ip of the volumes after restoring, as described in the link above;

• There are two separate scripts to restore either a full backup taken manually, or an incremental backup chain taken with a scheduled backup. These could be merged to avoid duplication.

Usage

Backup

velero-backup <backup name> <csv list of namespaces to back up>

Scheduling a backup

velero-schedule <schedule name> <csv list of namespaces to back up>

Restore of a full (manual) backup

velero-restore <backup name> <namespaces to restore>

Restore of a schedule incremental backup chain

velero-restore-scheduled <schedule name> <namespaces to restore> <*optional*: name of the last backup to restore>

The third parameter is to restore an incremental backup chain up to a specific backup, ignoring the following backups.