Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
I use this script to backup my QEMU/KVM/libVirt virtual machines. The script requires KVM 2.1+ since it uses the live blockcommit mode. This means the data in the snapshot disk is rolled back into the original instead of the other way around. Script does NOT handle spaces in paths.
#!/bin/bash
#
BACKUPDEST="$1"
DOMAIN="$2"
MAXBACKUPS="$3"
if [ -z "$BACKUPDEST" -o -z "$DOMAIN" ]; then
echo "Usage: ./vm-backup <backup-folder> <domain> [max-backups]"
exit 1
fi
if [ -z "$MAXBACKUPS" ]; then
MAXBACKUPS=6
fi
echo "Beginning backup for $DOMAIN"
#
# Generate the backup path
#
BACKUPDATE=`date "+%Y-%m-%d.%H%M%S"`
BACKUPDOMAIN="$BACKUPDEST/$DOMAIN"
BACKUP="$BACKUPDOMAIN/$BACKUPDATE"
mkdir -p "$BACKUP"
#
# Get the list of targets (disks) and the image paths.
#
TARGETS=`virsh domblklist "$DOMAIN" --details | grep ^file | awk '{print $3}'`
IMAGES=`virsh domblklist "$DOMAIN" --details | grep ^file | awk '{print $4}'`
#
# Create the snapshot.
#
DISKSPEC=""
for t in $TARGETS; do
DISKSPEC="$DISKSPEC --diskspec $t,snapshot=external"
done
virsh snapshot-create-as --domain "$DOMAIN" --name backup --no-metadata \
--atomic --disk-only $DISKSPEC >/dev/null
if [ $? -ne 0 ]; then
echo "Failed to create snapshot for $DOMAIN"
exit 1
fi
#
# Copy disk images
#
for t in $IMAGES; do
NAME=`basename "$t"`
cp "$t" "$BACKUP"/"$NAME"
done
#
# Merge changes back.
#
BACKUPIMAGES=`virsh domblklist "$DOMAIN" --details | grep ^file | awk '{print $4}'`
for t in $TARGETS; do
virsh blockcommit "$DOMAIN" "$t" --active --pivot >/dev/null
if [ $? -ne 0 ]; then
echo "Could not merge changes for disk $t of $DOMAIN. VM may be in invalid state."
exit 1
fi
done
#
# Cleanup left over backup images.
#
for t in $BACKUPIMAGES; do
rm -f "$t"
done
#
# Dump the configuration information.
#
virsh dumpxml "$DOMAIN" >"$BACKUP/$DOMAIN.xml"
#
# Cleanup older backups.
#
LIST=`ls -r1 "$BACKUPDOMAIN" | grep -E '^[0-9]{4}-[0-9]{2}-[0-9]{2}\.[0-9]+$'`
i=1
for b in $LIST; do
if [ $i -gt "$MAXBACKUPS" ]; then
echo "Removing old backup "`basename $b`
rm -rf "$b"
fi
i=$[$i+1]
done
echo "Finished backup"
echo ""
@vchakoshy

This comment has been minimized.

Copy link

vchakoshy commented Jun 16, 2016

Good job. ;)

@boulate

This comment has been minimized.

Copy link

boulate commented Oct 26, 2016

Thank you for this share ! I was looking for a script to do exactly this kind of work without having to do it by hand.

Juste one thing : I think there is a mistake at the end of the script :

    if [ $i -gt "$MAXBACKUPS" ]; then
        echo "Removing old backup "`basename $b`
        rm -rf "$b"
    fi

Must be :

    if [ $i -gt "$MAXBACKUPS" ]; then
        echo "Removing old backup "`basename $b`
        rm -rf "$BACKUPDOMAIN/$b"
    fi

If you want the script to take the originial path in consideration. No ? :)

@ASAPHAANING

This comment has been minimized.

Copy link

ASAPHAANING commented Mar 2, 2017

Thank you very much for this useful utility script. I've added a wrapper that will handle the domains using virsh list --all and loop the backup script for all of those. By default this keeps 1 copy.

#!/bin/bash
#
# Get the list of all domains, active and inactive, iterate over the list with the main backup-script such that all domains get backed up.
test="$(virsh list --all | awk {'print $2'} | tail -n +3)"
while read -r line; do
    bash <vm-backup.sh location> <backup dir> "$line" 1
done <<< "$test"
@Monkybusiness

This comment has been minimized.

Copy link

Monkybusiness commented Mar 7, 2017

Thanks for this script, very helpful. I had some problems though which turned out to be whenever there is a CDROM or FLOPPY in the hardware list for a VM. The problem is that the domblklist command used isn't filtered enough. It is filtered so only lines starting with "file" are used for building targets but the problem here is that things like a cdrom or a floppy will also be included and have no source. Output from the filtered domblklist can look like this:

Type Device Target Source

---------------------------------------------------

file disk hda /mnt/whatever/filename-flat.vmdk

file cdrom hdb -

This throws up an error as the snapshot command tries to create a snapshot for the cdrom without a source file. We need to get rid of any cdrom and floppy (could be other things, but this is what I had also starting with "file"), so I have substituted (in 3 places), where it says:

grep ^file

change to:

grep ^file | grep -v 'cdrom' | grep -v 'floppy'

After this, it works fine again. The alternative is of course to just remove all CDROM's and FLOPPY-units.

@hadrins

This comment has been minimized.

Copy link

hadrins commented Sep 17, 2017

Great script. It helped me get my own started. I need a script that would do a live backup on all the VM on the KVM. I was able to use your code and create a script that will backup all the running/paused VM.

I will have to fine tune it and post.
Thanks.

@shurak

This comment has been minimized.

Copy link

shurak commented Jan 21, 2018

Great script and great comments. FYI your script is running in a nuclear facility :>

@GerhardK90

This comment has been minimized.

Copy link

GerhardK90 commented Feb 28, 2018

Thank you for the script.
I agree with @boulate, without the modification the cleanup of old backups won't work properly.
Furthermore you should be aware, that the created backups might be in inconsistent states. While creating the snapshot it is not ensured, that the Disk is in a consistent state. This can lead to partial dataloss.
You should install qemu-agent on your VMs and run the snapshot with --quiesce.
Another option is to use virsh domfsfreeze and virsh domfsthaw.

@sebastiaanfranken

This comment has been minimized.

Copy link

sebastiaanfranken commented Mar 21, 2018

I've used your script as a starting point. My script does a backup of all domains on the KVM server it's on. I run this every week for our weekly backup cycle. Works like a charm so far.

#!/bin/bash

# Set the language to English so virsh does it's output
# in English as well
LANG=en_US

# Define the script name, this is used with systemd-cat to
# identify this script in the journald output
SCRIPTNAME=kvm-backup

# List domains
DOMAINS=$(virsh list | tail -n +3 | awk '{print $2}')

# Loop over the domains found above and do the
# actual backup

for DOMAIN in $DOMAINS; do

	echo "Starting backup for $DOMAIN on $(date +'%d-%m-%Y %H:%M:%S')" | systemd-cat -t $SCRIPTNAME

	# Generate the backup folder URI - this is something you should
	# change/check
	BACKUPFOLDER=/mnt/$DOMAIN/$(date +%d-%m-%Y)
	mkdir -p $BACKUPFOLDER

	# Get the target disk
	TARGETS=$(virsh domblklist $DOMAIN --details | grep disk | awk '{print $3}')

	# Get the image page
	IMAGES=$(virsh domblklist $DOMAIN --details | grep disk | awk '{print $4}')

	# Create the snapshot/disk specification
	DISKSPEC=""

	for TARGET in $TARGETS; do
		DISKSPEC="$DISKSPEC --diskspec $TARGET,snapshot=external"
	done

	virsh snapshot-create-as --domain $DOMAIN --name "backup-$DOMAIN" --no-metadata --atomic --disk-only $DISKSPEC 1>/dev/null 2>&1

	if [ $? -ne 0 ]; then
		echo "Failed to create snapshot for $DOMAIN" | systemd-cat -t $SCRIPTNAME
		exit 1
	fi

	# Copy disk image
	for IMAGE in $IMAGES; do
		NAME=$(basename $IMAGE)
		cp $IMAGE $BACKUPFOLDER/$NAME
	done

	# Merge changes back
	BACKUPIMAGES=$(virsh domblklist $DOMAIN --details | grep disk | awk '{print $4}')

	for TARGET in $TARGETS; do
		virsh blockcommit $DOMAIN $TARGET --active --pivot 1>/dev/null 2>&1

		if [ $? -ne 0 ]; then
			echo "Could not merge changes for disk of $TARGET of $DOMAIN. VM may be in invalid state." | systemd-cat -t $SCRIPTNAME
			exit 1
		fi
	done

	# Cleanup left over backups
	for BACKUP in $BACKUPIMAGES; do
		rm -f $BACKUP
	done

	# Dump the configuration information.
	virsh dumpxml $DOMAIN > $BACKUPFOLDER/$DOMAIN.xml 1>/dev/null 2>&1

	echo "Finished backup of $DOMAIN at $(date +'%d-%m-%Y %H:%M:%S')" | systemd-cat -t $SCRIPTNAME
done

exit 0
@nebbian

This comment has been minimized.

Copy link

nebbian commented May 4, 2018

Hey @sebastiaanfranken, thanks for posting. I'm manually going through this script to see how it works, and am a little confused by these lines:

BACKUPIMAGES=$(virsh domblklist $DOMAIN --details | grep disk | awk '{print $4}')

# Cleanup left over backups
	for BACKUP in $BACKUPIMAGES; do
		rm -f $BACKUP
	done

It seems to me that this script will delete the images from the running VMs, instead of the old backups. Is this a copy/paste issue from the lines above (getting the original list of images), or am I reading this wrong?

@tevkar

This comment has been minimized.

Copy link

tevkar commented May 12, 2018

@nebbian,

Setting BACKUPIMAGES before running blockcommit gives a list of snapshot image files which will be comitted. They can safely be removed if blockcommit operation was successful.

@v0112358

This comment has been minimized.

Copy link

v0112358 commented May 24, 2018

Great ! Thank you for your script. But I think you should use '--spare=always' option with cp command.

@churusaa

This comment has been minimized.

Copy link

churusaa commented Jun 11, 2018

Updated version that uses rsync instead of cp so that progress, elapsed time, rate, etc... are shown. Example of pv included (not tested) not used because it doesn't preserve permissions.


# Set the language to English so virsh does it's output
# in English as well
# LANG=en_US

# Define the script name, this is used with systemd-cat to
# identify this script in the journald output
SCRIPTNAME=kvm-backup

# List domains
DOMAINS=$(virsh list | tail -n +3 | awk '{print $2}')

# Loop over the domains found above and do the
# actual backup

for DOMAIN in $DOMAINS; do

	echo "Starting backup for $DOMAIN on $(date +'%d-%m-%Y %H:%M:%S')" | systemd-cat -t $SCRIPTNAME

	# Generate the backup folder URI - this is something you should
	# change/check
	BACKUPFOLDER=/mnt/backups/$DOMAIN/$(date +%d-%m-%Y)
	mkdir -p $BACKUPFOLDER

	# Get the target disk
	TARGETS=$(virsh domblklist $DOMAIN --details | grep disk | awk '{print $3}')

	# Get the image page
	IMAGES=$(virsh domblklist $DOMAIN --details | grep disk | awk '{print $4}')

	# Create the snapshot/disk specification
	DISKSPEC=""

	for TARGET in $TARGETS; do
		DISKSPEC="$DISKSPEC --diskspec $TARGET,snapshot=external"
	done

	virsh snapshot-create-as --domain $DOMAIN --name "backup-$DOMAIN" --no-metadata --atomic --disk-only $DISKSPEC 1>/dev/null 2>&1

	if [ $? -ne 0 ]; then
		echo "Failed to create snapshot for $DOMAIN" | systemd-cat -t $SCRIPTNAME
		exit 1
	fi

	# Copy disk image
	for IMAGE in $IMAGES; do
		NAME=$(basename $IMAGE)
                # cp $IMAGE $BACKUPFOLDER/$NAME
                # pv $IMAGE > $BACKUPFOLDER/$NAME
		rsync -ah --progress $IMAGE $BACKUPFOLDER/$NAME
	done

	# Merge changes back
	BACKUPIMAGES=$(virsh domblklist $DOMAIN --details | grep disk | awk '{print $4}')

	for TARGET in $TARGETS; do
		virsh blockcommit $DOMAIN $TARGET --active --pivot 1>/dev/null 2>&1

		if [ $? -ne 0 ]; then
			echo "Could not merge changes for disk of $TARGET of $DOMAIN. VM may be in invalid state." | systemd-cat -t $SCRIPTNAME
			exit 1
		fi
	done

	# Cleanup left over backups
	for BACKUP in $BACKUPIMAGES; do
		rm -f $BACKUP
	done

	# Dump the configuration information.
	virsh dumpxml $DOMAIN > $BACKUPFOLDER/$DOMAIN.xml 1>/dev/null 2>&1

	echo "Finished backup of $DOMAIN at $(date +'%d-%m-%Y %H:%M:%S')" | systemd-cat -t $SCRIPTNAME
done

exit 0
@dalu

This comment has been minimized.

Copy link

dalu commented Oct 28, 2018

@churusaa use rsync -ahW when you're not making a diff copy with rsync, it's a lot faster.
Also in some cases one would want to omit user and group ownership change, so the result is

rsync -ahW --no-o --no-g $IMAGE $BACKUPFOLDER/$NAME

@bttrfngrs

This comment has been minimized.

Copy link

bttrfngrs commented Nov 19, 2018

@churusaa I was loving this script .. until a recent upgrade on our ubuntu to the latest Bionic Beaver the script quickly returns to prompt without executing any commands..

@andreaswork

This comment has been minimized.

Copy link

andreaswork commented Jan 8, 2019

I had some problems with blockjob still being active when the script tried to blockcommit on VM's with more than 1 disk, had to manually abort the blockjob and redo blockcommit on both disks.

To prevent this issue, i added "--wait" option on line 60, so it actually checks and doesn't assume the blockjob is complete before starting blockcommit.

ps. this is not an issue with this script, this is an issue with Virsh tool, --wait fixes this problem.

@addictedtoflames

This comment has been minimized.

Copy link

addictedtoflames commented Jan 19, 2019

Loving this script, I just have one issue with it. Some of my VMs have physical disks attached to them meaning that this script returns the error: "error: unsupported configuration: source for disk 'sda' is not a regular file; refusing to generate external snapshot name"

I already have backups in place for the files on these disks so I don't need them included in the snapshot. Is there a way I can exclude block devices within the script?

@berlinguyinca

This comment has been minimized.

Copy link

berlinguyinca commented Feb 28, 2019

well done!

@reinistihovs

This comment has been minimized.

Copy link

reinistihovs commented Jul 25, 2019

Heres my version, which creates backups with rsync --sparse and --inplace.
What this means?
First time when script is running, the whole .qcow2 file is transfered,
Next time only the changes inside the .qcow2 file are synced into the backup file.
This makes the backup complete 10x faster.
This also saves a TON of space.

#!/bin/bash

#To exclude a domain, please add to its name "nobackup"
#First shutdown the guest, then use this command: virsh domrename oldname newname.

DATE=`date +%Y-%m-%d.%H:%M:%S`
LOG=/var/log/kvm-backup.$DATE.LOG
BACKUPROOT=/backup


DOMAINS=$(virsh list --all | tail -n +3 | awk '{print $2}')

for DOMAIN in $DOMAINS; do
        echo "-----------WORKER START $DOMAIN-----------" > $LOG
        echo "Starting backup for $DOMAIN on $(date +'%d-%m-%Y %H:%M:%S')"  >> $LOG

        if [[ $DOMAIN == *"nobackup"* ]];then
                echo "Skipping $DOMAIN , because its excluded." > $LOG
                exit 1
        fi

        VMSTATE=`virsh list --all | grep $DOMAIN | awk '{print $3}'`
        if [[ $VMSTATE != "running" ]]; then
                echo "Skipping $DOMAIN , because its not running." > $LOG
                exit 1
        fi

        BACKUPFOLDER=$BACKUPROOT/KVM-BACKUPS/$DOMAIN
        mkdir -p $BACKUPFOLDER
        TARGETS=$(virsh domblklist $DOMAIN --details | grep disk | awk '{print $3}')
        IMAGES=$(virsh domblklist $DOMAIN --details | grep disk | awk '{print $4}')
        DISKSPEC=""
        for TARGET in $TARGETS; do
                DISKSPEC="$DISKSPEC --diskspec $TARGET,snapshot=external"
        done

        virsh snapshot-create-as --domain $DOMAIN --name "backup-$DOMAIN" --no-metadata --atomic --disk-only $DISKSPEC >> $LOG
        if [ $? -ne 0 ]; then
                echo "Failed to create snapshot for $DOMAIN" > $LOG
                exit 1
        fi

        for IMAGE in $IMAGES; do
                NAME=$(basename $IMAGE)
                if test -f "$BACKUPFOLDER/$NAME"; then
                echo "Backup exists, merging only changes to image" > $LOG
                rsync -apvz --inplace $IMAGE $BACKUPFOLDER/$NAME >> $LOG
                else
                echo "Backup does not exist, creating a full sparse copy" > $LOG
                rsync -apvz --sparse $IMAGE $BACKUPFOLDER/$NAME >> $LOG
                fi

        done

        BACKUPIMAGES=$(virsh domblklist $DOMAIN --details | grep disk | awk '{print $4}')
        for TARGET in $TARGETS; do
                virsh blockcommit $DOMAIN $TARGET --active --pivot >> $LOG

                if [ $? -ne 0 ]; then
                        echo "Could not merge changes for disk of $TARGET of $DOMAIN. VM may be in invalid state." > $LOG
                        exit 1
                fi
        done

        for BACKUP in $BACKUPIMAGES; do
                if [[ $BACKUP == *"backup-"* ]];then

                echo "deleted temporary image $BACKUP" > $LOG
                rm -f $BACKUP
                fi
        done

        virsh dumpxml $DOMAIN > $BACKUPFOLDER/$DOMAIN.xml
        echo "-----------WORKER END $DOMAIN-----------" >> $LOG
        echo "Finished backup of $DOMAIN at $(date +'%d-%m-%Y %H:%M:%S')" >> $LOG
done

exit 0

@fuznutz04

This comment has been minimized.

Copy link

fuznutz04 commented Aug 2, 2019

@reinistihovs, I'll give your version a try. I have been using the original one posted in this thread, and it is successfully taking snapshots, transferring them etc. It is also generating a domain.xml file. However, the file is blank. If I manually do a dumpxml, it generates the proper configuration file. But it does not do so properly via the script. Any ideas?

@fuznutz04

This comment has been minimized.

Copy link

fuznutz04 commented Sep 5, 2019

Does anyone else have this issue? This will generate a domain.xml file, but the file is blank. If I manually do a dumpxml, it generates the proper configuration file. But it does not do so properly via the script. Any ideas?

@brentl99

This comment has been minimized.

Copy link

brentl99 commented Jan 9, 2020

Thank you for posting scripts, they are VERY VERY helpful. I have posted below a variation most suitable for my use:

#!/bin/bash
#
# This script backs up a list of VMs.
# An overview of the process is as follows:
# * invokes a "snapshot" which transfers VM disk I/O to new "snapshot" image file(s).
# * copy the VM's image file(s) to a backup
# * invoke a "blockcommit" which merges (or "pivots") the snapshot image back
#   to the VM's primary image file(s)
# * delete the snapshot image file(s)
# * make a copy of the VM define/XML file
#
# Note: On CentOS 7 snapshotting requires the "-ev" version of qemu.
#       yum install centos-release-qemu-ev qemu-kvm-ev libvirt
#
# The script uses gzip to compress the source image (e.g. qcow2) on the fly
# to the destination backup image. bzip2 was also tested, but bzip2 (and
# other compression utilities) provide better compression (15-20%) but gzip
# is 7-10 times faster.
#
# If the process fails part way through the snapshot, copy, or blockcommit,
# the VM may be left running on the snapshot file which is Not desirable.
#

# define an emergency mail recipient
EMR=myservermessages@mydomain.com
#
HOST="$(hostname)"
SHCMD="$(basename -- $0)"
BACKUPROOT=/mydata01/qemu_backup
[ ! -f $BACKUPROOT/logs ] && mkdir -p $BACKUPROOT/logs
DATE="$(date +%Y%m%d.%H%M%S)"
LOG="$BACKUPROOT/logs/qemu-backup.$(date +%Y%m%d).log"
ERRORED=0
BREAK=false

#Optionally list all VMs and back them all up
#DOMAINS=$(virsh list --all | tail -n +3 | awk '{print $2}')
DOMAINS="myVMa myVMb"

echo "$SHCMD: Starting backups on $(date +'%d-%m-%Y %H:%M:%S')"  >> $LOG
for DOMAIN in $DOMAINS; do
        BREAK=false

        echo "---- VM Backup start $DOMAIN ---- $(date +'%d-%m-%Y %H:%M:%S')"  >> $LOG

        VMSTATE=$(virsh list --all | grep [[:space:]]$DOMAIN[[:space:]] | awk '{print $3}')
        if [[ $VMSTATE != "running" ]]; then
                echo "Skipping $DOMAIN , because it is not running." >> $LOG
                continue
        fi

        BACKUPFOLDER=$BACKUPROOT/$DOMAIN
        [ ! -d $BACKUPFOLDER ] && mkdir -p $BACKUPFOLDER
        TARGETS=$(virsh domblklist $DOMAIN --details | grep disk | awk '{print $3}')
        IMAGES=$(virsh domblklist $DOMAIN --details | grep disk | awk '{print $4}')

        # check to make sure the VM is running on a standard image, not
        # a snapshot that may be from a backup that previously failed
        # Note: if you are going to change the naming of the snapshot temp file, 
        #           code must be changed in 4 (four) places - this is ONE
        for IMAGE in $IMAGES; do
                if [[ $IMAGE == *"snaptemp-"* ]]; then
                        ERR="$SHCMD: Error VM $DOMAIN is running on a snapshot disk image: $IMAGE"
                        echo $ERR >> $LOG
                        echo "$ERR
Host:       $HOST
Disk Image: $IMAGE
Domain:     $DOMAIN
Command:    virsh domblklist $DOMAIN --details" | mail -s "$SHCMD snapshot Exception for $DOMAIN" $EMR
                        BREAK=true
                        ERRORED=$(($ERRORED+1))
                        break
                fi
        done
        [ $BREAK == true ] && continue

        # gather all the disks being used by the VM so they can be collectively snapshotted
        # Note: if you are going to change the naming of the snapshot temp file, 
        #           code must be changed in 4 (four) places - this is TWO
        DISKSPEC=""
        for TARGET in $TARGETS; do
                if [[ $TARGET == *"snaptemp-"* ]]; then
                        ERR="$SHCMD: Error VM $DOMAIN is running on a snapshot disk image: $TARGET"
                        echo $ERR >> $LOG
                        echo "$ERR
Host:       $HOST
Disk Image: $IMAGE
Domain:     $DOMAIN
Command:    $CMD" | mail -s "$SHCMD snapshot Exception for $DOMAIN" $EMR
                        BREAK=true
                        break
                fi
                DISKSPEC="$DISKSPEC --diskspec $TARGET,snapshot=external"
        done
        [ $BREAK == true ] && continue

        # transfer the VM to snapshot disk image(s)
        # Note: if you are going to change the naming of the snapshot temp file, 
        #           code must be changed in 4 (four) places - this is THREE
        CMD="virsh snapshot-create-as --domain $DOMAIN --name snaptemp-$DOMAIN-$DATE --no-metadata --atomic --disk-only $DISKSPEC >> $LOG 2>&1"
        echo "Command: $CMD" >> $LOG 2>&1
        eval "$CMD"
        if [ $? -ne 0 ]; then
                ERR="Failed to create snapshot for $DOMAIN"
                echo $ERR >> $LOG
                echo "$ERR
Host:    $HOST
Domain:  $DOMAIN
Command: $CMD" | mail -s "$SHCMD snapshot Exception for $DOMAIN" $EMR
                ERRORED=$(($ERRORED+1))
                continue
        fi

        # copy/back/compress the VM's disk image(s)
        for IMAGE in $IMAGES; do
                echo "Copying $IMAGE to $BACKUPFOLDER" >> $LOG
                ZFILE="$BACKUPFOLDER/$(basename -- $IMAGE)-$DATE.gz"
                CMD="gzip < $IMAGE > $ZFILE 2>> $LOG"
                echo "Command: $CMD" >> $LOG
                SECS=$(printf "%.0f" $(/usr/bin/time -f %e sh -c "$CMD" 2>&1))
                printf '%s%dh:%dm:%ds\n' "Duration: " $(($SECS/3600)) $(($SECS%3600/60)) $(($SECS%60)) >> $LOG
                BYTES=$(stat -c %s $IMAGE)
                printf "%s%'d\n" "Source MB: " $(($BYTES/1024/1024)) >> $LOG
                printf "%s%'d\n" "kB/Second: " $(($BYTES/$SECS/1024)) >> $LOG
                ZBYTES=$(stat -c %s $ZFILE)
                printf "%s%'d\n" "Destination MB: " $(($ZBYTES/1024/1024)) >> $LOG
                printf "%s%d%s\n" "Compression: " $((($BYTES-$ZBYTES)*100/$BYTES)) "%" >> $LOG
        done

        # Update the VM's disk image(s) with any changes recorded in the snapshot
        # while the copy process was running.  In qemu lingo this is called a "pivot"
        BACKUPIMAGES=$(virsh domblklist $DOMAIN --details | grep disk | awk '{print $4}')
        for TARGET in $TARGETS; do
                CMD="virsh blockcommit $DOMAIN $TARGET --active --pivot >> $LOG 2>&1"
                echo "Command: $CMD" >> $LOG
                eval "$CMD"

                if [ $? -ne 0 ]; then
                        ERR="Could not merge changes for disk of $TARGET of $DOMAIN. VM may be in an invalid state."
                        echo $ERR >> $LOG
                        echo "$ERR
Host:    $HOST
Domain:  $DOMAIN
Command: $CMD" | mail -s "$SHCMD blockcommit Exception for $DOMAIN" $EMR
                        BREAK=true
                        ERRORORED=$(($ERRORED+1))
                        break
                fi
        done
        [ $BREAK == true ] && continue

        # Now that the VM's disk image(s) have been successfully committed/pivoted to
        # back to the main disk image, remove the temporary snapshot image file(s)
        # Note: if you are going to change the naming of the snapshot temp file, 
        #           code must be changed in 4 (four) places - this is FOUR
        for BACKUP in $BACKUPIMAGES; do
                if [[ $BACKUP == *"snaptemp-"* ]]; then
                        CMD="rm -f $BACKUP >> $LOG 2>&1"
                        echo " Deleting temporary image $BACKUP" >> $LOG
                        echo "Command: $CMD" >> $LOG
                        eval "$CMD"
                fi
        done

        # capture the VM's definition in use at the time the backup was done
        CMD="virsh dumpxml $DOMAIN > $BACKUPFOLDER/$DOMAIN-$DATE.xml 2>> $LOG"
        echo "Command: $CMD" >> $LOG
        eval "$CMD"
        echo "---- Backup done $DOMAIN ---- $(date +'%d-%m-%Y %H:%M:%S') ----" >> $LOG
done
echo "$SHCMD: Finished backups at $(date +'%d-%m-%Y %H:%M:%S')
====================" >> $LOG

exit $ERRORED
@panciom

This comment has been minimized.

Copy link

panciom commented Jan 29, 2020

Great job.
I have modified the script for making remote rsync backup via SSH. With bwlimit.
It require password-less login to remote server.
It enforce on VM (name not win as it not works very qell with guest agent installed) --quiesce option to flush cache to disk (guest agent required on VM).
It send a mail at the end of the script.
It works on running and stopped VM.

It's not very fast. Rsync on hundreds GB of data is slow only for reading all.
I'm thinking to use PHP to make a script for making multiple external live snapshot for transferring only changed data already on single little file.
PHP because i'm friendly with it.

Good backup.

#!/bin/bash

# Script: kvm-backup.sh
# By: Pancio <....@.....it>
# Date: 21/12/2019
#
# Descr: Live backup of KVM VM on host via SSH...
# Source: https://gist.github.com/cabal95/e36c06e716d3328b512b
# Credits: See link above...

# NOTES:
# To exclude a domain, please add to its name "nobackup"
# First shutdown the guest, then use this command: virsh domrename oldname newname.




###########################################################
################### START CONFIGURATION ###################

# FQDN server name...
SERVER_NAME=$(hostname -f)

# Mail sender and destination...
MAIL_TO=to@gmail.com
MAIL_FROM=from@gmail.com

# Path to log files...
LOG_PATH=/var/log/kvm-backup

# Path where to place backup files...
# Used with local or SSH remote...
# Use absolute path "/" or related to home "~/"...
BACKUP_PATH=/Backup-KVM

# If you have guest-agent installed on VM "--quiesce"
# can flush data to disk before taking snapshot... 
SNAP_CREATE_PARAMS="--quiesce"

# Disable --quiesce for win VM...
DISABLE_SNAP_CREATE_PARAMS_WIN=1

# Be verbose on pivoting snapshot to original disk-file...
SNAP_COMM_PARAMS="--verbose"

# SSH remote rsync backup these are examples...
# Remember to configure passordless login to remote server...
RSYNC_SSH_PARAMS='-p2222'
SCP_SSH_PARAMS='-P2222'
RSYNC_DEST="root@srv.remote.it"

# Max KB/s rsync data transfer (ex. 1200KB/s=10Mbps)...
RSYNC_SPEED="--bwlimit=2400"

# Path to tmp for storing dumpxml...
PATH_TMP_XML=/tmp

# Path to XML files for stopped VM...
PATH_LIBVIRT_QEMU="/etc/libvirt/qemu"

################### STOP CONFIGURATION ###################
###########################################################




DATE=`date +%Y-%m-%d_%H-%M-%S`
LOG=$LOG_PATH/kvm-backup.$DATE.log
WARNING=0


DOMAINS=$(virsh list --all 2> /dev/null | tail -n +3 | awk '{print $2}')

if [ -z "$DOMAINS" ]; then
  echo "Error extracting VM list from libvirt!" >> $LOG
  cat $LOG | mail -s "[KVM-Backup] Node='$SERVER_NAME' ERROR EXTRACTING VM LIST" $MAIL_TO -aFrom:$MAIL_FROM	
  exit 1
fi

NR_DOMAINS=$(echo "$DOMAINS" | wc -l)


for DOMAIN in $DOMAINS; do
  echo "" >> $LOG
  echo "-----------WORKER START FOR VM $DOMAIN-----------" >> $LOG
  echo "-> Starting backup for VM $DOMAIN on $(date +'%d-%m-%Y %H:%M:%S')"  >> $LOG

  if [[ $DOMAIN == *"nobackup"* ]];then
    echo "-> Skipping VM $DOMAIN because its excluded." >> $LOG
    continue
  fi

  VM_RUNNING=1
  VMSTATE=`virsh list --all | grep $DOMAIN | awk '{print $3}'`
	if [[ $VMSTATE != "in" ]]; then
    echo "-> VM $DOMAIN not running. No snapshot and blockcommit. Only rsync." >> $LOG
    VM_RUNNING=0
  fi

	MY_SNAP_CREATE_PARAMS=$SNAP_CREATE_PARAMS
  if [[ $DOMAIN == *"win"* ]];then
    echo "-> Skipping SNAP_CREATE_PARAMS for $DOMAIN because its *win*." >> $LOG
    MY_SNAP_CREATE_PARAMS=""
  fi

	# Images to copy...
	IMAGES=$(virsh domblklist $DOMAIN --details | grep disk | awk '{print $4}')

	if [[ $VM_RUNNING -eq 1 ]]; then
		TARGETS=$(virsh domblklist $DOMAIN --details | grep disk | awk '{print $3}')
		DISKSPEC=""

		for TARGET in $TARGETS; do
		  DISKSPEC="$DISKSPEC --diskspec $TARGET,snapshot=external"
		done

		virsh snapshot-create-as --domain $DOMAIN --name "backup-$DOMAIN" --no-metadata --atomic --disk-only $MY_SNAP_CREATE_PARAMS $DISKSPEC >> $LOG
		if [ $? -ne 0 ]; then
		  echo "-> Failed to create snapshot for VM $DOMAIN. Try verify GuestAgent is running inside." >> $LOG
		  WARNING=1    
		  continue
		fi
	fi


  BACKUPFOLDER=$BACKUP_PATH/$SERVER_NAME/$DOMAIN
  ssh $RSYNC_SSH_PARAMS $RSYNC_DEST ''mkdir -p $BACKUPFOLDER''

  for IMAGE in $IMAGES; do
    NAME=$(basename $IMAGE)
    DUMMY=$((ssh $RSYNC_SSH_PARAMS $RSYNC_DEST stat $BACKUPFOLDER/$NAME) 2>&1)

    if [ $? -eq 0 ]; then
      echo "-> Backup exists on $RSYNC_DEST:$BACKUPFOLDER/$NAME, merging only changes to image" >> $LOG
      rsync -apvz -e "ssh $RSYNC_SSH_PARAMS" $RSYNC_SPEED --inplace $IMAGE $RSYNC_DEST:$BACKUPFOLDER/$NAME >> $LOG
    else
      echo "-> Backup does not exist on $RSYNC_DEST:$BACKUPFOLDER/$NAME, creating a full sparse copy" >> $LOG
      rsync -apvz -e "ssh $RSYNC_SSH_PARAMS" $RSYNC_SPEED --sparse $IMAGE $RSYNC_DEST:$BACKUPFOLDER/$NAME >> $LOG
    fi
  done

	if [[ $VM_RUNNING -eq 1 ]]; then
		BACKUPIMAGES=$(virsh domblklist $DOMAIN --details | grep disk | awk '{print $4}')
		
		for TARGET in $TARGETS; do
		  virsh blockcommit $DOMAIN $TARGET --active --pivot $SNAP_COMM_PARAMS >> $LOG

		  if [ $? -ne 0 ]; then
		    echo "-> Could not merge changes for disk of $TARGET of VM $DOMAIN. VM may be in invalid state." >> $LOG
		    WARNING=1    
		    continue
		  fi
		done

		for BACKUP in $BACKUPIMAGES; do
		  if [[ $BACKUP == *"backup-"* ]];then
		    echo "-> Deleted temporary image $BACKUP" >> $LOG
		    rm -f $BACKUP
		  fi
		done
	fi

	if [[ $VM_RUNNING -eq 1 ]]; then
  	virsh dumpxml $DOMAIN > $PATH_TMP_XML/$DOMAIN.xml
		scp $SCP_SSH_PARAMS $PATH_TMP_XML/$DOMAIN.xml $RSYNC_DEST:$BACKUPFOLDER/$DOMAIN.xml >> $LOG
		rm $PATH_TMP_XML/$DOMAIN.xml
	else
		scp $SCP_SSH_PARAMS $PATH_LIBVIRT_QEMU/$DOMAIN.xml $RSYNC_DEST:$BACKUPFOLDER/$DOMAIN.xml >> $LOG
	fi

  echo "-> Finished backup of $DOMAIN at $(date +'%d-%m-%Y %H:%M:%S')" >> $LOG
  echo "-----------WORKER END FOR VM $DOMAIN-----------" >> $LOG
done

echo "" >> $LOG


# Send email of results for comodity...
cat $LOG | mail -s "[KVM-Backup] Node='$SERVER_NAME' VMcount='$NR_DOMAINS' Warning='$WARNING'" $MAIL_TO -aFrom:$MAIL_FROM


exit 0
@brentl99

This comment has been minimized.

Copy link

brentl99 commented Feb 5, 2020

I discovered a bug in posted sample scripts that include the following:
VMSTATE=`virsh list --all | grep $DOMAIN | awk '{print $3}'

If you have domain names like "windows10", "windows10vm", "dows10", etc, grep fails to extract the correct line from the virsh list.
In my example this occurs because "windows10" matches "windows10" and "windows10vm".
And "dows10" matches "windows10", "windows10vm" and "dows10".

The solution that I made to correct this was the following edit:
VMSTATE=`virsh list --all | grep [[:space:]]$DOMAIN[[:space:]] | awk '{print $3}'

@Ryushin

This comment has been minimized.

Copy link

Ryushin commented Feb 18, 2020

Here is a version that uses Borg Backup. I used brentl99's version for my template. You can configure Borg to use local storage or remote storage using SSH. I've added quite a few options. Including skipping specific domains for specific virtual disks. There are also options for checking the Borg Repositories on a day of week or specific weeks in the month. I've been running it in production for a couple of weeks now with good success.

#!/bin/bash
#
# This script backs up a list of VMs using Borg Backup.
# An overview of the process is as follows:
# * invokes a "snapshot" which transfers VM disk I/O to new "snapshot" image file(s).
# * Use borg to backup the VM's image file(s)
# * invoke a "blockcommit" which merges (or "pivots") the snapshot image back
#   to the VM's primary image file(s)
# * delete the snapshot image file(s)
# * make a copy of the VM define/XML file
#
# If the process fails part way through the snapshot, copy, or blockcommit,
# the VM may be left running on the snapshot file which is Not desirable.
#
# Note: Paths and the virtual domains cannot contain spaces

# Define email recipient
EMAIL_RECIPIENT="user@domain.net"

HOST="$(hostname)"
SHCMD="$(basename -- $0)"
LOGS_DIR=/var/log/vm_backups
DATE="$(date +%Y%m%d_%H%M)"
LOG="$LOGS_DIR/vm_backups.$DATE.log"
ERRORED=0
BREAK=false
QEMU_XML_BACKUPS="/var/log/vm_backups/qemu_xml_backups"

# List of any domains to not back up.  Separate with a pipe |
SKIP_DOMAINS=""

# List of specific virtual disks to not back up. Separate with a pipe |
SKIP_DISKS=""

# Virsh snapshot extension name
VIRSH_SNAP_NAME="snaptemp"

# Send summary email at the end of the backup?
EMAIL_SUMMARY="yes"

# Output borg summary at end of backup?
END_SUMMARY="yes"

# How many days to keep logs and qemu.xml files.
KEEP_FILES_FOR="14"

# Borg environment varibles
#export BORG_SSH_SERVER='borg-backup@nas.example.net'
export BORG_REPO='/volume1/borgbackuprepo'
#export BORG_REPO="ssh://$BORG_SSH_SERVER/volume1/BorgBackupRepo"
#export BORG_RSH='ssh -i /home/user/.ssh/id_ed25519 -o BatchMode=yes -o VerifyHostKeyDNS=yes'
# See https://borgbackup.readthedocs.io/en/stable/faq.html#it-always-chunks-all-my-files-even-unchanged-ones
export BORG_FILES_CACHE_TTL='100'

# Borg create options
# Note: ZFS and BTRFS should use native compression.  
#	ionice and nice is used with the borg create command.
BORG_CREATE_OPTIONS="--compression none --list --stats --files-cache=mtime,size --noctime --noatime"

# Borg init options
BORG_INIT_OPTIONS="--make-parent-dirs --encryption=none"

# How long to keep borg archives
# --keep-hourly=24 --keep-daily=14 --keep-weekly=4 --keep-monthly=2"
BORG_PRUNE_OPTIONS="-v --list --keep-daily=14"

# Borg mtime touch file
# See: https://borgbackup.readthedocs.io/en/stable/faq.html#i-am-seeing-a-added-status-for-an-unchanged-file
BORG_MTIME_FILE="borg_touch_mtime.txt"

# How often to perform full check on borg repositories.
# Day of the week to perform the full checK.  Use full weekday name (date %A).
CHECK_DOW="Friday"
# Which week in the month to perform the full check.  Put any number(s) 1-5.
CHECK_WEEKS="12345"
# Send email of check results?
EMAIL_CHECK_RESULTS="yes"
# Borg check options
BORG_CHECK_OPTIONS=""

##################  End Configuration Options  ##################

# Create directories if they are missing
[ ! -f $LOGS_DIR ] && mkdir -p $LOGS_DIR
[ ! -f $QEMU_XML_BACKUPS ] && mkdir -p $QEMU_XML_BACKUPS

# SKIP_DOMAINS and SKIP_DISKS cannot be blank or egrep will match everything
[ "$SKIP_DOMAINS" = "" ] && SKIP_DOMAINS="$(mktemp --dry-run XXXXXXXXXXXXXXXXXXXX)"
[ "$SKIP_DISKS" = "" ] && SKIP_DISKS="$(mktemp --dry-run XXXXXXXXXXXXXXXXXXXX)"

# Create list of domains to backup except those in SKIP_DOMAINS
DOMAINS=$(virsh list --all | tail -n +3 | egrep -v "$SKIP_DOMAINS" | awk '{print $2}' | sort | sed '/^[[:space:]]*$/d')

# Create summary temp file if required
[ "$EMAIL_SUMMARY" = "yes" -o "$END_SUMMARY" = "yes" ] && SUMMARY_FILE="$(mktemp /tmp/summary_XXXXXXX)"

# Check to see if the borg repo is using ssh and test connection.
echo "$BORG_REPO" | grep "ssh://" > /dev/null
if [ "$?" -eq "0" -a "$BORG_SSH_SERVER" != "" ]
then
	ssh -oBatchMode=yes $BORG_SSH_SERVER ls > /dev/null 2>&1
	if [ "$?" -ne "0" ]
	then
		ERR="$SHCMD: Error!  Cannot connect to $BORG_SSH_SERVER with SSH key."
		echo $ERR >> $LOG
		echo "$ERR
Host:       $HOST
Command:    ssh -oBatchMode=yes $BORG_SSH_SERVER ls" | mail -s "$SHCMD ssh connection failed" $EMAIL_RECIPIENT
	echo $ERR
	exit 1
	fi
fi

DAY_OF_WEEK="$(date +'%A')"
WEEK_IN_MONTH=$(echo $((($(date +%-d)-1)/7+1)))

echo "$SHCMD: Starting backups on $(date +'%d-%m-%Y %H:%M:%S')"  >> $LOG

# Check borg respositories if day of week and week in month match options.
echo $CHECK_WEEKS | grep -q $WEEK_IN_MONTH && CURRENT_WEEK="true"
if [ "$DAY_OF_WEEK" = "$CHECK_DOW" -a "$CURRENT_WEEK" = "true" ] 
then
	CHECK_RESULTS_FILE="$(mktemp /tmp/check_results_XXXXXXX)"
	echo -e "Perform full check of borg repositories\n" > $CHECK_RESULTS_FILE
	for DOMAIN in $DOMAINS; do
		borg info $BORG_REPO/$DOMAIN > /dev/null
		if [ "$?" -eq "0" ]
		then
			echo "Checking borg repository $BORG_REPO/$DOMAIN:" >> $CHECK_RESULTS_FILE
			borg --verbose check $BORG_CHECK_OPTIONS $BORG_REPO/$DOMAIN >> $CHECK_RESULTS_FILE 2>&1
			if [ "$?" -ne "0" ]
			then
				echo "Errors found in $BORG_REPO/$DOMAIN repository!" >> $CHECK_RESULTS_FILE
				echo "Manual intervention is required." >> $CHECK_RESULTS_FILE
				REPOSITORY_ERRORS="true"
			fi
		fi
	done
	if [ "$REPOSITORY_ERRORS" = "true" ]
	then
        echo -e "Borg repository errors found:\n\n
Host: $HOST

$(cat $CHECK_RESULTS_FILE)" | mail -s "Borg Repository Errors Found for $HOST" $EMAIL_RECIPIENT
	fi
	if [ "$EMAIL_CHECK_RESULTS" = "yes" -a "$REPOSITORY_ERRORS" != "true" ]
	then
        echo -e "Borg repository check results:\n\n
Host:       $HOST

$(cat $CHECK_RESULTS_FILE)" | mail -s "Borg Repository Check Results for $HOST" $EMAIL_RECIPIENT
	fi
fi


for DOMAIN in $DOMAINS; do
        BREAK=false

        echo "---- VM Backup start $DOMAIN ---- $(date +'%d-%m-%Y %H:%M:%S')"  >> $LOG

        VMSTATE=$(virsh list --all | grep [[:space:]]$DOMAIN[[:space:]] | awk '{print $3}')

        BORG_ARCHIVE_FOLDER=$BORG_REPO/$DOMAIN
        [ ! -d $BORG_ARCHIVE_FOLDER ] && mkdir -p $BORG_ARCHIVE_FOLDER
        TARGETS=$(virsh domblklist $DOMAIN --details | grep disk | awk '{print $3}')
        IMAGES=$(virsh domblklist $DOMAIN --details | grep disk | egrep -v "$SKIP_DISKS" | awk '{print $4}')
        # check to make sure the VM is running on a standard image, not
        # a snapshot that may be from a backup that previously failed
	if [ "$VMSTATE" = "running" ]
	then
	        for IMAGE in $IMAGES; do
	                if [[ $IMAGE == *"$VIRSH_SNAP_NAME-"* ]]; then
	                        ERR="$SHCMD: Error VM $DOMAIN is running on a snapshot disk image: $IMAGE"
	                        echo $ERR >> $LOG
	                        echo "$ERR
Host:       $HOST
Disk Image: $IMAGE
Domain:     $DOMAIN
Logfile:    $LOG
Command:    virsh domblklist $DOMAIN --details" | mail -s "$SHCMD snapshot Exception for $DOMAIN" $EMAIL_RECIPIENT
	                        BREAK=true
	                        ERRORED=$(($ERRORED+1))
	                        break
	                fi
	        done
	        [ $BREAK == true ] && continue

	        # gather all the disks being used by the VM so they can be collectively snapshotted
	        DISKSPEC=""
	        for TARGET in $TARGETS; do
	                if [[ $TARGET == *"$VIRSH_SNAP_NAME-"* ]]; then
	                        ERR="$SHCMD: Error VM $DOMAIN is running on a snapshot disk image: $TARGET"
	                        echo $ERR >> $LOG
	                        echo "$ERR
Host:       $HOST
Disk Image: $IMAGE
Domain:     $DOMAIN
Logfile:    $LOG
Command:    $CMD" | mail -s "$SHCMD snapshot Exception for $DOMAIN" $EMAIL_RECIPIENT
	                        BREAK=true
	                        break
	                fi
	                DISKSPEC="$DISKSPEC --diskspec $TARGET,snapshot=external"
	        done
	        [ $BREAK == true ] && continue

	        # Transfer the VM to snapshot disk image(s)
	        CMD="virsh snapshot-create-as --domain $DOMAIN --name $VIRSH_SNAP_NAME-$DOMAIN-$DATE --no-metadata --atomic --disk-only $DISKSPEC >> $LOG 2>&1"
	        echo "Command: $CMD" >> $LOG
	        eval "$CMD" >> $LOG 2>&1
	        if [ $? -ne 0 ]; then
	                ERR="Failed to create snapshot for $DOMAIN"
	                echo $ERR >> $LOG
	                echo "$ERR
Host:    $HOST
Domain:  $DOMAIN
Logfile: $LOG
Command: $CMD" | mail -s "$SHCMD snapshot Exception for $DOMAIN" $EMAIL_RECIPIENT
	                ERRORED=$(($ERRORED+1))
	                continue
	        fi
	fi

        # Use borg to backup the VM's disk image(s)
	if [ "$IMAGES" != "" ]
	then
		echo -e "\nUsing borg to backup $IMAGES to $BORG_ARCHIVE_FOLDER" >> $LOG
		# Check to see if the borg repo exists and if not, create it.
		echo -e "\nChecking to see if the borg $BORG_ARCHIVE_FOLDER repository exists using borg info..." >> $LOG
		CMD="borg info $BORG_ARCHIVE_FOLDER >> $LOG  2>&1"
		echo "Command: $CMD" >> $LOG
		eval "$CMD"
		if [ "$?" -ne "0" ]
		then
			echo -e "\nBorg Repository does not exist.  Creating $BORG_ARCHIVE_FOLDER" >> $LOG
			CMD="borg init $BORG_INIT_OPTIONS $BORG_ARCHIVE_FOLDER >> $LOG 2>&1"
			echo "Command: $CMD" >> $LOG
			eval "$CMD"
		fi

		# Backup using borg
		NOW=$(date +%Y%m%d_%H%M)
		echo -e "\nBacking up using borg.." >> $LOG
		echo "Create mtime file and wait two seconds." >> $LOG
		IMAGE_DIR=$(dirname $(echo $IMAGES | awk {'print $1'} | head -1))
		touch $IMAGE_DIR/$BORG_MTIME_FILE; sleep 2
		CMD="ionice -c2 -n7 nice -n19 borg create $BORG_CREATE_OPTIONS $BORG_ARCHIVE_FOLDER::${DOMAIN}_$NOW $IMAGE_DIR/${BORG_MTIME_FILE} ${IMAGES//$'\n'/ } >> $LOG 2>&1"
		echo "Command: $CMD" >> $LOG
		eval "$CMD"
		if [ "$?" -ne "0" ]
		then
			ERR="$SHCMD: Error!  Borg failed to backup $DOMAIN"
			echo $ERR >> $LOG
			echo "$ERR
Host:     $HOST
Domain:   $DOMAIN
Logfile:  $LOG
Command:  $CMD" | mail -s "$SHCMD borg backup failed" $EMAIL_RECIPIENT
			echo $ERR
		fi
		rm $IMAGE_DIR/$BORG_MTIME_FILE

		# Summary information about the last backup.
		echo -e "\nShow summary info about the last backup" >> $LOG
		CMD="borg info $BORG_ARCHIVE_FOLDER --last 1 >> $LOG 2>&1"
		echo "Command: $CMD" >> $LOG
		eval "$CMD"
                if [ "$?" -ne "0" ]
                then
                        ERR="$SHCMD: Error!  Borg summary failed for $DOMAIN"
                        echo $ERR >> $LOG
                        echo "$ERR
Host:     $HOST
Domain:   $DOMAIN
Logfile:  $LOG
Command:  $CMD" | mail -s "$SHCMD borg summary failed" $EMAIL_RECIPIENT
                        echo $ERR
                fi
		# Create summary temp file if required
		if [ "$EMAIL_SUMMARY" = "yes" -o "$END_SUMMARY" = "yes" ]
		then
			echo "Summary of $DOMAIN" >> $SUMMARY_FILE
			borg info $BORG_ARCHIVE_FOLDER --last 1 >> $SUMMARY_FILE
			echo -e "-----------------------------------------------\n\n" >> $SUMMARY_FILE
		fi

		# Prune borg archives
		echo -e "\nPrune borg archives older than $BORG_PRUNE_OPTIONS days." >> $LOG
		CMD="borg prune $BORG_PRUNE_OPTIONS $BORG_ARCHIVE_FOLDER >> $LOG 2>&1"
		echo "Command: $CMD" >> $LOG
		eval "$CMD"
		if [ "$?" -ne "0" ]
                then
                        ERR="$SHCMD: Error!  Failed pruning borg archive for $DOMAIN"
                        echo $ERR >> $LOG
                        echo "$ERR
Host:     $HOST
Domain:   $DOMAIN
Logfile:  $LOG
Command:  $CMD" | mail -s "$SHCMD borg prune failed" $EMAIL_RECIPIENT
                        echo $ERR
                fi
	fi

	if [ "$VMSTATE" = "running" ]
	then
	        # Update the VM's disk image(s) with any changes recorded in the snapshot
	        # while the copy process was running.  In qemu lingo this is called a "pivot"
	        BACKUP_IMAGES=$(virsh domblklist $DOMAIN --details | grep disk | egrep -v "$SKIP_DISKS" | awk '{print $4}')
	        for TARGET in $TARGETS; do
	                CMD="virsh blockcommit $DOMAIN $TARGET --active --pivot >> $LOG 2>&1"
	                echo "Command: $CMD" >> $LOG
	                eval "$CMD"
	                if [ $? -ne 0 ]; then
	                        ERR="Could not merge changes for disk of $TARGET of $DOMAIN. VM may be in an invalid state."
	                        echo $ERR >> $LOG
	                        echo "$ERR
Host:    $HOST
Domain:  $DOMAIN
Logfile: $LOG
Command: $CMD" | mail -s "$SHCMD blockcommit Exception for $DOMAIN" $EMAIL_RECIPIENT
        	                BREAK=true
        	                ERRORORED=$(($ERRORED+1))
	                        break
	                fi
	        done
	        [ $BREAK == true ] && continue

	        # Now that the VM's disk image(s) have been successfully committed/pivoted to
	        # back to the main disk image, remove the temporary snapshot image file(s)
	        for BACKUP in $BACKUP_IMAGES; do
	                if [[ $BACKUP == *"$VIRSH_SNAP_NAME-"* ]]; then
	                        CMD="rm -f $BACKUP >> $LOG 2>&1"
	                        echo " Deleting temporary image $BACKUP" >> $LOG
	                        echo "Command: $CMD" >> $LOG
	                        eval "$CMD"
	                fi
	        done
	fi

        # Capture the VM's definition in use at the time the backup was done
        CMD="virsh dumpxml $DOMAIN > $QEMU_XML_BACKUPS/$DOMAIN.xml.qemuconfig.$(date +'%d-%m-%Y') 2>> $LOG"
        echo "Command: $CMD" >> $LOG
        eval "$CMD"
        echo "---- Backup done $DOMAIN ---- $(date +'%d-%m-%Y %H:%M:%S') ----" >> $LOG
	# Prune old definition backup files
	echo "Remove old qemu definition files older than $KEEP_FILES_FOR days." >> $LOG
	find $QEMU_XML_BACKUPS -maxdepth 1 -mtime +$KEEP_FILES_FOR -name "*xml.qemuconfig" -exec rm -vf {} \; >> $LOG
done
# Remove old log files
echo "Remove log files older than $KEEP_FILES_FOR days" >> $LOG
find $LOGS_DIR -maxdepth 1 -mtime +$KEEP_FILES_FOR -name "*.log" -exec rm -vf {} \; >> $LOG

echo "$SHCMD: Finished backups at $(date +'%d-%m-%Y %H:%M:%S')
====================" >> $LOG

if [ "$EMAIL_SUMMARY" = "yes" ]
then

	echo -e "Summary of Borg Backup:\n\n
Host:       $HOST
Domains:    ${DOMAINS//$'\n'/ }

$(cat $SUMMARY_FILE)" | mail -s "Borg Backup Summary for $HOST" $EMAIL_RECIPIENT
fi

if [ "$END_SUMMARY" = "yes" ]
then
	echo -e "Borg Backup Summary:\n\n
Host:       $HOST
Domains:    ${DOMAINS//$'\n'/ }

$(cat $SUMMARY_FILE)\n"
fi

# Remove temp files
rm -f $SUMMARY_FILE $CHECK_RESULTS_FILE

exit $ERRORED
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.