I use this script to backup my QEMU/KVM/libVirt virtual machines. The script requires KVM 2.1+ since it uses the live blockcommit mode. This means the data in the snapshot disk is rolled back into the original instead of the other way around. Script does NOT handle spaces in paths.

vm-backup.sh

#!/bin/bash
#

BACKUPDEST="$1"
DOMAIN="$2"
MAXBACKUPS="$3"

if [ -z "$BACKUPDEST" -o -z "$DOMAIN" ]; then
    echo "Usage: ./vm-backup <backup-folder> <domain> [max-backups]"
    exit 1
fi

if [ -z "$MAXBACKUPS" ]; then
    MAXBACKUPS=6
fi

echo "Beginning backup for $DOMAIN"

#
# Generate the backup path
#
BACKUPDATE=`date "+%Y-%m-%d.%H%M%S"`
BACKUPDOMAIN="$BACKUPDEST/$DOMAIN"
BACKUP="$BACKUPDOMAIN/$BACKUPDATE"
mkdir -p "$BACKUP"

#
# Get the list of targets (disks) and the image paths.
#
TARGETS=`virsh domblklist "$DOMAIN" --details | grep ^file | awk '{print $3}'`
IMAGES=`virsh domblklist "$DOMAIN" --details | grep ^file | awk '{print $4}'`

#
# Create the snapshot.
#
DISKSPEC=""
for t in $TARGETS; do
    DISKSPEC="$DISKSPEC --diskspec $t,snapshot=external"
done
virsh snapshot-create-as --domain "$DOMAIN" --name backup --no-metadata \
	--atomic --disk-only $DISKSPEC >/dev/null
if [ $? -ne 0 ]; then
    echo "Failed to create snapshot for $DOMAIN"
    exit 1
fi

#
# Copy disk images
#
for t in $IMAGES; do
    NAME=`basename "$t"`
    cp "$t" "$BACKUP"/"$NAME"
done

#
# Merge changes back.
#
BACKUPIMAGES=`virsh domblklist "$DOMAIN" --details | grep ^file | awk '{print $4}'`
for t in $TARGETS; do
    virsh blockcommit "$DOMAIN" "$t" --active --pivot >/dev/null
    if [ $? -ne 0 ]; then
        echo "Could not merge changes for disk $t of $DOMAIN. VM may be in invalid state."
        exit 1
    fi
done

#
# Cleanup left over backup images.
#
for t in $BACKUPIMAGES; do
    rm -f "$t"
done

#
# Dump the configuration information.
#
virsh dumpxml "$DOMAIN" >"$BACKUP/$DOMAIN.xml"

#
# Cleanup older backups.
#
LIST=`ls -r1 "$BACKUPDOMAIN" | grep -E '^[0-9]{4}-[0-9]{2}-[0-9]{2}\.[0-9]+$'`
i=1
for b in $LIST; do
    if [ $i -gt "$MAXBACKUPS" ]; then
        echo "Removing old backup "`basename $b`
        rm -rf "$b"
    fi

    i=$[$i+1]
done

echo "Finished backup"
echo ""

Thanks @Ryushin this is really helpful for me especially for newbiees to bacula/bareos:) I will try bacup with bareos asap :)

Don't thank me yet. You might find my setup a bit complicated. I have scripts in bareos/scripts that I call for backups. Along with sending backups and important files to other hosts. Take your time and go through it all. Use my stuff to create your setup. I also call sudo for a few scripts so you will need to create a bareos sudo user to call certain programs.

Hi everyone, did anyone adopt this to backup disks, which are also connected from pools?

Hi guys

Below is my actual time-tested custom console backup solution to implement regular nighlty increment backups of live kvm machines w/o shutting the pool of virtual servers off in a completely automated non-interactive mode

The flow is mostly built upon great cli backup tool virtnbdbackup developed by @abbbi in Python with a number of extra related subroutines

Long story short

1. Every night the routine produces 3 fresh backup copies, 2 local and 1 remote. Local backups are handled by dedicated 2Tb nvme
   drive, mounted under /backup point. Remote copies occupy remote rented storage box.
2. Each vm is supplied with 2 virtual drives - vda with OS / data in qcow2 format and raw vdb utilized as swap partition. The last one is excluded from backups for obvious reasons.
3. On 1st night of each month the fresh full backup is created. Then night by night it gets incremented from the live running machines during a month till the next one, when the cycle repeats.
4. Primary local vm backups are stored under /backup/storage/kvm folder, the full path for each month is prepared automatically with this script by the scheme: /backup/storage/kvm/[guest-name]/MM.YYYY
5. On 1st night of a month the step before full backups are created is the only step when machines are shutting down for a short term to maintain their virtual vda drives. Each drive gets defragmented and the wasted space are reclaimed to speed up the storage for the next month. For safety reasons drives are not defragmented in place, a new defragmented copy for each drive is created under the /tmp/ folder. And only If the image defragmentation tool and another image tool, utilized for post-check, both report zero status, castling is taking place.
6. As soon as the process is fully automated, to insure vms against epic fail with a plot when despite zero statuses of tools defragmented drive appears broken and rejected by Qemu, the working origin of a drive in a state before the last defragmentaton is not wiped immediately, but instead is stored under /backup/storage/kvm/dev/block folder. A few days later, when admin feels completely assure a new copies of drives after defragmentation are working fine, she can drop these obsoleted extra backups by hands.
7. The fresh defragmented copy of drive from /tmp/ takes places of the old one which has been moved to /backup/storage/kvm/dev/block, vm is started and the new full backup is created.
8. Every night on the next step after backups are created or incremented, the whole tree under /backup/storage/kvm gets packed into squashfs entity and stored under the name /backup/storage/squashfs/kvm/kvm.sqsh to be the 2nd local copy in case the 1st one gets in a trouble.
9. Some time later close to the dawn lftp tool (which are not covered by this script since it runs it's own separate plot aimed for remote) will locate this fresh night build of kvm.sqsh and transfer it to the remote storage box with other fresh backups in squashfs containers, thus providing the 3rd remote backup copy in case some evil thing exterminates both local along with live origin

The described flow is implemented night by night by the script below in a completely automated non-interactive mode with neither excess nor complain for over 2 last years, so I suppose for this moment it has been tested live long enough to share for the community with no worry

Please feel free to utilize this stuff for own purposes, to ask questions, to request related to this routine extra info (kinda mentioned above automated non-interactive lftp plot to populate backups to the remote), to discuss alternatives, or just to say thanks or hi

Glory to Ukraine! 🇺🇦
Juliy V. Chirkov, https://t.me/juliyvchirkov

#!/usr/bin/env bash
#
# Implements completely automated non-interactive backup routine for live kvm pool
#
# Designed, developed and tested in a long run under Ubuntu 20.04.4 and 22.04.3 at
# 2021-2023 by Juliy V. Chirkov <juliyvchirkov@gmail.com> https://juliyvchirkov.github.io/
# under the MIT licence https://juliyvchirkov.mit-license.org/2021-2023
#

[[ -z "${SHDEBUG}" ]] || set -vx

virtnbdbackupSocket="/var/tmp/virtnbdbackup.sock"
virtnbdbackupBackupType="stream"
virtnbdbackupBackupLevel="inc"
virtnbdbackupExcludeDrives="vdb"
virtnbdbackupWorkers="$(/usr/bin/nproc)"

vmBackupRoot="/backup/storage/kvm"
vmBackupImagesB4Defragment="${vmBackupRoot}/dev/block"
vmBackupSubstorageThisMonth="$(/usr/bin/date +%m.%Y)"

SQUASHFS="/backup/storage/squashfs/kvm"

defragment() {
    local kvmImg="$(/usr/bin/virsh domblklist "${1}" | /usr/bin/grep -Po "vda\s+\K\N+")"
    local tmpImg="/tmp/${kvmImg##*/}"
    local elapsed=0
    local restartLibvirtGuests
    local restartLibvirtd

    /usr/bin/virsh shutdown "${1}" --mode=agent &>/dev/null

    while /usr/bin/virsh list | /usr/bin/grep "${1}" -q; do
        /usr/bin/sleep 1

        elapsed=$(( elapsed + 1 ))

        [[ "${elapsed}" -eq 180 ]] && /usr/bin/virsh shutdown "${1}" &>/dev/null
    done

    if /usr/bin/virt-sparsify --compress "${kvmImg}" "${tmpImg}" && 
       /usr/bin/qemu-img check "${tmpImg}" &>/dev/null; then
        /usr/bin/virsh checkpoint-delete "${1}" --checkpointname virtnbdbackup.0 --children

        /usr/bin/systemctl -q is-active libvirt-guests && {
            restartLibvirtGuests=1
            /usr/bin/systemctl stop libvirt-guests
        }

        /usr/bin/systemctl -q is-active libvirtd && {
            restartLibvirtd=1
            /usr/bin/systemctl stop libvirtd
        }

        [[ -d "/var/lib/libvirt/qemu/checkpoint/${1}" ]] &&
            /usr/bin/rm -rf "/var/lib/libvirt/qemu/checkpoint/${1}"

        /usr/bin/chown root:kvm "${tmpImg}"

        /usr/bin/mv "${kvmImg}" "${vmBackupImagesB4Defragment}/${1}.${vmBackupSubstorageThisMonth}.vda"

        /usr/bin/mv "${tmpImg}" "${kvmImg}"

        [[ -z "${restartLibvirtd}" ]] || /usr/bin/systemctl start libvirtd
        [[ -z "${restartLibvirtGuests}" ]] || /usr/bin/systemctl start libvirt-guests
    fi

    /usr/bin/virsh start "${1}"
    /usr/bin/sleep 30s
}

[[ -d "${vmBackupRoot}" ]] || /usr/bin/mkdir -pm755 "${vmBackupRoot}"
[[ -d "${SQUASHFS}" ]] || /usr/bin/mkdir -m755 "${SQUASHFS}"

mapfile -t vmlist < <(/usr/bin/virsh list | /usr/bin/grep -oP "[-\d]+\s+\K[^\s]+" || :)

for vm in "${vmlist[@]}"; do
    [[ -d "${vmBackupRoot}/${vm}" ]] || /usr/bin/mkdir -m755 "${vmBackupRoot}/${vm}"

    [[ -d "${vmBackupRoot}/${vm}/${vmBackupSubstorageThisMonth}" ]] || {
        /usr/bin/mkdir -m755 "${vmBackupRoot}/${vm}/${vmBackupSubstorageThisMonth}"
        virtnbdbackupBackupLevel="full"
        defragment "${vm}"
    }

    /usr/bin/virtnbdbackup --domain "${vm}" \
                           --socketfile "${virtnbdbackupSocket}" \
                           --level "${virtnbdbackupBackupLevel}" \
                           --type "${virtnbdbackupBackupType}" \
                           --exclude "${virtnbdbackupExcludeDrives}" \
                           --worker "${virtnbdbackupWorkers}" \
                           --output "${vmBackupRoot}/${vm}/${vmBackupSubstorageThisMonth}" \
                           --compress
done

[[ -f "${SQUASHFS}/kvm.sqsh" ]] && /usr/bin/rm "${SQUASHFS}/kvm.sqsh"

/usr/bin/mksquashfs "${vmBackupRoot}/" "${SQUASHFS}/kvm.sqsh" \
            -noI -noD -noF -noX -comp xz -Xbcj x86 -keep-as-directory

@juliyvchirkov thanks for sharing! Just a few notes from my side:

virtnbdbackupWorkers="$(/usr/bin/nproc)"

this doesnt make quite that much sense, virtnbdbackup uses one worker for each disk the virtual machine has attached, to speed up the backup process (process multiple disks in different threads).

So it is always limited to the amount of disks, not to the amounts of cpu's that your host system has. If you use the amount of cpus (which is most probably bigger than the amount of disks the virtual machine has) it will always default to the amount of disks, as the value is bigger ;)). So if you dont want to limit the amount of concurrent disks to process during backup, leave this value to default or limit it to a lower amount of workers if you want to throttle backup process.

virtnbdbackupSocket="/var/tmp/virtnbdbackup.sock"

by default virtnbdbackup uses a process related socket file, that allows to start multiple backups from different virtual
machines at the same time. If you ever want to enhance your script to backup multiple virtual machines at the same time (not sequentially as now) you dont want to set this option to a hardcoded socked.

Is there a reason you all do not use blockcopy? I can backup/clone complete VM's without shutting them down. A snapshot relies on the last backup.
So a mix of a weekly Full Backup and snapshots would be a smart thing right? I have to read up on it, but when I make 6 snapshots and one full backup, I need basically just the fullbackup plus the latest snapshot to restore my machine.

Is there a reason you all do not use blockcopy? I can backup/clone complete VM's without shutting them down. A snapshot relies on the last backup. So a mix of a weekly Full Backup and snapshots would be a smart thing right? I have to read up on it, but when I make 6 snapshots and one full backup, I need basically just the fullbackup plus the latest snapshot to restore my machine.

The point of this script (at least the last time I checked it), is to pivot your live OS to a temporary snapshot file, while it is still booted and running. This frees up the ability to copy the OS virtual disk without risk of data corruption while it is running live. Once the backup of the VM disk is finished, the live snapshot is pivoted back onto the VM disk file, and resumes (or rather, maintains) live running of the OS.

Example of how I automate this script in the following loop and it backs up all the VMs on the system. Note: I did have to make sure permissions were correct on the disk image files so the script would actually back them up, but besides that, it works great! Love it! Thanks for this.

I also rsync the /etc directory to the backup drive so I have a copy of the existing configs. I suppose I should copy the /etc directory into a date-stamped directory with the images, I just haven't written that yet.

#!/bin/bash
date
date > /etc/date.log

BACKUPDIR=/media/usbbackup
IMGDIR=$BACKUPDIR/image-backups
# However many backups you want to keep...
NUMBACKUPS=5

rsync -arp --delete /etc $BACKUPDIR

# Selects all domains (on or off) to begin the loop (i.e. VMs with a number: running, or VMs with a dash [-]: stopped.)
for VM in $( virsh list --all | awk '( $1 ~ /^[0-9]|-+$/ ) { print $2 }' )
do
        /usr/local/bin/vm-backup.sh $IMGDIR $VM $NUMBACKUPS
        #echo "Next"
done
date
exit

Example of how I automate this script in the following loop and it backs up all the VMs on the system. Note: I did have to make sure permissions were correct on the disk image files so the script would actually back them up, but besides that, it works great! Love it! Thanks for this.

I also rsync the /etc directory to the backup drive so I have a copy of the existing configs. I suppose I should copy the /etc directory into a date-stamped directory with the images, I just haven't written that yet.

#!/bin/bash
date
date > /etc/date.log

BACKUPDIR=/media/usbbackup
IMGDIR=$BACKUPDIR/image-backups

However many backups you want to keep...

NUMBACKUPS=5

rsync -arp --delete /etc $BACKUPDIR

Selects all domains (on or off) to begin the loop (i.e. VMs with a number: running, or VMs with a dash [-]: stopped.)

for VM in $( virsh list --all | awk '( $1 ~ /^[0-9]|-+$/ ) { print $2 }' )
do
/usr/local/bin/vm-backup.sh $IMGDIR $VM $NUMBACKUPS
#echo "Next"
done
date
exit

Instead of using awk, you can use virsh list --all --name.