Skip to content

Instantly share code, notes, and snippets.

@dguerri
Last active August 29, 2015 14:21
Show Gist options
  • Save dguerri/70efcfe0c3ae0fac8bd7 to your computer and use it in GitHub Desktop.
Save dguerri/70efcfe0c3ae0fac8bd7 to your computer and use it in GitHub Desktop.

Forward incremental Backup

Initial configuration. "Current image" pointer is an abstraction for the image currently used by qemu to run a given domain.

  Current image
        |
        v
+-----------------+
|   disk.qcow2    |
+-----------------+

Run fi-backup.

                             Current image
                                   |
                                   v
+------------------+   +-------------------------+
|    disk.qcow2    |<--| disk.qcow2-<timestamp1> |
+------------------+   +-------------------------+

After the first run, disk.qcow2 is the backing file of disk.qcow2-<timestamp1>. That means that disk.qcow2 is used in read-only mode by qemu. Thus, it can be safely copied to perform our backup.

                                                           Current image
                                                                 |
                                                                 v
+------------------+   +-------------------------+   +-------------------------+
|    disk.qcow2    |<--| disk.qcow2-<timestamp1> |<--| disk.qcow2-<timestamp2> |
+------------------+   +-------------------------+   +-------------------------+

After the second run, disk.qcow2 has not changed, so we can just copy disk.qcow2-<timestamp1>: this is the "key" of forward incremental backup.

Consolidation

Consolidation copies the content of backing files into the current image, allowing us to remove them. This reduces the number of image files, reduces the size of disk images and improves domain performances.

                                                           Current image
                                                                 |
                                                                 v
+------------------+   +-------------------------+   +-------------------------+
|    disk.qcow2    |<--| disk.qcow2-<timestamp1> |<--| disk.qcow2-<timestamp2> |
+------------------+   +-------------------------+   +-------------------------+

Running in consolidation mode will "copy" backing images content back into the current image.

                                                          Current image
                                        +-----bpull----+        |
                                        |              v        v
+---------------+   +-------------------------+   +-------------------------+
|  disk.qcow2   |<--| disk.qcow2-<timestamp1> | X | disk.qcow2-<timestamp2> |
+---------------+   +-------------------------+   +-------------------------+

After that, old backing files can be deleted.

       Current image
             |
             v
+-------------------------+
| disk.qcow2-<timestamp2> |
+-------------------------+

Restore

Restore procedure for a domain A, with images in directory B, backup directory C is the following. Thanks to svorobyov for these steps.

  1. clean up the directory B from all images for domain A;
  2. copy the desired chains of the backing images for domain A from C to B (although it is not recommended, chains can be of different lengths, e.g., recover disk 1 to the state two days back and disk 2 to the state three days);
  3. specify (using virsh, virt-manager, virt-install, ...) that the disks for A are the last images of the chains in 2 (this is important, otherwise the VM state description may be inconsistent)
  4. optionally, consolidate for domain A (which will use chains of images in B)
@svorobyov
Copy link

Thanks a lot, and a good description, I think.

A few corrections:

  1. "qcow2-" actually disappears in all occurrences of the kind qcow2-<timestamp...> in the drawings above;
  2. in the "Consolidation" section, replace "After that, old backing files can be deleted." with:

When consolidation finishes, all intermediate backing files are deleted automatically, and only two images remain: disk.qcow2 and disk. (a warning is printed).

After that disk.qcow2 may be deleted, and the current state becomes:

... your last drawing ...

  1. Actually, I think (I have not tested it yet), that what you describe in "Consolidation" only happens when consolidation is run non-live (on a shutdown VM). If you perform a live consolidation,
  • disk. is frozen (snapshot)
  • a new disk. is created (and being modified)
  • then everything happens as you describe, and in the end 3 files remain: disk.qcow2, disk., and disk., of which disk. is backing for disk.; disk.qcow2 can be deleted.

@dguerri
Copy link
Author

dguerri commented May 26, 2015

Doh, I forgot to remove the extension.
About live consolidation, blockpull is performed on the current image, so no additional disks are created

Domain creation

root@ubuntu:~/LibVirtKvm-scripts# ls /home/davide/a\ b/
disk1.qcow2  disk2.qcow2
root@ubuntu:~/LibVirtKvm-scripts# virt-install --virt-type qemu --import --name test-2disks --ram 1024 --disk path=/home/davide/a\ b/disk1.qcow2,device=disk,format=qcow2,bus=virtio --disk path=/home/davide/a\ b/disk2.qcow2,format=qcow2,bus=virtio --vcpus=1 --noautoconsole 

Starting install...
Creating domain...                                                                                                                                                                              |    0 B     00:00     
Domain creation completed. You can restart your domain by running:
  virsh --connect qemu:///system start test-2disks

root@ubuntu:~/LibVirtKvm-scripts# virsh -q -r domblklist test-2disks --details
file       disk       vda        /home/davide/a b/disk1.qcow2
file       disk       vdb        /home/davide/a b/disk2.qcow2

First backup

root@ubuntu:~/LibVirtKvm-scripts# ./fi-backup.sh -d test-2disks
[DEB] libVirt version '1.2.2' is supported
[DEB] /usr/bin/qemu-img version '2.0.0' is supported
[DEB] KVM version '2.0.0' is supported
[DEB] Snapshot for domain 'test-2disks' requested
[DEB] Using timestamp '20150526-215616'
[DEB] Snapshotting block devices for 'test-2disks' using suffix 'bimg-20150526-215616'
[VER] Snapshot for block devices of 'test-2disks' successful
[DEB] No backup directory specified

Second backup

root@ubuntu:~/LibVirtKvm-scripts# ./fi-backup.sh -d test-2disks
[DEB] libVirt version '1.2.2' is supported
[DEB] /usr/bin/qemu-img version '2.0.0' is supported
[DEB] KVM version '2.0.0' is supported
[DEB] Snapshot for domain 'test-2disks' requested
[DEB] Using timestamp '20150526-215619'
[DEB] Snapshotting block devices for 'test-2disks' using suffix 'bimg-20150526-215619'
[VER] Snapshot for block devices of 'test-2disks' successful
[DEB] No backup directory specified

Current images

root@ubuntu:~/LibVirtKvm-scripts# virsh -q -r domblklist test-2disks --details
file       disk       vda        /home/davide/a b/disk1.bimg-20150526-215619
file       disk       vdb        /home/davide/a b/disk2.bimg-20150526-215619

Consolidation

root@ubuntu:~/LibVirtKvm-scripts# ./fi-backup.sh -dc test-2disks
[DEB] libVirt version '1.2.2' is supported
[DEB] /usr/bin/qemu-img version '2.0.0' is supported
[DEB] KVM version '2.0.0' is supported
[DEB] Consolidation of block devices for 'test-2disks' requested
[DEB] Block devices to be consolidated:
  /home/davide/a b/disk1.bimg-20150526-215619 /home/davide/a b/disk2.bimg-20150526-215619
[DEB] Consolidation of block device: '/home/davide/a b/disk1.bimg-20150526-215619' for 'test-2disks'
[DEB] Parent block device: '/home/davide/a b/disk1.bimg-20150526-215616'
[VER] Consolidation of block device '/home/davide/a b/disk1.bimg-20150526-215619' for 'test-2disks' successful
[VER] Deleting old backup files for 'test-2disks'
[DEB] Processing old backing file '/home/davide/a b/disk1.bimg-20150526-215616' for 'test-2disks'
[DEB] Parent backing file: '/home/davide/a b/disk1.qcow2'
[VER] Deleting backing file '/home/davide/a b/disk1.bimg-20150526-215616'
[DEB] Next file in backing file chain: '/home/davide/a b/disk1.qcow2'
[DEB] Processing old backing file '/home/davide/a b/disk1.qcow2' for 'test-2disks'
[WAR] '/home/davide/a b/disk1.qcow2' doesn't seem to be a backup backing file image.
[WAR] Stopping backing file chain removal (manual intervetion might be required)
[WAR] This is expected if this is the first consolidation
[DEB] Consolidation of block device: '/home/davide/a b/disk2.bimg-20150526-215619' for 'test-2disks'
[DEB] Parent block device: '/home/davide/a b/disk2.bimg-20150526-215616'
[VER] Consolidation of block device '/home/davide/a b/disk2.bimg-20150526-215619' for 'test-2disks' successful
[VER] Deleting old backup files for 'test-2disks'
[DEB] Processing old backing file '/home/davide/a b/disk2.bimg-20150526-215616' for 'test-2disks'
[DEB] Parent backing file: '/home/davide/a b/disk2.qcow2'
[VER] Deleting backing file '/home/davide/a b/disk2.bimg-20150526-215616'
[DEB] Next file in backing file chain: '/home/davide/a b/disk2.qcow2'
[DEB] Processing old backing file '/home/davide/a b/disk2.qcow2' for 'test-2disks'
[WAR] '/home/davide/a b/disk2.qcow2' doesn't seem to be a backup backing file image.
[WAR] Stopping backing file chain removal (manual intervetion might be required)
[WAR] This is expected if this is the first consolidation

Current images

root@ubuntu:~/LibVirtKvm-scripts# virsh -q -r domblklist test-2disks --details
file       disk       vda        /home/davide/a b/disk1.bimg-20150526-215619
file       disk       vdb        /home/davide/a b/disk2.bimg-20150526-215619

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment