rdelta-ghetto.sh

#!/bin/bash
##############################################################
##
## rdelta-ghetto.sh - create reverse-delta incremental backups from 
## ghettoVCB ESXi backups.
##
## Author : A. Angelatos (@thanosa75)
## History:
##  - 2018/06/22 - first version
##  - 2018/06/26 - found/fixed issue with ghetto rotation
##
##
##          !!!!!!!! IMPORTANT WARNING !!!!!!!! 
##
## THIS SCRIPT ONLY WORKS IF ghettoVCB.sh IS ALTERED TO PROPERLY
## SORT THE DIRECTORIES ON ROTATION. IT DOES NOT WORK IF YOU 
## LEAVE IT AS-IS. 
## THE FIX IS SIMPLE: REPLACE 'ls -t' with 'ls | sort -r ' 
## 
## Licensed under the 3-Clause BSD License, see below:
## 
## Copyright 2018 Athanasios Angelatos
## Redistribution and use in source and binary forms, with or 
## without modification, are permitted provided that the following 
## conditions are met:
## 1. Redistributions of source code must retain the above 
##    copyright notice, this list of conditions and the following 
##    disclaimer.
## 2. Redistributions in binary form must reproduce the above 
##    copyright notice, this list of conditions and the following 
##    disclaimer in the documentation and/or other materials provided 
##    with the distribution.
## 3. Neither the name of the copyright holder nor the names of its 
##    contributors may be used to endorse or promote products derived 
##    from this software without specific prior written permission.
## THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 
## "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 
## LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 
## FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 
## COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 
## INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, 
## BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 
## LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 
## CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 
## LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY 
## WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY 
## OF SUCH DAMAGE._
## 
##############################################################
## Configuration section

# nice the process from default (10) to a lesser value.
NICELEVEL=15 

# MEMORY PARAMETERS
# Most of the parameters below affect memory use of the process
# window for delta calculation (16MB)
DELTAW=16777216

# instruction window (192k)
INSTRW=192000

# secondary compression to use (djw / lzma)
SECCOMP=lzma

# block size for matching, 128MB
BLOCKSIZ=128000000

# .vmdk files should be larger than this (512k) to be delta'ed  
VMDKSIZ=512000
##############################################################

## the 1st argument is the directory that contains the backups
## for the given VM. Script works on a VM basis.
BACKUP_DIR="$1"

if [ "xx$1" == "xx" ] ;
then
  echo "Usage: $0 <directory for VM backups of ghettoVCB>"
  exit 1
fi

## look for STATUS.ok (produced by ghettoVCB) and try to find
## which of the previous backups (going back in time) is the oldest
## that we can use to diff against. This will update 2 variables:
## LASTDIR and LASTP. 
##
findLastFull() {
  # find the oldest "STATUS.ok" file
  P=0
  THIS=$(date +%Y-%m-%d -d "$P day ago")
  DIR=$(find "$BACKUP_DIR" -type d -name "*${THIS}*")
  while [ -f "$DIR/STATUS.ok" ]
  do
    if [ -f "$DIR/STATUS.xdelta" ] ;
    then
      echo "1st delta found in $DIR";
      break;
    fi
    LASTDIR=$DIR
    LASTP=$P
    CURRP=$(( $P - 1 ))
    #echo "file exists at $P $THIS, going back"
    P=$(( $P + 1 ))
    THIS=$(date +%Y-%m-%d -d "$P day ago")
    DIR=$(find "$BACKUP_DIR" -type d -name "*${THIS}*")
  done
}


## using nice, this method takes two files (FROM,TO) and the 
## directory and filename to produce the xdelta file. 
## Executes xdelta3 with large windows and compression tuned 
## to produce the delta. It is agnostic to whether you do forward or reverse.
deltaFile() {
   FROM="$1"
   TO="$2"
   PATCH="$3/$4.xdelta3"
   echo "Delta $PATCH of $FROM vs $TO ... "
   XD="nice -n$NICELEVEL xdelta3 -W$DELTAW -I$INSTRW -S$SECCOMP -B$BLOCKSIZ -e -s $FROM $TO $PATCH"
   echo "Executing $XD"
   nice -n$NICELEVEL xdelta3 -W$DELTAW -I$INSTRW -S$SECCOMP -B$BLOCKSIZ -e -s "$FROM" "$TO" "$PATCH"
   RET=$?
   echo "Exit code $RET"
   return $RET
}

## loops through the BACKUP dir to find vmdk files for backup. 
## this method only checks files larger than 512k, so meta-vmdk files are
## ignored.
## 
doDeltas() {
  DF=$(date +%Y-%m-%d -d "$1 day ago")
  DT=$(date +%Y-%m-%d -d "$2 day ago")
  DIRF=$(find "$BACKUP_DIR" -type d -name "*${DF}*")
  DIRT=$(find "$BACKUP_DIR" -type d -name "*${DT}*")

  echo "Diff between $DIRF and $DIRT"
  for FILE in "$DIRF"/*vmdk; do
    FN=$( basename "$FILE" )
    if [[ $(find "$DIRT/$FN" -type f -size +${VMDKSIZ}c 2>/dev/null) ]];
    then
      echo "Processing $DIRT/$FN ..."
      deltaFile "$FILE" "$DIRT/$FN" "$DIRT" "$FN"
      if [ $? -eq 0 ];
      then
        rm "$DIRT/$FN"
        touch "$DIRT/STATUS.xdelta"
      fi
    else
      echo "$DIRT/$FN is too small to deltadiff, skipping"
    fi
  done
  echo "Completing the diff process"
}

## main 
findLastFull
if [ "$LASTP" == "0" ] ;
then
  echo "No previous backup found; no action."
  exit 0
fi
echo "Found full backup at $LASTDIR, $LASTP and next full at $CURRP"
echo "Starting the delta process..."
doDeltas $CURRP $LASTP
echo "Delta process done."

Once delta vdmk files are created, what would be the process to recover a VM back to a point in time (of course assuming that I have a full backup and diff files everyday since)?

IMHO does not make sense to use this for too far back. I use it for the deltas of the week, obviously can be used for deltas of months etc. The process -same as all delta methods- is to go backwards in time by applying the deltas in sequence: so from T0 (today) you get +delta for T-1, then +delta for T-2 and so on. The furthest jump possible is the delta you have so if lets say you have deltas for weeks and days, you could apply T0 plus delta of week 40 to get to week 40 then delta of days in that week to get to a specific day. Hope this makes sense. I only kept current week plus 2 weeks of full (not delta) backups. Στις Δευ, 21 Δεκ 2020, 04:49 ο χρήστης Daniel Healy < notifications@github.com> έγραψε:

…

***@***.**** commented on this gist. ------------------------------ Once delta vdmk files are created, what would be the process to recover a VM back to a point in time (of course assuming that I have a full backup and diff files everyday since)? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://gist.github.com/20c374fc43512472ad4730ac7029e2bd#gistcomment-3568278>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ7HV2TJEVSV427GMSXGITSV2ZUTANCNFSM4VDTHZPA> .

@thanosa75 - Thanks! My use case is that I have a separate device which is taking backups with ghettoVCB of a single VM. Then, I am rsyncing that entire backup over the WAN to an offsite location. I would like to continue taking daily backups, determining the diff (which practically should mean an incremental backup), then rsync the diff offsite. Repeat for 30 days, then restart the process with another full backup.

Would you think this is practical to use for a disaster recovery process? Also, if I understand the process correctly, ghettoVCB would need to take an entire backup everyday for the diff to work, correct?