Skip to content

Instantly share code, notes, and snippets.

@gersteba
Forked from Jip-Hop/autorun.sh
Created January 17, 2024 08:03
Show Gist options
  • Save gersteba/6b07be49aa94c8df1bb88e7db615987d to your computer and use it in GitHub Desktop.
Save gersteba/6b07be49aa94c8df1bb88e7db615987d to your computer and use it in GitHub Desktop.
Autorun Synology Hyper Backup and Integrity Check with Email Notifications
#!/bin/sh
# This script is to be used in combination with Synology Autorun:
# - https://github.com/reidemei/synology-autorun
# - https://github.com/Jip-Hop/synology-autorun
#
# You need to change the task_id to match your Hyper Backup task.
# Get it with command: more /usr/syno/etc/synobackup.conf
#
# I like to keep "Beep at start and end" disabled in Autorun, because I don't
# want the NAS to beep after completing (could be in the middle of the night)
# But beep at start is a nice way to confirm the script has started,
# so that's why this script starts with a beep.
#
# After the backup completes, the integrity check will start.
# Unfortunately in DSM you can't choose to receive email notifications of the integrity check results.
# So there's a little workaround, at the end of this script, to send an (email) notification.
# The results of the integrity check are taken from the synobackup.log file.
#
# In DSM -> Control Panel -> Notification I enabled email notifications,
# I changed its Subject to %TITLE% and the content to:
# Dear user,
#
# Integrity check for %TASK_NAME% is done.
#
# %OUTPUT%
#
# This way I receive email notifications with the results of the Integrity Check.
#
# Credits:
# - https://github.com/Jip-Hop
# - https://bernd.distler.ws/archives/1835-Synology-automatische-Datensicherung-mit-DSM6.html
# - https://www.beatificabytes.be/send-custom-notifications-from-scripts-running-on-a-synology-new/
task_id=6 # Hyper Backup task id, get it with command: more /usr/syno/etc/synobackup.conf
task_name="USB3 3TB Seagate" # Only used for the notification
/bin/echo 2 > /dev/ttyS1 # Beep on start
startTime=$(date +"%Y/%m/%d %H:%M:%S") # Current date and time
device=$2 # e.g. sde1, passed to this script as second argument
# Backup
/usr/syno/bin/synobackup --backup $task_id --type image
while sleep 60 && /var/packages/HyperBackup/target/bin/dsmbackup --running-on-dev $device
do
:
done
# Check integrity
/var/packages/HyperBackup/target/bin/detect_monitor -k $task_id -t -f -g
# Wait a bit before detect_monitor is up and running
sleep 60
# Wait until check is finished, poll every 60 seconds
/var/packages/HyperBackup/target/bin/detect_monitor -k $task_id -p 60
# Send results of integrity check via email (from last lines of log file)
IFS=''
output=""
title=
NL=$'\n'
while read line
do
# Compute the seconds since epoch for the start date and time
t1=$(date --date="$startTime" +%s)
# Date and time in log line (second column)
dt2=$(echo "$line" | cut -d$'\t' -f2)
# Compute the seconds since epoch for log line date and time
t2=$(date --date="$dt2" +%s)
# Compute the difference in dates in seconds
let "tDiff=$t2-$t1"
# echo "Approx diff b/w $startTime & $dt2 = $tDiff"
# Stop reading log lines from before the startTime
if [[ "$tDiff" -lt 0 ]]; then
break
fi
text=`echo "$line" | cut -d$'\t' -f4`
# Get rid of [Local] prefix
text=`echo "$text" | sed 's/\[Local\]//'`
if [ -z ${title} ]; then
title=$text
fi
output="$output${NL}$text"
done <<<$(tac /var/log/synolog/synobackup.log)
# Hijack the ShareSyncError event to send custom message.
# This event is free to reuse because I don't use the Shared Folder Sync (rsync) feature.
# More info on sending custom (email) notifications: https://www.beatificabytes.be/send-custom-notifications-from-scripts-running-on-a-synology-new/
/usr/syno/bin/synonotify "ShareSyncError" "{\"%OUTPUT%\": \"${output}\", \"%TITLE%\": \"${title}\", \"%TASK_NAME%\": \"${task_name}\"}"
# Sleep a bit more before unmounting the disk
sleep 60
# Unmount the disk
exit 100
@gersteba
Copy link
Author

Hi!

Now I reworked the script trying to adapt the advice given in @Jip-Hop's original autorun.sh project:

#!/bin/sh

# This script is to be called by a Scheduler task as root user,
# having 'Run command / User-defined script' filled in with your script's path.
# i. e. /bin/bash /volume1/Scripts/autobackup.sh
#
# You need to change the task_id to match your Hyper Backup task.
# Get it with command: more /var/packages/HyperBackup/etc/synobackup.conf
# You also need to change the location of USB device and name of the block device associated with 
# the filesystem partition on the USB disk. Find out with command 'df' having the USB device attached.
#
# I like to keep "Beep at start and end" disabled in Autorun, because I don't
# want the NAS to beep after completing (could be in the middle of the night)
# But beep at start is a nice way to confirm the script has started,
# so that's why this script starts with a beep.
#
# After the backup and the version rotation complete, the integrity check will start. 
# If you like to receive the log entries in an e-mail after this script finished,
# check 'Send run details by email' and fill in 'Email' in the Scheduler task settings.
#
# Tested with DSM 7.2-64570 Update 3 and Hyper Backup 4.1.0-3718.
#
# Credits:
# - https://gist.github.com/Jip-Hop/b9ddb2cc124302a5558659e1298c36ec
# - https://derwebprogrammierer.at/

function process_is_active() {
	local _process_name="$1"
	# Find one PID only (-s) across all shells (-x)
	pidof -s -x "/var/packages/HyperBackup/target/bin/${_process_name}" > /dev/null
	return $?
}

task_id=12 # Hyper Backup task id
task_name="[Omnia Auto-Backup]" # Only used for log entries

# Location of USB device and name of the block device associated with the filesystem partition on the USB disk. Find out with command 'df'.
USBDRV=/volumeUSB1/usbshare # See column 'Mounted on' in df result
device=sdq1 # See column 'Filesystem' in df result

#/bin/echo 2 > /dev/ttyS1 # Beep on start

startTime=$(date +"%Y/%m/%d %H:%M:%S")
echo -e "info\t${startTime}\tSYSTEM:\t${task_name} Started." >> /var/log/synolog/synobackup.log

# Backup - Begin
currTime=$(date +"%Y/%m/%d %H:%M:%S") # Current date and time
echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Backup start ..." >> /var/log/synolog/synobackup.log
/usr/syno/bin/synobackup --backup $task_id --type image

sleep 60
while /var/packages/HyperBackup/target/bin/dsmbackup --running-on-dev $device; do
	sleep 60
done
# Backup - End

## Version rotation - Begin
sleep 60
currTime=$(date +"%Y/%m/%d %H:%M:%S")
echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Rotation start ..." >> /var/log/synolog/synobackup.log
while process_is_active 'synoimgbkptool'; do
	sleep 60
done
## Version rotation - End

## Check integrity - Begin
sleep 60
currTime=$(date +"%Y/%m/%d %H:%M:%S")
echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check start ..." >> /var/log/synolog/synobackup.log
/var/packages/HyperBackup/target/bin/detect_monitor --task-id $task_id --trigger --full --guard

sleep 60
while process_is_active 'detect_monitor'; do
	sleep 60
done
## Check integrity - End

# Sleep a bit more before unmounting the disk
sleep 60

## Unmount USB device - Begin
sync
sleep 10
umount $USBDRV
umountResult=$(/usr/syno/bin/synousbdisk -umount $device; >/tmp/usbtab)
currTime=$(date +"%Y/%m/%d %H:%M:%S")
echo -e "info\t${currTime}\tSYSTEM:\t${umountResult}" >> /var/log/synolog/synobackup.log
## Unmount USB device - End

currTime=$(date +"%Y/%m/%d %H:%M:%S")
echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Finished." >> /var/log/synolog/synobackup.log

## Get results of auto backup (from last lines of log file) - Begin
IFS=''
output=()
NL=$'\n'

while read line
do
    
    # Compute the seconds since epoch for the start date and time
    t1=$(date --date="$startTime" +%s)
    
    # Date and time in log line (second column)
    dt2=$(echo "$line" | cut -d$'\t' -f2)
    # Compute the seconds since epoch for log line date and time
    t2=$(date --date="$dt2" +%s)
    
    # Compute the difference in dates in seconds
    let "tDiff=$t2-$t1"
    
    # Stop reading log lines from before the startTime
    if [[ "$tDiff" -lt 0 ]]; then
        break
    fi
    
    #text=`echo "$line" | cut -d$'\t' -f4`
    text=$(echo "$line" | cut -d$'\t' -f4)
    # Get rid of [Local] prefix
    text=$(echo "$text" | sed 's/\[Local\]//')
    # Add date and time
	text=$(echo "${dt2}  ${text}")
    
	output+=("$text")
    
done <<<$(tac /var/log/synolog/synobackup.log)

n=${#output[*]}
for (( i = n-1; i >= 0; i-- ))
do
    
	echo "${output[i]}"
	
done
## Get results ... - End

exit 0

Seems to work fine.
The only thing, I do not understand, yet:
Why does the rotation already start while dsmbackup is still running?
Compare script with following log entries:

2024/01/13 02:00:01  [Omnia Auto-Backup] Started.
2024/01/13 02:00:01  [Omnia Auto-Backup] Backup start ...
2024/01/13 02:00:15  [Omnia Backup] Backup task started.
2024/01/13 16:55:18  [Omnia Backup] Backup task finished successfully. [599281 files scanned] [71 new files] [9 files modified] [599201 files unchanged]
2024/01/13 16:55:19  [Omnia Backup] Trigger version rotation.
2024/01/13 16:55:57  [usbshare1] Version rotation started from ID [Omnia.hbk].
2024/01/13 16:56:57  [Omnia Auto-Backup] Rotation start ...
2024/01/13 18:16:29  [usbshare1] Rotate version [2023-12-09 02:00:39] from ID [Omnia.hbk].
2024/01/13 18:16:30  [usbshare1] Version rotation completed from ID [Omnia.hbk].
2024/01/13 18:16:59  [Omnia Auto-Backup] Integrity check start ...
2024/01/13 18:17:10  [Omnia Backup] Backup integrity check has started.
2024/01/14 06:48:30  [Omnia Backup] Data integrity check finished. 3574.4 GB data checked this time, accounting for 100.0% of the total data (3574.4 GB data in total, 100.0% checked already).
2024/01/14 07:34:47  [Omnia Backup] Backup integrity check is finished. No error was found.
2024/01/14 07:36:12  Unmount USB device sdq1 succeeded.
2024/01/14 07:36:12  [Omnia Auto-Backup] Finished.

When I tried to remove the
while process_is_active 'synoimgbkptool';
check, the integrity check failed, because the rotation was not finished, yet.
So this check is also necessary.

And:
How can I find out, whether the backup, the rotation and the integrity check worked fine or not, and create different texts in the last log entry accordingly?
i. e. 'Finished successfully.' or 'Failed during rotation.'

@JimMeLad
Copy link

Hi! See you've forked successfully :-)
Personally I'm not at all convinced that 'dsmbackup' and 'synobackup' are the right tools for the job. Certainly, on my system at least, if I start a backup process, neither of those commands gets called. Instead a task called 'img_backup' kicks off which in turns spawns a load of worker threads.
My suspicion is that 'dsmbackup' and 'synobackup' are old tools, probably now deprecated, but retained for backward compatibility purposes. I'm therefore not at all surprised that 'dsmbackup' reports that it has finished even though version rotation has kicked off as I guess 'dsmbackup' knows nothing about version rotation.
This, I hope, backs up what I've said on the original thread - I think you're in danger of building a fragile script that is going to be at the mercy of whatever Synology decide to do with HyperBackup in future.

So, having given it a bit of thought, I thought I'd be a bit more creative.
These are my requirements/problem statement for my system:

  1. My backups are the most important thing, whether scheduled or ad-hoc. I need those, and the attendant version rotations, to complete without error 100% of the time.
  2. As a consequence of point 1, I'm going to work with the way the Synology software works as far as I can
  3. I only want to automate the scheduled backup and integrity check tasks as these run unattended. Ad-hoc backups and its integrity check are run manually and be monitored by me.
  4. I will run scheduled backups daily and the integrity check (itgchk) weekly (largely because it takes such a long time to run)
  5. I have no way of reliably predicting how long a backup will take, nor the version rotation. I could take a guess but I know that sometimes that guess will be wrong and the itgchk will fail which is unacceptable to me.
  6. I need a way of holding up the itgchk task until the backup is fully complete, whether it takes 5 minutes or 5 hours

What I've decided to do it to treat the backup tasks as a 'black box' - I can't see into it, have no idea what it's doing under the covers (other than backing up data hopefully), and can't reliably predict when or how it will complete.

So what I need it a way of triggering the start of the itgchk based on a known state of the backup task, and it seems to me that the last thing my backup task does is eject the USB drive (because I've configured it to do so).
I'm then using the ejection of the USB drive as a trigger to start the itgchk task. These are the basic steps:

  1. Scheduled task runs at 00:01 to mount the attached USB drive
  2. Task starts at 00:03 to kick off the itgchk job. This checks that the USB drive is mounted (step 1), and then waits
  3. Backup tasks kicks off at 00:05 (scheduled through the HyperBackup UI). This then goes and does its thing, backing up, rotating versions etc, finally ejecting the USB drive once done
  4. The tasks that was kicked off in step 2 now sees the USB drive has disappeared, so...
  5. ...re-mounts it...
  6. ...calls the 'detect_monitor' to start the itgchk process...
  7. ...waits for 'synoimgbkptool' to finish...
  8. ...unmounts and ejects the USB drive...
  9. ...runs a script to report on the results from the 'synobackup.log' file

Not exactly an answer to your questions but I hope that it helps

@JimMeLad
Copy link

I thought I'd found something useful in here:
/volume1/@appdata/HyperBackup/last_result
I have two files, 'backup.last', and 'detect.last' that look like they are updated by HB, however, looking at the 'backup.last' file for example, the value for 'start_time' is the same as the value for 'end_time' (??!!), and the entry for 'last_backup_success_time' has the time that the backup finished, NOT backup and version rotation.

On my most recent backup run, the version rotation (according to the backup log) ran for approx. another 50 seconds after the 'last_backup_success_time' value was written.

I don't know where you're going to take this next, but if you want someone to bounce ideas off then I'd be happy to help.

@gersteba
Copy link
Author

gersteba commented Jan 24, 2024

  1. Scheduled task runs at 00:01 to mount the attached USB drive
  2. Task starts at 00:03 to kick off the itgchk job. This checks that the USB drive is mounted (step 1), and then waits
  3. Backup tasks kicks off at 00:05 (scheduled through the HyperBackup UI). This then goes and does its thing, backing up, rotating versions etc, finally ejecting the USB drive once done
  4. The tasks that was kicked off in step 2 now sees the USB drive has disappeared, so...
  5. ...re-mounts it...
  6. ...calls the 'detect_monitor' to start the itgchk process...
  7. ...waits for 'synoimgbkptool' to finish...
  8. ...unmounts and ejects the USB drive...
  9. ...runs a script to report on the results from the 'synobackup.log' file

Hi!

Thank you for your thoughts.
The steps you described above seem to make much sense.
There are just some things I don't understand/know:

  • Step 2: How can I check, that the USB drive ist mounted? And what shall the script wait for - the USB drive disappearing?
  • Step 5: How can I re-mount the USB drive in my script?

@JimMeLad
Copy link

Step 2: Add this function before or after your existing 'process_is_active':

function usb_drive_mounted() {
# ------------------------------------------------------------------------------------------------
# Function: usb_drive_mounted()
# Purpose : Check if USB drive is currently mounted to a file system
# Returns : 0=Mounted, 1=Not mounted
# ------------------------------------------------------------------------------------------------

local _usb_mount

# If we haven't been passed a mount point to look for, assume default
if (( $# == 0 )); then
	_usb_mount="/volumeUSB1/usbshare"
else
	_usb_mount="$1"
fi

# Command 'df...' returns 0 if file system is found (mounted), 1 otherwise
# Limit output of 'df' command to just list the filesystem target ('Mounted on')
if df --exclude-type=tmpfs --output=target | grep --quiet "${_usb_mount}"; then
	return 0
else
	return 1
fi

}

Then you can do something like:

_backup_wait_seconds=3600 # MAXIMUM time to wait for backup to complete
_end_time=$(( SECONDS + _backup_wait_seconds ))
while usb_drive_mounted && (( SECONDS < _end_time )); do
sleep 30s
done;

if usb_drive_mounted; then
echo 'Backup task did not eject USB drive before timeout occurred - quitting'
exit 1
fi

At this point, you know the backup has ejected the drive so you can move on to the next step.

@gersteba
Copy link
Author

Thank you for your code.
How can I (re-)mount the USB drive in my script?

@JimMeLad
Copy link

JimMeLad commented Jan 24, 2024

Well the quick and dirty method is to find the usb bus & port number that your usb drive is on (from a terminal session, type 'lsusb' and see if you can see the name of our drive. The value you need are the two numbers on the left side that looks a bit like this:
|__2-1 0bc2:ac35:1708 00 3.20 5000MBit/s 896mA 1IF (Seagate BUP Portable......)
In my case the value I need is '2-1'
Then in your script, at the point where you need to re-mount the drive, issue:

echo '2-1' > '/sys/bus/usb/drivers/usb/unbind'
sleep 2s
echo '2-1' > '/sys/bus/usb/drivers/usb/bind'

(obviously substitute your usb bus&port if not '2-1')
After a few seconds the usb device should appear on your desktop

This is a crude way of achieving the result as it relies on hard-coding the usb details but will do for now just so you can get up and running

@vorezal
Copy link

vorezal commented Feb 14, 2024

For the next person to find this, the re-mount script in the previous comment would be:

echo '2-1' > '/sys/bus/usb/drivers/usb/unbind'
sleep 2s
echo '2-1' > '/sys/bus/usb/drivers/usb/bind'

@vorezal
Copy link

vorezal commented Feb 21, 2024

I found the above scripts very helpful in writing a version for my own purposes, which I ended up completely over-engineering. I originally wanted to support running multiple integrity checks in sequence against multiple HyperBackup jobs with targets on more than one external drive. In the end I also arbitrarily decided I only wanted to have to pass a single argument matching a HyperBackup job name and have multiple instances of the script figure out the rest. Pretty sure I made it way too complicated, but just in case anyone finds it useful I'll post it here anyway.

If you use this, be sure to schedule schedule a job to mount your drive(s), then one job for each HyperBackup task you want to check that calls this script with a matching task name argument, and then your actual backup jobs with the setting to remove the destination external device when the backup has completed selected, in that order. Synology queues HyperBackup jobs if they are scheduled while another is already running, so if you have multiple backup tasks targeting one disk, like me, just schedule them in the proper order one right after the other and only set the final one accessing the target disk to remove it.

There is definitely a fair bit of cleanup I could do and improvements I could make, plus there's a less than perfect locking mechanism and some ideas I had for multiple disk support in the functions ended up being unnecessary but left in anyway for now. Oh well.

Click for code
#!/bin/bash

# This script is to be called by a Scheduler task as root user,
# having 'Run command / User-defined script' filled in with your script's path.
# i. e. /bin/bash /volume1/Scripts/autointegrity.sh "<task string>"
#
# You need to pass a task string to match against.
# This may be a partial string, but must match a HyberBackup job name uniquely.
#
# I like to keep "Beep at start and end" disabled in Autorun, because I don't
# want the NAS to beep after completing (could be in the middle of the night)
# But beep at start is a nice way to confirm the script has started,
# so that's why this script starts with a beep.
#
# The script will wait until any drives are unmounted (via the Synology backup task)
# then re-mount the disk(s) and begin the integrity check(s).
# If you want to receive the log entries in an e-mail after this script finished,
# check 'Send run details by email' and fill in 'Email' in the Scheduler task settings.
#
# Tested with DSM 7.2.1-69057 Update 4 and Hyper Backup 4.1.0-3718 on a DS1821+.
#
# Credits:
# - https://gist.github.com/gersteba/6b07be49aa94c8df1bb88e7db615987d
# - https://gist.github.com/Jip-Hop/b9ddb2cc124302a5558659e1298c36ec
# - https://derwebprogrammierer.at/

function integrity_process_is_active() {
	# ------------------------------------------------------------------------------------------------
	# Function: integrity_process_is_active()
	# Purpose : Determine an integrity check is already running, with or without specific task_id
	# Returns : 0=Running, 1=Not running
	# ------------------------------------------------------------------------------------------------

	local _task_id

	if (( $# == 0 )); then
		_task_id=""
	else
		_task_id=" $1"
	fi

	# Find process(es)
	/bin/ps aux | /bin/grep -v grep | /bin/grep --quiet "detect_monitor \-\-task\-id${_task_id}"
}

function integrity_script_waiting() {
	# ------------------------------------------------------------------------------------------------
	# Function: integrity_script_waiting()
	# Purpose : Determine if other instances of this script are waiting to run integrity checks
	# Returns : 0=Waiting, 1=Not waiting
	# ------------------------------------------------------------------------------------------------

	# Find process(es)
	/bin/ps aux | /bin/grep -v grep | /bin/grep --quiet "Waiting to start integrity checks"
}

function self_is_lowest_pid() {
	# ------------------------------------------------------------------------------------------------
	# Function: self_is_lowest_pid()
	# Purpose : Determine if this instance of the script has the lowest PID
	# Returns : 0=Is lowest, 1=Is not lowest
	# ------------------------------------------------------------------------------------------------

	running_script_pids=$(sub_pid="${BASHPID}"; /bin/ps aux | /bin/grep $0 | /bin/grep -v "grep" | /bin/grep -v "${sub_pid}" | /bin/awk '{print $2}')
	if [[ "$(head -n 1 <<< "${running_script_pids}")" == "${my_pid}" ]]; then
		return 0
	else
		return 1
	fi
}

function usb_drive_mounted() {
	# ------------------------------------------------------------------------------------------------
	# Function: usb_drive_mounted()
	# Purpose : Check if any USB drive is currently mounted to a file system matching string
	# Returns : 0=Mounted, 1=Not mounted
	# ------------------------------------------------------------------------------------------------

	local _search_text
	# If we haven't been passed a mount point to look for, assume default
	if (( $# == 0 )); then
		_search_text="USB"
	else
		_search_text="$1"
	fi

	# Command 'df...' returns 0 if any file system is found (mounted), 1 otherwise
	# Limit output of 'df' command to just list the filesystem target ('Mounted on')
	/bin/df --exclude-type=tmpfs | /bin/grep --quiet "${_search_text}"
}

function get_usb_devices() {
	# ------------------------------------------------------------------------------------------------
	# Function: get_usb_devices()
	# Purpose : Get USB block devices matching a passed string. Default to returning all USB devices
	# Output : USB block device IDs if found, empty string if not found
	# Returns : 0=Completed execution, 1=Error
	# ------------------------------------------------------------------------------------------------

	local _search_text
	local _usb_device_paths
	local _usb_device
	local _usb_devices

	# If we haven't been passed a mount point to look for, assume default
	if (( $# == 0 )); then
		_search_text="USB"
	else
		_search_text="$1"
	fi

	# Command 'df...' returns the USB block device ID, or empty string if not found
	if usb_drive_mounted "${_search_text}"; then
		_usb_device_paths=$(/bin/df --exclude-type=tmpfs | /bin/grep "${_search_text}" | /bin/awk '{print $1}')

		for _usb_device in $_usb_device_paths; do
			_usb_devices+=" ${_usb_device##*/}"
		done

		/bin/echo "${_usb_devices}" | /bin/sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'
	else
		/bin/echo ""
	fi
}

function get_usb_device_ports() {
	# ------------------------------------------------------------------------------------------------
	# Function: get_usb_device_ports()
	# Purpose : Get a USB device bus and port identifier(s) by search string
	# Output : Bus and Port identifier(s) if found, empty string if not found
	# Returns : 0=Completed execution, 1=Error
	# ------------------------------------------------------------------------------------------------

	local _search_text
	local _usb_device
	local _usb_devices
	local _usb_device_port
	local _usb_device_ports

	_search_text="$1"

	_usb_devices=$(get_usb_devices "${_search_text}")

	for _usb_device in ${_usb_devices}; do
		_usb_device_port=$(/bin/udevadm info -q path -n "${_search_text}" | grep -o "/[0-9]-[0-9]/" 2> /dev/null)

		if (( $? == 0 )); then
			_usb_device_ports+=" ${_usb_device_port//\/}"
		fi
	done
	/bin/echo "${_usb_device_ports}" | /bin/sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'
	return 0
}

function get_device_from_share() {
	# ------------------------------------------------------------------------------------------------
	# Function: get_device_from_share()
	# Purpose : Get the device name for a USB disk by remote share name
	# Output : USB device identifier if found, empty string if not found
	# Returns : 0=Completed execution, 1=Error
	# ------------------------------------------------------------------------------------------------

	local _remote_share
	local _disk_info
	local _usb_device_paths

	if (( $# == 0 )); then
		/bin/echo ""
		exit 0
	else
		_remote_share=" $1"
	fi

	_usb_device_paths=$(/bin/df --exclude-type=tmpfs | /bin/grep "/dev/usb" | /bin/awk '{print $1}')

	for _usb_device in ${_usb_device_paths}; do
		_disk_info=$(/usr/syno/bin/synousbdisk -info "${_usb_device##*/}")

		if /bin/grep --quiet "${_remote_share}" <<< "${_disk_info}"; then
			/bin/grep "information:" <<< "${_disk_info}" | awk '{print $2}'
		fi
	done
}

function usb_port_has_mounted_volumes() {
	# ------------------------------------------------------------------------------------------------
	# Function: usb_port_has_mounted_volumes()
	# Purpose : Check if any mounted USB volumes are on a specific USB bus and port
	# Returns : 0=Mounted, 1=Not mounted
	# ------------------------------------------------------------------------------------------------

	local _usb_port
	local _usb_devices

	if (( $# == 0 )); then
		return 1
	else
		_usb_port="$1"
	fi

	_usb_devices=$(get_usb_devices)

	for _usb_device in ${_usb_devices}; do
		local _usb_device_port
		_usb_device_port=$(get_usb_device_ports "${_usb_device}")
		if [[ "${_usb_port}" == "${_usb_device_port}" ]]; then
			return 0
		fi
	done

	return 1
}

function safe_remount_usb_port() {
	# ------------------------------------------------------------------------------------------------
	# Function: safe_remount_usb_port()
	# Purpose : Re-mount a USB port if all devices are unmounted
	# Returns : 0=Found and mounted or detected externally mounted, 1=Could not safely remount
	# ------------------------------------------------------------------------------------------------

	local _usb_port

	_usb_port="$1"

	if [[ -n "$(/usr/syno/bin/lsusb | /bin/grep "${_usb_port}")" ]]; then
		if ! usb_port_has_mounted_volumes "${_usb_port}"; then
			/bin/echo "${_usb_port}" > /sys/bus/usb/drivers/usb/unbind
			/bin/sleep 10
			/bin/echo "${_usb_port}" > /sys/bus/usb/drivers/usb/bind
		else
			return 1
		fi
	else
		return 1
	fi
}

function safe_unmount_usb_device() {
	# ------------------------------------------------------------------------------------------------
	# Function: safe_unmount_usb_device()
	# Purpose : Unmount a USB device
	# Returns : 0=Script ran successfully, 1=Error
	# ------------------------------------------------------------------------------------------------

	local _usb_device
	local _unmount_result

	_usb_device="$1"

	/bin/sync
	/bin/sleep 5
	if usb_drive_mounted "${_usb_device}"; then
		_unmount_result=$(/usr/syno/bin/synousbdisk -umount "${_usb_device}")
		code=$?

		sleep 5;
		if [[ "${code}" == 0 ]]; then
			remove_usb_from_gui "${_usb_device}"
		fi

		/bin/echo "${_unmount_result}"
	fi
}

function remove_usb_from_gui() {
	# ------------------------------------------------------------------------------------------------
	# Function: remove_usb_from_gui()
	# Purpose : Remove a specific USB device(s) from the DSM GUI, if found
	# Returns : 0=Completed execution, 1=Error
	# ------------------------------------------------------------------------------------------------

	local _usb_device
	local _usb_gui_reference

	for _usb_device in "$@"; do
		_usb_gui_reference="${_usb_device%p*}"
		/bin/echo 1 > /sys/block/${_usb_gui_reference}/device/delete;
	done
}

startTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")

if (( $# != 1 )); then
	currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
	(( debug == 1 )) && /bin/echo "This script requires one argument."
	/bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - This script requires one argument." >> /var/log/synolog/synobackup.log
	exit 1
fi

_backup_wait_seconds=7200 # MAXIMUM time to wait for backup to complete
_integrity_wait_seconds=14400 # MAXIMUM time to wait for all integrity checks

task_string="$1"
task_name="[$1]" # Only used for log entries
my_pid="${BASHPID}"
leader=0
serial_integrity=1

synobackup_config_json=$(/bin/cat /var/packages/HyperBackup/etc/synobackup.conf | /bin/awk '/\[global/{g=1} /\[task_[0-9]+|\[repo_[0-9]+/{f=1} !(g || f) { st = index($0,"="); printf "\"" substr($0,0,st-1) "\": " substr($0,st+1) ","; } g{ printf "{ \"global\": {"; g=0 } f{ s=substr($0, 2, length-2); printf "}, \"" s "\": {"; f=0 }; END { printf "}}"}' | /bin/sed 's/,}/}/g')

if ! /bin/jq type >/dev/null 2>&1 <<< "${synobackup_config_json}"; then
	currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
	(( debug == 1 )) && /bin/echo "Unable to parse synobackup.conf to JSON. Exiting."
	/bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - Unable to parse synobackup.conf to JSON. Exiting." >> /var/log/synolog/synobackup.log
	exit 1
fi

my_task_id=$(/bin/jq -r --arg task_string "${task_string}" 'to_entries[] | select(.value.name | strings | contains($task_string)) | .key' <<< "${synobackup_config_json}")
my_task_id_num="${my_task_id##*_}"

if [[ $(/bin/wc -l <<< "${my_task_id}") -gt 1 ]]; then
	currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
	(( debug == 1 )) && /bin/echo "Task string matched more than one HyperBackup job."
	/bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - Task string matched more than one HyperBackup job." >> /var/log/synolog/synobackup.log
	exit 1
fi

if [[  $(/bin/wc -l <<< "${my_task_id}") -eq 0 ]]; then
	currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
	(( debug == 1 )) && /bin/echo "Task string did not match a HyperBackup job."
	/bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - Task string did not match a HyperBackup job." >> /var/log/synolog/synobackup.log
	exit 1
fi

my_repo_id=repo_$(/bin/jq -r --arg task_id "${my_task_id}" '.[$task_id].repo_id' <<< "${synobackup_config_json}")
my_remote_share=$(/bin/jq -r --arg repo_id "${my_repo_id}" '.[$repo_id].remote_share' <<< "${synobackup_config_json}")
my_device=$(get_device_from_share "${my_remote_share}")

if [[ -z "${my_device}" ]]; then
	currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
	(( debug == 1 )) && /bin/echo "No USB device found. Exiting."
	/bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - No USB device found. Exiting." >> /var/log/synolog/synobackup.log
	exit 1
fi

my_device_port="$(get_usb_device_ports "${my_device}")"

if [[ -z "${my_device_port}" ]]; then
	currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
	(( debug == 1 )) && /bin/echo "No USB device port found. Exiting."
	/bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - No USB device port found. Exiting." >> /var/log/synolog/synobackup.log
	exit 1
fi

#/bin/echo 2 > /dev/ttyS1 # Beep on start

currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
(( debug == 1 )) && /bin/echo "Integrity check task started."
/bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - Started." >> /var/log/synolog/synobackup.log

## Get my USB bus and port
## Get all mounted shares on my usb port

## Backup - Begin
_end_time=$(( SECONDS + ${_backup_wait_seconds} ))
while usb_drive_mounted "${my_device}" && (( SECONDS < ${_end_time} )); do
	/bin/sleep 20
done

/bin/sleep 20

if usb_drive_mounted "${my_device}"; then
	currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
	(( debug == 1 )) && /bin/echo "Backup task did not eject USB drive before timeout occurred - exiting."
	/bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - Backup task did not eject USB drive before timeout occurred - exiting." >> /var/log/synolog/synobackup.log
	exit 1
fi
## Backup - End
/bin/sleep 20

## Device mount - Begin
## Remount devices (only one script per usb port, all drives must eject)

## Determine leader by lowest PID. If more than one script instance is scheduled, all should be running at this point.
if self_is_lowest_pid; then
	leader=1
fi

# If leader, wait for all devices on USB port to be unmounted
if [[ "${leader}" == "1" ]]; then
	_end_time=$(( SECONDS + ${_backup_wait_seconds} ))
	
	while usb_port_has_mounted_volumes "${my_device_port}" && (( SECONDS < ${_end_time} )); do
		/bin/sleep 20
	done
	
	/bin/sleep 10
	
	safe_remount_usb_port "${my_device_port}"
fi

## Check if mounted loop
_end_time=$(( SECONDS + ${_backup_wait_seconds} ))
while ! usb_drive_mounted "${my_device}" && (( SECONDS < ${_end_time} )); do
	/bin/sleep 20s
done

## Device mount - End

/bin/sleep $(( ${my_task_id_num} % 60 ))

## Check integrity - Begin
# Sleep for any other running task to run one at a time. Slight race condition possibility here.
# The subshell sleep/test is a very poor substitute for a locking mechanism.
# It can certainly fail, but should be very rare and without significant consequences.
# I also didn't really want to mess with a file based locking mechanism via flock, but it could be done.
if [[ "${serial_integrity}" == "1" ]]; then
	while integrity_process_is_active; do
		/bin/bash -c "/bin/sleep $(( (${my_task_id_num} % 60) * 3 + 30 )); test 'Waiting to start integrity checks'"
	done
fi

/var/packages/HyperBackup/target/bin/detect_monitor --task-id ${my_task_id_num} --trigger --full --guard

# Wait until check is finished, poll every 60 seconds
while integrity_process_is_active "${my_task_id_num}"; do
	/bin/sleep 60
done

## Check integrity - End
currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
(( debug == 1 )) && /bin/echo "Integrity check complete."
(( debug == 1 )) && /bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - Integrity check complete." >> /var/log/synolog/synobackup.log

# Sleep a bit more before attempting to unmount the disk
/bin/sleep 10

## Unmount USB device - Begin
## Wait until all integrity tasks are complete.

while integrity_process_is_active || integrity_script_waiting; do
	/bin/sleep 60
done

## Attempt unmount. If already unmounted, no action will be taken.
unmount_result=$(safe_unmount_usb_device "${my_device%p*}")

currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
(( debug == 1 )) && /bin/echo "Disk unmount result: ${unmount_result}"
(( debug == 1 )) && /bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - ${unmount_result}" >> /var/log/synolog/synobackup.log

## Unmount USB device - End

currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
(( debug == 1 )) && /bin/echo "Integrity check task finished."
/bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - Finished." >> /var/log/synolog/synobackup.log

## Get results of auto backup (from last lines of log file) - Begin
IFS=''
output=()
NL=$'\n'

while read line
do

    # Compute the seconds since epoch for the start date and time
    t1=$(/bin/date --date="${startTime}" +%s)

    # Date and time in log line (second column)
    dt2=$(/bin/echo "${line}" | cut -d$'\t' -f2)
    # Compute the seconds since epoch for log line date and time
    t2=$(/bin/date --date="${dt2}" +%s)

    # Compute the difference in dates in seconds
    let "tDiff=${t2}-${t1}"

    # Stop reading log lines from before the startTime
    if [[ "${tDiff}" -lt 0 ]]; then
        break
    fi

    text=$(/bin/echo "${line}" | /bin/cut -d$'\t' -f4)
    # Get only lines for this task
	text=$(echo "${text}" | sed 's/\[Local\]//')
    text=$(/bin/echo "${text}" | /bin/grep "\[${task_string}\]")
    # Add date and time
	text=$(/bin/echo "${dt2}  ${text}")

	output+=("${text}")

done <<<$(/bin/tac /var/log/synolog/synobackup.log)

n=${#output[*]}
for (( i = n-1; i >= 0; i-- ))
do

	/bin/echo "${output[i]}"

done
## Get results - End

## Hijack the USB Copy package's "Completed a Task" event to send a notification
## Though it does still show it is from the USB Copy application, the rest of the message is generic enough
## This will send a message at the INFO level. USBCOPYError would be a good tag for the WARN level

if /bin/grep --quiet "USBCOPYFinished=" /usr/syno/etc/notification/notification_filter.settings; then
	while ! self_is_lowest_pid; do
		sleep 120
	done

	/usr/syno/bin/synonotify USBCOPYFinished "{\"%COPY%\":\"${task_string} integrity check\"}"
fi

exit 0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment