PCS CMD error-handling howto

pcs.cmd..error-handling.sh

#!/bin/bash

# FOR TRAINING ans STUDYING ONLY
# SOURCE Gianpaolo Del Matto
# http://phaq.phunsites.net/2010/11/22/trap-errors-exit-codes-and-line-numbers-within-a-bash-script/
################################


# trap handler: print location of last error and process it further
#
function my_trap_handler()
{
        MYSELF="$0"              # equals to my script name
        LASTLINE="$1"            # argument 1: last line of error occurence
        LASTERR="$2"             # argument 2: error code of last command
        echo "${MYSELF}: line ${LASTLINE}: exit status of last command: ${LASTERR}"

        # do additional processing: send email or SNMP trap, write result to database, etc.
}

# trap commands with non-zero exit code
#
trap 'my_trap_handler ${LINENO} $?' ERR


# we need to process the DIRECTORY at /ksdjhfskdfkshd
#
my_directory=/ksdjhfskdfkshd


# test if the directory exists
#
echo -n "check if directory exists: '${my_directory}': "

if [ -d "${my_directory}" ]; then
        echo "ok."

        # try to delete the file
        #
        rmdir "${my_directory}"

        # try to handle delete failure (exit != 0)
        #
        if [ "$?" = "0" ]; then
                echo "ok: directory deleted."
        else
                echo "failed: directory not deleted."
        fi
else
        echo "failure, directory does not exist or is not a directory."
fi

# we're done
#
exit

This would result in the following output. The trap is completely circumvented by testing the directory for it’s existence before deleting it.

$ bash test.sh
check if file exists: '/ksdjhfskdfkshd': failure, directory does not exist or is not a directory.

One more thing to consider is what happens, if a trap is caught within a control structure block (IF, WHILE, etc).

To illustrate this, I create the directory I wanted to delete, so the script will dive into the IF-THEN-ELSE-FI block.

$ mkdir /ksdjhfskdfkshd

I then changed the “rmdir” command within the script to “rmdiir” (misspelled), so it WILL fail upon execution.
Please note the script output we get this time:

$ bash test.sh
check if file exists: '/ksdjhfskdfkshd': ok.
test.sh: line 34: rmdiir: command not found
test.sh: line 45: exit status of last command: 127
failed: directory not deleted.

While Bash itself states the misspelled command being on line 34, the trap catches the error on line 45.

Now, why is this?

The reason for this is very simple: Any control structure is regarded as some sort of multi-line command within the script.
So when the trap catches the erronous command on line 34, it sees it’s origin on line 45 because the “IF-THEN-ELSE-FI” clause ends on line 45.

The same happens if you use any other control structure.

The trap in this case is only capable of outlining the “general direction” to where the error happened, but it cannot pin-point to it.
So it’s still recommended to also capture the script output, either by redirecting the script output manually from the shell, or more elegantly by adding some lines to the script, which always redirect the output (STDERR and STDOUT) to a logfile automatically.

In the end, the trap handler could be setup to send and email using both information from the intercepted trap and the logfile.

So the final script may look like this:

#!/bin/bash


# initialize upon startup
#
my_temp_dir=`mktemp -d /tmp/test.XXXXXX`		# we want a unique temp dir
my_log_file=${my_temp_dir}/output.log
my_out_pipe=${my_temp_dir}/output.pipe


# initialize output redirection
#
mkfifo ${my_out_pipe}                                   # we need a FIFO for the output pipe
exec 3>&1                                               # assign a new file descriptor 3 to STDOUT
tee ${my_log_file} &3 &               # background redirect file descriptor 3 throught the FIFO
tee_pid=$!                                              # store the PID for 'tee'
exec 2>&1 > ${my_out_pipe}                              # redirect STDERR and STDOUT to the output pipe


# do some final cleanup upon exit
#
function my_exit_handler()
{
     exec 1>&3 3>&-                                    # restore file descriptors
     wait ${tee_pid}                                   # wait for 'tee' to exit

     # remove temp dir upon exit
     #
     #[ -d ${my_temp_dir} ] && rm -rf ${my_temp_dir}
}

# trap handler: print location of last error and process it further
#
function my_trap_handler()
{
        MYSELF="$0"               # equals to my script name
        LASTLINE="$1"            # argument 1: last line of error occurence
        LASTERR="$2"             # argument 2: error code of last command
        echo "script error encountered at `date` in ${MYSELF}: line ${LASTLINE}: exit status of last command: ${LASTERR}"

        # do additional processing: send email or SNMP trap, write result to database, etc.
	#
	# let's assume we send an email message with
	# subject: "script error encountered at `date` in ${MYSELF}: line ${LASTLINE}: exit status of last command: ${LASTERR}"
	# with additional contents of ${my_log_file} as email body
}


# trap commands with non-zero exit code
# trap script EXIT
#
trap 'my_trap_handler ${LINENO} $?' ERR
trap 'my_exit_handler' EXIT


# we need to process the DIRECTORY at /ksdjhfskdfkshd
#
my_directory=/ksdjhfskdfkshd

# test if the file exists
#
echo -n "check if file exists: '${my_directory}': "

if [ -d "${my_directory}" ]; then
	echo "ok."

	# try to delete the file
	#
	rmdir "${my_directory}"

	# try to handle delete failure (exit != 0)
	#
	if [ "$?" = "0" ]; then
	        echo "ok: directory deleted."
	else
	        echo "failed: directory not deleted."
	fi
else
	echo "failure, directory does not exist or is not a directory."
fi


# we're done
#
exit

To conclude this: Adding some further logic to a script using output redirection and traps adds some great debugging aid, especially when it comes to unknown and yet-undiscovered errors.
The implementation requires just a few additional lines to work with any script and will save countless hours worth of debugging.
Also, the sample trap handler presented herein can be extended to do virtually anything, from adding additional information like the environment, to submitting errors into a MySQL database, sending SNMP traps, or whatever you could imagine.