Bash idioms and tools

bash4thewin.md

This page collects a set of bash scripting idioms and tools for easy reference.

Improved Error Detection: set -e and set -u

With -e (errexit), bash exits immediately if an unchecked command exits with a nonzero value. A checked command is part of a while, until, if, elif, or an expression with && or ||. Everything else is unchecked. For a pipeline, only the last command is considered. Adding "or no-op", || :, ensures a command is safe, e.g., do-something | grep 'may_not_match' || :. NB. Calling a bash function in a checked context effectively disables -e for all commands in the function. Use the optional shellcheck, check-set-e-suppressed, to alert on this.

With -u (nounset), bash exits immediately if there is an unset variable. To permit an optional variable, use ${var:-}, e.g., ${1:-} for an optional function argument.

Idioms

idiom	description
${var:-default}	value of var if var is set, default otherwise
${var:+alt_value}	alt_value if var is set, nothing otherwise, e.g., ${PORT:+-p $PORT}
${var#pattern}	value with matching prefix removed
${var##pattern}	value with longest matching prefix removed
${var%pattern}	value with matching suffix removed
${var%%pattern}	value with longest matching suffix removed
${var%%$'\n'*}	the first line (value with everything after the first newline removed)
${var,,}	value converted to lower case
printf '%(%s)T' -1	print epoch time, seconds since 1970-01-01 00:00:00 UTC
printf -v var '%(%s)T' -1	set var to epoch time
$EPOCHSECONDS	epoch time, available since bash v5.0
do_something 2>&-	discard command output to stderr
local -n var_ref=$1	pass by reference, access a variable in the caller
${0##*/}	script name without directory, same as basename $0
${0%/+([^/])}	directory containing the script, needs shopt -s extglob
files=(glob); ${#files[*]}	count of files matching glob pattern, needs shopt -s nullglob
var=$(<a_file)	set var to contents of a_file
${var:0:1}	the first character of the value
${var%%*(/)}	value with zero or more trailing slashes removed, needs shopt -s extglob
$'\t'	a literal tab character

Longer examples

With -e beware of ending a function with an && or || expression unless the function represents a predicate.

#!/bin/bash

# GOOD
function log1 () {
    if (( VERBOSE )); then
        echo "$*"
    fi
}

# BAD
function log2 () {
    (( VERBOSE )) && echo "$*"
}

set -eu

VERBOSE=0
log1 "this works"
log2 "this trips -e"
echo done  # never reached

The following needs the associative array declared with -A and lastpipe enabled or it fails because of an unbound variable.

#!/bin/bash
# shellcheck disable=SC2034

setup_bash () {
	# catch unexpected errors
	set -eu
	# allow the tail of a pipe to set values in this shell
	shopt -s lastpipe
}

# Snapshot running services (pid != 0) with name and start time
# Use sed to convert stanzas to tabular data
# Populate a dictionary named by $1 with the key/value pairs
load_start_times () {
	local -n dict=$1
	local pid start id

	(systemctl show -p Id,MainPID,ExecMainStartTimestampMonotonic \*; echo) |
	sed -n '/^$/{x; s/^\n//; s/\n/\t/g; /^[1-9]/p; b}; s/.*=//; H' |
	while read -r pid start id; do
		dict[$id]=$start
	done
}

main () {
	setup_bash
	local -A start_times
	load_start_times start_times
	echo "${start_times[cron.service]}"
}

main "$@"

Combining sleep and wait with background processes provides simple concurrency.

#!/bin/bash
# demonstrate child processes that repeatedly do something while parent is alive
# while kill -s 0 "$parent" 2>&-; do something; done

# Function to snapshot memory usage of a pid to a file once each SAMPLE_INTVL
function sample_statm () {
    local parent=$1 pid=$2
    exec > "$pid".statm
    cd /proc/"$pid"
    while kill -s 0 "$parent" 2>&-; do
        echo -n "$EPOCHSECONDS "
        cat statm
        sleep "$SAMPLE_INTVL"
    done
}

function get_max_memory () {
    local file=$1 column=$2
    local -n _max=$3 _time=$4
    local -a f
    _max=0
    while read -ra f; do
        if (( f[column] > _max )); then
            _max=${f[$column]} _time=${f[0]}
        fi
    done < "$file"
}

function main () {
    set -eu
    DURATION=${DURATION:-60}
    SAMPLE_INTVL=${SAMPLE_INTVL:-1}
    local -i start_time=$EPOCHSECONDS
    local pid

    if (( ! $# )); then
        echo "usage: $0 pid [pid...]" >&2
        exit 1
    fi

    sleep "$DURATION" &
    local sleep_pid=$(jobs -p %sleep)
    for pid in "$@"; do
        sample_statm "$sleep_pid" "$pid" &
    done

    # Of the background processes, all but sleep self-terminate.
    # At shell exit, kill sleep.  The "or no-op" preserves the prior exit value.
    trap 'kill %sleep 2>&- || :' EXIT
    if ! wait -n; then
        echo "child error; exit 1" >&2
        exit 1
    fi

    local -i max_memory max_time
    for pid in "$@"; do
        get_max_memory "$pid".statm 2 max_memory max_time
        echo "$pid: max RSS: $max_memory at $(( max_time - start_time )) seconds"
    done
}

main "$@"

Trace files illuminate the sometimes vexing nuances of errexit.

# capture a trace of the current script
# $1 is an optional directory to contain the output file
# opens filehandle 4 for append
function start_trace () {
    local dir=${1:-.}
    mkdir -p "$dir"
    exec 4>> "$dir/xtrace"
    BASH_XTRACEFD=4
    PS4='+$$@$LINENO: '
    set -x
    date >&4
    declare -p >&4
}

sed

sed is a non-interactive editor that uses regular expressions to transform text. Many bash scripts use sed making it worthwhile to cover less-used sed functions and idioms.

function/idiom	description
b label	branch to label; if label is omitted, branch to end of script.
h H	copy/append pattern space to hold space.
g G	copy/append hold space to pattern space.
x	exchange the contents of the hold and pattern spaces.
n N	read/append the next line of input into the pattern space.
sed -n '/pat/{h; n; G; p}'	print a matching line and the following line in reverse order
ls | sed 'h; s/foo/bar/; x; G; s/\n/ /; s/^/mv /' | sh	rename files
sed -n '/^$/{x; s/^\n//; s/\n/\t/g; p; b}; H'	convert stanzas to tab-separated values (input must end with a blank line)

Tools

ShellCheck

Available from https://www.shellcheck.net/ and packaged in many Linux distributions, shellcheck runs lint-like checks for common bash pitfalls. The following additon to the top of a script enables all optional checks except two:

# Enable all optional shellcheck checks except the
# two that encourage superfluous braces and quoting.
# shellcheck enable=all
# shellcheck disable=SC2248,SC2250

Kcov

kcov provides line-based code coverage reports for bash scripts and many other languages. Find it at https://simonkagstrom.github.io/kcov/ or packaged from your Linux distribution.

ShellMetrics

From https://github.com/shellspec/shellmetrics, this tool measures cyclomatic complexity for shell scripts.

BASH Debugger

Single step through a bash script, set breakpoints, show a backtrace - all this and more from https://bashdb.sourceforge.net/
NB. For versions of bash newer than 5.0, try building from the matching branch of the git repo.

Further Reading

• Advanced Bash-Scripting Guide: https://tldp.org/LDP/abs/html/
• sed & awk: Chapter 6. Advanced sed Commands preview at Google books