Skip to content

Instantly share code, notes, and snippets.

@vncsna
Forked from mohanpedala/bash_strict_mode.md
Created June 6, 2021 01:59
Show Gist options
  • Save vncsna/64825d5609c146e80de8b1fd623011ca to your computer and use it in GitHub Desktop.
Save vncsna/64825d5609c146e80de8b1fd623011ca to your computer and use it in GitHub Desktop.
set -e, -u, -o, -x pipefail explanation

set -e, -u, -o, -x pipefail

The set lines

  • These lines deliberately cause your script to fail. Wait, what? Believe me, this is a good thing.
  • With these settings, certain common errors will cause the script to immediately fail, explicitly and loudly. Otherwise, you can get hidden bugs that are discovered only when they blow up in production.
  • set -euxo pipefail is short for:
set -e
set -u
set -o pipefail
set -x
  • set -e The set -e option instructs bash to immediately exit if any command [1] has a non-zero exit status. You wouldn't want to set this for your command-line shell, but in a script it's massively helpful. In all widely used general-purpose programming languages, an unhandled runtime error - whether that's a thrown exception in Java, or a segmentation fault in C, or a syntax error in Python - immediately halts execution of the program; subsequent lines are not executed.

    • By default, bash does not do this. This default behavior is exactly what you want if you are using bash on the command line
    • you don't want a typo to log you out! But in a script, you really want the opposite.
    • If one line in a script fails, but the last line succeeds, the whole script has a successful exit code. That makes it very easy to miss the error.
    • Again, what you want when using bash as your command-line shell and using it in scripts are at odds here. Being intolerant of errors is a lot better in scripts, and that's what set -e gives you.
  • set -x enables a mode of the shell where all executed commands are printed to the terminal. In your case it's clearly used for debugging, which is a typical use case for set -x : printing every command as it is executed may help you to visualize the control flow of the script if it is not functioning as expected.

  • set -u affects variables. When set, a reference to any variable you haven't previously defined - with the exceptions of $* and $@ - is an error, and causes the program to immediately exit. Languages like Python, C, Java and more all behave the same way, for all sorts of good reasons. One is so typos don't create new variables without you realizing it. For example:

#!/bin/bash
firstName="Aaron"
fullName="$firstname Maxwell"
echo "$fullName"
  • Take a moment and look. Do you see the error? The right-hand side of the third line says "firstname", all lowercase, instead of the camel-cased "firstName". Without the -u option, this will be a silent error. But with the -u option, the script exits on that line with an exit code of 1, printing the message "firstname: unbound variable" to stderr.

  • This is what you want: have it fail explicitly and immediately, rather than create subtle bugs that may be discovered too late.

  • set -o pipefail this setting prevents errors in a pipeline from being masked. If any command in a pipeline fails, that return code will be used as the return code of the whole pipeline. By default, the pipeline's return code is that of the last command even if it succeeds. Imagine finding a sorted list of matching lines in a file:

$ grep some-string /non/existent/file | sort
grep: /non/existent/file: No such file or directory
% echo $?
0
  • Here, grep has an exit code of 2, writes an error message to stderr, and an empty string to stdout.
  • This empty string is then passed through sort, which happily accepts it as valid input, and returns a status code of 0.
  • This is fine for a command line, but bad for a shell script: you almost certainly want the script to exit right then with a nonzero exit code... like this:
$ set -o pipefail
$ grep some-string /non/existent/file | sort
grep: /non/existent/file: No such file or directory
$ echo $?
2

Setting IFS The IFS variable - which stands for Internal Field Separator - controls what Bash calls word splitting. When set to a string, each character in the string is considered by Bash to separate words. This governs how bash will iterate through a sequence. For example, this script:

#!/bin/bash
IFS=$' '
items="a b c"
for x in $items; do
    echo "$x"
done

IFS=$'\n'
for y in $items; do
    echo "$y"
done
... will print out this:

a
b
c
a b c
  • In the first for loop, IFS is set to $' '. (The $'...' syntax creates a string, with backslash-escaped characters replaced with special characters - like "\t" for tab and "\n" for newline.) Within the for loops, x and y are set to whatever bash considers a "word" in the original sequence.
  • For the first loop, IFS is a space, meaning that words are separated by a space character.
  • For the second loop, "words" are separated by a newline, which means bash considers the whole value of "items" as a single word. If IFS is more than one character, splitting will be done on any of those characters.
  • Got all that? The next question is, why are we setting IFS to a string consisting of a tab character and a newline? Because it gives us better behavior when iterating over a loop. By "better", I mean "much less likely to cause surprising and confusing bugs". This is apparent in working with bash arrays:
#!/bin/bash
names=(
  "Aaron Maxwell"
  "Wayne Gretzky"
  "David Beckham"
)

echo "With default IFS value..."
for name in ${names[@]}; do
  echo "$name"
done

echo ""
echo "With strict-mode IFS value..."
IFS=$'\n\t'
for name in ${names[@]}; do
  echo "$name"
done

## Output
With default IFS value...
Aaron
Maxwell
Wayne
Gretzky
David
Beckham

With strict-mode IFS value...
Aaron Maxwell
Wayne Gretzky
David Beckham

Or consider a script that takes filenames as command line arguments:

for arg in $@; do
    echo "doing something with file: $arg"
done
  • If you invoke this as myscript.sh notes todo-list 'My Resume.doc', then with the default IFS value, the third argument will be mis-parsed as two separate files - named "My" and "Resume.doc". When actually it's a file that has a space in it, named "My Resume.doc".

  • Which behavior is more generally useful? The second, of course - where we have the ability to not split on spaces. If we have an array of strings that in general contain spaces, we normally want to iterate through them item by item, and not split an individual item into several.

  • Setting IFS to $'\n\t' means that word splitting will happen only on newlines and tab characters. This very often produces useful splitting behavior.

  • By default, bash sets this to $' \n\t' - space, newline, tab - which is too eager.

Full Reference Click Here

@suryapandian
Copy link

Thanks, this was useful

@aedavids
Copy link

nice job!

@skykit-alejandrorodriguez

thanks for the explanation! very helpful

@aedavids
Copy link

super helpful!

@crshnbrn66
Copy link

Nice writeup you explained it very well

@visaals
Copy link

visaals commented Jun 27, 2022

Thanks, this is great!

@lynxtaa
Copy link

lynxtaa commented Aug 6, 2022

Thanks, helped me a lot!

@jakejacobs
Copy link

Well written. Easy to userstand. Thank you.

@mrclrchtr
Copy link

Thank you very much!

@p-kenne
Copy link

p-kenne commented May 8, 2023

thank you . well explained

@fllrrt
Copy link

fllrrt commented Jun 26, 2023

Many thanks. Nice job.

@mathieu-aubin
Copy link

Good explanations with use cases examples

@evatosco
Copy link

evatosco commented Aug 2, 2023

This is incredibly helpful. Understanding how it works makes it easier to use and interpret its output. This is awesome for beginners like me. Thanks a lot!!

@ernstki
Copy link

ernstki commented Aug 15, 2023

Unfortunately, the blog post1 that served as raw material for this Gist is now gone. So it's no longer possible to debate with the author about his use of IFS. With the benefit of hindsight, though, I no longer think it's a good idea for the use case he presented. It breaks the normal and very useful default behavior of "$*" and "${array[*]}", for one thing.

If you double-quote your variables when they're used, including array variables, there's no need to muck with IFS to preserve the internal whitespace of array elements. Double-quoting variable expansions in Bash scripts is a best practice anyway—even after you've become proficient enough to understand when that's not strictly required, in cases such as "simple assignment."

The only downside is it requires you, the programmer, to exercise a little more discipline.

array=("the rain" "in Spain" lies mainly "on the plain")

# yields each array element individually double-quoted
for var in "${array[@]}"; do
    touch "$var.txt"  # or "${var}.txt"
done

Result:

$ ls -1 *.txt
in Spain.txt
lies.txt
mainly.txt
on the plain.txt
the rain.txt

It also doesn't hurt to have a basic understanding of Bash's quoting and word splitting behavior. Without that understanding, you will continue to write buggy scripts (for example) that can't handle filenames with literal whitespace characters in them, without ever quite understanding why.

Valid use cases for setting IFS

That said, there are still occasions where it's useful to set IFS, such as parsing delimited text

$ while IFS=: read u _ _ _ _ h _; do echo "$u's home is $h"; done < /etc/passwd
root's home is /root
bin's home is /bin
daemon's home is /sbin
adm's home is /var/adm

…or using it as a poor man's join operator

$ slugify() ( IFS=-; echo "${*,,}" )  # note outer parens, runs in subshell
$ slugify My blog post title
my-blog-post-title

$ addpaths=(/path/one /path/two)
$ PATH=$(IFS=:; echo "${addpaths[*]}"):$PATH
$ echo $PATH
/path/one:/path/two:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Footnotes

  1. "Use the Unofficial Bash Strict Mode (Unless You Looove Debugging)" by Aaron Maxwell (Wayback Machine link)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment