Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
set -e, -u, -o, -x pipefail explanation

Table of Contents

set -e, -u, -x, -o pipefail

  • The set lines
    • These lines deliberately cause your script to fail. Wait, what? Believe me, this is a good thing.

    • With these settings, certain common errors will cause the script to immediately fail, explicitly and loudly. Otherwise, you can get hidden bugs that are discovered only when they blow up in production.

      set -euxo pipefail is short for:

      set -e
      set -u
      set -o pipefail
      set -x
      

set -e

  • The set -e option instructs bash to immediately exit if any command [1] has a non-zero exit status. You wouldn't want to set this for your command-line shell, but in a script it's massively helpful. In all widely used general-purpose programming languages, an unhandled runtime error
  • whether that's a thrown exception in Java, or a segmentation fault in C, or a syntax error in Python - immediately halts execution of the program; subsequent lines are not executed.

    • By default, bash does not do this. This default behavior is exactly what you want if you are using bash on the command line
    • you don't want a typo to log you out! But in a script, you really want the opposite.
    • If one line in a script fails, but the last line succeeds, the whole script has a successful exit code. That makes it very easy to miss the error.
    • Again, what you want when using bash as your command-line shell and using it in scripts are at odds here. Being intolerant of errors is a lot better in scripts, and that's what set -e gives you.

set -x

  • Enables a mode of the shell where all executed commands are printed to the terminal. In your case it's clearly used for debugging, which is a typical use case for set -x : printing every command as it is executed may help you to visualize the control flow of the script if it is not functioning as expected.

set -u

  • Affects variables. When set, a reference to any variable you haven't previously defined - with the exceptions of $* and $@ - is an error, and causes the program to immediately exit. Languages like Python, C, Java and more all behave the same way, for all sorts of good reasons. One is so typos don't create new variables without you realizing it. For example:

    #!/bin/bash
    firstName="Aaron"
    fullName="$firstname Maxwell"
    echo "$fullName"
    
  • Take a moment and look. Do you see the error? The right-hand side of the third line says "firstname", all lowercase, instead of the camel-cased "firstName". Without the -u option, this will be a silent error. But with the -u option, the script exits on that line with an exit code of 1, printing the message "firstname: unbound variable" to stderr.

  • This is what you want: have it fail explicitly and immediately, rather than create subtle bugs that may be discovered too late.

set -o pipefail

  • This setting prevents errors in a pipeline from being masked. If any command in a pipeline fails, that return code will be used as the return code of the whole pipeline. By default, the pipeline's return code is that of the last command even if it succeeds. Imagine finding a sorted list of matching lines in a file:

    $ grep some-string /non/existent/file | sort
    grep: /non/existent/file: No such file or directory
    % echo $?
    0
    
  • Here, grep has an exit code of 2, writes an error message to stderr, and an empty string to stdout.

  • This empty string is then passed through sort, which happily accepts it as valid input, and returns a status code of 0.

  • This is fine for a command line, but bad for a shell script: you almost certainly want the script to exit right then with a nonzero exit code... like this:

    $ set -o pipefail
    $ grep some-string /non/existent/file | sort
    grep: /non/existent/file: No such file or directory
    $ echo $?
    2
    

Setting IFS

  • The IFS variable - which stands for Internal Field Separator - controls what Bash calls word splitting. When set to a string, each character in the string is considered by Bash to separate words. This governs how bash will iterate through a sequence. For example, this script:

    #!/bin/bash
    IFS=$' '
    items="a b c"
    for x in $items; do
        echo "$x"
    done
    
    IFS=$'\n'
    for y in $items; do
        echo "$y"
    done
    ... will print out this:
    
    a
    b
    c
    a b c
    
  • In the first for loop, IFS is set to $' '. (The $'...' syntax creates a string, with backslash-escaped characters replaced with special characters - like "\t" for tab and "\n" for newline.) Within the for loops, x and y are set to whatever bash considers a "word" in the original sequence.

  • For the first loop, IFS is a space, meaning that words are separated by a space character.

  • For the second loop, "words" are separated by a newline, which means bash considers the whole value of "items" as a single word. If IFS is more than one character, splitting will be done on any of those characters.

  • Got all that? The next question is, why are we setting IFS to a string consisting of a tab character and a newline? Because it gives us better behavior when iterating over a loop. By "better", I mean "much less likely to cause surprising and confusing bugs". This is apparent in working with bash arrays:

    #!/bin/bash
    names=(
    "Aaron Maxwell"
    "Wayne Gretzky"
    "David Beckham"
    )
    
    echo "With default IFS value..."
    for name in ${names[@]}; do
    echo "$name"
    done
    
    echo ""
    echo "With strict-mode IFS value..."
    IFS=$'\n\t'
    for name in ${names[@]}; do
    echo "$name"
    done
    
    
    ## Output
    With default IFS value...
    Aaron
    Maxwell
    Wayne
    Gretzky
    David
    Beckham
    
    With strict-mode IFS value...
    Aaron Maxwell
    Wayne Gretzky
    David Beckham
    

Or consider a script that takes filenames as command line arguments:

```
for arg in $@; do
    echo "doing something with file: $arg"
done
```
  • If you invoke this as myscript.sh notes todo-list 'My Resume.doc', then with the default IFS value, the third argument will be mis-parsed as two separate files - named "My" and "Resume.doc". When actually it's a file that has a space in it, named "My Resume.doc".

  • Which behavior is more generally useful? The second, of course - where we have the ability to not split on spaces. If we have an array of strings that in general contain spaces, we normally want to iterate through them item by item, and not split an individual item into several.

  • Setting IFS to $'\n\t' means that word splitting will happen only on newlines and tab characters. This very often produces useful splitting behavior.

  • By default, bash sets this to $' \n\t' - space, newline, tab - which is too eager.

Original Reference

@mtskillman

This comment has been minimized.

Copy link

@mtskillman mtskillman commented Nov 20, 2020

thank you for this!

@mohanpedala

This comment has been minimized.

Copy link
Owner Author

@mohanpedala mohanpedala commented Nov 20, 2020

you are welcome

@DanielAdeniji

This comment has been minimized.

Copy link

@DanielAdeniji DanielAdeniji commented Nov 28, 2020

Mohan, please place "Here, grep has an exit code of 2" outside of the code block.


% grep some-string /non/existent/file \| sort
--
  | grep: /non/existent/file: No such file or directory
  | % echo $?
  | 0

  | (% is the bash prompt.) Here, grep has an exit code of 2, writes an error message to stderr, and an empty string to stdout. This empty string is then passed through sort, which happily accepts it as valid input, and returns a status code of 0. This is fine for a command line, but bad for a shell script: you almost certainly want the script to exit right then with a nonzero exit code... like this:


  |  
  | % set -o pipefail
  | % grep some-string /non/existent/file \| sort
  | grep: /non/existent/file: No such file or directory
  | % echo $?
  | 2
  | 
@Carmelly212

This comment has been minimized.

Copy link

@Carmelly212 Carmelly212 commented Jan 6, 2021

thank you very much!
so basically "pipefail" makes sure errors are not masked and returns with a nonzero exit code?

@sumedhak27

This comment has been minimized.

Copy link

@sumedhak27 sumedhak27 commented Jan 13, 2021

Awesome discription!!
Thnaks 😄 !!

@heyrun

This comment has been minimized.

Copy link

@heyrun heyrun commented Jan 24, 2021

Great teacher.

can you please explain the role of the @ symbol?

@downthecode

This comment has been minimized.

Copy link

@downthecode downthecode commented Feb 21, 2021

Thanks

@mikekeke

This comment has been minimized.

Copy link

@mikekeke mikekeke commented Mar 30, 2021

Very useful. Ty!

@smac89

This comment has been minimized.

Copy link

@smac89 smac89 commented Apr 20, 2021

I would also add set -x, which prints each line that is being executed in the script. This is useful when building a docker image because docker just prints the name of the script but you are left wondering which part of the script is currently running.

@mohanpedala

This comment has been minimized.

Copy link
Owner Author

@mohanpedala mohanpedala commented Apr 21, 2021

I would also add set -x, which prints each line that is being executed in the script. This is useful when building a docker image because docker just prints the name of the script but you are left wondering which part of the script is currently running.

Thank you added it..

@mohanpedala

This comment has been minimized.

Copy link
Owner Author

@mohanpedala mohanpedala commented Apr 21, 2021

Great teacher.

can you please explain the role of the @ symbol?

${names[@]}: All elements, space-separated
$@: All postitional arguments (as separate strings)

@mohanpedala

This comment has been minimized.

Copy link
Owner Author

@mohanpedala mohanpedala commented Apr 21, 2021

Mohan, please place "Here, grep has an exit code of 2" outside of the code block.


% grep some-string /non/existent/file \| sort
--
  | grep: /non/existent/file: No such file or directory
  | % echo $?
  | 0

  | (% is the bash prompt.) Here, grep has an exit code of 2, writes an error message to stderr, and an empty string to stdout. This empty string is then passed through sort, which happily accepts it as valid input, and returns a status code of 0. This is fine for a command line, but bad for a shell script: you almost certainly want the script to exit right then with a nonzero exit code... like this:


  |  
  | % set -o pipefail
  | % grep some-string /non/existent/file \| sort
  | grep: /non/existent/file: No such file or directory
  | % echo $?
  | 2
  | 

Thanks Daniel, I have updated the file

@Silcet

This comment has been minimized.

Copy link

@Silcet Silcet commented Apr 21, 2021

Great stuff. I'll write this in stone and make it the new commandments at work

@jhult

This comment has been minimized.

Copy link

@jhult jhult commented Apr 22, 2021

Interesting take on using set -e: https://mywiki.wooledge.org/BashFAQ/105

GreyCat's personal recommendation is simple: don't use set -e. Add your own error checking instead.

rking's personal recommendation is to go ahead and use set -e, but beware of possible gotchas. It has useful semantics, so to exclude it from the toolbox is to give into FUD.

geirha's personal recommendation is to handle errors properly and not rely on the unreliable set -e.

@berkant

This comment has been minimized.

Copy link

@berkant berkant commented May 14, 2021

set -euo pipefail does not include -x, which you said shorthand for the following lines of individual commands that wrongly includes -x as well. Change to -euxo pipefail?

@mohanpedala

This comment has been minimized.

Copy link
Owner Author

@mohanpedala mohanpedala commented May 16, 2021

set -euo pipefail does not include -x, which you said shorthand for the following lines of individual commands that wrongly includes -x as well. Change to -euxo pipefail?

thank you added it..

@awolad

This comment has been minimized.

Copy link

@awolad awolad commented Jul 1, 2021

image

I'm getting the error from Github actions.

./scripts/deploy.sh: 2: set: Illegal option -o pipefail

@berkant

This comment has been minimized.

Copy link

@berkant berkant commented Jul 1, 2021

@awolad

This comment has been minimized.

Copy link

@awolad awolad commented Jul 1, 2021

@berkant Previously I was executing the script using sh ./scripts/deploy.sh Now it's working using bash ./scripts/deploy.sh Thank you so much :)

@cpauya

This comment has been minimized.

Copy link

@cpauya cpauya commented Aug 18, 2021

Thanks very much! The -u and -o pipefail are new to me.

Just a trivial one: maybe change title/heading from set -e, -u, -o, -x pipefail to set -e, -u, -x, -o pipefail to be more consistent/correct? 🙏 😺

@leiless

This comment has been minimized.

Copy link

@leiless leiless commented Aug 24, 2021

Thanks very much! The -u and -o pipefail are new to me.

Just a trivial one: maybe change title/heading from set -e, -u, -o, -x pipefail to set -e, -u, -x, -o pipefail to be more consistent/correct?

Upvote!

@mohanpedala

This comment has been minimized.

Copy link
Owner Author

@mohanpedala mohanpedala commented Aug 24, 2021

Thanks very much! The -u and -o pipefail are new to me.
Just a trivial one: maybe change title/heading from set -e, -u, -o, -x pipefail to set -e, -u, -x, -o pipefail to be more consistent/correct?

Upvote!

I have chnaged it thank you

@thingsiplay

This comment has been minimized.

Copy link

@thingsiplay thingsiplay commented Sep 15, 2021

Previously I was executing the script using sh ./scripts/deploy.sh Now it's working using bash ./scripts/deploy.sh Thank you so much :)

I am a bit late, but just found this wonderful tutorial. And by accident I just found in the man bash that using bash as sh would put it into strict POSIX mode, which is similar to dash and does not support any of the bash features. So if you have bash set as your sh interpreter and use the sh command, then it goes into this strict mode. That is why it did not work. This is the line from the manual in question:

If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible, while conforming to the POSIX standard as well.

@bkahlert

This comment has been minimized.

Copy link

@bkahlert bkahlert commented Oct 20, 2021

Nice that this gist receives so much positive feedback. I just ask myself if all that gratitude shouldn't be better directed to Aaron Maxwell, the author of the original article to be found at http://redsymbol.net/articles/unofficial-bash-strict-mode/.
This gist is close to a 1:1 copy.
I recommend to original article for reading as it also describes what to do in cases were the "strict mode" causes problems.

@cvega21

This comment has been minimized.

Copy link

@cvega21 cvega21 commented Oct 22, 2021

thank you Aaron Maxwell for this, and thank you mohan for compiling it here! very clear explanations and super useful knowledge.

@kayomarz

This comment has been minimized.

Copy link

@kayomarz kayomarz commented Oct 26, 2021

Very useful gist but as @bkahlert pointed out its almost a copy of the original author @redsymbol who deserves a mention.

Nice that this gist receives so much positive feedback. I just ask myself if all that gratitude shouldn't be better directed to Aaron Maxwell, the author of the original article to be found at http://redsymbol.net/articles/unofficial-bash-strict-mode/. This gist is close to a 1:1 copy. I recommend to original article for reading as it also describes what to do in cases were the "strict mode" causes problems.

@mohanpedala

This comment has been minimized.

Copy link
Owner Author

@mohanpedala mohanpedala commented Oct 26, 2021

Nice that this gist receives so much positive feedback. I just ask myself if all that gratitude shouldn't be better directed to Aaron Maxwell, the author of the original article to be found at http://redsymbol.net/articles/unofficial-bash-strict-mode/. This gist is close to a 1:1 copy. I recommend to original article for reading as it also describes what to do in cases were the "strict mode" causes problems.

Hi @bkahlert , Original reference has been added at the bottom of the gist when I created it ("Full Reference Click Here"). Please read the full gist before pointing out. and for your reference I keep my daily notes and findings as a gist.
I like to see most of the content in markdown so I have created the content I need in the markdown version. I really appreciate Aaron Maxwell article that is the reason why I did-not change the names or content in the gist.

@mohanpedala

This comment has been minimized.

Copy link
Owner Author

@mohanpedala mohanpedala commented Oct 26, 2021

Very useful gist but as @bkahlert pointed out its almost a copy of the original author @redsymbol who deserves a mention.

Nice that this gist receives so much positive feedback. I just ask myself if all that gratitude shouldn't be better directed to Aaron Maxwell, the author of the original article to be found at http://redsymbol.net/articles/unofficial-bash-strict-mode/. This gist is close to a 1:1 copy. I recommend to original article for reading as it also describes what to do in cases were the "strict mode" causes problems.

Hi @kayomarz Original reference has been added at the bottom of the gist when I created it ("Full Reference Click Here"). Please read the full gist before pointing out. and for your reference I keep my daily notes and findings as a gist.
I like to see most of the content in markdown so I have created the content I need in the markdown version. I really appreciate Aaron Maxwell article that is the reason why I did-not change the names or content in the gist.

@bkahlert

This comment has been minimized.

Copy link

@bkahlert bkahlert commented Oct 29, 2021

Very useful gist but as @bkahlert pointed out its almost a copy of the original author @redsymbol who deserves a mention.

Nice that this gist receives so much positive feedback. I just ask myself if all that gratitude shouldn't be better directed to Aaron Maxwell, the author of the original article to be found at http://redsymbol.net/articles/unofficial-bash-strict-mode/. This gist is close to a 1:1 copy. I recommend to original article for reading as it also describes what to do in cases were the "strict mode" causes problems.

Hi @kayomarz Original reference has been added at the bottom of the gist when I created it ("Full Reference Click Here"). Please read the full gist before pointing out. and for your reference I keep my daily notes and findings as a gist. I like to see most of the content in markdown so I have created the content I need in the markdown version. I really appreciate Aaron Maxwell article that is the reason why I did-not change the names or content in the gist.

Hi @mohanpedala, I did read your gist completely and well noticed your "Full Reference Click Here" link. That's what made me comment. I consider such a generic reference to a 1:1 copied article inappropriate. Especially in the light of its perceived value. You might want to read Citation styles guide: Choosing a style and citing correctly.

@mohanpedala

This comment has been minimized.

Copy link
Owner Author

@mohanpedala mohanpedala commented Oct 29, 2021

Very useful gist but as @bkahlert pointed out its almost a copy of the original author @redsymbol who deserves a mention.

Nice that this gist receives so much positive feedback. I just ask myself if all that gratitude shouldn't be better directed to Aaron Maxwell, the author of the original article to be found at http://redsymbol.net/articles/unofficial-bash-strict-mode/. This gist is close to a 1:1 copy. I recommend to original article for reading as it also describes what to do in cases were the "strict mode" causes problems.

Hi @kayomarz Original reference has been added at the bottom of the gist when I created it ("Full Reference Click Here"). Please read the full gist before pointing out. and for your reference I keep my daily notes and findings as a gist. I like to see most of the content in markdown so I have created the content I need in the markdown version. I really appreciate Aaron Maxwell article that is the reason why I did-not change the names or content in the gist.

Hi @mohanpedala, I did read your gist completely and well noticed your "Full Reference Click Here" link. That's what made me comment. I consider such a generic reference to a 1:1 copied article inappropriate. Especially in the light of its perceived value. You might want to read Citation styles guide: Choosing a style and citing correctly.

Thank you @bkahlert I hope this looks good 😊

@bkahlert

This comment has been minimized.

Copy link

@bkahlert bkahlert commented Nov 11, 2021

Wonderful

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment