Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
set -e, -u, -o, -x pipefail explanation

Table of Contents

set -e, -u, -x, -o pipefail

  • The set lines
    • These lines deliberately cause your script to fail. Wait, what? Believe me, this is a good thing.

    • With these settings, certain common errors will cause the script to immediately fail, explicitly and loudly. Otherwise, you can get hidden bugs that are discovered only when they blow up in production.

      set -euxo pipefail is short for:

      set -e
      set -u
      set -o pipefail
      set -x
      

set -e

  • The set -e option instructs bash to immediately exit if any command [1] has a non-zero exit status. You wouldn't want to set this for your command-line shell, but in a script it's massively helpful. In all widely used general-purpose programming languages, an unhandled runtime error
  • whether that's a thrown exception in Java, or a segmentation fault in C, or a syntax error in Python - immediately halts execution of the program; subsequent lines are not executed.

    • By default, bash does not do this. This default behavior is exactly what you want if you are using bash on the command line
    • you don't want a typo to log you out! But in a script, you really want the opposite.
    • If one line in a script fails, but the last line succeeds, the whole script has a successful exit code. That makes it very easy to miss the error.
    • Again, what you want when using bash as your command-line shell and using it in scripts are at odds here. Being intolerant of errors is a lot better in scripts, and that's what set -e gives you.

set -x

  • Enables a mode of the shell where all executed commands are printed to the terminal. In your case it's clearly used for debugging, which is a typical use case for set -x : printing every command as it is executed may help you to visualize the control flow of the script if it is not functioning as expected.

set -u

  • Affects variables. When set, a reference to any variable you haven't previously defined - with the exceptions of $* and $@ - is an error, and causes the program to immediately exit. Languages like Python, C, Java and more all behave the same way, for all sorts of good reasons. One is so typos don't create new variables without you realizing it. For example:

    #!/bin/bash
    firstName="Aaron"
    fullName="$firstname Maxwell"
    echo "$fullName"
    
  • Take a moment and look. Do you see the error? The right-hand side of the third line says "firstname", all lowercase, instead of the camel-cased "firstName". Without the -u option, this will be a silent error. But with the -u option, the script exits on that line with an exit code of 1, printing the message "firstname: unbound variable" to stderr.

  • This is what you want: have it fail explicitly and immediately, rather than create subtle bugs that may be discovered too late.

set -o pipefail

  • This setting prevents errors in a pipeline from being masked. If any command in a pipeline fails, that return code will be used as the return code of the whole pipeline. By default, the pipeline's return code is that of the last command even if it succeeds. Imagine finding a sorted list of matching lines in a file:

    $ grep some-string /non/existent/file | sort
    grep: /non/existent/file: No such file or directory
    % echo $?
    0
    
  • Here, grep has an exit code of 2, writes an error message to stderr, and an empty string to stdout.

  • This empty string is then passed through sort, which happily accepts it as valid input, and returns a status code of 0.

  • This is fine for a command line, but bad for a shell script: you almost certainly want the script to exit right then with a nonzero exit code... like this:

    $ set -o pipefail
    $ grep some-string /non/existent/file | sort
    grep: /non/existent/file: No such file or directory
    $ echo $?
    2
    

Setting IFS

  • The IFS variable - which stands for Internal Field Separator - controls what Bash calls word splitting. When set to a string, each character in the string is considered by Bash to separate words. This governs how bash will iterate through a sequence. For example, this script:

    #!/bin/bash
    IFS=$' '
    items="a b c"
    for x in $items; do
        echo "$x"
    done
    
    IFS=$'\n'
    for y in $items; do
        echo "$y"
    done
    ... will print out this:
    
    a
    b
    c
    a b c
    
  • In the first for loop, IFS is set to $' '. (The $'...' syntax creates a string, with backslash-escaped characters replaced with special characters - like "\t" for tab and "\n" for newline.) Within the for loops, x and y are set to whatever bash considers a "word" in the original sequence.

  • For the first loop, IFS is a space, meaning that words are separated by a space character.

  • For the second loop, "words" are separated by a newline, which means bash considers the whole value of "items" as a single word. If IFS is more than one character, splitting will be done on any of those characters.

  • Got all that? The next question is, why are we setting IFS to a string consisting of a tab character and a newline? Because it gives us better behavior when iterating over a loop. By "better", I mean "much less likely to cause surprising and confusing bugs". This is apparent in working with bash arrays:

    #!/bin/bash
    names=(
    "Aaron Maxwell"
    "Wayne Gretzky"
    "David Beckham"
    )
    
    echo "With default IFS value..."
    for name in ${names[@]}; do
    echo "$name"
    done
    
    echo ""
    echo "With strict-mode IFS value..."
    IFS=$'\n\t'
    for name in ${names[@]}; do
    echo "$name"
    done
    
    
    ## Output
    With default IFS value...
    Aaron
    Maxwell
    Wayne
    Gretzky
    David
    Beckham
    
    With strict-mode IFS value...
    Aaron Maxwell
    Wayne Gretzky
    David Beckham
    

Or consider a script that takes filenames as command line arguments:

```
for arg in $@; do
    echo "doing something with file: $arg"
done
```
  • If you invoke this as myscript.sh notes todo-list 'My Resume.doc', then with the default IFS value, the third argument will be mis-parsed as two separate files - named "My" and "Resume.doc". When actually it's a file that has a space in it, named "My Resume.doc".

  • Which behavior is more generally useful? The second, of course - where we have the ability to not split on spaces. If we have an array of strings that in general contain spaces, we normally want to iterate through them item by item, and not split an individual item into several.

  • Setting IFS to $'\n\t' means that word splitting will happen only on newlines and tab characters. This very often produces useful splitting behavior.

  • By default, bash sets this to $' \n\t' - space, newline, tab - which is too eager.

Original Reference

@mikekeke
Copy link

mikekeke commented Mar 30, 2021

Very useful. Ty!

@smac89
Copy link

smac89 commented Apr 20, 2021

I would also add set -x, which prints each line that is being executed in the script. This is useful when building a docker image because docker just prints the name of the script but you are left wondering which part of the script is currently running.

@mohanpedala
Copy link
Author

mohanpedala commented Apr 21, 2021

I would also add set -x, which prints each line that is being executed in the script. This is useful when building a docker image because docker just prints the name of the script but you are left wondering which part of the script is currently running.

Thank you added it..

@mohanpedala
Copy link
Author

mohanpedala commented Apr 21, 2021

Great teacher.

can you please explain the role of the @ symbol?

${names[@]}: All elements, space-separated
$@: All postitional arguments (as separate strings)

@mohanpedala
Copy link
Author

mohanpedala commented Apr 21, 2021

Mohan, please place "Here, grep has an exit code of 2" outside of the code block.


% grep some-string /non/existent/file \| sort
--
  | grep: /non/existent/file: No such file or directory
  | % echo $?
  | 0

  | (% is the bash prompt.) Here, grep has an exit code of 2, writes an error message to stderr, and an empty string to stdout. This empty string is then passed through sort, which happily accepts it as valid input, and returns a status code of 0. This is fine for a command line, but bad for a shell script: you almost certainly want the script to exit right then with a nonzero exit code... like this:


  |  
  | % set -o pipefail
  | % grep some-string /non/existent/file \| sort
  | grep: /non/existent/file: No such file or directory
  | % echo $?
  | 2
  | 

Thanks Daniel, I have updated the file

@Silcet
Copy link

Silcet commented Apr 21, 2021

Great stuff. I'll write this in stone and make it the new commandments at work

@jhult
Copy link

jhult commented Apr 22, 2021

Interesting take on using set -e: https://mywiki.wooledge.org/BashFAQ/105

GreyCat's personal recommendation is simple: don't use set -e. Add your own error checking instead.

rking's personal recommendation is to go ahead and use set -e, but beware of possible gotchas. It has useful semantics, so to exclude it from the toolbox is to give into FUD.

geirha's personal recommendation is to handle errors properly and not rely on the unreliable set -e.

@berkant
Copy link

berkant commented May 14, 2021

set -euo pipefail does not include -x, which you said shorthand for the following lines of individual commands that wrongly includes -x as well. Change to -euxo pipefail?

@mohanpedala
Copy link
Author

mohanpedala commented May 16, 2021

set -euo pipefail does not include -x, which you said shorthand for the following lines of individual commands that wrongly includes -x as well. Change to -euxo pipefail?

thank you added it..

@awolad
Copy link

awolad commented Jul 1, 2021

image

I'm getting the error from Github actions.

./scripts/deploy.sh: 2: set: Illegal option -o pipefail

@berkant
Copy link

berkant commented Jul 1, 2021

@awolad
Copy link

awolad commented Jul 1, 2021

@berkant Previously I was executing the script using sh ./scripts/deploy.sh Now it's working using bash ./scripts/deploy.sh Thank you so much :)

@cpauya
Copy link

cpauya commented Aug 18, 2021

Thanks very much! The -u and -o pipefail are new to me.

Just a trivial one: maybe change title/heading from set -e, -u, -o, -x pipefail to set -e, -u, -x, -o pipefail to be more consistent/correct? 🙏 😺

@leiless
Copy link

leiless commented Aug 24, 2021

Thanks very much! The -u and -o pipefail are new to me.

Just a trivial one: maybe change title/heading from set -e, -u, -o, -x pipefail to set -e, -u, -x, -o pipefail to be more consistent/correct?

Upvote!

@mohanpedala
Copy link
Author

mohanpedala commented Aug 24, 2021

Thanks very much! The -u and -o pipefail are new to me.
Just a trivial one: maybe change title/heading from set -e, -u, -o, -x pipefail to set -e, -u, -x, -o pipefail to be more consistent/correct?

Upvote!

I have chnaged it thank you

@thingsiplay
Copy link

thingsiplay commented Sep 15, 2021

Previously I was executing the script using sh ./scripts/deploy.sh Now it's working using bash ./scripts/deploy.sh Thank you so much :)

I am a bit late, but just found this wonderful tutorial. And by accident I just found in the man bash that using bash as sh would put it into strict POSIX mode, which is similar to dash and does not support any of the bash features. So if you have bash set as your sh interpreter and use the sh command, then it goes into this strict mode. That is why it did not work. This is the line from the manual in question:

If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible, while conforming to the POSIX standard as well.

@bkahlert
Copy link

bkahlert commented Oct 20, 2021

Nice that this gist receives so much positive feedback. I just ask myself if all that gratitude shouldn't be better directed to Aaron Maxwell, the author of the original article to be found at http://redsymbol.net/articles/unofficial-bash-strict-mode/.
This gist is close to a 1:1 copy.
I recommend to original article for reading as it also describes what to do in cases were the "strict mode" causes problems.

@cvega21
Copy link

cvega21 commented Oct 22, 2021

thank you Aaron Maxwell for this, and thank you mohan for compiling it here! very clear explanations and super useful knowledge.

@kayomarz
Copy link

kayomarz commented Oct 26, 2021

Very useful gist but as @bkahlert pointed out its almost a copy of the original author @redsymbol who deserves a mention.

Nice that this gist receives so much positive feedback. I just ask myself if all that gratitude shouldn't be better directed to Aaron Maxwell, the author of the original article to be found at http://redsymbol.net/articles/unofficial-bash-strict-mode/. This gist is close to a 1:1 copy. I recommend to original article for reading as it also describes what to do in cases were the "strict mode" causes problems.

@mohanpedala
Copy link
Author

mohanpedala commented Oct 26, 2021

Nice that this gist receives so much positive feedback. I just ask myself if all that gratitude shouldn't be better directed to Aaron Maxwell, the author of the original article to be found at http://redsymbol.net/articles/unofficial-bash-strict-mode/. This gist is close to a 1:1 copy. I recommend to original article for reading as it also describes what to do in cases were the "strict mode" causes problems.

Hi @bkahlert , Original reference has been added at the bottom of the gist when I created it ("Full Reference Click Here"). Please read the full gist before pointing out. and for your reference I keep my daily notes and findings as a gist.
I like to see most of the content in markdown so I have created the content I need in the markdown version. I really appreciate Aaron Maxwell article that is the reason why I did-not change the names or content in the gist.

@mohanpedala
Copy link
Author

mohanpedala commented Oct 26, 2021

Very useful gist but as @bkahlert pointed out its almost a copy of the original author @redsymbol who deserves a mention.

Nice that this gist receives so much positive feedback. I just ask myself if all that gratitude shouldn't be better directed to Aaron Maxwell, the author of the original article to be found at http://redsymbol.net/articles/unofficial-bash-strict-mode/. This gist is close to a 1:1 copy. I recommend to original article for reading as it also describes what to do in cases were the "strict mode" causes problems.

Hi @kayomarz Original reference has been added at the bottom of the gist when I created it ("Full Reference Click Here"). Please read the full gist before pointing out. and for your reference I keep my daily notes and findings as a gist.
I like to see most of the content in markdown so I have created the content I need in the markdown version. I really appreciate Aaron Maxwell article that is the reason why I did-not change the names or content in the gist.

@bkahlert
Copy link

bkahlert commented Oct 29, 2021

Very useful gist but as @bkahlert pointed out its almost a copy of the original author @redsymbol who deserves a mention.

Nice that this gist receives so much positive feedback. I just ask myself if all that gratitude shouldn't be better directed to Aaron Maxwell, the author of the original article to be found at http://redsymbol.net/articles/unofficial-bash-strict-mode/. This gist is close to a 1:1 copy. I recommend to original article for reading as it also describes what to do in cases were the "strict mode" causes problems.

Hi @kayomarz Original reference has been added at the bottom of the gist when I created it ("Full Reference Click Here"). Please read the full gist before pointing out. and for your reference I keep my daily notes and findings as a gist. I like to see most of the content in markdown so I have created the content I need in the markdown version. I really appreciate Aaron Maxwell article that is the reason why I did-not change the names or content in the gist.

Hi @mohanpedala, I did read your gist completely and well noticed your "Full Reference Click Here" link. That's what made me comment. I consider such a generic reference to a 1:1 copied article inappropriate. Especially in the light of its perceived value. You might want to read Citation styles guide: Choosing a style and citing correctly.

@mohanpedala
Copy link
Author

mohanpedala commented Oct 29, 2021

Very useful gist but as @bkahlert pointed out its almost a copy of the original author @redsymbol who deserves a mention.

Nice that this gist receives so much positive feedback. I just ask myself if all that gratitude shouldn't be better directed to Aaron Maxwell, the author of the original article to be found at http://redsymbol.net/articles/unofficial-bash-strict-mode/. This gist is close to a 1:1 copy. I recommend to original article for reading as it also describes what to do in cases were the "strict mode" causes problems.

Hi @kayomarz Original reference has been added at the bottom of the gist when I created it ("Full Reference Click Here"). Please read the full gist before pointing out. and for your reference I keep my daily notes and findings as a gist. I like to see most of the content in markdown so I have created the content I need in the markdown version. I really appreciate Aaron Maxwell article that is the reason why I did-not change the names or content in the gist.

Hi @mohanpedala, I did read your gist completely and well noticed your "Full Reference Click Here" link. That's what made me comment. I consider such a generic reference to a 1:1 copied article inappropriate. Especially in the light of its perceived value. You might want to read Citation styles guide: Choosing a style and citing correctly.

Thank you @bkahlert I hope this looks good 😊

@bkahlert
Copy link

bkahlert commented Nov 11, 2021

Wonderful

@Gboom345
Copy link

Gboom345 commented Mar 14, 2022

very good explanation

@bolshakoff
Copy link

bolshakoff commented Mar 30, 2022

Nice.

@fazlearefin
Copy link

fazlearefin commented Apr 1, 2022

@mohanpedala Consider using -E as well as mentioned in https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/ ?

So it becomes

set -Eeuxo pipefail

@patrikjuvonen
Copy link

patrikjuvonen commented Apr 6, 2022

https://blog.cloudflare.com/pipefail-how-a-missing-shell-option-slowed-cloudflare-down/

Interesting real-life scenario post about how missing the pipefail option caused a major cascading failure in Cloudflare's production.

@rim8
Copy link

rim8 commented May 3, 2022

i didn't get : set -o pipefail

@icntrs
Copy link

icntrs commented Jun 16, 2022

Little tinny thing :
$ grep some-string /non/existent/file | sort
grep: /non/existent/file: No such file or directory
% echo $? --> $ echo $?
0

very useful info, thanks :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment