A few days ago, I mentioned to Rob Ballou, that there's just something very satisfying about shell scripting. Once you've groked all its little idiosyncracies, it's really quite addictive. As developers, I think we all get some gratification from getting things done, but also from building elegant, complete systems. For me, shell scripting really scratches all those itches. I get to build a system from start to finish, completely self-contained and solve a real problem.
Elegant, complete systems might at first seem antithetical to what most people think of shell scripting. Shell scripting is plagued by weird quoting, strange unfamiliar constructions, and a lack useful data types. However, once you embrace the things that shell scripting is good at and learn a few strategies for writing scripts tidily, it can become a fantastic tool for solving real, every day problems.
So, what are those strategies and what are things that the shell does well?
#!/bin/bash
set -e
main () {
}
main $@For a very long time, I used to write shell scripts from top to bottom. A long list of commands to run, in order. I made little use of functions and I didn't have a clear sense of an entry point to my tool. The pattern above is tremendously helpful, because it frames shell scripting in a context we're used to seeing. By starting every shell script with the template above, we immediately get a few things. First, set -e turns on better error reporting. It tells the script to fail when it encounters and error rather than just chugging along. Sweet. Next, we have the function main (). This is the real win with this template. It immediately puts us in familiar territory. The opening and closing curly braces trigger our instincts to create small, single purpose functions instead of long unorganized sets of instructions. From main we can should call just a few functions of our own creation to accomplish our tasks. Don't let main grow beyong 5-10 lines. Finally, main $@, this should alwasy be the last line of your script after all your other functions. This calls your main function with all of the arguments to your script.
I don't have a single useful script that I don't start with this template any more.
Shell scripting doesn't work like other languages we're used to using. Scripts, commands, and functions don't return data. Instead, they stream data out. Thus, you don't typically write anything like my_var=some_command. That is, the results of some_command can't just be assigned to a variable without some acrobatics. Why? Imagine the fake script below (it wouldn't work):
my_var=some_command # We're trying to take the result of some_command and assign it to my_var
my_result=some_other_command my_var # We're trying to take the previous results and pass it to another command.
echo $my_resultIn shell, the above would never work and it completely missed the things that shell does well. That is, working in streams. We should rewrite the above like this:
some_command | some_other_commandThe above streams the results of some_command directly to some_other_command. By default, your shell will write the output of any command to the terminal unless you capture it with something else, this makes the echo in our bad example completely useless.
Just as shell functions don't return data, you shouldn't pass data to your functions as arguments. Instead, your data should be streamed into the function. Arguments are for altering how you deal with that data, not for providing data. In our bad example, we did some_other_command my_var. Don't do that. As we saw, we can just stream the data from the last command into another command using a pipe(|).
This essence of shell scripting is composition. At the end of the day, all we're doing is manipulating strings. Most, (if not all) systems can be thought of as data that comes in and data that should come out. What happens in between is the black box where developers spend their day. That black box is the composition of many functions and tools that we build to manipulate data as it comes in to produce that data that eventually makes it out the other end.
Shell is this concept taken to the extreme. We really want our scripts to work like foo | bar | baz | zap | etc.. These chains of small, single purpose functions can be composed into complex workflows that can do powerful things. When your pipelines get too long, simply break chunks of that pipeline into small functions. Our pipeline above could be broken up like so:
main () {
foo | bar_zap | etc.
}
bar_zap () {
bar | baz | zap
}Here, we composed bar, baz and zap into the single function bar_zap. Build lots of little functions.
Finally, don't try to make a single script do it all. Don't be afraid to make lots of little scripts that you can compose on the terminal to accomplish what you need to do. It's perfectly okay to do this:
#> ./a_lil_script.sh | ./anothuh_script.sh | ./final_script.shIf you really must have a single file, make a script that basically just does the above.
This one's short and sweet. Please, please, please don't have your scripts write to a file if you don't have to. Take your data in on stdin and print out to stdout. If you need to write to a file, just do that at the terminal level like so: ./my_script.sh > your_file.txt. Now your script is a sensible citizen in the scripting world. Your script will play nicely with others.
Before we exit, don't forget your exit codes! When your script dies, don't just let it end like nothing happened. Die hard. exit 1 is how to do it.
Finally, use stderr. If you want to log a warning or print some information that's superfluous to the data you're supposed to be emitting. Then write that to stderr with >&2. This will print it out the terminal, but it won't mess up your anything writing to a pipe or muddle up a file.
That's all for now. I hope this gave you a few ideas to try out and some rules to live by. When you need to get something done, don't be afraid of the shell. It's probably the best productivity tool on your computer today.
This is a really well written post. Not enough people realize how much you can accomplish without having to use Java, C, Python etc. and if they did they would save a lot of time, not to mention prototype more effectively in the rare case they need a more robust tool.