Command Line Tools and Unix I/O

command_line.md

Intro: Command Line Tools

• Types of programs: Side-effect-y (mkdir, touch) vs Function-y (cat, grep, wc) -- Produce output
• Basic Unix "contract": Take in text stream and output text stream
• Somewhat janky but works fairly well

Examples

• head - show me first lines of a file

head /usr/share/dict/words

• default: this information appears in your terminal

Pipes and Redirection

• Default behavior: output goes to your terminal and gets printed
• But sometimes we want to customize this
• example: 2 programs head and wc

Print number of words:

head /usr/share/dict/words | wc

Or, lines (with a flag):

head /usr/share/dict/words | wc -l

Command-line Args vs. text input

• head <filename> vs cat /some/file | head

Redirection Scenarios

• Send this output to another program
• Send this output to a file
• Unix "pipe" -- Send STDOUT from this program to STDIN of the next program
• Fundamental command-line building block for connecting multiple programs to do useful stuff
• Somewhat similar to chaining enumerables in ruby, e.g.:

[1,2,3,4].map { |i| i * 2 }.map { |i| i - 1 }.select { |i| i.odd? }

Common "Pipe" Scenarios

• How long is this input
• Select lines in this input matching a string or regex
• Replace a certain text pattern in this input with another

STDIN / STDOUT / STDERR

• Most programs work with 3 standard input / output streams
• "Standard In": Text that was given to me
• "Standard Out": Text that I am producing for consumption
• "Standard Error": A second output stream, but often used for debug messages, warnings, etc

In Ruby, for example

• $stdin, $stdout, $stderr
• $stdin is default target of gets
• $stdout is default target of puts

ruby -e "10.times { puts 'hello' }" | wc -l

ruby -e "10.times { \$stderr.puts 'hello' }" 2> /tmp/my_err_lines.txt
cat /tmp/my_err_lines.txt

Power of piped IO: Stream processing

• This abstraction lets us process things in small chunks, i.e. "lazily" or "streaming"
• If written correctly, can allow programs to process large volumes of data which would never fit into memory
• Common convention is line-based processing: read characters until you hit a newline, process that line, then continue

Unix Tour: Some Handy Tools

• echo
• head
• tail
• cat
• sort
• uniq
• wc
• curl
• jq
• sed
• grep
• pv
• cut

These can also be combined with custom programs of our own

e.g. processing stdin with ruby:

head /usr/share/dict/words | ruby -e "count = 0; while line = \$stdin.gets;  count += 1; end; puts count"

Executables, Chmod, Shebang, PATH

• Why can you just type some commands into your shell? (ruby, node, cat, etc)
• Unix "Permissions": OS's way of managing what actions can be performed to what files
• Read, Write, or "Execute"