Unix Cookbook

unix_cookbok.md

Unix Cookbook

System

Find system naming:

uname  # os
uuname -l  # short
uname -a  # detailed
uname -n  # hostname only
hostname  # same

Finding executable locations:

type topic  # default
type -a topic  # all
whence topic  # same
which topic  # same

Find terminal logins for self and other:

tty  # self
who  # other

System runs this for user login:

login

File Descriptors

These are usually pointed at /dev/tty (users current terminal).

File Descriptor	Description
FD0	STDIN
FD1	STDOUT
FD2	STDERR

Redirecting output:

command 2>errorfile 1>&2  # STDERR to file, STDOUT to wherever STDERR is pointing (same file)

File description program input:

asker < /dev/null
many_inputs < yes

Shell

General Shortcuts

Key	Action
<CTRL-A>, <HOME>	Beginning of line
<CTRL-E>, <END>	End of line
<CTRL-L>	Clear full screen
<CTRL-T>	Transpose (switch) last 2 Characters after mistype
<CTRL-U>	Clear line back to beginning
<ESC-T>	Transpose (switch) last 2 Words

History

History Shortcuts:

Key	Action
<CTRL-R>	History search
!!	Entire Last command line
!:s/his/her/	Last command with substitution
!:0*	All things/words from Last entire command (same as previous)
!*	Last command but skip first thing/words (0 index), usually last program name
!:1*	Same as previous
!:2*	Skip first two things/words
!$	Last single thing/word on the previous command line

List and Navigate:

history  # all history
history 5  # last 5 commands

History Substitution:

ls -al1
^al1^o  # Same as: ls -o

ls gulpfileee.js
^ee  # eee => e, gulpfile.js

Files and Directories

Wildcards

Wildcard	Matches
?	Any single character
*	Any group of zero or more characters
[ab]	Either 'a' or 'b'
[a-z]	Any character in range between 'a' and 'z' inclusive

Wildcard shell string expansions:

ls /usr/local/{bin,lib}  # List both directories
echo mv file{1,2}.ext  # Output: "mv file1.ext file2.ext"
mv file{1,2}.ext  # Same as: mv file1.ext file2.ext
echo hello{1..5}  # Range zsh Output: "hello1 hello2 hello3 hello4 hello5"

Listing

# sort by time, most recent first
ls -t  # Last Modified time
ls -u  # Access time
ls -c  # Inode change time

# sort by size, largest first
ls -S  # File size

# Modify visual output:
ls -l  # detailed output
ls -1  # list in single column
ls -C  # list in multiple columns

# Extending output
ls -a  # show hidden files
ls -R  # list subdirs and subfile recursively

Locating

updatedb  # required before first-run only
locate filename

Finding

Calling Format

find <path(s)> <-matcher(s)> <-action(s)>

Single Matching

find .  # recursive by default
find . -maxdepth 1  # current directory only
find . -name filename  # filename only, NOT path
find . -path pathname  # full pathname
find . -perm mode  # octal i.e. '0644'
find . -type c  # codes f=file l=symlink d=directory p=pipefile s=socket
find . -user name  # name or id
find . -group name  # name or id
find . -size n  # up to n blocks long (1 block = 512 bytes)
find . -size +n  # over n blocks long
find . -atime n  # accessed time up to n days ago
find . -atime +n  # accessed time over n days ago
find . -mtime n  # modified time up to n days ago
find . -mtime +n  # modified time over n days ago
find . -newer file  # newer than file

Multiple Matching

find . operator1 operator2  # AND (default)
find . operator1 -a operator2  # AND (explicit)
find . operator1 -o operator2  # OR
find . !operator  # NOT
find . \(expression\)  # evaluation

Match Actions

find . operator -print  # default SOMETIMES
find . operator -ls  # output of 'ls -l'
find . operator -exec command \;  # match filename without path, '\;' ends command string 
find . operator -exec command {} \;  # match filename with full path
find . operator -ok command {} \;  # same as -exec but with prompt

Watching

head file  # see top of file
tail file  # see bottom of file, can watch grow

Content

Filtering

grep "query" pathinfo  # single lines at a time
grep -i "query" pathinfo  # case-insensitive
grep -v "query" pathinfo  # invert match
grep -C "query" pathinfo  # return context before and after match, specify integer param for number of lines
grep -B "query" pathinfo  # return context before, ""
grep -A "query" pathinfo  # return context after, ""
agrep "query" pathinfo  # can do multi-line searches
grep "query" pathinfo | sed -e /js/d -e /node/d  # remove any lines with 'js' or 'node' in them

Sorting

By default, sort divides lines into fields at whitespace, and sorts lines by field, from left to right. Field 0 is leftmost.

sort +2 -3 filename  # Start sorting on field[2], stop sorting on field[3], can use multiple
sort +1n filename  # Do a numeric sort on field[1]
sort -t: /etc/passwd  # Use a different field separator character (colon ':')
sort -f filename  # Fold uppercase and lowercase together
sort -l filename  # Use locale sort order instead of default
sort -g filename  # Sort numeric with scientific notation
sort -b filename  # Ignore leading blanks in sort
sort -u filename  # Eliminate duplicate lines in sort, unique results only (better than `uniq`)
sort -d filename  # Dictionary order, eliminate all punctuation in sort
sort -M filename  # Sort by Month
sort -r filename  # Reverse sort order

Counting

wc filename  # lines words characters filename
wc -l filename  # lines

Comparing

Differences

diff file1 file2
cmp file1 file2  # bytewise

Similarities

comm file1 file2

Uniqueness

uniq -u filename  # unique lines output
uniq -d filename  # duplicated lines output
uniq -c filename  # show count
uniq -n filename  # ignore certain amount first (also +n)

Editing

Details below on awk, sed, fmt, etc.

General

fmt 80 longfile.txt  # reformat wide file to 80-chars wide
cat filename | tr '[a-z]' '[A-Z]'  # capitalize translate file
uname -a | cut -d' ' -f3  # using space separator char, get third field
uname -a | cut -c1  # first column of file
dd bs=100 skip=1 if=filename  # delete first 100 bytes of file
dd bs=1 skip=100 if=filename  # ""
join -t, file1.csv file2.csv  # join two CSV files on ID

awk

Pull apart a line of text into sequence of elements.

Basic usage:

awk [options] 'script' [var=value] [file(s)]
awk [options] -f scriptfile [var=value] [files(s)]

Internal system variables:

Variable	Description
FILENAME	Current filename
FS	Field separator (whitespace)
NF	Number of fields in current record
NR	Number of the current record
OFMT	Output format for numbers (%.6g)
OFS	Output field separator (blank)
ORS	Output record separator (newline)
RS	Record separator (newline)
$o	Entire input record
$n	Nth field in current record

An awk script is made up of patterns and procedures.

# If pattern is missing, {procedure} is applied to all records.
# If {procedure} is missing, matched record written to stdout.
pattern {procedure}

Builtin commands:

Group	Commands
Math	sin cos atan exp int log rand sqrt
String	index length match split sub substr tolower toupper
Control Flow	break continue do/while exit for if return
I/O	close delete getline next print printf sprintf system

Examples:

uname -a | awk '{print $3}'  # print 3rd text 'column'
awk -F: '{print $1}' filename  # use a custom field separator ':' instead of default whitespace
awk -f scriptfile filename  # execute file instead of inline script
awk -v var=value '{print $var}' filename  # pre-set variable
awk '/pattern/{print $1}' filename
awk '{print NF}' filename  # print number of fields
uname -a | awk 'NF > 14'  # print if number of fields > 14
uname -a | awk '{print length($2)}'  # field $2 is 26 chars long
awk -F, '{total += $2} END {print total}' file1.csv  # total column of CSV file

sed

Stream editor.

command1 | sed 's/old/new/' | command2
grep "query" pathinfo | sed -e /js/d -e /node/d  # remove any lines with 'js' or 'node' in them

Processes

Processes can be accessed via virtual files in Linux /proc filesystem.

Signals

Name	Number	Meaning and usage
HUP	1	Hangup - stop running, log out or disconnect modem
INT	2	Interrupt - stop running,
QUIT	3	Stop running and dump core, <CTRL->
KILL	9	Emergency stop
SEGV	11	Segmentation violation - tried to access illegal memory
TERM	15	Terminate gracefully if possible
STOP	17	Emergency stop but can CONT-inue
TSTP	18	Suspend stop, ready to continue in background,
CONT	19	Continue executing after STOP or TSTP
CHLD	20	Child process status has changed

Job Control

command &  # run command in background
<CTRL-C>  # kill current foreground job with INTR signal
<CTRL-Z>  # suspend current fg job with TSTP signal
suspend  # suspend current shell
stop  # suspend a background job
bg %num  # Let stopped job <num> continue in bg
fg %num  # Put a bg job into the fg
kill %num  # Kill job num
jobs  # list jobs
set notify  # immediate job-state change notices
stty tostop  # auto-stop bg processes if they try writing to screen
nice -n 19 command  # run command nicely, won't take up monster cpu loads

ps tool

Common ps output columns:

Column	Contents
USER	Username of process owner
UID	User ID of process owner
PID	Process ID
%CPU	Fraction of CPU consumed
%MEM	Fraction of system memory consumed
SZ	Virtual memory used in k or pages
RSS	Real memory used
TT,TTY	Terminal port associated with process
STAT,S	Process state (R)unnable, (S)leeping, (I)dle, S(T)opped, (Z)ombie, (D)isk wait, (P)age wait, S(W)apped
TIME	Total CPU time used
PPID	Parents PID
PRI	Actual execution priority
NI	Process nice number
WCHAN	Event process is waiting for

Pipes

command1 -list `command2`  # command1 output to command1 input, can run into length problems
command2 | xargs command1 -list  # same thing with xargs, no length problems

Named Pipes

FIFO named pipes:

mkfifo /tmp/fifo
tail -f /tmp/fifo &  # tail will not see anything until the writer process (`echo`) dies on the next line
echo "holla holla holla" > /tmp/fifo  # when echo is done the `tail` above will output

Process Substitution

The <(command) expression tells the interpreter to run command and make its output appear as a file:

diff <(sort file1) <(sort file2)

Manuals

Path lookup environment variable:

export MANPATH=$MANPATH:/new/path/to/man/files

Lookup manual documentation for a program or topic:

man topic
info topic  # more?

Target a specific section of a manual page:

man -s section topic
man section topic

Find ideal manual topic page from many options:

man -k topic  # long form
apropos topic  # same
man -f topic  # short form
whatis topic  # same

Write your own man pages (see also: troff, nroff):

NAME
  One line summary of what it does.

SYNOPSIS
  How to invoke the program, including all arguments and options (including [optional] arguments).
  
DESCRIPTION
  What the program does.
  
OPTIONS
  Explanation of each option.

EXAMPLES
  Simply show how to use this program.
  
ENVIRONMENT
  Environment variable concerns for this program.

FILES
  Files the program internals may read or write, including temp files. Doesn't include files on the command line.

BUGS
  Known bugs.

AUTHOR
  Who wrote the program.