Skip to content

Instantly share code, notes, and snippets.

@danoneata

danoneata/README.md

Created June 21, 2019 12:23
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save danoneata/aad80b29d383b044565d776b4c7e7a00 to your computer and use it in GitHub Desktop.
Save danoneata/aad80b29d383b044565d776b4c7e7a00 to your computer and use it in GitHub Desktop.
Download ZIP
Unix shell intro
Raw
README.md

Intro

  • shell takes commands and passes to the operating system
  • most popular shell is bash (Bourne Again Shell): a replacement of sh, the origianl Unix shell, which was written by Steve Bourne
  • motivating examples:
# Copies all HTML files to `destination`,
# but only those that do not exist or are newer
cp -u *.html destination
# Finds the ten most common words in a text file
curl http://composingprograms.com/shakespeare.txt | \
tr -d '[[:punct:]]' | tr ' ' '\n' | sort | uniq -c | sort -nr | head
# Finds all the function definitions in a Python file 
find . -name '*.py' -exec grep -H '^def ' {} \;
who | grep -i blonde | date; cd ~; unzip; touch; strip; finger; mount; gasp; yes; uptime; umount; sleep
  • powerful suite of commands; examples: man cal date free exit

Navigation and system exploration

Navigation.

  • cd change directory

  • ls lists directory contents

  • pwd prints working directory

  • files are organized in a tree

  • / root directory

  • . current directory

  • .. parrent directory

  • ~ home directory

  • ~username username's home directory

Some important folders:

  • / root
  • /bin programs, common binaries
  • /etc system configurations
  • /home users' home
  • /tmp, /var temporary files
  • /usr user-installed programs

Exploring the system.

  • ls lists direcotry contents

  • file determines file type

  • less view file contents

  • Many of these commands accept arguments or parameters

  • Options can be combined: ls -l -a -r can be written as ls -lar

  • Help --help or -help or -h

Manipulating files and directories

  • cp copies files and directories
  • mv moves files and directories
  • mkdir creates directories
  • rm removes files and directories
  • touch create a new empty file or updates its last-modified time stamp
  • ln create hard and symbolic links

Warning! The above command do not prompt for confirmation It makes it easy to overwrite/delete a file; use the -i flag to overwrite this setting.

Flags:

  • -i, --interative
  • -r, --recursive
  • -u, --update

Wildcards:

Wildcard Matches
* any character
? any single character
[characters] any
[!characters] any
[[:class:]] any character that is a member of the specified class

Classes are:

  • [:alnum:]
  • [:alpha:]
  • [:digit:]
  • [:lower:]
  • [:upper:]

Redirection

Get images from here

Standard input, output, error.

  • Redirecting standard output > and >>
ls -l /usr/bin
ls -l /usr/bin > ls-output.txt
ls -l ls-output.txt
less ls-output.txt
ls -l /bin/usr > ls-output.txt
> ls-output.txt
ls -l /usr/bin >> ls-output.txt
ls -l /usr/bin >> ls-output.txt
  • Redirecting standard error 1> stdout and 2> stderr:
ls -l /bin/usr 2> ls-error.txt
  • Redirecting both standard output and standard error to one file &>
ls -l /bin/usr &> ls-output.txt

  • cat concatenate files

  • head outputs the first part of a file

  • tail outputs the last part of a file

  • sort sort lines of text

  • uniq report or omit repeated lines

  • wc counts newlines, words and bytes for each file

  • tee read from standard input and write to standard output and files

  • grep prints lines matching a pattern

  • sort sorts lines of text

  • find finds files in a given directory

  • regex101: Online regex tester and debugger

  • regexone: Interactive tutorial for regex

Pipelines.

command1 | command2
  • Dissecting the motivating example:
curl http://composingprograms.com/shakespeare.txt \
| tr -d '[[:punct:]]' | tr ' ' '\n' | sort \
| uniq -c | sort -nr | head

Unix philosophy:

  • Write programs that do one thing and do it well.
  • Write programs to work together.
  • Write programs to handle text streams, because that is a universal interface.

Keyboard shortcuts and miscellany

  • clear clears the screen

  • history display the contents of the history list

  • My favorite: ALT+. -- place the argument of the most recent command on the shell

mv a{,.bk}
!!  # execute last command
sudo !!

Access Rights and Intro to Processes

File system access rights

  • In your directory, type ls -l (for long listing)
  • You will see that you now get lots of details about the contents of your directory, similar to the example below:
-rwxrw-r-- 1 marius speed   5538367 Apr 21 11:35 'LectureX.pdf'

Each file (and directory) has associated access rights: in the left-hand column is a 10 symbol string consisting of the symbols d, r, w, x, -, and, occasionally, s or S:

  • If d is present, it will be at the left hand end of the string, and indicates a directory: otherwise - will be the starting symbol of the string. The 9 remaining symbols indicate the permissions, or access rights, and are taken as three groups of 3.
  • The left group of 3 gives the file permissions for the user that owns the file (or directory) (marius in the above example);
  • The middle group gives the permissions for the group of people to whom the file (or directory) belongs (speed in the above example);
  • The rightmost group gives the permissions for all others.

The symbols r, w, etc., have slightly different meanings depending on whether they refer to a simple file or to a directory.

Access rights on files:

  • r (or -), indicates read permission (or otherwise), that is, the presence or absence of permission to read and copy the file;
  • w (or -), indicates write permission (or otherwise), that is, the permission (or otherwise) to change a file;
  • x (or -), indicates execution permission (or otherwise), that is, the permission to execute a file, where appropriate

Access rights on folders

  • r allows users to list files in the directory;
  • w means that users may delete files from the directory or move files into it;
  • x means the right to access files in the directory. This implies that you may read files in the directory provided you have read permission on the individual files.

Example:

-rw-------
#  means a file that only the owner can read and write, no-one else can read or write and no-one has execution rights (e.g. a mailbox file).

Changing access rights

chmod: changing a file mode. Only the owner of a file can use chmod to change the permissions of a file.

  • Symbols: u user, g group, o other, a all, r read, w write (and delete), x execute (and access directory), + add permission, - take away permission.

Example:

chmod go-rwx some_file
# removes read write and execute permissions on the file for the group and others. It will leave the other permissions unaffected.
chmod a+rw some_file
# gives read and write permissions on the file to all.

Processes and Jobs

A process is an executing program identified by a unique PID (process identifier). To see information about your processes, with their associated PID and status, type ps.

A process may be

  • in the foreground,
  • in the background, or
  • be suspended.

In general the shell does not return the UNIX prompt until the current process has finished executing. Some processes take a long time to run and hold up the terminal. Backgrounding a long process has the effect that the UNIX prompt is returned immediately, and other tasks can be carried out while the original process continues executing.

Running background processes

To background a process, type an & at the end of the command line. For example, the command sleep waits a given number of seconds before continuing.

sleep 10
# will wait 10 seconds before returning the command prompt. Until the command prompt is returned, you can do nothing except wait.

sleep 10 & 
[1] 27349
# will run sleep in the background.
# the sleep command is typed in by the user; the next line, indicating job number and PID, is returned by the machine.

The & runs the job in the background and returns the prompt straight away, allowing you do run other programs while waiting for that one to finish.

The user is returned a job number (numbered from 1) enclosed in square brackets, together with a PID and is notified when a background process is finished. Backgrounding is useful for jobs which will take a long time to complete (e.g., training a ML system).

Backgrounding a current foreground process

  • Type sleep 1000
  • You can suspend the process running in the foreground by typing ^Z, i.e., hold down the [Ctrl] key and type [z]. Then to put it in the background, type bg.

Listing suspended and background processes

When a process is running, backgrounded or suspended, it will be entered onto a list along with a job number. To examine this list, type jobs. An example output could be

[1] Suspended sleep 1000
[2] Running firefox
[3] Running matlab

To restart (foreground) a suspended processes:

# fg %jobnumber

fg 1 
# will bring sleep 1000 back
# typing fg w/o a job number foregrounds the last suspended process

Killing a process

kill: terminate or signal a process.

It is sometimes necessary to kill a process (for example, when an executing program is in an infinite loop). To kill a job running in the foreground, type ^C.

kill jobnumber
# kills a suspended or background process 

sleep 100 &
jobs
# say it is job number 2, for example
kill 2

ps: process status

Alternatively, processes can be killed by finding their process numbers (PIDs) and using kill PID_number.

sleep 100 & 
ps 
PID TT S TIME COMMAND
20077 pts/5 S 0:05 sleep 100
21563 pts/5 T 0:00 firefox
21873 pts/5 S 0:25 gedit
# let's kill the sleep 100 process 
kill 20077 
# and check if it worked
ps

If a process refuses to be killed, use the -9 option: kill -9 20077.

Optional

Play with htop: https://codeahoy.com/2017/01/20/hhtop-explained-visually/

Further resources

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment