Big Data Basics - tools - Unix shell and commands overview

tools-unix_shell_and_commands.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# tools - Unix shell and shell commands"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Based in part on http://nbviewer.ipython.org/url/ibug-um.github.io/2014-summer-camp/setup/the-command-line.ipynb by Matt Burton_\n",
    "\n",
    "When working on a server, one should at least know of and be able to access the command line on the server."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Use SSH to connect to unix shell on server"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First thing we'll test is opening a command shell on the server using SSH.\n",
    "\n",
    "In order to do this:\n",
    "\n",
    "- On Windows, you can either use:\n",
    "\n",
    "    - **putty**, a tried-and-true Windows SSH client that is part of a suite of a light-weight, standalone security programs that can be downloaded from this odd-looking but totally legit web address: [http://www.chiark.greenend.org.uk/~sgtatham/putty/](http://www.chiark.greenend.org.uk/~sgtatham/putty/)\n",
    "    \n",
    "        - download the putty.exe executable - [http://tartarus.org/~simon/putty-snapshots/x86/putty.exe](http://tartarus.org/~simon/putty-snapshots/x86/putty.exe)\n",
    "        - run it.  An initial configuration window will come up:\n",
    "        \n",
    "        <img src=\"http://data.jrn.cas.msu.edu/images/putty_config.png\" />\n",
    "        \n",
    "        - In that window:\n",
    "        \n",
    "            - in the \"Host Name (or IP address)\" field, enter \"bigdataforsocialscience.com\" (without the quotation marks).\n",
    "            - make sure the \"Port\" is 22 and the \"Protocol\" us SSH.\n",
    "            - enter a name in the \"Saved Sessions\" text box and click the \"Save\" button (on subsequent use, you'll just click this name in the list, click \"Load\", then click the \"Open\" button).\n",
    "            - finally, click \"Open\".\n",
    "            - You will be prompted for a username, then a password.  Enter each when it is requested.\n",
    "            \n",
    "    - or the bash shell included in the program **Github for Windows**, available here: [https://desktop.github.com/](https://desktop.github.com/)\n",
    "    \n",
    "        - Once you have installed the Github for Windows program, make sure that the \"Git Shell\" program defaults to creating a bash shell (the \"Git Bash\" option) in the options:\n",
    "        \n",
    "        <img src=\"http://data.jrn.cas.msu.edu/images/git_bash_shell.png\" />\n",
    "        \n",
    "        - Run the Windows \"Git Shell\" program.\n",
    "        - type `ssh <username>@bigdataforsocialscience.com`\n",
    "        - enter your password when prompted.\n",
    "\n",
    "- On Mac, you'll open the \"Terminal\" program:\n",
    "\n",
    "    - Open the \"Terminal\" application, /Applications/Utilities/Terminal, or search for \"Terminal\" in Launchpad.\n",
    "    - type `ssh <username>@bigdataforsocialscience.com`\n",
    "    - enter your password when prompted.\n",
    "\n",
    "\n",
    "Regardless of connection method, the first time you connect to a given server, you'll likely be asked if you want to store the server's keys.  When asked, you should say yes.\n",
    "\n",
    "After entering your password, you should be connected to the server and see a shell prompt.  It will look something like this:\n",
    "\n",
    "    jmorgan@ip-172-31-36-239:~$\n",
    "\n",
    "To start, we'll try a few commands to make sure your shell is working OK:\n",
    "\n",
    "- _`pwd`_ - gets **_p_**resent **_w_**orking **_d_**irectory.\n",
    "- _`ls`_ - **_l_**i**_s_**t directory contents.\n",
    "- _`ls -al`_ - **_l_**i**_s_**t directory contents (\"-al\" = **_a_**ll files, and in **_l_**ist mode).\n",
    "- `python --version` - get the version of **_python_**.\n",
    "- `pip -V` - get the version of the **_pip_** python package manager.\n",
    "\n",
    "And we'll sneak in some commands to see if jupyterhub is working:\n",
    "- `nbgrader extension install --symlink`\n",
    "- `nbgrader extension activate`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. shell commands"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Anatomy of a shell command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`<command> <flags_and_parameters> <arguments>`\n",
    "\n",
    "- Example: `ls -al /home/jonathanmorgan`\n",
    "\n",
    "WHERE:\n",
    "\n",
    "- `<command>` is the name of the command you are running (\"`ls`\" in the above example).\n",
    "- `<flags_and_parameters>` are basic instructions you pass to the command to tell it precisely how you want it to work.\n",
    "    - Flags are generally either a letter preceded by a single hyphen (\"-\") or a longer, more descriptive string preceded by two hyphens.  In the example above, \"`-al`\" contains two flags - \"`-a`\", which tells `ls` to output all files, not just those that are not hidden, and \"`-l`\", which tells the command to output the directory listing as a list, one file to a line, with details on each file.  You can sometimes combine flags together, as in this case, by entering a single hyphen, then all the flags you want to pass to the command.  Some commands expect each flag to be separate, however, with each preceded by a hyphen and separated by a space.\n",
    "    - Parameters are similar to flags, but they have additional values that follow the parameter or flag and provide more detail on what you are asking the command to do.  For example, you might tell a command to store the results of running it to a file with a \"`-f`\" parameter, where the \"`-f`\" is followed by the path to the location where you want the output file stored.\n",
    "        - a common example: `tar -cvzf <file_name> <directory_you_are_tarring>`\n",
    "- `<arguments>` are other values you pass to the command that further specify how the command will run.  In the \"`ls`\" example above, \"`/home/jonathanmorgan`\" is the sole argument to the \"`ls`\" command, telling it to list the contents of the directory \"`/home/joanthanmorgan`\".\n",
    "\n",
    "Different commands can make very different choices about how features are exposed, in particular which things are flags, parameters, and arguments.  So, how do you know how a given command works?\n",
    "\n",
    "- In any mac/linux/unix shell (there are more than just bash - csh, ksh, dash, sh, etc.), to find documentation on a command, use the \"man\" (for manual) command:\n",
    "\n",
    "        man <command_you_want_info_on>\n",
    "        # example: man ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example - Working with directories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `pwd` - to Print Working Directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    jmorgan@ip-172-31-36-239:~$ pwd\n",
    "    /home/jmorgan/work/python_class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `ls` - to LiSt directory contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "jmorgan@ip-172-31-36-239:~$ ls\n",
    "python_class.sqlite python_utilities    uofm_2015_spring\n",
    "jmorgan@ip-172-31-36-239:~$ \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Use the `-l` parameter to do a long listing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "jmorgan@ip-172-31-36-239:~$ ls -al\n",
    "total 272\n",
    "drwxr-xr-x   6 jmorgan  jmorgan     204 Jan 22 00:05 .\n",
    "drwxr-xr-x   9 jmorgan  jmorgan     306 Jan 15 17:24 ..\n",
    "-rw-r--r--@  1 jmorgan  jmorgan    6148 Jan 15 14:09 .DS_Store\n",
    "-rw-r--r--@  1 jmorgan  jmorgan  131072 Jan 15 14:29 python_class.sqlite\n",
    "drwxr-xr-x  22 jmorgan  jmorgan     748 Jan 13 20:28 python_utilities\n",
    "drwxr-xr-x   7 jmorgan  jmorgan     238 Jan 15 18:12 uofm_2015_spring\n",
    "jmorgan@ip-172-31-36-239:~$ \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `cd <filepath>` - to Change Directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "jmorgan@ip-172-31-36-239:~$ pwd\n",
    "/home/jmorgan/work/python_class\n",
    "\n",
    "jmorgan@ip-172-31-36-239:~$ ls -al\n",
    "total 272\n",
    "drwxr-xr-x   6 jmorgan  jmorgan     204 Jan 22 00:05 .\n",
    "drwxr-xr-x   9 jmorgan  jmorgan     306 Jan 15 17:24 ..\n",
    "-rw-r--r--@  1 jmorgan  jmorgan    6148 Jan 15 14:09 .DS_Store\n",
    "-rw-r--r--@  1 jmorgan  jmorgan  131072 Jan 15 14:29 python_class.sqlite\n",
    "drwxr-xr-x  22 jmorgan  jmorgan     748 Jan 13 20:28 python_utilities\n",
    "drwxr-xr-x   7 jmorgan  jmorgan     238 Jan 15 18:12 uofm_2015_spring\n",
    "\n",
    "jmorgan@ip-172-31-36-239:~$ cd uofm_2015_spring/\n",
    "\n",
    "jmorgan@ip-172-31-36-239:~$ pwd\n",
    "/home/jmorgan/work/python_class/uofm_2015_spring\n",
    "\n",
    "jmorgan@ip-172-31-36-239:~$ \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `mv <existing filepath> <desired filepath>` - to MoVe files or folders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### File Paths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### `/foo/bar` - the *bar* is a subdirectory of *foo*, or a file inside *foo*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`/Users/mcburton/Dropbox/si/bigdatabootcamp/2014-summer-camp/setup`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### \"`.`\" - means the current directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usually used when running commands or shell scripts that are in the same directory as your shell:\n",
    "\n",
    "```\n",
    "jmorgan@ip-172-31-36-239:~$ pwd\n",
    "/home/jmorgan/work/python_class/uofm_2015_spring\n",
    "\n",
    "jmorgan@ip-172-31-36-239:~$ ls -al\n",
    "total 32\n",
    "drwxr-xr-x   8 jmorgan  jmorgan  272 Jan 22 00:19 .\n",
    "drwxr-xr-x   6 jmorgan  jmorgan  204 Jan 22 00:05 ..\n",
    "drwxr-xr-x  15 jmorgan  jmorgan  510 Jan 15 18:12 .git\n",
    "-rw-r--r--   1 jmorgan  jmorgan  675 Jan 15 14:08 .gitignore\n",
    "-rw-r--r--   1 jmorgan  jmorgan  104 Jan 15 14:09 README.md\n",
    "drwxr-xr-x   3 jmorgan  jmorgan  102 Jan 15 18:12 exercises\n",
    "-rw-rw-r--   1 jmorgan  jmorgan   48 Jan 15 13:53 hello_world.py\n",
    "-rwxrwxr-x   1 jmorgan  jmorgan   17 Jan 22 00:19 test.sh\n",
    "\n",
    "jmorgan@ip-172-31-36-239:~$ cat test.sh\n",
    "#!/bin/sh\n",
    "ls -al\n",
    "\n",
    "jmorgan@ip-172-31-36-239:~$ test.sh\n",
    "-bash: test.sh: command not found\n",
    "\n",
    "jmorgan@ip-172-31-36-239:~$ ./test.sh\n",
    "total 32\n",
    "drwxr-xr-x   8 jmorgan  jmorgan  272 Jan 22 00:19 .\n",
    "drwxr-xr-x   6 jmorgan  jmorgan  204 Jan 22 00:05 ..\n",
    "drwxr-xr-x  15 jmorgan  jmorgan  510 Jan 15 18:12 .git\n",
    "-rw-r--r--   1 jmorgan  jmorgan  675 Jan 15 14:08 .gitignore\n",
    "-rw-r--r--   1 jmorgan  jmorgan  104 Jan 15 14:09 README.md\n",
    "drwxr-xr-x   3 jmorgan  jmorgan  102 Jan 15 18:12 exercises\n",
    "-rw-rw-r--   1 jmorgan  jmorgan   48 Jan 15 13:53 hello_world.py\n",
    "-rwxrwxr-x   1 jmorgan  jmorgan   17 Jan 22 00:19 test.sh\n",
    "jmorgan@ip-172-31-36-239:~$ \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### \"`..`\" - means up/back one directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "jmorgan@ip-172-31-36-239:~$ pwd\n",
    "/home/jmorgan/work/si/bigdatabootcamp/2014-summer-camp/setup\n",
    "\n",
    "jmorgan@ip-172-31-36-239:~$ ls -l\n",
    "total 3280\n",
    "-rw-r--r--@ 1 jmorgan  jmorgan  218212 May 16 12:31 OSX-anaconda-launcher.png\n",
    "-rw-r--r--@ 1 jmorgan  jmorgan  175418 May 18 22:17 anaconda-command-line.png\n",
    "-rw-r--r--@ 1 jmorgan  jmorgan  113444 May 16 12:37 anaconda-launcher-launch.png\n",
    "-rw-r--r--@ 1 jmorgan  jmorgan  114809 May 16 12:38 anaconda-launcher.png\n",
    "-rw-r--r--@ 1 jmorgan  jmorgan   77454 May 18 22:07 dir-tree.png\n",
    "-rw-r--r--@ 1 jmorgan  jmorgan  244584 May 16 14:17 sqlite-manager-install.png\n",
    "-rw-r--r--@ 1 jmorgan  jmorgan   79532 May 16 14:25 sqlite-manager.png\n",
    "-rw-r--r--@ 1 jmorgan  jmorgan  253725 May 18 22:15 terminal-location.png\n",
    "-rw-r--r--@ 1 jmorgan  jmorgan   44467 May 18 22:09 terminal.png\n",
    "-rw-r--r--  1 jmorgan  jmorgan    8531 May 19 09:40 the-command-line.ipynb\n",
    "-rw-r--r--  1 jmorgan  jmorgan    6787 May 18 13:13 the-setup.ipynb\n",
    "-rw-r--r--@ 1 jmorgan  jmorgan  277489 May 16 12:33 windows-8-anaconda-launcher.png\n",
    "-rw-r--r--@ 1 jmorgan  jmorgan   41325 May 19 09:33 windows-commands.png\n",
    "\n",
    "jmorgan@ip-172-31-36-239:~$ cd ..\n",
    "\n",
    "jmorgan@ip-172-31-36-239:~$ pwd\n",
    "/home/jmorgan/work/si/bigdatabootcamp/2014-summer-camp\n",
    "\n",
    "jmorgan@ip-172-31-36-239:~$ ls -l\n",
    "total 40\n",
    "-rw-r--r--   1 jmorgan  jmorgan  2838 Apr 24 15:14 LICENSE.md\n",
    "-rw-r--r--   1 jmorgan  jmorgan   248 Apr 24 22:20 README.md\n",
    "-rw-r--r--   1 jmorgan  jmorgan   321 Apr 24 16:06 _config.yml\n",
    "drwxr-xr-x   7 jmorgan  jmorgan   238 Apr 24 15:14 _includes\n",
    "drwxr-xr-x   4 jmorgan  jmorgan   136 Apr 24 16:57 _layouts\n",
    "drwxr-xr-x  11 jmorgan  jmorgan   374 May 19 09:40 _site\n",
    "drwxr-xr-x   6 jmorgan  jmorgan   204 Apr 24 17:20 css\n",
    "drwxr-xr-x   9 jmorgan  jmorgan   306 May  9 12:10 img\n",
    "-rw-r--r--   1 jmorgan  jmorgan  6717 May 18 22:00 index.html\n",
    "drwxr-xr-x   4 jmorgan  jmorgan   136 Apr 24 15:14 js\n",
    "drwxr-xr-x  17 jmorgan  jmorgan   578 May 19 09:33 setup\n",
    "drwxr-xr-x   5 jmorgan  jmorgan   170 May 18 21:48 slides\n",
    "\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}