Skip to content

Instantly share code, notes, and snippets.

@jonathanmorgan

jonathanmorgan/class3-bash_ipython_design.ipynb

Last active August 29, 2015 14:13
Show Gist options
  • Save jonathanmorgan/05ad42fae0586f05ffde to your computer and use it in GitHub Desktop.
Save jonathanmorgan/05ad42fae0586f05ffde to your computer and use it in GitHub Desktop.
Download ZIP
IPython Notebook that goes over bash shell, IPython, IPython Notebook, and Software Design
Display the source blob
Display the rendered blob
Raw
class3-bash_ipython_design.ipynb
{
"metadata": {
"name": "",
"signature": "sha256:6d06f3e5bd0128552ca00277447fba119f33a2ac62231ba85921249fec26156a"
},
"nbformat": 3,
"nbformat_minor": 0,
"worksheets": [
{
"cells": [
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Week 3: bash, IPython, IPython Notebook and software design"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"_Based on http://nbviewer.ipython.org/url/ibug-um.github.io/2014-summer-camp/setup/the-command-line.ipynb by Matt Burton_"
]
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Before we begin:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- How to use bash shell instead of PowerShell in Git for Windows (and my first annotated image!):\n",
"\n",
"<img src=\"http://data.jrn.cas.msu.edu/images/git_bash_shell.png\" width=\"800\" />\n",
"\n",
"- I've started a few pages in the wiki on ctools:\n",
"\n",
" - interesting data\n",
" - visualization\n",
" - python information\n",
" \n",
"Check them out and add things if you get a chance. Feel free to make new pages, as well, if you are so inclined. To do this:\n",
"\n",
"- edit the page you want to link to your new page.\n",
"- in the page, place a single set of square brackets around the name you want for the new page.\n",
"- Save the page.\n",
"- in the view of the page, click the link you just created. This will create the new page and take you to it.\n",
" \n",
"So, for example:\n",
" \n",
"- `[new sample page]`\n",
" \n",
"Would link to a page named \"new sample page\"."
]
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"1. The bash shell"
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Getting to the bash shell"
]
},
{
"cell_type": "heading",
"level": 3,
"metadata": {},
"source": [
"On OSX"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Terminal is located at `/Applications/Utilities/Terminal`.\n",
"- Or, open Launcher, then in the search bar at the top, type \"Terminal\"\n",
"- On all my machines, I've added it to the Dock, as well. Its super useful..."
]
},
{
"cell_type": "heading",
"level": 3,
"metadata": {},
"source": [
"On Windows (using Git for Windows)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Make sure in Git for Windows Options that you've set your \"Default shell\" to \"git bash\".\n",
"- Run the program \"Git Shell\"."
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Anatomy of a shell command"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`<command> <flags_and_parameters> <arguments>`\n",
"\n",
"- Example: `ls -al /home/jonathanmorgan`\n",
"\n",
"WHERE:\n",
"\n",
"- `<command>` is the name of the command you are running (\"`ls`\" in the above example).\n",
"- `<flags_and_parameters>` are basic instructions you pass to the command to tell it precisely how you want it to work.\n",
" - Flags are generally either a letter preceded by a single hyphen (\"-\") or a longer, more descriptive string preceded by two hyphens. In the example above, \"`-al`\" contains two flags - \"`-a`\", which tells `ls` to output all files, not just those that are not hidden, and \"`-l`\", which tells the command to output the directory listing as a list, one file to a line, with details on each file. You can sometimes combine flags together, as in this case, by entering a single hyphen, then all the flags you want to pass to the command. Some commands expect each flag to be separate, however, with each preceded by a hyphen and separated by a space.\n",
" - Parameters are similar to flags, but they have additional values that follow the parameter or flag and provide more detail on what you are asking the command to do. For example, you might tell a command to store the results of running it to a file with a \"`-f`\" parameter, where the \"`-f`\" is followed by the path to the location where you want the output file stored.\n",
" - a common example: `tar -cvzf <file_name> <directory_you_are_tarring>`\n",
"- `<arguments>` are other values you pass to the command that further specify how the command will run. In the \"`ls`\" example above, \"`/home/jonathanmorgan`\" is the sole argument to the \"`ls`\" command, telling it to list the contents of the directory \"`/home/joanthanmorgan`\".\n",
"\n",
"Different commands can make very different choices about how features are exposed, in particular which things are flags, parameters, and arguments. So, how do you know how a given command works?\n",
"\n",
"- In bash on mac or linux, to find documentation on a command, use the \"man\" (for manual) command:\n",
"\n",
" man <command_you_want_info_on>\n",
" # example: man ls\n",
" \n",
"- On Windows, in git bash, man isn't installed by default, but a given command often provides help if you invoke it with either the \"`--help`\" or \"`--usage`\" flags. Example:\n",
"\n",
" ls --help"
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Working with Directories"
]
},
{
"cell_type": "heading",
"level": 3,
"metadata": {},
"source": [
"`pwd` - to Print Working Directory"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" JSM-2013-MBP:python_class jonathanmorgan$ pwd\n",
" /Users/jonathanmorgan/Documents/work/python_class"
]
},
{
"cell_type": "heading",
"level": 3,
"metadata": {},
"source": [
"`ls` - to LiSt directory contents"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```\n",
"JSM-2013-MBP:python_class jonathanmorgan$ ls\n",
"python_class.sqlite python_utilities uofm_2015_spring\n",
"JSM-2013-MBP:python_class jonathanmorgan$ \n",
"```"
]
},
{
"cell_type": "heading",
"level": 4,
"metadata": {},
"source": [
"Use the `-l` parameter to do a long listing"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```\n",
"JSM-2013-MBP:python_class jonathanmorgan$ ls -al\n",
"total 272\n",
"drwxr-xr-x 6 jonathanmorgan staff 204 Jan 22 00:05 .\n",
"drwxr-xr-x 9 jonathanmorgan staff 306 Jan 15 17:24 ..\n",
"-rw-r--r--@ 1 jonathanmorgan staff 6148 Jan 15 14:09 .DS_Store\n",
"-rw-r--r--@ 1 jonathanmorgan staff 131072 Jan 15 14:29 python_class.sqlite\n",
"drwxr-xr-x 22 jonathanmorgan staff 748 Jan 13 20:28 python_utilities\n",
"drwxr-xr-x 7 jonathanmorgan staff 238 Jan 15 18:12 uofm_2015_spring\n",
"JSM-2013-MBP:python_class jonathanmorgan$ \n",
"```"
]
},
{
"cell_type": "heading",
"level": 3,
"metadata": {},
"source": [
"`cd <filepath>` - to Change Directory"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```\n",
"JSM-2013-MBP:python_class jonathanmorgan$ pwd\n",
"/Users/jonathanmorgan/Documents/work/python_class\n",
"\n",
"JSM-2013-MBP:python_class jonathanmorgan$ ls -al\n",
"total 272\n",
"drwxr-xr-x 6 jonathanmorgan staff 204 Jan 22 00:05 .\n",
"drwxr-xr-x 9 jonathanmorgan staff 306 Jan 15 17:24 ..\n",
"-rw-r--r--@ 1 jonathanmorgan staff 6148 Jan 15 14:09 .DS_Store\n",
"-rw-r--r--@ 1 jonathanmorgan staff 131072 Jan 15 14:29 python_class.sqlite\n",
"drwxr-xr-x 22 jonathanmorgan staff 748 Jan 13 20:28 python_utilities\n",
"drwxr-xr-x 7 jonathanmorgan staff 238 Jan 15 18:12 uofm_2015_spring\n",
"\n",
"JSM-2013-MBP:python_class jonathanmorgan$ cd uofm_2015_spring/\n",
"\n",
"JSM-2013-MBP:uofm_2015_spring jonathanmorgan$ pwd\n",
"/Users/jonathanmorgan/Documents/work/python_class/uofm_2015_spring\n",
"\n",
"JSM-2013-MBP:uofm_2015_spring jonathanmorgan$ \n",
"```"
]
},
{
"cell_type": "heading",
"level": 3,
"metadata": {},
"source": [
"`mv <existing filepath> <desired filepath>` - to MoVe files or folders"
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"File Paths"
]
},
{
"cell_type": "heading",
"level": 3,
"metadata": {},
"source": [
"`/foo/bar` - the *bar* is a subdirectory of *foo*, or a file inside *foo*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`/Users/mcburton/Dropbox/si/bigdatabootcamp/2014-summer-camp/setup`"
]
},
{
"cell_type": "heading",
"level": 3,
"metadata": {},
"source": [
"\"`.`\" - means the current directory"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Usually used when running commands or shell scripts that are in the same directory as your shell:\n",
"\n",
"```\n",
"JSM-2013-MBP:uofm_2015_spring jonathanmorgan$ pwd\n",
"/Users/jonathanmorgan/Documents/work/python_class/uofm_2015_spring\n",
"\n",
"JSM-2013-MBP:uofm_2015_spring jonathanmorgan$ ls -al\n",
"total 32\n",
"drwxr-xr-x 8 jonathanmorgan staff 272 Jan 22 00:19 .\n",
"drwxr-xr-x 6 jonathanmorgan staff 204 Jan 22 00:05 ..\n",
"drwxr-xr-x 15 jonathanmorgan staff 510 Jan 15 18:12 .git\n",
"-rw-r--r-- 1 jonathanmorgan staff 675 Jan 15 14:08 .gitignore\n",
"-rw-r--r-- 1 jonathanmorgan staff 104 Jan 15 14:09 README.md\n",
"drwxr-xr-x 3 jonathanmorgan staff 102 Jan 15 18:12 exercises\n",
"-rw-rw-r-- 1 jonathanmorgan staff 48 Jan 15 13:53 hello_world.py\n",
"-rwxrwxr-x 1 jonathanmorgan staff 17 Jan 22 00:19 test.sh\n",
"\n",
"JSM-2013-MBP:uofm_2015_spring jonathanmorgan$ cat test.sh\n",
"#!/bin/sh\n",
"ls -al\n",
"\n",
"JSM-2013-MBP:uofm_2015_spring jonathanmorgan$ test.sh\n",
"-bash: test.sh: command not found\n",
"\n",
"JSM-2013-MBP:uofm_2015_spring jonathanmorgan$ ./test.sh\n",
"total 32\n",
"drwxr-xr-x 8 jonathanmorgan staff 272 Jan 22 00:19 .\n",
"drwxr-xr-x 6 jonathanmorgan staff 204 Jan 22 00:05 ..\n",
"drwxr-xr-x 15 jonathanmorgan staff 510 Jan 15 18:12 .git\n",
"-rw-r--r-- 1 jonathanmorgan staff 675 Jan 15 14:08 .gitignore\n",
"-rw-r--r-- 1 jonathanmorgan staff 104 Jan 15 14:09 README.md\n",
"drwxr-xr-x 3 jonathanmorgan staff 102 Jan 15 18:12 exercises\n",
"-rw-rw-r-- 1 jonathanmorgan staff 48 Jan 15 13:53 hello_world.py\n",
"-rwxrwxr-x 1 jonathanmorgan staff 17 Jan 22 00:19 test.sh\n",
"JSM-2013-MBP:uofm_2015_spring jonathanmorgan$ \n",
"```"
]
},
{
"cell_type": "heading",
"level": 3,
"metadata": {},
"source": [
"\"`..`\" - means up/back one directory"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```\n",
"pilemian:setup mcburton$ pwd\n",
"/Users/mcburton/Dropbox/si/bigdatabootcamp/2014-summer-camp/setup\n",
"\n",
"pilemian:setup mcburton$ ls -l\n",
"total 3280\n",
"-rw-r--r--@ 1 mcburton staff 218212 May 16 12:31 OSX-anaconda-launcher.png\n",
"-rw-r--r--@ 1 mcburton staff 175418 May 18 22:17 anaconda-command-line.png\n",
"-rw-r--r--@ 1 mcburton staff 113444 May 16 12:37 anaconda-launcher-launch.png\n",
"-rw-r--r--@ 1 mcburton staff 114809 May 16 12:38 anaconda-launcher.png\n",
"-rw-r--r--@ 1 mcburton staff 77454 May 18 22:07 dir-tree.png\n",
"-rw-r--r--@ 1 mcburton staff 244584 May 16 14:17 sqlite-manager-install.png\n",
"-rw-r--r--@ 1 mcburton staff 79532 May 16 14:25 sqlite-manager.png\n",
"-rw-r--r--@ 1 mcburton staff 253725 May 18 22:15 terminal-location.png\n",
"-rw-r--r--@ 1 mcburton staff 44467 May 18 22:09 terminal.png\n",
"-rw-r--r-- 1 mcburton staff 8531 May 19 09:40 the-command-line.ipynb\n",
"-rw-r--r-- 1 mcburton staff 6787 May 18 13:13 the-setup.ipynb\n",
"-rw-r--r--@ 1 mcburton staff 277489 May 16 12:33 windows-8-anaconda-launcher.png\n",
"-rw-r--r--@ 1 mcburton staff 41325 May 19 09:33 windows-commands.png\n",
"\n",
"pilemian:setup mcburton$ cd ..\n",
"\n",
"pilemian:2014-summer-camp mcburton$ pwd\n",
"/Users/mcburton/Dropbox/si/bigdatabootcamp/2014-summer-camp\n",
"\n",
"pilemian:2014-summer-camp mcburton$ ls -l\n",
"total 40\n",
"-rw-r--r-- 1 mcburton staff 2838 Apr 24 15:14 LICENSE.md\n",
"-rw-r--r-- 1 mcburton staff 248 Apr 24 22:20 README.md\n",
"-rw-r--r-- 1 mcburton staff 321 Apr 24 16:06 _config.yml\n",
"drwxr-xr-x 7 mcburton staff 238 Apr 24 15:14 _includes\n",
"drwxr-xr-x 4 mcburton staff 136 Apr 24 16:57 _layouts\n",
"drwxr-xr-x 11 mcburton staff 374 May 19 09:40 _site\n",
"drwxr-xr-x 6 mcburton staff 204 Apr 24 17:20 css\n",
"drwxr-xr-x 9 mcburton staff 306 May 9 12:10 img\n",
"-rw-r--r-- 1 mcburton staff 6717 May 18 22:00 index.html\n",
"drwxr-xr-x 4 mcburton staff 136 Apr 24 15:14 js\n",
"drwxr-xr-x 17 mcburton staff 578 May 19 09:33 setup\n",
"drwxr-xr-x 5 mcburton staff 170 May 18 21:48 slides\n",
"\n",
"```"
]
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"2. the IPython Interactive Interpreter"
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Launching the IPython Interactive Interpreter"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"```\n",
"JSM-2013-MBP:uofm_2015_spring jonathanmorgan$ ipython\n",
"Python 2.7.9 |Anaconda 2.1.0 (x86_64)| (default, Dec 15 2014, 10:37:34) \n",
"Type \"copyright\", \"credits\" or \"license\" for more information.\n",
"\n",
"IPython 2.3.1 -- An enhanced Interactive Python.\n",
"Anaconda is brought to you by Continuum Analytics.\n",
"Please check out: http://continuum.io/thanks and https://binstar.org\n",
"? -> Introduction and overview of IPython's features.\n",
"%quickref -> Quick reference.\n",
"help -> Python's own help system.\n",
"object? -> Details about 'object', use 'object??' for extra details.\n",
"\n",
"In [1]: \n",
"```"
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Basic commands"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- introduction: [http://ipython.org/ipython-doc/2/interactive/tutorial.html](http://ipython.org/ipython-doc/2/interactive/tutorial.html)\n",
"- `?` = overview of features\n",
"- `%quickref` = Python quick reference.\n",
"- add `\u201c?\u201d` after a variable reference - details on that variable.\n",
"- `\u201c??\u201d` = extra details.\n",
"- `help` = python\u2019s built-in help system."
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Running Python Code in IPython"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For more information on running Python Code in IPython, see the week 2 tech notes: [http://nbviewer.ipython.org/gist/jonathanmorgan/c1a84e3b42c92662bfd2#IPython](http://nbviewer.ipython.org/gist/jonathanmorgan/c1a84e3b42c92662bfd2#IPython)"
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Magic Commands"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"IPython magic commands let you run more advanced functionality than just python code. Examples:\n",
"\n",
"- `%run <python_file_path>` - runs the file whose path you enter inside IPython as a python program, leaving all variables in the program still available in IPython, for debugging purposes.\n",
"\n",
"The `cd`, `pwd`, `mkdir`, etc. commands that we ran in IPython last week that worked just like the ones in the shell are actually magic commands, too. The default behavior of IPython is to check commands to see if they are magic, even if they don't have a \"%\" at the beginning, and run the magic command if it finds a match. Be aware.\n",
"\n",
"There are many more magic commands built into IPython, and you also have the ability to add 3rd party magic commands, or create your own. If you are interested in learning more about IPython magic commands:\n",
"\n",
"- IPython tip sheet by Chris Myers of Cornell - [http://pages.physics.cornell.edu/~myers/teaching/ComputationalMethods/python/ipython.html](http://pages.physics.cornell.edu/~myers/teaching/ComputationalMethods/python/ipython.html)\n",
"- IPython magic command documentation - [http://ipython.org/ipython-doc/stable/interactive/reference.html#magic-command-system] (http://ipython.org/ipython-doc/stable/interactive/reference.html#magic-command-system)"
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"An example IPython workflow"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To start:\n",
"- put code in a file.\n",
"- `%run <file_name>` the file.\n",
"- then, interact with the results.\n",
"- make changes as needed and repeat.\n",
"- example: show file for my current work, run it, briefly interact with the variables that result.\n",
"\n",
"Implementing new features over time:\n",
"- sometimes, for simple things (testing API libraries for example), I will go line by line in IPython. As soon as you start looping or defining functions, however, I break it out into a file.\n",
"- Then, once it is working, I take the contents of the file and integrate them into project - function, methods, or objects, depending. \n"
]
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"3. IPython Notebook"
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"What is IPython Notebook?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"IPython Notebook is a little more complicated than the IPython Interpreter.\n",
"\n",
"- It consists of two pieces:\n",
" - an IPython server that runs your python code for you and stores changes to your ipython notebook. In a standard install, this server will run on port 8888.\n",
" - a web app that you run in a web browser that talks to the IPython Notebook server, lets you view and interact with IPython Notebooks.\n",
"- an IPython Notebook is still saved in a text file, but not an easily readable one - JSON structured data.\n",
"- IPython notebook captures all input and output, giving you better logging than %logstart in hte IPython Interpreter.\n",
"- IPython Notebooks are useful for mixing formatted output, text, images, etc. with code and logged output:\n",
" - you can insert pictures, and can include rendered graphics generated by python code.\n",
" - you can even auto-generate tables of contents (though it is a bit of a challenge to install):\n",
" - [https://github.com/minrk/ipython_extensions](https://github.com/minrk/ipython_extensions)"
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"IPython Notebook startup and shutdown"
]
},
{
"cell_type": "heading",
"level": 3,
"metadata": {},
"source": [
"IPython Notebook startup"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Anaconda:\n",
"\n",
"- open the Anaconda Launcher.\n",
"- click on the \"Launch\" button for ipython-notebook. This will:\n",
"\n",
" - open a command/terminal window to run the server component on your local machine.\n",
" - open your default web browser and point it at ipython notebook server home page (a listing of what is in your home directory, by default): [http://localhost:8888/tree](http://localhost:8888/tree)\n",
"\n",
"Terminal:\n",
"\n",
"- alternately, you can open a command terminal and run:\n",
"\n",
" cd <wherever_you_want>\n",
" ipython notebook\n",
"\n",
" This is the same as using the Launcher, except it will start the notebook anchored in the directory your terminal is in when you run the command. It should also launch a browser window to [http://localhost:8888/tree](http://localhost:8888/tree), just like starting IPython Notebook using the Anaconda Launcher. If not, just open a browser window and go to [http://localhost:8888/tree](http://localhost:8888/tree)."
]
},
{
"cell_type": "heading",
"level": 3,
"metadata": {},
"source": [
"IPython Notebook shutdown"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the command/terminal window in which the IPython Notebook server is running (either opened when you Launched ipython-notebook, or the shell in which you ran `ipython notebook`), press `Ctrl-C` twice. This will kill the server part of the IPython notebook, which was running in that window. Your ipython notebook pages in browser windows will not close automatically, but they will no longer work correctly if the server is not listening, so you should close them out once you shut down the server."
]
},
{
"cell_type": "heading",
"level": 3,
"metadata": {},
"source": [
"Monitoring and Troubleshooting the server"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- while ipython notebook is running, you can look in the command/terminal window that opened when you started it to see logging.\n",
"- in this window at startup, the ipython notebook will tell you the profile directory it is using, information on other profiles it loads, and the URL to use to connect to the server."
]
},
{
"cell_type": "heading",
"level": 3,
"metadata": {},
"source": [
"IPython Notebook Basics"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are three basic types of cells in an IPython Notebook:\n",
"\n",
"- **code** - can contain Python commands and/or IPython magic commands, used just as you would a basic IPython Interactive Interpreter. To run the commands in a cell, press the black triangular \"Play\" button in the menu-bar at at the top of the web app. The commands will run and their output will be placed inline after the cell.\n",
" - more information - IPython documentation on \"Running Code\": [http://nbviewer.ipython.org/github/ipython/ipython/blob/master/examples/Notebook/Running%20Code.ipynb](http://nbviewer.ipython.org/github/ipython/ipython/blob/master/examples/Notebook/Running%20Code.ipynb)\n",
"- **markdown** - contains text in the markdown formatting language. The contents will be rendered when you run the cell.\n",
" - more info on markdown cells: [http://nbviewer.ipython.org/github/ipython/ipython/blob/master/examples/Notebook/Working%20With%20Markdown%20Cells.ipynb](http://nbviewer.ipython.org/github/ipython/ipython/blob/master/examples/Notebook/Working%20With%20Markdown%20Cells.ipynb)\n",
" - markdown syntax: [http://daringfireball.net/projects/markdown/syntax](http://daringfireball.net/projects/markdown/syntax)\n",
"- **headers** - headers turn into links when they are run, and they can be used to generate tables of contents.\n",
"\n",
"You can insert cells into a notebook, then move them up and down or copy and paste them. To edit an existing cell, double-click on the cell. To run the code in a cell, or to render the markdown, or to format a header and convert it to a link, click the black \"Play\" triangle button in the IPython Notebook menu bar.\n",
"\n",
"Good beginner documentation:\n",
"\n",
"- IPython Notebook basics: [http://nbviewer.ipython.org/github/ipython/ipython/blob/master/examples/Notebook/Notebook%20Basics.ipynb](http://nbviewer.ipython.org/github/ipython/ipython/blob/master/examples/Notebook/Notebook%20Basics.ipynb)\n",
"- IPython Notebook examples: [http://nbviewer.ipython.org/github/ipython/ipython/tree/master/examples/Notebook/](http://nbviewer.ipython.org/github/ipython/ipython/tree/master/examples/Notebook/)"
]
},
{
"cell_type": "heading",
"level": 3,
"metadata": {},
"source": [
"IPython Notebook Files"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- IPython Notebook files have the file extension of \"`*.ipynb`\".\n",
"- They are saved in their entirety to the file system as JSON (a way of using plain text to record structured data).\n",
"\n",
" - look at what the week 2 IPython Notebook looked like internally - [https://gist.github.com/jonathanmorgan/c1a84e3b42c92662bfd2](https://gist.github.com/jonathanmorgan/c1a84e3b42c92662bfd2).\n",
"\n",
"- These files can be downloaded and interacted with using your local Anaconda-provided IPython Notebook.\n",
"- You can also use the online notebook viewer at http://nbviewer.ipython.org to view IPython Notebooks stored as a github GIST (a single file repository) or placed on a web server, so they can be viewed even if the user doesn't have the IPython server on their local computer.\n",
"\n",
" - when viewing IPython Noteboooks in this way, you can download a copy to your local desktop so you can look at how it is coded, use it as a template, etc.\n",
" - _suggestion:_ Download IPython Notebook files you see and think are useful, in case they either disappear or are subsequently altered beyond recognition.\n",
"\n",
"- Notebooks can also be exported to flat HTML, but these transformations can be a little hard to set up. You'll need pandoc or node.js installed, and even then, you might have to do some relatively substantial troubleshooting. I got it to work on my mac by installing pandoc using homebrew, but I am still working on getting it to work in windows."
]
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"4. The Software Development Process and YOU!"
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Basic steps in software development"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- **project statements, goals, and requirements** - briefly describe the project, including goals for the project, technical and otherwise, and requirements that the project must meet.\n",
"- **high-level design** - outline all the high level parts of a system, the things that have to do work, what they do (described in sentences), and how they interact.\n",
"- **low-level design and pseudocode** - starts to get down into the details of implementation - identify tasks that need to get done, then arrange them in the order you want them done, then in a separate document or graph, illustrate how all the things work together.\n",
"- **incremental implementation and testing** - for each step in your low-level design:\n",
" - create code\n",
" - test\n",
" - integrate into overall project\n",
" - test overall project to make sure you haven\u2019t broken anything from previous steps.\n",
" - repeat\n",
"- **end-to-end testing**\n",
"\n",
"Notes:\n",
"\n",
"- recursive and iterative process - always can (and should) loop back to a previous step as understanding of project changes, to see how new information affects everything. Also be on the lookout for unknowns and keep a running list of questions you need answered to be able to finish the project.\n",
"- everyone has their own process, and there are lots of methodologies.\n",
" - extreme programming\n",
" - agile development\n",
" - etc.\n",
" - many of these are marketed as revolutions, but most are evolutionary, still trains that are dispatched or routed differently on the tracks of the overall process outlined above.\n",
"\n",
"- most important thing is to have a process other than \u201cwork real hard until it is done\u201d.\n",
"\n",
"_More information:_ good wikipedia links for more information on software development methodologies (and to see how easy it is to get lost in the weeds here):\n",
"- [https://en.wikipedia.org/wiki/Software_development_process](https://en.wikipedia.org/wiki/Software_development_process)\n",
"- https://en.wikipedia.org/wiki/Software_development_process](https://en.wikipedia.org/wiki/Software_development_process)\n",
"- [https://en.wikipedia.org/wiki/Agile_software_development](https://en.wikipedia.org/wiki/Agile_software_development)"
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"Our process"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"what we\u2019ll do:\n",
"- a little more basic than full-on software life cycle...\n",
"- project description, goals and requirements\n",
"- high level design that makes a list of technical tasks you'll need to do and includes some pseudo code so you start to capture the order in which you'll have to do the tasks to gather your data.\n",
"- as you discover unknowns, holes in your plan, or things you don't know make to implement, make a note of them in your plan and make a separate list of questions that need to be answered.\n",
"\n",
"When doing pseudocode, there are two basic ways to control the flow of a program you should use:\n",
"\n",
"- **loops** - looping through lists of data or lines in a file.\n",
"- **conditionals** - if <some criteria> do one thing <else> do another thing.\n",
"\n",
"Also, as we design, we'll keep an eye out for tasks that look like they might have been common enough that someone would have created a program, package, or library we can use rather than implementing everything ourselves."
]
},
{
"cell_type": "heading",
"level": 2,
"metadata": {},
"source": [
"In-class Exercise"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Time permitting, we'll pick a partner or two and make a design/plan for the following project:\n",
"\n",
"- **project**: You have a corpus of emails in mbox format (the enron emails - [http://www.cs.cmu.edu/~enron/](http://www.cs.cmu.edu/~enron/)), one folder per account, and you want to parse the emails, database them, look for names and other personal references to make two types of ties - direct communication (to and from) and referred to within email, but not in to or from - then detect valence of each email to examine positive or negative attitudes toward people based on mentions in emails.\n",
"\n",
"Take this project idea, state it briefly, outline the goals and requirements, then design a breakdown of work and high-level logic the data collection will follow. It will likely help to download and look at the contents of the files. For each part of implementation, look for libraries that might be helpful for that step. Also, it is fine if you don't know how to do something. For tasks you don't know how you'd implement, at least capture them, and make a note that you haven't the foggiest how to implement. Having a list of what you don't know is useful in and of itself.\n",
"\n",
"We'll take 15 or 20 minutes to do this, then we'll go over the plan I made for this project and see how our plans compare.\n",
"\n",
"If at this point we're almost out of time, however, we'll just walk a sample design for this project."
]
},
{
"cell_type": "heading",
"level": 1,
"metadata": {},
"source": [
"Homework Exercise 3"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Take your class project idea, state it briefly, outline the goals and requirements for it, then follow the process outlined in class to design a breakdown of work and high-level logic the data collection will follow. For each part of this implementation plan, make a note of tasks you don't know how to do and look for libraries that might be helpful for that step.\n",
"\n",
"To reiterate, it is fine (for now, at least) if you don't know how to do something in your plan. For tasks you don't know how you'd implement, at least capture them, and make a note that you haven't the foggiest how to implement. Having a list of what you don't know is useful in and of itself, will give you a better idea how challenging the idea might be.\n",
"\n",
"Deliverables:\n",
"\n",
"- Capture your design in a Word document and hand it in through CTools."
]
},
{
"cell_type": "raw",
"metadata": {},
"source": [
"%nbtoc"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [],
"language": "python",
"metadata": {},
"outputs": []
}
],
"metadata": {}
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment