IPython Notebook with brief tests for software we'll use in class, to make sure it is installed correctly and working.

class2-test_drive.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:f00657ceb841a344a7aa68c8fa49601d0e3075ec0092f6ad22215075ee173d00"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Test Drive! - 2015/01/15"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Shell"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Open a bash shell.\n",
      "\n",
      "On Windows, you'll open \"Git Shell\", installed with Github for Windows.\n",
      "\n",
      "On Mac, you'll open the \"Terminal\" program.\n",
      "\n",
      "To start, we'll try a few commands to make sure your shell is working OK:\n",
      "\n",
      "- `pwd`\n",
      "- `ls`\n",
      "- `ls -al`\n",
      "- `conda -V`\n",
      "- `pip -V`\n",
      "- `python --version`\n",
      "\n",
      "Next, we'll test out installing an external python package (beautiful soup 4) using conda:\n",
      "\n",
      "- `conda install beautiful-soup`\n",
      "\n",
      "You can also install packages using `pip`:\n",
      "\n",
      "- `pip install beautifulsoup4`\n",
      "\n",
      "conda and pip each install to the same place and play nice with each other, so installing with one won't cause problems if you try to subsequently install with the other (which is convenient, since the two sometimes have different names for the same package, as we've seen).\n",
      "\n",
      "Finally, we'll run a few `conda` commands to make sure that everything is up to date:\n",
      "\n",
      "- `conda update conda`\n",
      "- `conda update ipython`"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "IPython"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Next, we will launch and try out the IPython interactive shell.  To run the interactive shell, you have a few choices:\n",
      "\n",
      "- 1) type `ipython` in your bash shell.\n",
      "- 2) open the Anaconda \"Launcher\" app and run \"QT Console\"."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In the IPython shell, we'll next try out a few basic commands that relate to directories, so we can create a work folder to hold our classwork. These commands (pwd, cd, etc.) work the same as if you'd have run them in a bash shell, though you can't guarantee that all, or even most, shell commands will work in IPython.  We'll look at the current directory, change directories, then make a work directory.  This example makes a work directory \"python_class\" in \"Documents/work\".  You can place your work directory anywhere you want, but make sure you know where it is since you'll need to get back there later.\n",
      "\n",
      "The commands:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pwd"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 2,
       "text": [
        "u'/Users/jonathanmorgan/Dropbox/Documents/work/teaching/UofM/2015-01-Spring/SURV699Y/week2'"
       ]
      }
     ],
     "prompt_number": 2
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "cd"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "/Users/jonathanmorgan\n"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "cd Documents"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "/Users/jonathanmorgan/Documents\n"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "cd work"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "/Users/jonathanmorgan/Documents/work\n"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "mkdir python_class"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "cd python_class"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "/Users/jonathanmorgan/Documents/work/python_class\n"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Next, we'll try a few simple python commands, to make sure python is installed correctly.\n",
      "\n",
      "- First, we will print something to the screen:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print( \"Something to the screen\" )"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Something to the screen\n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Next, we'll declare a variable, then print its contents to the screen."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "test_text = \"Hello!\"\n",
      "print( test_text )"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Hello!\n"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "And then we'll see what simply entering a variable name at the prompt does:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "test_text"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 10,
       "text": [
        "'Hello!'"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Test to see if we installed beautiful soup correctly such that we can import it:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from bs4 import BeautifulSoup"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "No errors = success (more on Beautiful Soup later)!"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Editor"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Next, we try coding in our editor of choice.  In your text editor:\n",
      "\n",
      "- Create a new file.\n",
      "- In that file, enter the following code:\n",
      "\n",
      "        my_message = \"Hello, world!\"\n",
      "        print( my_message )\n",
      "        \n",
      "- Save this file as \"hello_world.py\" in your work directory."
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "IPython again!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now, back in IPython:\n",
      "\n",
      "- cd into your work directory if you aren't already there (check with pwd)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pwd"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 12,
       "text": [
        "u'/Users/jonathanmorgan/Documents/work/python_class'"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "- look to make sure your file is there:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ls"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "hello_world.py\r\n"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "- run your `hello_world.py` file using the `%run` command:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%run hello_world.py"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Hello, world!\n"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "If you see \"Hello, world!\", everything is working correctly (and you now know how to run an external file in IPython - more on that later, as well, but this is a key reason IPython is great)."
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "github.com and git client"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "github.com"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "As this class progresses, we will be storing our code in a git version control repository on github.com.  So, before we do anything else with git, if you don't have a github account, go now and create one:\n",
      "\n",
      "- [http://github.com](http://github.com).\n",
      "- enter a username, your email address, and a password, then click \"Sign up for GitHub\".\n",
      "- there will probably be more instructions.  Follow them.  Write down your password.\n",
      "- Also, consider setting up two-factor authentication and then creating an application password you'll use when you check things in should you do so from the command line.\n",
      "\n",
      "Once you have a github.com account, log in at [http://github.com](http://github.com).\n",
      "\n",
      "We will now create a repository in which you will be versioning the code you hand in for this class (and potentially other documents where being able to see changes over time might be valuable - data output files, perhaps, or text documents).  \n",
      "\n",
      "- _Aside on versioning:_ File versioning keeps track of multiple states of files over time, so you can see what you changed when and potentially revert to an older state if you really screw something up and don't know how to fix it.  We'll talk more about versioning soon, but for now, just trust me that you should seriously consider versioning absoutely everything you do, and, at the least, your code files.\n",
      "\n",
      "To create a new public repository on github follow the instructions here (including screenshots and stuff):\n",
      "\n",
      "- [https://help.github.com/articles/create-a-repo/](https://help.github.com/articles/create-a-repo/)\n",
      "\n",
      "In summary (no screenshots - sorry):\n",
      "\n",
      "- log in to [http://github.com](http://github.com)\n",
      "- in the upper right corner, click the plus sign and choose \"New Repository\".\n",
      "- Enter a name and a description.\n",
      "- Make the repository public (yup, get used to it - open access science - well, and free versus having to pay for private repositories).\n",
      "- select \"Initialize this repository with a README\".\n",
      "- click \"Create Repository\"\n",
      "- email Cliff, Frauke, and Jon the URL of your new repository."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "GitHub GUI client"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now, open up your GUI github client (It will just be named \"GitHub\", regardless of OS).\n",
      "\n",
      "In the github client:\n",
      "\n",
      "- log into your github account inside the GitHub client.\n",
      "    - _Windows_ - click the little gear icon in the upper right corner of the window, then login in the settings.\n",
      "    - _Mac_ - in the menu bar, click \"GitHub\"-->\"Preferences\", then in the preferences window, go to the \"Accounts\" tab and log in there.\n",
      "- click on the plus sign (\"+\") in the upper left corner of the app.\n",
      "- In the little sub-window that opens subsequently, click \"Clone\" at the top.  This will pull up a list of all of your repositories (just the one at this point, unless you are already a github user and have more than one).\n",
      "- select your repository, then click \"Clone <your_repo_name>\".\n",
      "- this will bring up a file save dialog box appropriate to your operating system.  Choose the folder where you want the local copy of your repository to live, then click the \"Clone\" button.\n",
      "\n",
      "    - _Note:_ I'd just use the work folder you created above for now.  You can always move it later.\n",
      "    \n",
      "A copy of your repository is now on your computer, in the folder you specified.  The repository is a folder named the same as you named your repository on github.  Inside, you'll find a file README.md (the readme file you told it to make when you created the repository).\n",
      "\n",
      "This repository is also now known to the GUI git client, should you choose to continue to use it (it provides much more polite views of history and changes to files.  I use the command line for most things, but I use the prettiest, easiest GUI I can find for viewing history and diffs)."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Committing and pushing changes"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now, we'll make some changes, add a file (your \"Hello, world!\" program), and commit, then push your changes.\n",
      "\n",
      "We'll talk more about git and versioning in a future class, but here's a brief overview of how git works:\n",
      "\n",
      "- In git, you have a local copy of your repository in which you make changes, tell git that you are ready to commit the changes (using the \"add\" command), then commit them.\n",
      "- Adding and committing a change adds it to the official change log for the file.  This changelog keeps track of all changes you make, lets you look at what you changed, and lets you roll back to a previous version if you need to.\n",
      "- With an online repository site like github, you also have a remote version of your repository that you can use to keep a backup, to clone your repository to multiple computers and keep them all in sync, or to allow multiple people to clone and work on together on a project.\n",
      "- The local and remote repos are connected, but they don't automatically update each other.  Changes added and committed locally don't automatically make their way up to the remote repository, and changes pushed from another computer or by another user don't automatically make their way down to your local copy of the repository.  In order to retrieve changes on the server that aren't on your computer, you use a command called \"pull\" to \"pull\" the changes down.  In order to get commits from your server added to the remote repository, you have to \"push\" your changes.\n",
      "\n",
      "So, to make sure git is working on our machines, we'll update our repository's README.md file and add our hello_world.py file, commit our changes, then push them up to github.  To do this:\n",
      "\n",
      "- In your text editor, go to your repository's folder on your computer and open up the file README.md for editing.\n",
      "- Inside, your repository's name and description will be at the top of the file, in Markdown (more on this later if you are interested).  Go down a line or two past the description and add something more profound to the page.  To start, it can be very brief.  Like, a sentence.  I don't care what, but it is public, so make it reasonably professional, witty if you can, perhaps with some panache.  Or not.  Doesn't matter.  Just add something, then save.\n",
      "\n",
      "Then, in a file browser or shell, move your \"hello_world.py\" file from wherever it is now into the root of your repository.\n",
      "\n",
      "Now, we'll commit and push our changes to github using the GitHub GUI:\n",
      "\n",
      "- first, go back to the GitHub GUI program.\n",
      "- click on your repository on the left.\n",
      "- Look at your changes:\n",
      "\n",
      "    - _In the Mac client,_ click the \"Changes\" tab at the top to pull up your changes.\n",
      "    - _In the Windows client,_ a message \"Uncommitted changes\" will appear at the top of the second column from the left when you've made changes.  Click the \"Show\" link to show them.\n",
      "    \n",
      "- commit your changes to master (actually adds, then commits - we'll do this from command line soon, so you can see what happens behind the scenes).  The applications look a bit different for Windows and Mac, but the steps are the same:\n",
      "\n",
      "    - make sure the checkboxes are checked next to all the changed files (should just be README.md and hello_world.py).\n",
      "    - enter a commit message summary and description (and make sure they are accurate - no \"changes\" and \"mades some changes\".  Describe what you did: \"filled in README and added hello_world.py\"; \"added something amazing to README.md, cuz it looked sort of pitiful with just the repository name and description, and added my first program of this class, hello world.\"\n",
      "    - commit\n",
      "\n",
      "        - _Mac:_ click \"Commit\".\n",
      "        - _Windows:_ click \"Commit to master\".\n",
      "\n",
      "    - sync (pull, then push) the changes to github.\n",
      "    \n",
      "        - Click the \"Sync\" button in the upper right corner of the application.\n",
      "        - _Note for later:_ this will pull, then push.  You will always pull, then push - \"pull\" to get changes made since last you retrieved changes, so you can deal with conflicts locally, then \"push\", to place changes and resolutions to any conflicts in the repository on the server.\n",
      "        \n",
      "- go to the github.com page for your repository and check it out!  Your hello_world.py file should now be in the list of files for the repository, and your README changes should appear on the front page as well, below the list of files."
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "SQLite"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "And, finally, we'll briefly put the database we'll all start with through its paces.  To do this, we'll open the SQLite Manager inside Firefox, create a database, create a table, place a few rows into the table, then try a few queries.\n",
      "\n",
      "For today, you'll just be copying SQL from this page and pasting it.  You don't need yet to know anything about SQL.  That is coming, however, along with options for avoiding it should you choose.\n",
      "\n",
      "- _Aside_ - I personally am competent at SQL, but find crafting and maintaining complex SQL to be like giving yourself root canals, over and over, while also periodically banging your foot with a sledge hammer - doable for a time, but thoroughly unpleasant.  I prefer to use SQL to read and write data, then put any logic for processing that data in whatever programming language I'm using - python, R, Java, whatever - rather than having logic in SQL.  There is an element of personal preference here, however, and also assessment of what any collaborators are comfortable with.  Your mileage may vary.\n",
      "\n",
      "So, let's do some SQLite!"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Create a database"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "- open firefox\n",
      "- open SQLite Manager\n",
      "- Open the \"New Database\" dialogue:\n",
      "\n",
      "    - _In Windows,_ in the menu at the top of the SQLite Manager window, go to \"Database\" --> \"New Database\".\n",
      "    - _In Mac,_ the menu items are up in the menu bar for \"Firefox\", but only when the SQLite Manager window is the active Window.\n",
      "    \n",
      "- enter a database name (\"test\" is fine, for now).\n",
      "\n",
      "    - _Note:_ it is trivial to rename a database, or just make a new one - each database is just a single file, and there isn't anything fancy about them..\n",
      "\n",
      "- In the file dialogue box that subsequently opens, choose to save the database in your work directory."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Make a table"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now, we'll create a table, insert some data, then query, to make sure we can make a database.\n",
      "\n",
      "- click the \"Execute SQL\" tab.\n",
      "- The Execute SQL window is broken into three parts:\n",
      "\n",
      "    - the SQL entry frame, at the top.\n",
      "    - buttons and a space for error messages.\n",
      "    - a larger frame to hold output of the latest query.\n",
      "\n",
      "- In the SQL entry frame, enter the SQL to create a simple table:\n",
      "\n",
      "        CREATE  TABLE \"main\".\"test_table\" (\"id\" INTEGER PRIMARY KEY  AUTOINCREMENT  NOT NULL  UNIQUE , \"name\" VARCHAR, \"value\" VARCHAR);\n",
      "\n",
      "- Then, one INSERT at a time, paste and run the SQL INSERT statements below to add some data to your table:\n",
      "\n",
      "        INSERT INTO \"main\".\"test_table\" ( name, value ) VALUES ( 'name1', 'value1' );\n",
      "        INSERT INTO \"main\".\"test_table\" ( name, value ) VALUES ( 'name2', 'value2' );\n",
      "        INSERT INTO \"main\".\"test_table\" ( name, value ) VALUES ( 'name10', 'value10' );\n",
      "\n",
      "- Now, one at a time, do a couple of simple queries:\n",
      "        \n",
      "        SELECT * FROM \"main\".\"test_table\" WHERE name = 'name2';\n",
      "        SELECT * FROM \"main\".\"test_table\" WHERE name LIKE 'name1%';\n",
      "\n",
      "If all that worked, then the software on your computer is set up appropriately.  Now to start using it..."
     ]
    }
   ],
   "metadata": {}
  }
 ]
}