PySparkSklearn.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:d719decce772384d9fcaec86da411b55dd346af1d89327269c5709ddda6d2961"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Run pyspark in a notebook\n",
      "\n",
      "According to the Apache Spark [documentation](https://spark.apache.org/docs/latest/programming-guide.html), you can do this:\n",
      "\n",
      "```\n",
      "IPYTHON_OPTS=\"notebook --pylab inline\" ./bin/pyspark\n",
      "```\n",
      "\n",
      "What follows supposes you have installed [scikit-learn](http://scikit-learn.org) and assembled Spark. Let's see if we can run some ML algorithm from the Scikit-Learn library on top of a Spark cluster (directly, without the need to **copy and paste everything**)."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Check if Spark is okay"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "txtFile = \"/Users/botta/Downloads/vocab.nytimes.txt\"\n",
      "words = sc.textFile(txtFile)\n",
      "words.filter(lambda w: w.startswith(\"spa\")).take(10)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 6,
       "text": [
        "[u'spa',\n",
        " u'space',\n",
        " u'spacecraft',\n",
        " u'spaced',\n",
        " u'spaceflight',\n",
        " u'spaceport',\n",
        " u'spacer',\n",
        " u'spaces',\n",
        " u'spaceship',\n",
        " u'spacesuit']"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Now let's play with Scikit-Learn"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from sklearn.datasets import make_classification\n",
      "from sklearn.ensemble import ExtraTreesClassifier\n",
      "\n",
      "# Build a classification task using 3 informative features\n",
      "X, y = make_classification(n_samples=3000,\n",
      "                           n_features=10,\n",
      "                           n_informative=3,\n",
      "                           n_redundant=0,\n",
      "                           n_repeated=0,\n",
      "                           n_classes=2,\n",
      "                           random_state=0,\n",
      "                           shuffle=False)\n",
      "\n",
      "# Build a forest and compute the feature importances\n",
      "forest = ExtraTreesClassifier(n_estimators=1000,\n",
      "                              random_state=0)\n",
      "\n",
      "forest.fit(X, y)\n",
      "importances = forest.feature_importances_\n",
      "std = np.std([tree.feature_importances_ for tree in forest.estimators_],\n",
      "             axis=0)\n",
      "indices = np.argsort(importances)[::-1]\n",
      "\n",
      "# Print the feature ranking\n",
      "print(\"Feature ranking:\")\n",
      "\n",
      "for f in range(10):\n",
      "    print(\"%d. feature %d (%f)\" % (f + 1, indices[f], importances[indices[f]]))\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Feature ranking:\n",
        "1. feature 1 (0.308214)\n",
        "2. feature 0 (0.296262)\n",
        "3. feature 2 (0.146073)\n",
        "4. feature 9 (0.036513)\n",
        "5. feature 5 (0.036396)\n",
        "6. feature 6 (0.035698)\n",
        "7. feature 3 (0.035383)\n",
        "8. feature 8 (0.035337)\n",
        "9. feature 7 (0.035248)\n",
        "10. feature 4 (0.034876)\n"
       ]
      }
     ],
     "prompt_number": 52
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Plot the feature importances of the forest\n",
      "plt.figure()\n",
      "plt.title(\"Feature importances\")\n",
      "plt.bar(range(10), importances[indices],\n",
      "       color=\"r\", yerr=std[indices], align=\"center\")\n",
      "plt.xticks(range(10), indices)\n",
      "plt.xlim([-1, 10])\n",
      "plt.show()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEKCAYAAAAGvn7fAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGeZJREFUeJzt3X+QHPV95vH3wwoBEgQw5octhOUCxUDFXCCOohgDY6xQ\nC/4hX1w5WWeHGHxYdzkZLrZTRKmUWZ+rzkXqODsJF47EsotgHF2MgZPrwAhChlA2AeTwK0HiEKBD\nEhgLYWTA4SxZz/0xvTAMs/Njd3Zmt+d5VU2pe7q//fn2aveZ3m93b8s2ERFRHvsNugMREdFbCfaI\niJJJsEdElEyCPSKiZBLsERElk2CPiCiZBHsMFUlrJP3loPsRMZ2U69ijU5K2AkcBPy/eMvCLtn84\nxW1eaPuOKXdwlpE0Bhxv+7cH3ZcolzmD7kDMKgY+0OMQNqDJNpY0Yvvn7decWSTlZy+mTYZiYsok\nHSppraSnJW2X9EVJ+xXLjpd0h6TnJO2U9A1JhxbLrgWOA74j6UVJn5NUkbStYftbJZ1dTI9Jul7S\ntZJ2A7/Tqn6Tvo4VdZG0SNI+SZ+Q9JSkXZL+vaRflfSQpB9L+rO6tp+Q9D1JfybpBUmbxvtVLH+r\npPXFdh6T9O8a6tb3exWwBlhR7Pv9xXoXSHpE0k8kPS7pU3XbqBT79xlJzxb7+4m65QdJuqL4er0g\n6S5JBxbLlkr6frFPD0g6q2G/Hi9qPiHp33b5LRAzje288uroBTwJvK/J+zcCVwEHAUcC9wCfKpYd\nD7wP2B94M3An8OWGbZ5dN18BtjWpe3YxPQb8DPhQMX9gq/pN+noZcG0xvQjYB/w5MBf4DeD/Fdt7\nM/BW4FngzGL9TwB7gEuAEeDfAC8AhxXL/x64stjWvwJ+BLy3Rb8vA/6qoX/nAW8vps8EXgZOrfva\n7Cm2NQKcWyw/tFj+34E7gLdQO2hbWvRlAfAcMFqst6yYPwKYD+wGFhfLjgZOHvT3Wl5Te+WIPboh\n4KbiqO/Hkm6QdDS1gPk92/9ieyfwFeCjALYft/23tvfYfg74MnDWhBU6833b64vpQ1vVn2AfGn3R\n9s9s3wa8CHzT9nO2nwbuAk6tW/dHtv/E9s9t/w3wKPABSQuBdwOXFtt6EPgqcH6zftt+pejL6/pj\n+2bbTxbTfw9sAM6oW2UP8J+L+rcALwHvKH5DuQC4xPYztvfZ/gfbPwM+Dtxs+7vFdm8HNgLvpzYU\ntg94p6SDbD9r+5EJvnYxS2ScL7phYLnrxtglLaF2NP6M9GpG7Qc8VSw/GvgT4D3AIcWy56fYj+11\n029rVb9Dz9ZN/0uT+fl18zsa2v5fakfIbwGet/1y3bKngHdN0O+mJJ1L7Uh+MbX9mAc8VLfKLtv7\n6uZ/ChxM7TeMA4HHm2z2bcBvSfpg3XtzgDts/1TSCuBzwFpJ3wM+a/vRdn2NmStH7DFV26gNXxxh\n+/DidajtdxbL/wu1q2h+yfahwG/z+u+7xsuyXqYWZkDt5Ci14ZV69W3a1W801cvAFjTMvw14uni9\nSdLBdcuO4/Vh3li7PqCRdADwbeCPgaNsHw7cTGcnl58DXgFOaLLsKWrDT4fXvQ6x/ccAtjfYPgc4\nBtgM5HLQWS7BHlNi+xlqwwX/TdIhkvYrTpieWaxyMLWw/omkBcDvN2ziWWrj8OP+D3CgpPMk7Q/8\nEXDAFOo3mswVOPVtjpJ0saT9Jf0WcCK1YY7twPeBL0k6QNIpwIXAN1ps91lgkV77VWNu8XoO2Fcc\nvZ/TSQeLo/ivUfs6vEXSiKRflzS36MMHJZ1TvH9gcSJ2gaSjJC2XNJ/aMM/LvHY5a8xSCfbohfOp\nBdIj1IZZvkXt6A/gC8Bp1E7QfYfaEWn9keuXgD8qxuw/Y3s38LvUxqe3UxtDrr9KxrzxyLdV/UaN\n7Ts5gq9f5x5qwyQ7gS8CH7H942LZSmonZJ8GbgA+Xzds1azf3yr+3SVpo+0XgYuBvyn2YyXwv1r0\npdHngIeB+4Bd1L62+xUfOsuBP6R2Qvcp4LPUPrD2A36P2hDTLmrj+f+hRY2YBdreoCRplNrJqBHg\nq7Yvn2C9XwXuBlbY/nY3bSNmg+LSwk/aPqPduhGD1PKIvRjfvBIYBU4GVko6aYL1Lge+223biIjo\nrXZDMUuALba32t4DrKP2K12jTwPXU/v1tNu2EbNFs+GUiBmnXbAv4PXjm9tpuCqgOCG2nNoNIvDa\nN37bthGzie1rbE90UjZixmgX7J0cnXwF+APXBuvrb7jIkU1ExAC0u0FpB7Cwbn4hb7zJ4leAdcUV\nW28GzpW0p8O2SMoHQETEJNhufvluq783QC34H6d2Cddc4AHgpBbrfx34zW7a1rowGJdddtlQ1R1k\n7WGrO8ja2efhqF1kZ9MsbnnEbnuvpNXArdQuWVxre5OkVcXyq7tt2/YjKCIipqTt34px7Q8N3dLw\nXtNAt31Bu7YRETG9hvrO00qlMlR1B1l72OoOsnb2eXhqT2Tgj8aT5EH3ISJitpE04cnToT5ij4go\nowR7RETJJNgjIkomwR4RUTJD92i8arX2Gp8eP6Fdqbw2HRExmw31VTES5IKciJiNclVMRMQQSbBH\nRJRMgj0iomQS7BERJZNgj4gomQR7RETJJNgjIkomwR4RUTJDd+fpoOSO14jol9x5OoDSueM1IqYq\nd55GRAyRtsEuaVTSZkmPSbq0yfLlkh6UdL+kH0g6u27ZVkkPFcvu7XXnIyLijVoOxUgaAR4FlgE7\ngPuAlbY31a0z3/bLxfQ7gRttn1DMPwn8iu3nW9TIUExERJemMhSzBNhie6vtPcA6YHn9CuOhXjgY\neK6xfpf9jYiIKWgX7AuAbXXz24v3XkfShyVtAm4BLq5bZOB2SRslXTTVzkZERHvtLnfsaMDA9k3A\nTZLOAK4F3lEsOt32M5KOBG6TtNn2XZPvbkREtNMu2HcAC+vmF1I7am/K9l2S5kg6wvYu288U7++U\ndCO1oZ03BPvY2Nir05VKhUou7I6IeJ1qtUp1/GaYNtqdPJ1D7eTp+4CngXt548nT44EnbFvSacC3\nbB8vaR4wYvtFSfOBDcAXbG9oqJGTpxERXWp18rTlEbvtvZJWA7cCI8Ba25skrSqWXw18BDhf0h7g\nJeCjRfNjgBskjde5rjHUIyKi93LnaY7YI2IWyp2nERFDJMEeEVEyCfaIiJJJsEdElEyCPSKiZBLs\nERElk2CPiCiZBHtERMkk2CMiSibBHhFRMgn2iIiSSbBHRJRMgj0iomQS7BERJZNgj4gomQR7RETJ\nJNgjIkomwR4RUTIJ9oiIkmkb7JJGJW2W9JikS5ssXy7pQUn3S/qBpLM7bdtLkrp+TaZdRMRM1/Jh\n1pJGgEeBZcAO4D5gpe1NdevMt/1yMf1O4EbbJ3TStmjTk4dZS6LbrQhjOg9rAb3pax5mHRFTM5WH\nWS8BttjeansPsA5YXr/CeKgXDgae67RtRET0XrtgXwBsq5vfXrz3OpI+LGkTcAtwcTdtIyKit+a0\nWd7RgIHtm4CbJJ0BXCvpxG46MTY29up0pVKhUql00zwiovSq1SrVarWjdduNsS8FxmyPFvNrgH22\nL2/R5nFqwzCLO2mbMfaIiO61GmNvd8S+EVgsaRHwNLACWNmw8eOBJ2xb0mkAtndJ2t2ubVl0f7WM\nu27Tiw+UiBgOLYPd9l5Jq4FbgRFgre1NklYVy68GPgKcL2kP8BLw0VZtp29XBqub2NUk1o+I6FTL\noZi+dKAEQzHd1h7UEFBElMdULneMiIhZJsEeEVEyCfaIiJJJsEdElEyCPSKiZBLsERElk2CPiCiZ\nBHtERMkk2CMiSibBHhFRMgn2iIiSSbBHRJRMgj0iomQS7BERJZNgj4gomQR7RETJJNgjIkomwR4R\nUTIJ9oiIkmkb7JJGJW2W9JikS5ss/5ikByU9JOl7kk6pW7a1eP9+Sff2uvMREfFGc1otlDQCXAks\nA3YA90lab3tT3WpPAGfa3i1pFPgLYGmxzEDF9vO973pERDTT7oh9CbDF9lbbe4B1wPL6FWzfbXt3\nMXsPcGzDNpo+RTsiIqZHu2BfAGyrm99evDeRTwI3180buF3SRkkXTa6LERHRjZZDMdSCuSOS3gtc\nCJxe9/bptp+RdCRwm6TNtu9qbDs2NvbqdKVSoVKpdFo2ImIoVKtVqtVqR+vKnji7JS0FxmyPFvNr\ngH22L29Y7xTgBmDU9pYJtnUZ8JLtKxred6s+dEpS559C420w7mKkSECzvnZbu1d1I2J4ScJ20yBp\nNxSzEVgsaZGkucAKYH3Dxo+jFuofrw91SfMkHVJMzwfOAR6e/G5EREQnWg7F2N4raTVwKzACrLW9\nSdKqYvnVwOeBw4GrJAHssb0EOAa4oXhvDnCd7Q3TticREQG0GYrpSwcyFDPpuhExvKYyFBMREbNM\ngj0iomQS7BERJZNgj4gomaE7eVrlLKpUiukKFaoAVKhS4c7WNcjJ04iYGVqdPB26YJ9SDRLsETEz\n5KqYiIghkiP2bmow+SP26RgCiojhlaGYHunVUEyv6kbE8MpQTETEEEmwR0SUTII9IqJkEuwRESWT\nYI+IKJkEe0REySTYIyJKJsEeEVEyCfaIiJJJsEdElEzbYJc0KmmzpMckXdpk+cckPSjpIUnfk3RK\np20jIqL3Wv6tGEkjwKPAMmAHcB+w0vamunV+HXjE9m5Jo8CY7aWdtC3a52/FTLJuRAyvqfytmCXA\nFttbbe8B1gHL61ewfbft3cXsPcCxnbaNiIjeaxfsC4BtdfPbi/cm8kng5km2jYiIHpjTZnnHv/9L\nei9wIXB6t23HxsZena5UKlQqlU6bRkQMhWq1SrVa7WjddmPsS6mNmY8W82uAfbYvb1jvFOAGYNT2\nli7bZox9knUjYnhNZYx9I7BY0iJJc4EVwPqGjR9HLdQ/Ph7qnbaNiIjeazkUY3uvpNXArcAIsNb2\nJkmriuVXA58HDgeukgSwx/aSidpO475ERAR5NF53NchQTETMDHk0XkTEEEmwR0SUTLvLHaMEqtXa\na3x6/GrSSuW16Ygoj4yxd1OD2T/GLkGG6yNmv4yxR0QMkQR7RETJJNgjIkomwR4RUTIJ9oiIkkmw\nR0SUTII9IqJkEuwRESWTYI+IKJkEe0REySTYIyJKJsEeEVEyCfaIiJJJsEdElEyCPSKiZNoGu6RR\nSZslPSbp0ibLT5R0t6RXJH22YdlWSQ9Jul/Svb3seERENNfyCUqSRoArgWXADuA+Settb6pbbRfw\naeDDTTZhoGL7+R71NyIi2mh3xL4E2GJ7q+09wDpgef0Ktnfa3gjsmWAbTZ/wERER06NdsC8AttXN\nby/e65SB2yVtlHRRt52LiIjutXuY9VSfjnm67WckHQncJmmz7bsaVxobG3t1ulKpUMkTliMiXqda\nrVIdfyp9Gy0fZi1pKTBme7SYXwPss315k3UvA16yfcUE22q6PA+znnzdSW0rD7OOKIWpPMx6I7BY\n0iJJc4EVwPqJ6jQUnSfpkGJ6PnAO8HBXPY+WJHX9mky7iJhdWg7F2N4raTVwKzACrLW9SdKqYvnV\nko4B7gN+Adgn6RLgZOAo4IYiGOYA19neMH27Mpy6PfhWl20S6xGzT8uhmL50IEMxfa0rjLuI614O\nA0VE70xlKCYiImaZBHtERMkk2CMiSibBHhFRMgn2iIiSSbBHRJRMgj0iomQS7BERJZNgj4gomQR7\nRETJJNgjIkomwR4RUTIJ9oiIkkmwR0SUTII9IqJkEuwRESWTYI+IKJkEe0REySTYIyJKpm2wSxqV\ntFnSY5IubbL8REl3S3pF0me7aRsREb3XMtgljQBXAqPAycBKSSc1rLYL+DTwXyfRNiIiemxOm+VL\ngC22twJIWgcsBzaNr2B7J7BT0vu7bRv9UeUsqlQAOIsqY1wGQIUqFe4cYM8iYjq0C/YFwLa6+e3A\nr3W47am0jR6qcGddgH9hoH2JiOnXLtg9hW133HZsbOzV6UqlQqVSmULZiIjyqVarVKvVjtaVPXH+\nSloKjNkeLebXAPtsX95k3cuAl2xf0U1bSW7Vh05JmtKnUEc1gGZ9ne7ag6rbqnZEDJYkbKvZsnZX\nxWwEFktaJGkusAJYP1GdKbSNiIgeaTkUY3uvpNXArcAIsNb2JkmriuVXSzoGuA/4BWCfpEuAk22/\n1KztdO5MRES0GYrpSwcyFDNj67aqHRGDNZWhmIiImGUS7BERJZNgj4gomQR7RETJJNgjIkomwR4R\nUTIJ9oiIkkmwR0SUTII9IqJkEuwRESWTYI+IKJkEe0REySTYIyJKJsEeEVEyCfaIiJJJsEdElEyC\nPSKiZBLsEREl0zbYJY1K2izpMUmXTrDOnxbLH5R0at37WyU9JOl+Sff2suMREdFcy4dZSxoBrgSW\nATuA+yStr38otaTzgBNsL5b0a8BVwNJisYGK7eenpfcREfEG7Y7YlwBbbG+1vQdYByxvWOdDwDUA\ntu8BDpN0dN3ypg9bjYiI6dEu2BcA2+rmtxfvdbqOgdslbZR00VQ6GhERnWk5FEMtmDsx0VH5e2w/\nLelI4DZJm23f1Xn3IiKiW+2CfQewsG5+IbUj8lbrHFu8h+2ni393SrqR2tDOG4J9bGzs1elKpUKl\nUumo8xERw6JarVKtVjtaV/bEB+WS5gCPAu8DngbuBVY2OXm62vZ5kpYCX7G9VNI8YMT2i5LmAxuA\nL9je0FDDrfrQKUkd/3ox6RpAs75Od+1B1W1VOyIGSxK2m46WtDxit71X0mrgVmAEWGt7k6RVxfKr\nbd8s6TxJW4CXgQuK5scAN0gar3NdY6hHRETvtTxi70sHcsQ+Y+u2qj2TVau11/j0+MhepfLadMRs\n1+qIPcHeTQ0S7N2YCQErQT+/xQe1z4P8Wg/jPs8ECfYeSbBPYTt9DthB1x1k7ezz9JsJHyoJ9h5J\nsE9hO0PyAz8Tamefh6P2pE+eRkykOCneBXfdZqIPs+muO1HtiNkiwR6T1k30aRLrD6Juq9pl/jCb\n6INs2PZ5MgcBvfx690qCPaILZfwwaxdJw7bPkzkQ6OXXuxfy99gjIkomJ0+7qUFOnnZTu8pZVKkU\n0xUqVAGoUKXCndNW9/XbMe7yGGlQtWda3UHWni11e1m7W7kqpkcS7P2rPagPlEHWnml1B1l7Jted\nrtrdSrD3yGwMudlaO/s8+LqDrF3Wuq1qd72dFsGeMfaIiJJJsEdElEyCPSKiZBLsERElk2CPiCiZ\nBHtERMkk2CMiSibBHhFRMgn2iIiSaRvskkYlbZb0mKRLJ1jnT4vlD0o6tZu2ERHRWy2DXdIIcCUw\nCpwMrJR0UsM65wEn2F4MfAq4qtO2g1YdsrqDrD1sdQdZe1B1B1l7UHUHXXsi7Y7YlwBbbG+1vQdY\nByxvWOdDwDUAtu8BDpN0TIdtB6o6ZHUHWXvY6g6y9qDqDrL2oOoOuvZE2gX7AmBb3fz24r1O1nlr\nB20jIqLH2gV7p3+CrB8PBYmIiA60/LO9kpYCY7ZHi/k1wD7bl9et8z+Aqu11xfxm4Czg7e3aFu/n\nqcEREZMw0Z/tbffM043AYkmLgKeBFcDKhnXWA6uBdcUHwQu2n5W0q4O2E3YsIiImp2Ww294raTVw\nKzACrLW9SdKqYvnVtm+WdJ6kLcDLwAWt2k7nzkRExAx4glJERPTW0N15Kulrkp6V9PCA6vf9pi1J\nCyX9naR/lvRPki7uR92i9iWSHi7qXtKvukXtrZIeknS/pHv7WPcwSddL2iTpkWKIsh91D5R0j6QH\nirpf6kfdovaa4vvrYUnflHRAn+q+o/j/HX/t7vP390hR9zv9qtkR20P1As4ATgUeHkDtEWALsAjY\nH3gAOKkPdY8BfrmYPhh4tE91fwl4GDiw2PfbgOP7+PV+EnjTAP6frwEuLKbnAIf2sfa8urr/ALyn\nDzUXAU8ABxTz/xP4nQF83fcDngEW9rHmZ4DrgPX93t9Wr6E7Yrd9F/DjAZUfyE1btn9o+4Fi+iVg\nE7X7DKbbicA9tl+x/XPgTuA3+1C3Xl9Pzks6FDjD9tegdq7J9u5+1bf902JyLrUP0+f7UPYnwB5g\nnqQ5wDxgRx/qNloGPG57W9s1e0DSscB5wFeZYZd8D12wD1gnN3xNq+IqpVOBe/pQ7p+AMyS9SdI8\n4P3AsX2oO87A7ZI2SrqoTzXfDuyU9HVJ/yjpL4t97wtJ+0l6AHgW+Dvbj0x3TdvPA1cAT1G7Au4F\n27dPd90mPgp8s4/1vgz8PrCvjzU7kmDvr4GeqZZ0MHA9cElx5D6tbG8GLgc2ALcA99PfH4LTbZ8K\nnAv8R0ln9KHmHOA04M9tn0btSrE/6ENdAGzvs/3L1D5Az5RUme6ako4H/hO1IZm3AgdL+th0123o\nw1zgg8C3+lTvA8CPbN/PDDtahwR7v+0AFtbNL6R21D7tJO0PfBv4hu2b+lETwPbXbL/L9lnAC9TG\n9/tV+5ni353AjdSGwqbbdmC77fuK+eupBX1fFcM//xt4Vx/KvQv4vu1dtvcCNwDv7kPdeucCPyj+\nr/vh3cCHJD0J/DVwtqS/6lPtthLs/fXqDV/FEcYKajd4TStJAtYCj9j+ynTXa6h9VPHvccC/pk+/\nKkuaJ+mQYno+cA61E7nTyvYPgW2SfrF4axnwz9NdF0DSmyUdVkwfBPwGtd+SpttmYKmkg4rvtWXA\ntA8BNVhJLWD7wvYf2l5o++3UhoDusH1+v+q30+7O09KR9NfU/uTBEZK2AZ+3/fV+1Pbgbto6Hfg4\n8JCk8R/0Nba/24fa10s6gtrJtd+1/ZM+1AQ4GrixljPMAa6zvaFPtT8NXFd8eD9OcdNeH7wFuEbS\nftQO2q61/bfTXdT2g8XR6kZqQ23/CPzFdNcdV3xwLwP6dR6lmRl1Q1BuUIqIKJkMxURElEyCPSKi\nZBLsERElk2CPiCiZBHtERMkk2CMiSibBHhFRMgn2iIiS+f+ddsM4xr6aPAAAAABJRU5ErkJggg==\n",
       "text": [
        "<matplotlib.figure.Figure at 0x111efe050>"
       ]
      }
     ],
     "prompt_number": 55
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "### What about a \"Sparkization\""
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "rddX = sc.parallelize(X)\n",
      "rddX.take(2)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 15,
       "text": [
        "[array([ 1.48006984, -1.35738004,  0.46249366,  0.74380015,  0.59214909,\n",
        "        -0.78646842, -1.17647315, -1.28080671,  1.66165186, -0.06794512]),\n",
        " array([-0.55838227,  0.24923089, -0.19986189,  2.36022859,  0.5555456 ,\n",
        "         0.43952232,  0.30627249,  0.99914985, -0.96606319,  2.16001311])]"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "So you can turn a Python Numpy array into an RDD. That's nice, you can play with your data with PySpark as well!!! But how can we apply some magic voodoo to it?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      " # Partition data\n",
      "def dataPart(X, y, start, stop): return dict(X=X[start:stop, :], y=y[start:stop])\n",
      "\n",
      "# We will do this as if we have 3 files containing data and we need to parallize them\n",
      "data = [dataPart(X, y, 0, 1000), dataPart(X,y,1000,2000), dataPart(X,y,2000,3000)]\n",
      "\n",
      "from sklearn.svm import LinearSVC\n",
      "def train(data):\n",
      "    X = data['X']\n",
      "    y = data['y']\n",
      "    #return ExtraTreesClassifier(n_estimators=100,random_state=0).fit(X,y)\n",
      "    return LinearSVC('l2').fit(X, y)\n",
      "\n",
      "# Merge 2 Models\n",
      "from sklearn.base import copy\n",
      "def merge(left,right):\n",
      "    new = copy.deepcopy(left)\n",
      "    new.coef_ += right.coef_\n",
      "    new.intercept_ += right.intercept_\n",
      "    #new.feature_importances_ += left.feature_importances_    \n",
      "    return new\n",
      "\n",
      "clf = sc.parallelize(data).map(train).reduce(merge)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 53
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "clf.coef_"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 54,
       "text": [
        "array([[ 0.93477661,  2.37358951,  0.82155397, -0.12406194,  0.09275819,\n",
        "        -0.09508289,  0.02561639,  0.1018192 , -0.04830787, -0.13229071]])"
       ]
      }
     ],
     "prompt_number": 54
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "importances = clf.coef_[0]\n",
      "indices = np.argsort(importances)[::-1]\n",
      "\n",
      "# Print the feature ranking\n",
      "print(\"Feature ranking:\")\n",
      "\n",
      "for f in range(10):\n",
      "    print(\"%d. feature %d (%f)\" % (f + 1, indices[f], importances[indices[f]]))\n",
      "\n",
      "# Plot the weights assigned to the features\n",
      "plt.figure()\n",
      "plt.title(\"Weights assigned to the features\")\n",
      "plt.bar(range(10), importances[indices],\n",
      "       color=\"r\", align=\"center\")\n",
      "plt.xticks(range(10), indices)\n",
      "plt.xlim([-1, 10])\n",
      "plt.show()\n",
      "\n",
      "# actually, we should plot the absolute values..."
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Feature ranking:\n",
        "1. feature 1 (2.373590)\n",
        "2. feature 0 (0.934777)\n",
        "3. feature 2 (0.821554)\n",
        "4. feature 7 (0.101819)\n",
        "5. feature 4 (0.092758)\n",
        "6. feature 6 (0.025616)\n",
        "7. feature 8 (-0.048308)\n",
        "8. feature 5 (-0.095083)\n",
        "9. feature 3 (-0.124062)\n",
        "10. feature 9 (-0.132291)\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEKCAYAAADpfBXhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGB5JREFUeJzt3Xu0JWV95vHvAw0KolxEW6BbWrkkqKxAJIAgchbDEGix\ndbKI0TELJS5JjCjqJENwnNBZoyHGcSQExyACg5eABCLBDBGJ4TSoAeXSXBsGEKSbS4s0dxRoeOaP\nehs2u/e5dNfZdU6/5/msdVbXrnp3/d7adfZTtd+qfVq2iYiIumw03R2IiIipl3CPiKhQwj0iokIJ\n94iICiXcIyIqlHCPiKhQwn0WkPQ+SRdPsu0HJF0+7D5NBUnHSzptGuqOSFreYb0p3SeSzpS0StIV\nU7XOmHkS7jNUCa6L+ubdNsa8d4+3LtvftP3bU9SvUUkfnIp1tWX7RNsfmu5+9JP0nKTXr+dzF5Tn\nD+W9KekA4GBge9v7tlzXBnMiMBsl3GeuJcB+kgQgaTtgDrDHmjd+mbcTcFmH/cq33iZH0/z8sewI\n3GX7V0Na/6RJmjPdfahZwn3mugrYBNijPD4AuBT4f33z7rB9v6QtJZ0u6V5JKyT9j56DwIvOsCQd\nIulWSQ9L+pKkJf1n45I+Xz66/1TSoWXeZ0vNUyQ9JunkMv+LklZKekTS9ZLeOGiDJB0l6WZJj0q6\nQ9LRPcu2lfTPkh6S9KCky3qWHVe26VFJt0g6qMxfLOnrPe2OlPQzSb+Q9GlJd/W1PVfSWWU9N0p6\nc89zt5d0vqSfl23+aM+yzST9n/J63AT81lg7raff15XX6HfL/A+VT1kPSvqncmAeZM3zHy793Jdy\nQB20T8r8Mfd9X98+CJwGvKX07YQy/3BJS8tr/0NJu/c8588k3V76cpOkd5X5uwFf7lnXqjL/RZ/s\nBvzuPSfpjyXdBtw6ifoD931Mgu38zNAf4N+Aj5fpU4CjgM/0zftqmf42zZttM+BVwJXA0WXZB4DL\ny/S2wCPAu2gO7h8Dngb+oKft08AHac4e/wi4p6dPl65pWx7/Ns2B6BXl8a8BrxljexYCryvTbwOe\nAPYoj08s/d+4/Ozfs76716wTeC3w+jJ9AvD1Mv0G4DFgP5qD4ufLdhxUli8GfgkcWrbrL4F/L8s2\nAq4GPk3z6eh1wB3AIWX5X9F8ktoKmAfcCNw9zn57bk0fy+ODgAdoDsqbAicDS8Z47o7l+Rv1zJto\nn4y57wes//1rfhfK4z2BlTQHLAFHAncCm5TlR/S89u8GHgfmDlrXGL8fH+ir9xxwcXktXzJe/fH2\nfX4m/smZ+8y2hCYEAd5Kc1Z3ec+8A4AlkuYChwGfsP1L2w8AJwHvGbDOhcCNti+w/Zztk4H7+9r8\nzPbpbt5RXwO2k/TqnuW9QwZPAy8HdpO0ke1bbfevDwDbF9m+s0xfBnyvZ1ueBrYDFth+1vYPy/xn\naULgjZI2sX237Z8O6McRwIW2f2T7GeDPWXsI6XLb3y3b9Q3gN8r83wK2tf0Z26tLH7/KC6/f7wKf\ntf2w7RXA3/TVnsj7gNNtL7X9NHA8zRnvawe0HWu9A/fJOu77Qes/GjjV9k/c+BrwFPAWANvnrdmf\nts8FbgP2maCvEzmxvJZPTVB/NWPv+5hAwn1muwx4q6StgVfZvgP4d5qx+K2BN5Y2O9Kc6dxXPto+\nBPwdzVlcv+2BFX3z+h8/H862nyyTW/Qsd8/yS2k+QXwJWCnpVEkvH7Qxkg6TdEUZmniI5kDzyrL4\n88DtwPfKkM1xZf23Ax+nOfNeKensMYY0XrRdtn8JPNjXZmXP9JPAS8vwxY7A9mteu9K344E1B7Tt\ngd67Y+4etH3j2A74WU/fnih922Ed1jHWPlmXfT/IjsB/6dv2eaXPa4a6ru1Z9iZe2Gfrq/e1HLN+\n+X2fzL6PARLuM9sVwJbAh4AfAth+FLiX5oznXts/o3mzPAW80vbW5WdL27sPWOe9NG8eACSp9/Ek\nrHVB1fbf2t6LZmhkV+BP+9tIeglwPvDXwKttbw1cRDn7s/247T+xvROwCPjkmvFV22fbPoAmCAx8\nbhLbtRmTD6HlwJ09r93Wtl9h+/Cy/D6aIYE1Bp1xj+deYEFP315W+nbPgLbresF6Xfb9IHfTfCrp\n3fYtbH9L0o7AV4CPANuUfXYjL5yxD+rrE8DLeh6/ZkCb3ueNWR8mve9jgIT7DFbOPq8CPsmL74j5\nQZm3pLS7j2aI439JermkjSTtJOlt/eukCdTdJb1Tzd0KH2HwG3AsK2nu0AFA0l6S9pG0Cc3Z8K9o\nhlL6bVp+fgE8J+kw4JCe9RwuaedysHm0rONZSbtKOqgcHJ4aZ/3nA++Q9BZJm9Kc7U122ODHwGOS\n/mu5eLqxpDdJ2qssPxc4XtJWkuYBHx17VUDfawScDRwl6TfKdvwlcIXtQZ8AHqAZl95pwLK1rOO+\nH+Q04I8k7a3GyyS9XdIWNCFtmn22kaSjaM7ce7dzXtn3aywFfqe8jjvTXCdYr/rrsO9jgIT7zLeE\n5iP2D3rmXU5zYbQ38I+kCc+bgVXAP/BCaLv8YPsXNGPIf03zpt2N5gDyVH/bHr2P/wY4Qs1dGycB\nr6A5u1sF3FXW+fn+jbD9GM3F23NL2/cC/9TTZGfgEpqLoj8CvmR7Cc2Y64k0oXdf2e7jB2zXTTSh\new7NmfJjwM8ns122nwUOp7ng+dNS6ytl2wD+gmZY5U7guzRj3uOdYS8GzirDDEfY/j7w32kOQPfS\nXLAdOCZehlw+C/ywvMb7jNf3Yrx9v1aJ3ufavprmk+Ep5bm3lfVh+2bgCzRDgffTBHvv7+H3gZuA\n+yX9vMz7Is31k5XAmTTXNnr7+qLtGK8+4+/7mICa6zPr+WRpPs0v+qtpdtpXygW63jYjNG/iNRdC\nzrf9mfUuGlOqjDkvB/5zCdMqlDPPh4Cdy9BVxKzS9ksEz9BcpV9a3kxXS7rE9rK+dktsL2pZK6aI\npENohiJ+yQvj4xv8V9ElvYPmbFLA/wSuT7DHbNVqWMb2/baXlunHgWU0dxb0G9a37WL9vIXmzpQH\ngLcD7yq3pW3oFtFcpLyHZsx6rNsBI6rXaljmRSuSFtCMD7+xBP2a+QcC/0hzm9o9wJ+UsbyIiBiS\nKfnbDmVI5jzg2N5gL64B5tt+stwhcQHN7XIRETEkrc/cy21Q/wz8i+2TJtH+TuDNtlf1zc8fpIqI\nWEe2Bw57txpzL/cknw7cPFawS5pb2iFpb5oDyqpBbT0Nf3/hhBNOmFV1s82zo262eXbUHU/bYZn9\ngd8Hrpd0bZn3Kco3+GyfSvM3Pz4saTXNl1xykSsiYshahbvtHzDB2b/tL9H83ZGIiOjIrP+G6sjI\nyKyqO521s82zo3a2eWaYslsh25LkmdKXiIgNgSQ8jAuqERExMyXcIyIqlHCPiKhQdf/7eLmlfqhy\nbSAiZrrqwh3W/b+yWRf5C2gRsSHIsExERIUS7hERFUq4R0RUKOEeEVGhhHtERIUS7hERFUq4R0RU\nKOEeEVGhhHtERIUS7hERFUq4R0RUKOEeEVGhhHtERIUS7hERFUq4R0RUKOEeEVGhhHtERIUS7hER\nFUq4R0RUKOEeEVGhhHtERIVahbuk+ZIulXSTpBslfWyMdidLuk3SdZL2bFMzIiImNqfl858BPmF7\nqaQtgKslXWJ72ZoGkhYCO9veRdI+wJeBfVvWjYiIcbQ6c7d9v+2lZfpxYBmwfV+zRcBZpc2VwFaS\n5rapGxER45uyMXdJC4A9gSv7Fu0ALO95vAKYN1V1IyJibVMS7mVI5jzg2HIGv1aTvseeiroRETFY\n2zF3JG0CnA98w/YFA5rcA8zveTyvzFvL4sWLn58eGRlhZGSkbfciIqoxOjrK6OjopNrKXv+TaEmi\nGU9/0PYnxmizEDjG9kJJ+wIn2V7rgqokt+lLz3qG+rFAwFT0MyKiLUnY7h8ZaZa1DPe3ApcB1/PC\nUMungNcC2D61tDsFOBR4AjjK9jUD1pVwj4hYB0ML96mUcI+IWDfjhXu+oRoRUaGEe0REhRLuEREV\nSrhHRFQo4R4RUaGEe0REhRLuEREVSrhHRFQo4R4RUaGEe0REhRLuEREVSrhHRFQo4R4RUaGEe0RE\nhRLuEREVSrhHRFQo4R4RUaGEe0REhRLuEREVSrhHRFQo4R4RUaGEe0REhRLuEREVSrhHRFQo4R4R\nUaGEe0REhRLuEREVSrhHRFQo4R4RUaHW4S7pDEkrJd0wxvIRSY9Iurb8fLptzYiIGN+cKVjHmcDf\nAl8bp80S24umoFZERExC6zN325cDD03QTG3rRETE5HUx5m5gP0nXSbpI0hs6qBkRMatNxbDMRK4B\n5tt+UtJhwAXAroMaLl68+PnpkZERRkZGOuheRMSGYXR0lNHR0Um1le3WBSUtAL5je/dJtL0TeLPt\nVX3zPUV9of1axlk/MBX9jIhoSxK2Bw57D31YRtJcSSrTe9McUFZN8LSIiGih9bCMpLOBA4FtJS0H\nTgA2AbB9KnAE8GFJq4Engfe0rRkREeObkmGZqZBhmYiIdTOtwzIREdG9hHtERIUS7hERFUq4R0RU\nKOEeEVGhhHtERIUS7hERFUq4R0RUKOEeEVGhhHtERIUS7hERFUq4R0RUKOEeEVGhhHtERIUS7hER\nFUq4R0RUKOEeEVGhhHtERIUS7hERFUq4R0RUKOEeEVGhhHtERIUS7hERFUq4R0RUKOEeEVGhhHtE\nRIUS7hERFUq4R0RUqFW4SzpD0kpJN4zT5mRJt0m6TtKebepFRMTktD1zPxM4dKyFkhYCO9veBTga\n+HLLehERMQmtwt325cBD4zRZBJxV2l4JbCVpbpuaERExsWGPue8ALO95vAKYN+SaERGz3pwOaqjv\nscdquHjx4uenR0ZGGBkZGU6PIiI2QKOjo4yOjk6qrewxs3ZyK5AWAN+xvfuAZX8HjNo+pzy+BTjQ\n9soBbd22L2U9Yx89poCAqehnRERbkrDdfwINDH9Y5kLgyNKJfYGHBwV7RERMrVbDMpLOBg4EtpW0\nHDgB2ATA9qm2L5K0UNLtwBPAUW07HBERE2s9LDNVMiwTEbFuxhuW6eKC6qwgDXx9p1wOLBExGQn3\nKTTs2O3m8BERNcjflomIqFDCPSKiQgn3iIgKJdwjIiqUcI+IqFDCPSKiQrkVsgJd3GOf++sjNiwJ\n90oM+1u5EbFhybBMRESFEu4RERVKuEdEVCjhHhFRoYR7RESFEu4RERVKuEdEVCjhHhFRoYR7RESF\nEu4RERVKuEdEVCjhHhFRoYR7RESFEu4RERVKuEdEVCjhHhFRoYR7RESFEu4RERVqHe6SDpV0i6Tb\nJB03YPmIpEckXVt+Pt22ZkREjK/V/6EqaWPgFOBg4B7gJ5IutL2sr+kS24va1IqIiMlre+a+N3C7\n7btsPwOcA7xzQLv8H8sRER1qG+47AMt7Hq8o83oZ2E/SdZIukvSGljUjImICrYZlaIJ7ItcA820/\nKekw4AJg10ENFy9e/Pz0yMgIIyMjLbsXEVGP0dFRRkdHJ9VW9mTyeYwnS/sCi20fWh4fDzxn+3Pj\nPOdO4M22V/XNd5u+9KxnUkec9V4/MKifw647nbXHqhsR00sStgcOe7cdlrkK2EXSAkmbAr8HXNhX\nfK4klem9aQ4oq9ZeVURETJVWwzK2V0s6BrgY2Bg43fYySX9Ylp8KHAF8WNJq4EngPS37HBERE2g1\nLDOVMiwzc2tnWCZiZhrmsExERMxACfeIiAol3CMiKpRwj4ioUMI9IqJCCfeIiAol3CMiKpRwj4io\nUMI9IqJCCfeIiAol3CMiKpRwj4ioUMI9IqJCCfeIiAol3CMiKpRwj4ioUMI9IqJCCfeIiAol3CMi\nKpRwj4ioUMI9IqJCCfeIiAol3CMiKpRwj4ioUMI9IqJCc6a7A7HhktRJHdud1ImoScI9Whl27I51\n+OjiwJKDSmzIEu6xwRpm9HbzmSRieFqPuUs6VNItkm6TdNwYbU4uy6+TtGfbmhERMb5W4S5pY+AU\n4FDgDcB7Je3W12YhsLPtXYCjgS+3qRkRERNre+a+N3C77btsPwOcA7yzr80i4CwA21cCW0ma27Ju\nRESMo+2Y+w7A8p7HK4B9JtFmHrCyZe2IzuVCbmwo2ob7ZH8L+98R+e2NDVYu5MaGoG243wPM73k8\nn+bMfLw288q8tSxevPj56ZGREUZGRtarU9P1BpnON2a2ue660/mdgun6tDIbt3kio6OjjI6OTqqt\n2nwElDQHuBX4D8C9wI+B99pe1tNmIXCM7YWS9gVOsr3vgHU5H0cjBpPUyXcKxgq6YX9amY6601l7\nrLrrvB4J2wOPRK3O3G2vlnQMcDGwMXC67WWS/rAsP9X2RZIWSrodeAI4qk3NiIiYWKsz96mUM/eI\nsU33EEXO3Lupu87rGdaZe0R0Iyc+sa7yVyEjIiqUcI+IqFCGZSJiQrn/fsOTcI+IcWW8f8OUcI+I\nGWs2fkluqiTcI2JGms5PDDV8WskF1YiICiXcIyIqlHCPiKhQwj0iokIJ94iICiXcIyIqlHCPiKhQ\nwj0iokIJ94iICiXcIyIqlHCPiKhQwj0iokIJ94iICiXcIyIqlHCPiKhQwj0iokIJ94iICiXcIyIq\nlHCPiKhQwj0iokIJ94iICs1Z3ydK2gb4FrAjcBfwbtsPD2h3F/Ao8CzwjO2917dmRERMTpsz9z8D\nLrG9K/D98ngQAyO295yJwT46Ojqr6k5n7Wzz7KidbZ4Z2oT7IuCsMn0W8K5x2qpFnaGajb8M2eb6\n605n7WzzzNAm3OfaXlmmVwJzx2hn4F8lXSXpQy3qRUTEJI075i7pEuA1Axb9t94Hti3JY6xmf9v3\nSXoVcImkW2xfvn7djYiIyZA9ViZP8ETpFpqx9PslbQdcavvXJ3jOCcDjtr8wYNn6dSQiYhazPXDY\ne73vlgEuBN4PfK78e0F/A0mbAxvbfkzSy4BDgL9Ylw5GRMS6a3Pmvg1wLvBaem6FlLQ9cJrtt0t6\nPfCP5SlzgG/aPrF9tyMiYjzrHe4RETFzzdpvqEo6Q9JKSTdMQ+1DJd0i6TZJx3VYd76kSyXdJOlG\nSR/rqO6vSbq25+eRrmqX+huXut/pqmapu5Wk8yQtk3SzpH07qnt82cc3SPp7SS/pom6pfZek68vr\n/eOOar5U0pWSlpbXubPRAUnHltf5RknHdlV3UmzPyh/gAGBP4IaO624M3A4sADYBlgK7dVT7NcAe\nZXoL4Nauavf0YSPgPmB+hzU/CXwTuLDjbT0L+IMyPQfYsoOaC4CfAi8pj78FvL/Dbb4T2KbL17nU\n3bzndb4CeGsHNd8E3AC8tLyvLwF26nrbx/qZtWfubm7HfGgaSu8N3G77LtvPAOcA7+yisO37bS8t\n048Dy4Dtu6jd42DgDtvLuygmaR6wEPgqHX6ZTtKWwAG2zwCwvdr2Ix2UfhR4Bthc0hxgc+CeDur2\n6vzmCNtPlslNaYJ2VQdlfx240vavbD8LLAF+p4O6kzJrw30a7QD0BtuKMq9TkhbQfHK5suPS7wH+\nvsN6XwT+FHiuw5oArwMekHSmpGsknVbuHhsq26uALwB3A/cCD9v+12HX7e0C0/ClRUkbSVpK84XK\nS23f3EHZG4EDJG1T9u3bgXkd1J2UhHv3pv0KtqQtgPOAY8sZfFd1NwXeAfxDR/UOB35u+1q6P5uc\nA/wm8L9t/ybwBGP//aUpI2kn4OM0wzPbA1tIet+w6/bY3/aewGHARyQd0EVR28/Z3oMmXN8maaSD\nmrfQ3Ar+PeBfgGvp/iRiTAn37t0DzO95PJ/m7L0TkjYBzge+YXut7yYM2WHA1bYf6KjefsAiSXcC\nZwMHSfpaR7VXACts/6Q8Po8m7IdtL+BHth+0vZrmVuT9OqgLgO37yr8PAN+mGYbsTBn6+r80r0MX\n9c6wvZftA4GHaa5jzQgJ9+5dBewiaUE5k/09mi+EDZ0kAacDN9s+qYuafd5LE7KdsP0p2/Ntv45m\nOOjfbB/ZUe37geWSdi2zDgZu6qD0LcC+kjYr+/tgoIshCiRtLunlZXrNlxaHfjeapG0lbVWmNwP+\nI81Z9NBJenX597XAf6LbIcdxtfmG6gZN0tnAgcArJS0H/tz2mcOua3u1pGOAi2ku/Jxue9mw6xb7\nA78PXC9pzS//8ba/O+zC5c1+MDCdfzyu6yGxjwLfLAfxO4Cjhl3Q9nXl08lVNEME1wBfGXbdYi7w\n7eaY8vyXFr/XQd3tgLMkbURzwvp129/voC7AeZJeSXMR+49tP9pR3QnlS0wRERXKsExERIUS7hER\nFUq4R0RUKOEeEVGhhHtERIUS7hERFUq4R0RUKOEeEVGh/w/39oOpdc+jIAAAAABJRU5ErkJggg==\n",
       "text": [
        "<matplotlib.figure.Figure at 0x112317290>"
       ]
      }
     ],
     "prompt_number": 67
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## And voil\u00e0!"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "![](http://weknowmemes.com/wp-content/uploads/2011/12/frogman-happy-240x180.jpg)"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}