Polyglot Data Science with IPython

README.md

Polyglot Data Science with IPython & friends

Author: Fernando Pérez.

A demonstration of how to use Python, Julia, Fortran and R cooperatively to analyze data, in the same process.

This is supported by the IPython kernel and a few extensions that take advantage of IPython's magic system to provide low-level integration between Python and other languages.

See the companion notebook for data preparation and setup.

Used for a lecture at the Berkeley Institute for Data Science. The lecture video has a live demo of this material.

License: CC-BY.

data.csv

polyglot-ds-prep.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Data generation for polyglot data science example\n",
    "\n",
    "In order to run the full [Polyglot Data Science with IPython notebook](polyglot-ds.ipynb), you will need to install [Julia](https://julialang.org/downloads), and then the following (assuming a conda-based deployment that will automatically pull in R, otherwise you also need ot install R):\n",
    "\n",
    "```\n",
    "conda install jupyter cython pandas matplotlib seaborn\n",
    "conda install rpy2\n",
    "pip install julia fortran-magic \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We generate synthetic data according to\n",
    "\n",
    "$$\n",
    "y(x) = a x + b x^2 + c \\sin(x^2) + \\cal{N}(0, \\epsilon)\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnX+MHOd537/PHVfinmTraJiopZVoKr+oQqbFCw+OGgFppTimC1nOxrLNGIpR9AeEAk0QscK1dCxElKHABA4OHbRFC8F2m0CqTdmyL5JplHIgFW6E0vVRR5qmRQaGZVFauTBT85RIdxL3eE//uJvj3t68M+/svDPzvrPfDyCIdzc7++7szDPPPM/3eR5RVRBCCKkPI1UvgBBCiFto2AkhpGbQsBNCSM2gYSeEkJpBw04IITWDhp0QQmoGDTshhNQMGnZCCKkZNOyEEFIzNlXxpu985zt1+/btVbw1IYQEy/Hjx/9WVbembVeJYd++fTtmZ2ereGtCCAkWEXnJZjuGYgghpGbQsBNCSM2gYSeEkJpBw04IITWDhp0QQmpGJaoYQlwzM9fB9NGzeHV+EdeNNzG1ZwfaE62ql0VIJdCwk+CZmevgU18/hcXuJQBAZ34Rn/r6KQCgcSdDCUMxJHimj55dM+oRi91LmD56tqIVEVItNOwkeF6dX8z0e0LqDg07CZ7rxpuxvx8RwY37j+C2g89gZq5T8qoIqQ4adhI8U3t2oNkY3fD7S6pQXI6507iTYYGGnQRPe6KFz35kJ1rjTQiAUZEN2zDmToYJqmJILWhPtNYUMDfuPxK7DWPuZFigx05qhynmbvo9IXWDHjsJnv7ipNtv2oonjnfWSSCbjVFM7dlR4SoJKQ8adhI0ccVJjx47h7HGCLaMNTC/0GUlKhk6aNgDhOXzl4krTgKAhe4yFIJDe3cN7bEhwwtj7IEReaid+UVK+ZCcEKUShgwrNOyBwfL59aQlRH1UwszMdXDbwWdYPEUKg6GYwGD5/PpQ1DXNBhqjgu4ljd3WNyUMG5aRMqBh9xRTHP268SY6MUbcNwNWFP2GcX6xa9y22RjF7TdtxW0Hn/EmH5H0xEXDTlzBUIyHJMXR48rnh0nK99BTp2OTpRFRzWlrvIm7d7fwxPGOV/kI05NVZ36RYRniDBp2D0nz6nrL51vjTXz2IzuHwtubmevgwoLZQwcAxcoxeW7/HXj2zHnv8hFJT1Y+3HhIPaBh95A0r27f4RMAgEN7d+G5/XcMhVEHYG2Qo+PnYz7C1LAsouobD6kHNOweYvLqBPAqrFA2tgY5On4+thbofeIyMUyJcFIMNOweEufVCVbCDL2keXd1k9XZGGQB1vINvuYj2hMtPLf/DqNxH5ZEOCkOGnYPiYujx4v5zN5dHQuZ0sIYAuCeW7ethaai4zjebKxts7nhzynv642HhA/ljp7S24YWAG47+EwmmWMdZXXRuns17CJI7Qfz1tLy2r8vLHS90Y33fx4f5JikHuQ27CJyA4C/APAuAMsAHlHVP8u732EmrlvhG28tbdguSaftY+LQBf03vH76j93CxSWvb3Bpn4eQQXDhsS8BuF9VnxeRtwE4LiLfVtUfOtj30GHqVtjPlrEG7nzvteva0/ZWMQ5jIVPcsTMR+g2OkCRyBxxV9aeq+vzqv/8ewAsA6IIMiKlbYT9jV2xK1GkPY/zW9tgB9b7BEeI0kyQi2wFMAPiuy/0OCzNznUQvs5dX5xeN23bmFzF99Czu3t0aqkImWy+87jc4QpwlT0XkagBPALhPVf8u5u/3ArgXALZt2+bqbWtDFEawZXNjBG92l41qmc78Ip443qmVMU/rQ28KP403G7jqyk1eJSjZU58UiaiaTEOGnYg0AHwTwFFV/dO07ScnJ3V2djb3+4ZK3EU9ffSstbeehai8PmRm5jo48OTpDQ2/mo3RdTeu/hh73DY+EMo6iX+IyHFVnUzbLncoRkQEwBcBvGBj1Icdk768CKMOJCcQQyA6XnFdHPsLtELpo8Oe+qRoXIRibgPwSQCnROTE6u/+SFW/5WDftcN0UY+K4JKDp6d+BCvG0TfjZktaQrQ/rp4kH/Ql/JHWC6jq9ZHwcaGK+WtVFVV9r6ruWv2PRt2A6aIexKhL+iZQ2DfP8pG0hKitusWnSlzbXkD3HT6Bic88Xdoa69aCYpjxp746cGwuipm5DkbExhyvJ+4VzcYo7rl121q/kaS9hqzZTjLcjVGxVrf4FP6w7QUEXK6ULdrI+nTjI/lhSwEH2Iw7i7bJ6plHSTUgufR8Zq6D+x8/Gbv/kDXbU3t2YN/hE7FG76orNsWGKuJCLj5V4sa1EkjKhZRRKVvHFhTDDA27A2wuiizFM71sboxg3+ETqTHX6PdxaouQNdvtiRbuO3wi9m+vxSRUTTfZsStG8cbFjce/7Jte/03n0N5daE+0jL2AIoq+Afl04yP5oWF3gM1FMcgFIsDaxKCkocf9w503N0ZSG2OFQPS5TMQZZdNNNo4soRwXPDBzCo8dO7f29NH7nU7t2bHhptxL0Tcg01PDNT2dMUk4MMbuAJuBDoNcmEn916OY/vb9R7Dv8Im12Oj8YhdvdpfXpisBCDIh1hvzjcP0JJLlBtq9pJg+eraUYzIz11ln1CP6Rx42Y9oKl/HUNbVnBxojGzM1b1xcCuacIZehYXeATV8WVxfmq/OLG4yeyVjEJcTKVloMSlLoKkmfnvUGWlaScProWcue+uuNqwC4e3fxHSDbEy1cvXnjA3x08yNhQcPuAJvCmPZEC1vG8j/WXjfetIrXv7raLyZuu7KUFnkwed4CJM55TRvGEUcZ6pikJwnFylPVQ0+d3vB9KYBnz5wvdG0R84ZB4aEXuQ0jjLE7wqZPeJwgxiRzG2828NbScmwidJ8hmdjLdePNRGPiu+Jh0LbD0ecxKYRMFJ0kTFO++NBi2LTG0IvchhF67CVgKovfMtbAPbduiw3jHPjwzcangDTjFt0A0rbzWfGQp+1we6KFz338llituImik5ODPElElKXcmdqzI/YYhV7kNozQYy+QSNVh8sbGrtiEh9s7Mfnudxg16nFeUpyCIvL8W32vT1JajIjgxv1HvFTP5B0bZ3o9UI0ktH89ts8SZcpVk6SlDMeEhZPujlkZhu6OcR38+hEALx68c+D9m4xev/zx4tIlLHSXE/c3TN0FfegZk6ZbB4BREXzu47eUsrY0J0SANc09qQ7b7o407AVhc+EW0Sfc1BL27t0tPHvmfOKa6tDiNxSKvvG7XgvA88MHSmvbS+KxiV+/cXHJeW8OU4HOs2fO4/abtta2p0xo9CqpTJQVW7etiub5EQ407AWRdlEKVjTCvdjI7tKajSW1hI0rkMmyZuKW9kTLWBhUZlWsrcHm+REOTJ46oD9me/tNW/HGW0vG7Rsjgu5yvIlNushsmo2ZJGtp/d5D7ykTKtNHz8aeC6YGZ0WQJsUEeH6EBj32nMRVdz567NwGaWPklLXGm7EVfhFJXpFN61mTTDDJqI+KeJc4Lao3uG89x0038rgGZ0URd840RgRbxhpeT6IiZuix58Q2PnntNZcTTzfuP2LcLskrsmk2ZpL5mRQPApSmvLAl7snkvsMn8NBTp/HgXTcPvFabJ56yGbQQyyV5paXEP2jYc2Ibn4zGnkWFQ3EX83izkXgx2RqB/irYmblObGhIANxz6zbvLuC0VgjAYIa4yp7jJollXE1CFWGPtMppEhYMxeQki2cVeYi337TVWG2axCDVmElVr4f27sLD7Z3W6y8Lm1YILvcb3XSLCsskTScKZQA3CQt67DlJ66PdTyQ9/OxHdmZ+9B3kkdnk/Y6VmJyzJfJq0yorBpXdJSUJiwzLpD0p+Owt+1DMRbJDw56RmbkODjx5es0D3jLWwN27W3j02Dnrfbw6vzjwxZz1daFMxrEtkgEGjz+n3YQXu5dw3+ETmD56NrcB6zWIdu16/cPHnASxg4Y9AzNzHUx99eQ6edqFhS4Of+9lbBlrrE07SqPMxFhSXN4nb8w2CZ0n/tz7xJPWaTGPAbO9SfmuC+cc1HBhjD0DDz11OlZz3L2kUE3uHhhRdmLMFJe//aatXk2lt/FeXcSf2xMtPLf/jtTe+Hli+TY3KYG74StFkZSTIH5Dw27JzFwn0SOfX+wmxoarSoyZknPPnjkf640dePJ0aWvrJc17jfqUuDh2M3MdvP6muYAsYtBQic3rFH6EM5J0/eOGm1/Un534C0Mxlth4b6ahGVU3T4qLy5uGdcwvdjHxmadz6cUHIRogEnf8XHu3pmrPfgYNldhUcib1iCmD/lwRsD4EBcB484v6s/twYyLx0GO3xNYL68fXUuwko3VhoYt9h0/ggZlTxm1c055o4Z5bt20IZxWhtbf5LvN8b6aBFf3bVIVJAgtcDkGl3fx8T/wOOzTslmTx3kIoxU4zLArgsWPnSnvknpnr4Nkz56FYaXEArBy/IrT2ad/llrFG7u8t6XkgrRANKLb1QVoOoDO/mGq4R0QYjvEYhmJS6B1AYAq19DN2xSbM/fEHil5aLtoTLTz01OnEvIFiZXZotH1R9KtILqmuecxFvK9J9rhlrJE7BBV9FhMCpBaiFS0zTDPagpX4etK5cUmV0kePoceeQG/FILBi6GyUL6E8pj54182pczijC7hI78ymuZlL4hLKn9+7C3N//IHcRirJG7YNKxV9PNKeWBSAKlLPjSK/I5IPeuwJxF1gNh67bxpxE9F6+pNo/RSpXZ6Z6xgTjUXeIIuq9kxas+1ouaKLyqb27DDONo14bbGLQ3t3lV5kFVcAWHYivw448dhF5Esi8jMR+YGL/fnCICetjxrxJNoTLRz48M1oNpJPhSKMbFrYwvcCnjhMa26NN62Nk2kfruLa7YlWqo7/utX1Prf/Drx48E6jisfldxQVAPY6GRcWupj62kkvrx2fcRWK+W8APuhoX96Q9aSNkm4mjbiPj62X47nJw66LMLJJYQtf1URpDNKozWYfwEpYzJVaKSkMF7deF58rDZMSp3tJvbx2fMZJKEZVvyMi213sq0riJiE9cbyTOnC4P9Ri0oj7GHu3qZIsysgmHQ9f1URpuOhtHm17/+MnNwxIidRKk+9+R+4K3GidnfnFtQlbLcN6y+jZnnQ++Hjt+Axj7KvEKRGeON7B3btb+PJ3X46dQGQqPPJheIItaRdMkTFO03HKErbwERfx+/ZEy+gguCoQyrrOortQJhV2+Xjt+ExpqhgRuVdEZkVk9vz582W9rTUmJcKXv/syPvFrN2R6DC3jsdUVaRfMmykhmjyEdJyqIOm7qaMH68Ng77pQmmFX1UdUdVJVJ7du3VrW21pjulAuqa557rbDEEIanmCK50aULTus4jj5Ngc1IqmCtY4ebHuihemP3YLx5uXE7paxBqY/6tfoxhAQTRhynGlHKzH2b6rqe9K2nZyc1NnZWSfv64rbDj6T2N9jVATLqt5KF/PQW4QVhwB48eCd5S6qJOJa7DYbo97ciB+YOYXHjp1bJzesYn0hyHeHARE5rqqTadu5kjt+GcD/BrBDRF4RkX/pYr9lkua5XlL1Xro4KJGsrQxJm2+4LgZy7f0/3N6JQ3t3rfNiN6dIU12TNNqvyPf08SkqFJycIar6CVW9VlUbqnq9qn7RxX7LJAoLRH1KkvBVupiXYYx5uywGKtIAvrV0OdcRDfUuy9iVXRlcxY2kblAV00P0aGkz/aaOyasyJG3A+tBPmsyuaFwqmB566nQhE4eqnmRU9njFqj9vHRi6XjGmR7zo9/sOn8CVm0YQk5xfR13DE1FY5tDeXQBWNPkuH4X7++9EMtKqvDJTCG7h4lKmtSQNYslrAKueW2s614u6Bqr+vHVgqDx2U9e82Zd+vq4QKalvClD/8ESR3QWTCqKq8MpM/XKicEfvNkkkhSXyGsCq6yLiumG6vAb6E7OmzpKKFZEDE7fpDJXHnqRVtxmkDKyoY3xRTBRFkTHVNK+rCq+sPdHCVVdu9HGyjApMWndeA1h17qNIWWpcPP31N5fQGI1/ZGa83Y6h8tiTtOo2+CSDK5IiH4XTxsZVFeIyfbb5xS5m5jqp37npc9kM1UijrNxH2hqKeL84J6K7rBhvNnDVlZtijynj7ekMlcduMhomIcx4s1F58UwRpEnJkroL5pWfmaoLgWorDJNuKDZPKiavOm2ohi29nRbzDvX2SUpouqG+ttjFc/vvMBZoMd6ezFAZ9riLrzEixpNHZOU1Li4mX7CRkiV1F8wrP2tPtHD15vgHxaiLXxWGJumGYmNEfKmiTcM3KWFaYrbsxG1dGKpQTNwj7cLFJaOaIWsCLQRspGQ2AziyPg73JsiSAl+ux8DZkjQq0NaIFN0kC8hfAZpXSui6AjUtMVt04rauDJVhBzZefDfuP5K4fd3ieVni571FMVn21Uv/RBwbqjrmD95188BGpIySexdqpTz5kyLUUjb5gys3jay9Jycq2TF0hr2ftGQeUK94nq10zqZPe5onG9eHxZaq1DFA9iRl0cOnI1wU7uSRThZVOGR60ok7f4rsNlonhirGHkdajxigXvE8W+lcmmG18WRtbg4mqjrmvUnKqT07MH30bGqSsaySexdqpTzSSddqqbQkbtmtDOrE0HjscUNy73zvtWtj7KLSdgE2dNKrUzzP1itNepJJK/9P6xbZvx8fY6hZvPCyKiVdFCrlkU6a3v+aZvL81Djiju/UV0/ioadOY36hm3j+1ekJuihqZ9jjYrpjjRG8ubSM3nGKFxa6ePTYubWfL6mi2RjF3btbePbM+Vq3J7VJ8pkMbprSI0v4ZeHiEoCVMXi+tYTNEnYoqzLU1U1w0CTv1J4dmPrqyQ1zSd9Ybb+QN4nbXda15LWPtQ4hUatQTNyUcwBY6K436iYWu5fwzZM/LbRXSij0yveAlYrbyLA9MHPK+AidJfzSqzpypdF2RRYvvMzK0Cs3Xb5ko+HpZR0vk1R1kGHTg3rdgvyVvMNArQy7acp5FuYXu3hg5pRXWt+qaE+01oxWb7OuR4+dMx6bpAs2riWyrzHTLPrpMjTs0ZNQr9NSRSJx3lGjs0G9bkV9pMdFUivD7ir2Ftc7xlcDVDRxrWj76T02pgu2Nd7EsqF1g48x06xeuMvK0Dh8SSS6KhiyES3EYRoGQ9ZTK8PuKvZm6h3jowEqkqRWtP1ExybJIIZURehbJakvrWxdhZ36Q302+JBUD4VaJU9NyZ04RICxxijeuLjRG40UMv34aICKJIs3GPWRuW68mZiA9lEB00+cgqrqhG7eBK2r4SYuG5JFSdy4hLsA+PVffAd+8v8WvUqqh0Jwhj2pws+mFH4NBf7kd3bGGpq7d7fW9WePfu+bASqaLN5gfwwe2CiL9KFLYRpRAr7XObiw0MXU104CqC6+m0cR0284+4ebANk+l+vWCSGcF6Ehatmy1iWTk5M6Ozub+XVxd/bGiODqzZvWtK/9J8TEZ56ODSe0xpt4bv8dxhsFp7KvDDVI06KnEVqr46TPHJ0zVTHoOZn2PVb9uYg9InJcVSfTtgvKY7fRvvZ6IDNzHbz+5tKG/fS2hzV5H2U0dPKdOC8xK6H12kl6Sqk6x9Ibupg+ehb7Dp/A9NGzqQbex+EmpFiCMuw2J2CvITHJH6+6YlMwhqZK+h+RB322C8lwJFU8RvHsfs/59pu2llbUNkhfGl+Gm/ApuDyCUsXYnoCRIUlq4u/TsAGf6ZXxDSo1CynpbBoEEj3lxfUzT9L1u2YQ2WOStLCs3JFvfeDrTlCG3Vb7OiKCmbmO0aBc02zwJBuAQbXHnfnFYG6e7YkWpj92C8Z7+p9sGWtg+qO3rPVsz6Lrd80gsse4KmKgXAmn6YZ03+ETmPjM05WfG3Vz9IJKngL2DaaS1C2bGyOJCVVipvdx+ppmAxeXLmHBsgIytERqPzNzHdx3+ITVtgLgxYN3Ol9DWnLX1/DGjfuPJIbyGqOydvMsmzhRhq/nqm3yNCiPHbgcGvj83l2J3uNi9xKePXM+tsjEVVn0MBId/0N7d+GtpWVrow6sfCf3P34yWK8oixdeVPgp6akp7cnTpVeadV9px2OQfjOuiHsKC73SPKjkaS+9ib2k9p5x6hbTa0KKBVeNqdnXqAiWV2ejxpFXQ10ltjf+Iodyp533JhWSy2Egg+zLRmFVhWOVVF0dsqMXjMce5yFE3qMpqWcy1GV246srppvpsipePHgntoyl9+gOzSuyvfEXrbqKznvTEPY4g+Sy18wg+4ri/HGN4CKqcKyS1hyyoxeEYU/LqA/SsMmnPiChMTPXMRqV68abxvqBOELximbmOnjjLbvP9FqG+a55yNJ7x2WvmUH31Z5o4XMfvyVRdVQmM3OdxFxdyI5eEKEYm6EHWQfesgBpcKaPno0NtUS9srO0T/bdKxpkGHdZnylLmwGXw0Dy7Cuu7UcVA6ojZ9HEeLMRtH1wYthF5IMA/gzAKIAvqOpBF/uNSPIQOPC2fEzfR9Qre5+lcsT38Ncgw7jL/ExZeqy4HEGYd18+OFVJA2GajVEc+PDNJa/ILbkNu4iMAvhPAH4LwCsAviciT6rqD/PuO8LkISiA+x8/uaETY2hl7KFh+j6iXEdapSNQjZeWlazDuEdFSg/p2RpJ110ZXe2rKpLCRnUIy7rw2N8H4Eeq+mMAEJGvAPhtAM4Me1JGnb3TyyfNY4v7ezQk3GetdT9Zz6FlVa8/l0tPedB9+dJWwOR8mFO7YeHCsLcAvNzz8ysAfs3BftewkTb243vsNmTSPLY6eHSA3ZNH//ZlYWsgfTGk0VpcSS7zrsOUCFegFk/7uStPReRjAPao6r9a/fmTAN6nqn/Qt929AO4FgG3btu1+6aWXBnq/tAo24HLVGBC+cSHVkSXGXmalomkwxT23bsPD7Z2J21VZUWmqmi2z4tvmOy2qatgFZbbtfQXADT0/Xw/g1f6NVPURAI8AKy0FBn2zNC8qetQH4IV3QMLF9kmx7PBSXOxfATx67ByOfP+na7MJFi4uxarJDjx5upJrwIfxfjZ5kzo87bvQsX8PwC+LyI0icgWA3wXwpIP9xpKUeRdgbZCwL8N/SdikFQP1nnNlkWQILyx012o9TBWV86vdTcvGh5m3aTcRQVhN60zkNuyqugTg9wEcBfACgMdV9XTe/ZpoT7SMVY29J4gP3gGpDz4YJZfvWYWD40PF93hKRXQUSgi946uTylNV/Zaq/oqq/qKq/omLfSbx4F03p54gPl2IJHx8MEq9a8mr3qjCwclT8e2igVmWimgg7Cf8ICpPI/pbxm5ujBhnnbosyCDEJ6VPe6KF2Zd+jseOnUsVEogAcfqIqhycQWSSrtQ0WSqiI/LO/K2KYAx7/5c7v9hFszGKQ3t3rX25/dKuu3e3ShtZRuqPDxWTEQ+3d2Ly3e9YS+xGdQK9CIBf/4V34PlzrwXt4Ni0FEkjrS/MlrFGbE5CVl/ry/duSxBNwID0jnJxjcKeON7B1J4dePHgnaUnuEh98HW6TnuitRYiivNDFcDz517D3btbQTe8y5svS+sL0xpv4sG7bo4Nb0W69tAIxmNP+3Jd3NUJ6ceXohoTafK9aOBMyJPB8jYwS+sLEz3Jm6ZjhSi4CMZjT0uGUgVDisB32azN+d2ZX/RiruigmKZGLVxcsvpMtn1hss518JlgDHuaKoEqGFIEvjsMtuf3hYUupr52MkjjHqlpegeMAyufyUaSaDpGrfHmBsGFL8qnvARj2NOkUnX6Uog/+O4wZJE+FjFX1Db/kDdP0Z5oIW74ks3Tk61tqNMAnty9YgZhcnJSZ2dnne/Xp4ZHpB741m8ljgdmTllJHwG3fVBMfVf6WzK7OIYzcx1jDNzmM9XFNpTZK8YbfJKjkXrgk37dRK/08dX5RYyIGNtZu3zSMCUloxAJgNT2HrbHMe9s0mGzDbUy7IQUQQhGoXeNM3MdTH315IZiHNdzRZPyDL2G20WeImlbhls3EkyMnRBiR3uihemP3bIu2bhlrIHpj97i9AaV5ilHxthFnsK0beizSYuCHjshNaSMp4ykyWbAZWPsor2HaR9FzCatQzyehp2QmlGuYYqP5feGfVzkKVzsw+a4+F6QZkutVDGEDDtlqXjSJhGNNxs48eAHnL1fXmyPi6spT0XdXIdSFUPIsFNWa420VgavLV5uqOXayA2yP9vj4iLR64PXz+QpITWirErZtP1F8fW45nx5BlgMuj/b4+Ii0etDGwoadkJqRFmVskn7602MujZyg+7P9ri4qGD3oQ0FDTshNaKs1hqmxlxbxhrr4taujdyg+yuzrYAPbSgYYyekRpRVKWv7Pnlb7sa9bpD9ZTkueaWiJhlo1I2yjDg7VTGEECfEJTUBOFXphNC7B4jW+X0sdpfX/T6adNUa8IZrq4phKIYQkhtTUhOA046JIXVgfLPPqAOXVf95k8hp0GMnhOTGlf67CKqoJDUdj36yHh/q2AkhpeGDEiSOqjTltp+7qOPDUAwhJDfG5KWg0qlNDz11uhJNuW1yuCilDA07ISQ3U3t2oDGyccSRKiobyTcz18GFhW7s34p+kjDJQXspcsIbDTshJDftiRau3hwf2S1iJJ8NeYdz5CEuyft7t24rLenLGDshxAnzBu8YgFUi0TV5hnO4SLhWOaCFHjshxAlJXrCg/Fj7oMM5XPe3ifaZZ5h3VmjYCSFOMMXZgRX9dtnhGFMbgbThHK772xRxo0iDhp0Q4oRoJJ+JssMxgxYzuZZuVtHtMVeMXUQ+BuAAgH8I4H2qyqojQoaY9kQL00fPGo34AzOn8HB7Z6nryRrndt3fpgqNf16P/QcAPgLgOw7WQgipAVN7diA+IAM8duxcpbp2G/J0yJyZ62DXQ09j+/4j2L7/CCY+8zTGxxqx2xapzMll2FX1BVUtX8dECPGW9kTLMAm1mlh7VgYN4czMdTD11ZOY75kedWGhi9cWu2iMrr/VFalhByh3JIQUQMsQzgDchCCK7v8ySAhn+uhZdJc33tKWFXj7FZtw1ZWbSutXk2rYReSvALwr5k+fVtW/tH0jEbkXwL0AsG3bNusFEkLCY2rPDuw7fCLWc88bgvBhpmgcSTes1xYB8ZhnAAAJHElEQVS7pQ73TjXsqvp+F2+kqo8AeARY6e7oYp+EED9pT7Qw+9LP8dixc+uMu4sQhEllcv/jJ9feuwpMSdfob2VCuSMhpBAebu/Eob27nJfRmzzjS6rr9OFlFwWZdPyNUSk0nh5HXrnj7wD4DwC2AjgiIidUdY+TlRFCgqeIsvokz7hXH152uCba74EnT68lULeMNfDgXTeX/hTBQRuEkKCIG4/XiwAYH2vEdnb0YfBHHjhogxBSS6L4/aPHzsX+XQEn7XqrmLzkChp2QkhwPHvm/ECvs01i+qq8sYXJU0JIcAyqhbdNYlbR38UlNOyEkOAYRD6Y1q63F19nuNpCw04ICQ6b0XO92LTr7cV04yhbjz4oNOyEkODo7+cy3mxs6McS/TSIfj5PIzAfYPKUEBIk/Rp5lyqW6HWhqmKoYyeE1IqQZYppUMdOCBk6QpcpuoKGnRBSG5JkioMY9lC9fxp2QkjQ9BpfU2B5EJliyN4/DTshJFjS+sZEDCJTzOL9++bZ07ATQoIlzvj2I7CvOO3FtkjJR8+eOnZCSLDYhFgUgxlYk5c/IrKut7uP7Qdo2AkhwWITYmkNWC1qqm7tH+hhurmYesaXAQ07ISRY0loL5KkWjapbR2XjVKRej9x0cxGg8KlNJmjYCSHBEhnf8WZj7XfRdDoXo/jaEy0sG4o4I099as8ObDT9KyGgqsIxNOyEkOB5a2l57d/LetlTd5G8TGsI1p5oOZVZuoCGnRASNEUnL20agpni+FV1g6RhJ4QETdG90/s7ScaFeHzrBkkdOyEkaK4bb8YqUFx6y/2dJOP+DvjTDZKGnRASNFN7dmyoPq3CW04z/mVCw04ICRrfvGUfoGEnhARPld6yb31iABp2QggZGB/7xAA07IQQkpnIS49L2ubp/+4KGnZCCMmATavgqgqTIqhjJ4SQDNi0Cq6qMCmChp0QQjKQ5o1XWZgUQcNOCCEZSPLGXTQec0Euwy4i0yJyRkS+LyLfEJFxVwsjhBAfMbUP+PzeXXhu/x2VG3Ugv8f+bQDvUdX3AvgbAJ/KvyRCCPGPmbkObjv4DPYdPoErN41gy1jD2DumanKpYlT16Z4fjwH4aL7lEEKIf/QrYeYXu2g2RnFo7y6vDHqES7njvwBw2OH+CCHEC0ytgQ88edq7qlPAwrCLyF8BeFfMnz6tqn+5us2nASwBeCxhP/cCuBcAtm3bNtBiCSGkCkxKmPnFLuYXuwD8qToFLAy7qr4/6e8i8s8AfAjAb6oaZkit7OcRAI8AwOTkpHE7QgjxiZm5DkZEcMls3tbwoeoUyBmKEZEPAvj3AP6xqi64WRIhhPhBFFu3MeoRVVedAvlVMf8RwNsAfFtETojIf3GwJkII8QJTlemoCLaMNWJeUX3VKZDTsKvqL6nqDaq6a/W/f+1qYYQQUjUm73tZFQ/edbNX4/B6YeUpIYQYMHnfCuDAk6chuByi2TLW8EbPTsNOCCEG4qpMI+YXu1joLq/9/GbPv6uGhp0QQgy0J1r47Ed2omURN48UMT5Aw04IIQm0J1p4bv8dEIttfVDEADTshBBihY3axQdFDEDDTgghViTF2wF/FDEAR+MRQogVkdol6g1zTbMBEWB+oetVnxiAhp0QQqxpT7S8Md5JMBRDCCE1g4adEEJqBkMxhBCSgZm5jpc92HuhYSeEEEv6Jyn51IO9F4ZiCCHEEtMkJV8qTiNo2AkhxBJTZakvFacRNOyEEGKJqbLUl4rTCBp2QgixJK761KeK0wgmTwkhxJL+6lOqYgghpAaEUH3KUAwhhNQMGnZCCKkZNOyEEFIzaNgJIaRm0LATQkjNEFUt/01FzgN4acCXvxPA3zpcThWE/hm4/uoJ/TOEvn6gms/wblXdmrZRJYY9DyIyq6qTVa8jD6F/Bq6/ekL/DKGvH/D7MzAUQwghNYOGnRBCakaIhv2RqhfggNA/A9dfPaF/htDXD3j8GYKLsRNCCEkmRI+dEEJIAkEZdhH5oIicFZEficj+qteTFRH5koj8TER+UPVaBkFEbhCRZ0XkBRE5LSJ/WPWasiAim0Xk/4jIydX1P1T1mgZBREZFZE5Evln1WgZBRH4iIqdE5ISIzFa9nqyIyLiIfE1EzqxeC/+o6jX1E0woRkRGAfwNgN8C8AqA7wH4hKr+sNKFZUBEfgPA6wD+QlXfU/V6siIi1wK4VlWfF5G3ATgOoB3KdyAiAuAqVX1dRBoA/hrAH6rqsYqXlgkR+bcAJgG8XVU/VPV6siIiPwEwqapB6thF5M8B/C9V/YKIXAFgTFXnq15XLyF57O8D8CNV/bGqXgTwFQC/XfGaMqGq3wHw86rXMSiq+lNVfX71338P4AUAfvcv7UFXeH31x8bqf2F4NquIyPUA7gTwharXMoyIyNsB/AaALwKAql70zagDYRn2FoCXe35+BQEZlbohItsBTAD4brUrycZqGOMEgJ8B+LaqBrV+AJ8H8O8ALFe9kBwogKdF5LiI3Fv1YjLyCwDOA/ivq+GwL4jIVVUvqp+QDLvE/C4ob6suiMjVAJ4AcJ+q/l3V68mCql5S1V0ArgfwPhEJJiQmIh8C8DNVPV71WnJym6r+KoB/CuDfrIYoQ2ETgF8F8J9VdQLAGwC8y/eFZNhfAXBDz8/XA3i1orUMLaux6ScAPKaqX696PYOy+vj8PwF8sOKlZOE2AB9ejVF/BcAdIvJotUvKjqq+uvr/nwH4BlbCrKHwCoBXep70voYVQ+8VIRn27wH4ZRG5cTVh8bsAnqx4TUPFavLxiwBeUNU/rXo9WRGRrSIyvvrvJoD3AzhT7arsUdVPqer1qrodK+f/M6r6exUvKxMictVq4h2rIYwPAAhGJaaq/xfAyyISTa/+TQDeiQeCmXmqqksi8vsAjgIYBfAlVT1d8bIyISJfBvBPALxTRF4B8KCqfrHaVWXiNgCfBHBqNU4NAH+kqt+qcE1ZuBbAn68qrEYAPK6qQUoGA+YfAPjGio+ATQD+u6r+j2qXlJk/APDYqoP5YwD/vOL1bCAYuSMhhBA7QgrFEEIIsYCGnRBCagYNOyGE1AwadkIIqRk07IQQUjNo2AkhpGbQsBNCSM2gYSeEkJrx/wE9KWMh+9Jd+AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "npts = 300\n",
    "eps = 0.2  # noise\n",
    "a, b, c  = 1, -0.2, 1 # model coefficients\n",
    "\n",
    "np.random.seed(1234)\n",
    "x = np.linspace(0, 2*np.pi, npts)\n",
    "y = a*x + b*x**2 + c*np.sin(x**2) + np.random.normal(scale=eps, size=npts)\n",
    "plt.plot(x, y, 'o');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write it to a CSV file for convenient retrieval in a \"typical\" workflow, Pandas does the job nicely:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.094287</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.021014</td>\n",
       "      <td>-0.216828</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.042028</td>\n",
       "      <td>0.329982</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          x         y\n",
       "0  0.000000  0.094287\n",
       "1  0.021014 -0.216828\n",
       "2  0.042028  0.329982"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.DataFrame({'x':x, 'y':y})\n",
    "data.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x,y\n",
      "0.0,0.09428703274649862\n",
      "0.02101399768287487,-0.21682787079387694\n"
     ]
    }
   ],
   "source": [
    "data.to_csv('data.csv', index=False)\n",
    "!head -3 data.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sanity check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.094287</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.021014</td>\n",
       "      <td>-0.216828</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.042028</td>\n",
       "      <td>0.329982</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          x         y\n",
       "0  0.000000  0.094287\n",
       "1  0.021014 -0.216828\n",
       "2  0.042028  0.329982"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data2 = pd.read_csv('data.csv')\n",
    "data2.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "x    5.894937e-14\n",
       "y    1.431537e-14\n",
       "dtype: float64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(data2-data).abs().sum()"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernel_info": {
   "name": "python3"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "nteract": {
   "version": "0.8.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

polyglot-ds.ipynb

This code is part of a demo presented during a lecture at BIDS, the Berkeley Institute for Data Science.

This is really nice work! I look forward to more of this kind of interoperability.

Nice work, thanks !

• I had to install PyCall in julia with Pkg.add("PyCall")
• Minor error in cell [25] sin(x**2) instead of sin(xx**2)