PkgIt Notebook

PkgIt presentation.ipynb

{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "PkgIt"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Packaging tool"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Packaging problem"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* Softwares use many dependencies (Qt4, PyQt4, cgal, ann, numpy, scipy, matplotlib\u2026).\n",
      "* Most of the time, we need to use versions packaged with the **same** compiler.\n",
      "\n",
      "So we need a tool:\n",
      "\n",
      "* to download, install, compile and package dependencies\n",
      "* to easily create an installer.\n",
      "* automatic\n",
      "* reproducible\n"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Solutions on different OS\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "* Ubuntu:\n",
      "    + apt-get, aptitude\n",
      "* Fedora: \n",
      "    + yum\n",
      "* Mac OS X: \n",
      "    + MacPort\n",
      "    + HomeBrew\n",
      "* Windows:\n",
      "    + Nothing today!\n"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "PkgIt solution"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "> pkgit --help"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from pkgit.easy_pkg import parse\n",
      "parse().print_help()"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "usage: pkgit [-h] [-v] [-p PACKAGE] [-i INSTALL] [-w WININST] [-c CREATE]\n",
        "             [-y PACKAGED] [--without WITHOUT] [--deps DEPS] [--versions]\n",
        "             [--force] [--dest-dir DEST_DIR] [--rm-tmp] [--rm-tmp-all]\n",
        "             [--dry-run]\n",
        "\n",
        "Permit to package modules thanks to formulas. Main commands are --package,\n",
        "--wininst, --create (package existing formula, create windows installer,\n",
        "create new formula).\n",
        "\n",
        "optional arguments:\n",
        "  -h, --help            show this help message and exit\n",
        "  -v, --version         show program's version number and exit\n",
        "  -p PACKAGE, --package PACKAGE\n",
        "                        Package formula named PACKAGE. Available formulas are:\n",
        "                        'alinea', 'ann', 'bisonflex', 'boost', 'cgal',\n",
        "                        'cmake', 'configobj', 'dateutil', 'gnuplot', 'inno',\n",
        "                        'ipython', 'lpy', 'matplotlib', 'mingw', 'mingw_rt',\n",
        "                        'mtg', 'numpy', 'oalab', 'openalea', 'pillow',\n",
        "                        'plantgl', 'pygments', 'pylsm', 'pyopengl',\n",
        "                        'pyparsing', 'pyqglviewer', 'pyqscintilla', 'pyqt4',\n",
        "                        'python', 'pywin', 'qglviewer', 'qhull', 'qscintilla',\n",
        "                        'qt4', 'qt4_dev', 'r', 'rpy2', 'scipy', 'scons',\n",
        "                        'setuptools', 'sip', 'skimage', 'svn', 'vplants'.\n",
        "  -i INSTALL, --install INSTALL\n",
        "                        Package formula named INSTALL if necessary and install\n",
        "                        it.\n",
        "  -w WININST, --wininst WININST\n",
        "                        Create Windows installer for formula WININST.\n",
        "  -c CREATE, --create CREATE\n",
        "                        Create a new formula named CREATE.\n",
        "  -y PACKAGED, --packaged PACKAGED\n",
        "                        Display if formula is yet packaged.\n",
        "  --without WITHOUT     Omit to package WITHOUT and WITHOUT's dependencies.\n",
        "                        Works with --package\n",
        "  --deps DEPS           Display dependencies of formula named DISP_DEPS.\n",
        "  --versions            Display all formulas available and versions of\n",
        "                        packages.\n",
        "  --force               Force packaging. Works with --package\n",
        "  --dest-dir DEST_DIR   copy result to DIR. Works with --package\n",
        "  --rm-tmp              Remove temporary files after packaging(except\n",
        "                        download). Works with --package\n",
        "  --rm-tmp-all          Remove all temporary files after packaging. Works with\n",
        "                        --package\n",
        "  --dry-run             Don't do anything. Works with --package\n"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "pkgit --package\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "> pkgit --package cgal\n",
      "\n",
      "* Each formula come from class Formula in file formula.py\n",
      "* A formula has predefined methods and flags. When you use \u201ceasy_pkg package something\u201d, following methods are called if corresponding flags are set to True.\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Methods (in the good order) and corresponding flags:\n",
      "\n",
      "* download()\t\t(DOWNLOAD) \n",
      "* unpack()\t\t(UNPACK)\n",
      "* install()     \t(INSTALL)\n",
      "* configure()\t\t(CONFIGURE) \n",
      "* make()\t\t(MAKE) \n",
      "* make_install()\t(MAKE_INSTALL)\n",
      "* bdist_egg()  \t(EGGIFY) \n",
      "* copy_installer() \t(COPY_INSTALLER) \n",
      "* post_install()\t(POST_INSTALL) \t"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "\n",
      "Options:\n",
      "\n",
      "* --without WITHOUT\n",
      "* --force\n",
      "* --dest-dir DEST_DIR\n",
      "* --rm-tmp\n",
      "* --rm-tmp-all\n",
      "* --dry-run\t\n"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "\n",
      "pkgit --create"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Create a new formula.\n",
      "\n",
      "One dependency = One formula file \n",
      "\n",
      "To create a new formula to dependency \u201cmy_amazing_package\u201d:\n",
      "\n",
      "> pkgit --create my_amazing_package\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#my_amazing_package.py\n",
      "\n",
      "from openalea.pkgit.formula import Formula\n",
      "\n",
      "class My_amazing_package(Formula):\n",
      "    download_url = \"http://...\"\n",
      "    download_name = \"my_amazing_package.zip\"\n",
      "    version = \"1.42.0\"\n"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "pkgit --install\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Package formula withh 'pkgit --package' and then install it.\n",
      "\n",
      "> pkgit --install ann\n",
      "\n"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "pkgit --versions"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Dsiplay available formulas and corresponding versions.\n",
      "\n",
      "> pkgit --versions"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from pkgit.utils import versions\n",
      "versions()"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 1,
       "text": [
        "['alinea            1.1',\n",
        " 'ann               1.1.2',\n",
        " 'bisonflex         2.4.1',\n",
        " 'boost             1.48.0',\n",
        " 'cgal              4.2',\n",
        " 'cmake             2.8.11.2',\n",
        " 'configobj         4.7.2',\n",
        " 'dateutil          2.2',\n",
        " 'gnuplot           4.6.3',\n",
        " 'inno              5.5.3',\n",
        " 'ipython           1.1.0',\n",
        " 'lpy               1.1',\n",
        " 'matplotlib        1.3.1',\n",
        " 'mingw             5.1.4_4c',\n",
        " 'mingw_rt          5.1.4_4c',\n",
        " 'mtg               1.1',\n",
        " 'numpy             1.8.0',\n",
        " 'oalab             0.1',\n",
        " 'openalea          1.1',\n",
        " 'pillow            2.1.0',\n",
        " 'plantgl           1.1',\n",
        " 'pygments          1.6',\n",
        " 'pylsm             0.1-r34',\n",
        " 'pyopengl          3.0.2',\n",
        " 'pyparsing         1.5.7',\n",
        " 'pyqglviewer       0.11',\n",
        " 'pyqscintilla      1.0',\n",
        " 'pyqt4             4.10.3',\n",
        " 'python            2.7.6',\n",
        " 'pywin             218',\n",
        " 'qglviewer         1.0',\n",
        " 'qhull             2012.1',\n",
        " 'qscintilla        2.6.2',\n",
        " 'qt4               4.8.5',\n",
        " 'qt4_dev           4.8.5',\n",
        " 'r                 2.15.3',\n",
        " 'rpy2              2.3.6',\n",
        " 'scipy             0.13.2',\n",
        " 'scons             2.3.0',\n",
        " 'setuptools        0.6c11',\n",
        " 'sip               4.15.2',\n",
        " 'skimage           0.9.3',\n",
        " 'svn               1.8.5',\n",
        " 'vplants           1.1']"
       ]
      }
     ],
     "prompt_number": 1
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "pkgit --deps"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Display dependencies of selected formula\n",
      "\n",
      "> pkgit --deps openalea"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from pkgit.utils import deps\n",
      "deps(\"openalea\")"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 3,
       "text": [
        "['python',\n",
        " 'pywin',\n",
        " 'mingw',\n",
        " 'mingw_rt',\n",
        " 'qt4',\n",
        " 'numpy',\n",
        " 'scipy',\n",
        " 'matplotlib',\n",
        " 'qscintilla',\n",
        " 'pyqscintilla',\n",
        " 'setuptools',\n",
        " 'pillow',\n",
        " 'pylsm',\n",
        " 'svn',\n",
        " 'dateutil',\n",
        " 'pygments',\n",
        " 'ipython',\n",
        " 'inno',\n",
        " 'openalea']"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "pkgit --packaged"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Check if formula named \u201cformula_name\u201d is yet packaged (locally).\n",
      "\n",
      "> pkgit --packaged pyqscintilla"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from pkgit.utils import packaged\n",
      "packaged(\"pyqscintilla\")"
     ],
     "language": "python",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "pyqscintilla NOT packaged\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 4,
       "text": [
        "False"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "TODO"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "1. Increase robustness (create release openalea/vplants/alinea 1.1 and test it!)\n",
      "2. Adapt system for Multi-OS\n"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Other OS"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {
      "slideshow": {
       "slide_type": "fragment"
      }
     },
     "source": [
      "Proposed solutions:\n",
      "\n",
      "1. Use only existing systems : HomeBrew, MacPorts, Apt-get, Aptitude, Yum. No easy_pkg\u2026 Pb: need to invest in other systems.\n",
      "2. Use only easy_pkg. Pb: re-invent the wheel! (Bad idea !)\n",
      "3. **Mix: Use existing systems in referencing in easy_pkg exact names / commands.** Here, we have a single formula file but 1 class by os with a generator which choose the good class.\n"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}