JSS autowig examples

basic.ipynb

dependent.ipynb

index.ipynb

run.bat

echo OFF

echo "AutoWIG Examples"
echo "================"
echo ""
echo "This script will run the four examples presented in the"
echo "accompanying paper. Corresponding Jupyter notebooks can"
echo "also be run individually, echo using, the following"
echo "commands,"
echo ""

echo ON

jupyter nbconvert --to notebook --execute basic.ipynb --allow-errors --inplace --ExecutePreprocessor.timeout=3600 --NotebookApp.kernel_spec_manager_class='environment_kernels.EnvironmentKernelSpecManager'
jupyter nbconvert --to notebook --execute subset.ipynb --allow-errors --inplace --ExecutePreprocessor.timeout=3600 --NotebookApp.kernel_spec_manager_class='environment_kernels.EnvironmentKernelSpecManager'
jupyter nbconvert --to notebook --execute template.ipynb --allow-errors --inplace --ExecutePreprocessor.timeout=3600 --NotebookApp.kernel_spec_manager_class='environment_kernels.EnvironmentKernelSpecManager'
jupyter nbconvert --to notebook --execute dependent.ipynb --allow-errors --inplace --ExecutePreprocessor.timeout=3600 --NotebookApp.kernel_spec_manager_class='environment_kernels.EnvironmentKernelSpecManager'

echo OFF

echo "In order to visualize the Jupyter notebooks execute the "
echo "folowing command"
echo ""
echo "jupyter notebook index.ipynb"

run.sh

set +x
set +e

echo "AutoWIG Examples"
echo "================"
echo ""
echo "This script will run the four examples presented in the"
echo "accompanying paper. Corresponding Jupyter notebooks can"
echo "also be run individually, echo using, the following"
echo "commands,"
echo ""

set -x

# git clone http://github.com/StatisKit/AutoWIG.git --depth=1
# mv AutoWIG/doc/examples/basic basic
# rm -rf AutoWIG
# jupyter nbconvert --to notebook --execute basic.ipynb --allow-errors --inplace --ExecutePreprocessor.timeout=3600 --NotebookApp.kernel_spec_manager_class='environment_kernels.EnvironmentKernelSpecManager'
# git clone http://github.com/StatisKit/PyClangLite.git --depth=1
# jupyter nbconvert --to notebook --execute subset.ipynb --allow-errors --inplace --ExecutePreprocessor.timeout=3600 --NotebookApp.kernel_spec_manager_class='environment_kernels.EnvironmentKernelSpecManager'
# rm -rf PySTL
# git clone http://github.com/StatisKit/PySTL.git --depth=1
# jupyter nbconvert --to notebook --execute template.ipynb --allow-errors --inplace --ExecutePreprocessor.timeout=3600 --NotebookApp.kernel_spec_manager_class='environment_kernels.EnvironmentKernelSpecManager'
git clone http://github.com/pfernique/StructureAnalysis.git
cd StructureAnalysis
git checkout -b remove_tool origin/remove_tool
cd ..
jupyter nbconvert --to notebook --execute dependent.ipynb --allow-errors --inplace --ExecutePreprocessor.timeout=3600 --NotebookApp.kernel_spec_manager_class='environment_kernels.EnvironmentKernelSpecManager'

set -e
set +x

echo "In order to visualize the Jupyter notebooks execute the "
echo "folowing command"
echo ""
echo "jupyter notebook index.ipynb"

subset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wrapping a subset of a very large library\n",
    "\n",
    "Sometimes, for a very large library, only a subset of available *C++* components is useful for end-users.\n",
    "Wrapping such libraries therefore requires **AutoWIG** to be able to consider only a subset of the *C++* components during the `Generate` step.\n",
    "The **Clang** library is a complete *C*/*C++* compiler.\n",
    "**Clang** is a great tool, but its stable *Python* interface (i.e., **libclang**) is lacking some useful features that are needed by **AutoWIG**.\n",
    "In particular, class template specializations are not available in the abstract syntax tree.\n",
    "Fortunately, most of the classes that would be needed during the traversal of the *C++* abstract syntax tree are not template specializations.\n",
    "We therefore proposed to bootstrap the **Clang** *Python* bindings using the `libclang parser` of **AutoWIG**.\n",
    "This new **Clang** *Python* interface is called **PyClangLite** and is able to parse class template specializations.\n",
    "As for **libclang**, this interface is proposed only for a subset of the **Clang** library sufficient enough for proposing the new `pyclanglite parser`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preamble\n",
    "\n",
    "The source code is available in the `PyClangLite` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!pygmentize PyClangLite/src/cpp/tool.h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This directory already contains wrappers, we therefore need to remove them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from path import path\n",
    "srcdir = path('PyClangLite')/'src'/'py'\n",
    "for wrapper in srcdir.walkfiles('*.cpp'):\n",
    "    wrapper.unlink()\n",
    "for wrapper in srcdir.walkfiles('*.h'):\n",
    "    wrapper.unlink()\n",
    "wrapper = srcdir/'clanglite'/'_clanglite.py'\n",
    "if wrapper.exists():\n",
    "    wrapper.unlink()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we need to install and compile the *C++* library.\n",
    "This is done using available **Conda** recipes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!conda build PyClangLite/conda/libclanglite -c statiskit -c conda-forge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapping the *C++* library\n",
    "\n",
    "Once these preliminaries done, we can proceed to the actual generation of wrappers for the **ClangLite** *C++* library.\n",
    "For this, we import **AutoWIG** and create an empty Abstract Semantic Graph (ASG)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import autowig\n",
    "asg = autowig.AbstractSemanticGraph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we parse headers with relevant compilation flags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "import sys\n",
    "prefix = path(sys.prefix)\n",
    "autowig.parser.plugin = 'libclang'\n",
    "asg = autowig.parser(asg, [prefix/'include'/'clanglite'/'tool.h'],\n",
    "               flags = ['-x', 'c++', '-std=c++11',\n",
    "                        '-D__STDC_LIMIT_MACROS',\n",
    "                        '-D__STDC_CONSTANT_MACROS',\n",
    "                        '-I' + (prefix/'include').abspath()],\n",
    "               libpath = prefix/'lib'/'libclang.so',\n",
    "               bootstrap = False,\n",
    "               silent = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since most of **AutoWIG** guidelines are respected in the **Clang** library, the `default` `controller` implementation could be suitable.\n",
    "Nevertheless, since we only need to wrap a subset of the *C++* library, we need to implements a new `controller` which explicits *C++* components that should be wrapped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def clanglite_controller(asg):\n",
    "    \n",
    "    for node in asg['::boost::python'].classes(nested = True):\n",
    "        node.is_copyable = True\n",
    "        \n",
    "    for node in asg.classes():\n",
    "        node.boost_python_export = False\n",
    "    for node in asg.enumerations():\n",
    "        node.boost_python_export = False\n",
    "    for node in asg.enumerators():\n",
    "        if node.parent.boost_python_export:\n",
    "            node.boost_python_export = False\n",
    "    for node in asg.typedefs():\n",
    "        node.boost_python_export = False\n",
    "            \n",
    "    from autowig.default_controller import refactoring\n",
    "    asg = refactoring(asg)\n",
    "    for fct in asg['::clanglite'].functions():\n",
    "        if not fct.localname == 'build_ast_from_code_with_args':\n",
    "            fct.parent = fct.parameters[0].qualified_type.desugared_type.unqualified_type\n",
    "            \n",
    "    for node in asg.functions(free = True):\n",
    "        node.boost_python_export = False\n",
    "    for node in asg.variables(free = True):\n",
    "        node.boost_python_export = False\n",
    "    \n",
    "    asg['class ::boost::python::api::object'].boost_python_export = True\n",
    "    asg['class ::boost::python::list'].boost_python_export = True \n",
    "    asg['class ::boost::python::str'].boost_python_export = True \n",
    "\n",
    "    subset = []\n",
    "    classes = [asg['class ::clang::QualType'],\n",
    "               asg['class ::clang::Type'],\n",
    "               asg['class ::clang::Decl']]\n",
    "    asg['class ::clang::QualType'].is_abstract = False\n",
    "    asg['class ::clang::QualType'].is_copyable = True\n",
    "    subset += classes\n",
    "    for cls in classes:\n",
    "        subset += cls.subclasses(recursive=True)\n",
    "    subset.append(asg['class ::llvm::StringRef'])\n",
    "    asg['class ::llvm::StringRef'].is_abstract = False\n",
    "    asg['class ::llvm::StringRef'].is_copyable = True\n",
    "    subset.append(asg['class ::clang::ASTUnit'])\n",
    "    subset.append(asg['class ::clang::ASTContext'])\n",
    "    subset.append(asg['class ::clang::SourceManager'])\n",
    "    for mtd in asg['class ::clang::ASTContext'].methods(pattern='.*getSourceManager.*'):\n",
    "        if mtd.return_type.globalname == 'class ::clang::SourceManager &':\n",
    "                mtd.boost_python_export = True\n",
    "                break\n",
    "    subset.append(asg['class ::clang::FileID'])\n",
    "    asg['class ::clang::FileID'].is_abstract = False\n",
    "    asg['class ::clang::FileID'].is_copyable = True\n",
    "    subset.append(asg['class ::clang::SourceLocation'])\n",
    "    asg['class ::clang::SourceLocation'].is_abstract = False\n",
    "    asg['class ::clang::SourceLocation'].is_copyable = True\n",
    "    subset.append(asg['class ::clang::CXXBaseSpecifier'])\n",
    "    subset.append(asg['class ::clang::DeclContext'])\n",
    "    subset.append(asg['class ::clang::TemplateArgument'])\n",
    "    asg['class ::clang::TemplateArgument'].is_abstract = False\n",
    "    asg['class ::clang::TemplateArgument'].is_copyable = True\n",
    "    subset.append(asg['class ::clang::TemplateArgumentList'])\n",
    "    subset.append(asg['enum ::clang::Type::TypeClass'])\n",
    "    subset.append(asg['enum ::clang::AccessSpecifier'])\n",
    "    subset.append(asg['enum ::clang::LinkageSpecDecl::LanguageIDs'])\n",
    "    subset.append(asg['enum ::clang::BuiltinType::Kind'])\n",
    "    subset.append(asg['enum ::clang::TemplateArgument::ArgKind'])\n",
    "    subset.append(asg['enum ::clang::Decl::Kind'])\n",
    "    subset.extend(asg['::boost::python'].classes(nested = True))\n",
    "    subset.extend(asg['::boost::python'].enumerations(nested = True))\n",
    "    subset.extend(asg.nodes('::clanglite::build_ast_from_code_with_args'))\n",
    "\n",
    "    for node in subset:\n",
    "        node.boost_python_export = True\n",
    "\n",
    "    if autowig.parser.plugin == 'libclang':\n",
    "        for node in (asg.functions(pattern='.*(llvm|clang).*_(begin|end)')\n",
    "                     + asg.functions(pattern='.*(llvm|clang).*getNameAsString')\n",
    "                     + asg.nodes('::clang::NamedDecl::getQualifiedNameAsString')\n",
    "                     + asg.nodes('::clang::ObjCProtocolDecl::collectInheritedProtocolProperties')\n",
    "                     + asg.nodes('::clang::ASTUnit::LoadFromASTFile')\n",
    "                     + asg.nodes('::clang::ASTUnit::getCachedCompletionTypes')\n",
    "                     + asg.nodes('::clang::ASTUnit::getBufferForFile')\n",
    "                     + asg.nodes('::clang::CXXRecordDecl::getCaptureFields')\n",
    "                     + asg.nodes('::clang::ASTContext::SectionInfos')\n",
    "                     + asg.nodes('::clang::ASTContext::getAllocator')\n",
    "                     + asg.nodes('::clang::ASTContext::getObjCEncoding.*')\n",
    "                     + asg.nodes('::clang::ASTContext::getAllocator')\n",
    "                     + asg.nodes('::clang::QualType::getAsString')\n",
    "                     + asg.nodes('::clang::SourceLocation::printToString')\n",
    "                     + asg['class ::llvm::StringRef'].methods()):\n",
    "            node.boost_python_export = False\n",
    "            \n",
    "    import sys\n",
    "    from path import path\n",
    "    for header in (path(sys.prefix)/'include'/'clang').walkfiles('*.h'):\n",
    "        asg[header.abspath()].is_external_dependency = False\n",
    "    \n",
    "    return asg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This `controller` is then dynamically registered and used on the ASG."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "autowig.controller['clanglite'] = clanglite_controller\n",
    "autowig.controller.plugin = 'clanglite'\n",
    "asg = autowig.controller(asg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the libarary subset has been defined, we need to select the `boost_python_pattern` `generator` implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "autowig.generator.plugin = 'boost_python_pattern'\n",
    "wrappers = autowig.generator(asg,\n",
    "                  module = srcdir/'_clanglite.cpp',\n",
    "                  decorator = srcdir/'clanglite'/'_clanglite.py',\n",
    "                  closure = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The wrappers are only generated in-memory.\n",
    "It is therefore needed to write them on the disk to complete the process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "wrappers.write()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is an example of the generated wrappers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!pygmentize PyClangLite/src/py/_clanglite.cpp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the wrappers are written on disk, we need to compile and install the *Python* bindings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!conda build PyClangLite/conda/python-clanglite -c statiskit -c conda-forge\n",
    "!conda install python-clanglite --use-local -c statiskit -c conda-forge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the *C++* library *Python* bindings\n",
    "\n",
    "Finally, we can hereafter use the *C++* library in the *Python* interpreter.\n",
    "For examples take a look on the following examples using the `pyclanglite parser` implementation:\n",
    "\n",
    "* [Wrapping a template library](template.ipynb)\n",
    "* [Wrapping dependent libraries](dependent.ipynb)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}

template.ipynb