cython tricks

cython_tricks.md

Cython

Cython has two major benefits:

1. Making python code faster, particularly things that can't be done in scipy/numpy
2. Wrapping/interfacing with C/C++ code

Cython gains most of it's benefit from statically typing arguments. However, statically typing is not required, in fact, regular python code is valid cython (but don't expect much of a speed up). By incrementally adding more type information, the code can speed up by several factors. This gist just provides a very basic usage of cython.

For a video tutorial vist http://conference.scipy.org/scipy2013/tutorial_detail.php?id=105

Cython install

$ pip install cython

.pyx files

.pyx files contain cython code. If you want the function to be visible from python then define the function with def keyword.

def hello_world():
    print("hello world")

Functions can also have local scope by using the cdef keyword in the function signature:

cdef int return_one():
    cdef int a = 1
    return a

Notice cdef is also used in a different context to declare static typing. You can declare several statically type variables in this way:

def myfunc():
    cdef:
        int a = 1, b = 2
        double c = 3.0
        char *d
    return a + b + c

A valid type in C/C++ should be available after using the cdef key word. Additionally, function parameters can be statically typed.

def myfunc(int n):
    return n

Notice that only for function parameters that the keyword cdef can be left off.

Using external C/C++ modules

To import a function from a header file we need to use the extern keyword:

cdef extern from "string.h":
    # describes the interface for the function used
    int strlen(char *c)

# since cdef functions can't be used from python, we need to define a `def` wrapper function
def get_len(char *message):
    return strlen(message)

Basic STL data structures can also be imported by using cimport

from libcpp.map cimport map
from libcpp.vector cimport vector

def myfunc():
    cdef: 
        map[int, int] a  # map between ints
        vector[int] vect = xrange(1, 10, 2)  # basic int vector

Importing a Cython Module

There are two ways to import a cython module:

1. The easy way -- using pyximport to auto-compile code
2. The harder way -- using disutils with a setup.py file

Pyximport

# easy way to import .pyx files, auto-compiles
import pyximport; pyximport.install()
import mycython_module

Setup.py and Disutils

Importing after creating a setup.py script is very straight forward:

import mycython_module

However, you need to create a setup.py that compiles your code into an extension module. This takes more effort than using pyximport which just does this for you. The setup.py/disutils method does provide more flexibility, however, which you will see below.

This is a standard setup.py file that will build the extension module (Requires Installed Cython):

from disutils.core import setup
from disutils.extension import Extension
from Cython.Disutils import build_ext

setup(
      cmdclass = {'build_ext': build_ext},
      ext_modules = [Extension("hello_world",
                               [hello_world.pyx"],
                               language='c++',
                               include_dir=[...])]
)

Building the extension module:

$ python setup.py build_ext --inplace

However people receiving the code will likely not have Cython installed. To avoid this issue, just provide the .c or .cpp files so that the user just needs to compile the C/C++ code. Notice since cython already generate the c/cpp files, the user does not need cython. A minor change to the setup.py script will allow the user to compile directly from the generated .c or .cpp files.

from distutils.core import setup
from distutils.extension import Extension
import sys

if '--use-cython' in sys.argv:
    USE_CYTHON = True
    sys.argv.remove('--use-cython')
else:
    USE_CYTHON = False
ext = '.pyx' if USE_CYTHON else '.cpp'
extensions = [Extension("src.python.cython_utils",
                        ["src/python/cython_utils"+ext],
                        language='c++',
                        include_dirs=['src/cpp/'])]

if USE_CYTHON:
    from Cython.Build import cythonize
    extensions = cythonize(extensions)

setup(
    ext_modules = extensions
)

Using the above setup.py script will be default just compile the specified .cpp files using the same command as before:

$ python setup.py build_ext --inplace

However if they do have cython (like a developer), then they can use the --use-cython CLI option to both generate the .cpp files and compile them.

$ python setup.py build_ext --inplace --use-cython

Makefile for streamlining build

I use a Makefile to specifically clean the cython build products. The clean* commands remove the results of a single cython extension called cython_utils.pyx. The build* commands will build from both the .pyx file (cython-build) and .c/.cpp files (build). In my case, I had some unit tests in the tests directory so I made an additional command to run nosetests from the python nose library to execute tests. I use the hash linux command to find if the nosetests command is missing.

clean:
	rm -f -r build/
	rm -f src/python/cython_utils.so

clean-cpp:
	rm -f src/python/cython_utils.cpp

clean-all: clean clean-cpp

.PHONY: build
build: clean
	python setup.py build_ext --inplace

.PHONY: cython-build
cython-build: clean clean-cpp
	python setup.py build_ext --inplace --use-cython

.PHONY: tests
tests:
	hash nosetests 2>/dev/null || { echo -e >&2 "############################\nI require the python library \"nose\" for unit tests but it's not installed.  Aborting.\n############################\n"; exit 1; } 
	nosetests --logging-level=INFO tests/

With the above Makfile, users can just type make build if I distribute the C/C++ files with the software.

$ make build

If C/C++ files aren't distributed, then a full build needing Cython is required:

$ make cython-build

They can then check if the unit tests pass:

$ make tests