Hybrid programming with C++ and Python

PyConZA 2012 - Cape TownJames Saunders - BusinessOptics

Overview

● Talk should be:○ Not super technical○ Some C++ knowledge required○ No advanced understanding of python required

● Won't dive too deeply into specific code● Will try to be pragmatic● Not exhaustive● Should give an idea of how to expose a

complex C++ library to python and vice versa

Some ProvisosI am talking about cPython (not applicable to other python implementations)

We use linux (though most things are cross platform)

Compile our code with GCC (Which shouldn't matter but probably does sometimes)

Mileage may vary (Like always)

Why would you want to do this?

● Write performance intensive code in C++, but use Python for everything else

● Use cool libraries in C++/Python from Python/C++

● Embed a scripting language● Easily extend complex behaviours in Python● Interact with legacy code (especially when

webifying something)

Our Problem

Django Webstack

C++ Computation Engine

Machine learning in scikit-learn

Using C++ in python (extending)

class FortuneTeller {

public:

FortuneTeller(int luckiness);

vector<int> get_lottery();}

from mycpplib import FortuneTeller

obj = FortuneTeller(5)

lucky_numbers = obj. get_lottery()

for num in lucky_numbers:

print num

The Fundamentals

● Python extension modules are shared libraries○ *.so in linux○ *.dll in windows○ I know nothing about Macs

● cPython is written in C and has "native" support for being extended in C

● Accessed through Python.h

Python.hstatic PyObject *my_callback = NULL;

static PyObject *my_set_callback(PyObject *dummy, PyObject *args){ PyObject *result = NULL; PyObject *temp;

if (PyArg_ParseTuple(args, "O:set_callback", &temp)) { if (!PyCallable_Check(temp)) { PyErr_SetString(PyExc_TypeError, "parameter must be callable"); return NULL; } Py_XINCREF(temp); /* Add a reference to new callback */ Py_XDECREF(my_callback); /* Dispose of previous callback */ my_callback = temp; /* Remember new callback */ /* Boilerplate to return "None" */ Py_INCREF(Py_None); result = Py_None; } return result;}

BLEHG!● Low Level● C based● Lots of

boilerplate

boost::python

● Higher level pure C++○ No silly IDL

● Works nicely with the rest of boost (Awesome C++ libraries for everything)

● Takes care of lots of details for you

boost::python example#include <boost/python.hpp>using namespace boost::python;

BOOST_PYTHON_MODULE(mycpplib){ class_<FortuneTeller>("FortuneTeller") .def("get_lottery", &FortuneTeller::get_lottery);}

But when you have a big library this is still pretty boring

Py++ and automated wrapper generation

● Python package● Takes in C++ header files● Generates boost::python code (C++) to wrap

the given header files.● Pretty comprehensive

● Reasonably stable (v1.0.0)● Not very active (New maintainer anyone?)● Stack overflow is your friend

Basic Strategy

C++ header

files*.hpp

Py++ script

C++wrapping Source

files*.cpp

CompilePython

Extension module

*.so

Our first attempt (The horror)

● 2500 line monolithic C++ module● Took forever to compile● Had to be hand tweaked to get it to compile● Changes involved generating a new version

and copying over sections of code from the old version

● Terrifying to make changes to the C++● Massively slowed down development

Worst thing ever!

Making things actually work

Find a better solution

or

Hire a full time trauma counselor for the dev team

Write a little (sort of) DSL

● Declarative● Abstract the lower-level py++ methods● Explicit and clear● Basically a couple of python functions

○ def limit_class(classname, function_names):...

○ def exclude_class(classname):...

● Clear process to add and change wrappings● Leave full capabilities of Py++ when needed

Expose only what you need

● By default py++ will expose every class and all public/protected methods

● This means even small changes to the C++ can mess with the wrappings

● Explicitly choose what to expose to python○ A public interface

● Makes changes easier to reason about● Limits unexpected changes in the python

interface (use of which is not statically type checked)

Convert where you can

● Sometimes its easier to automatically convert between C++ types and Python types○ Some types are just too difficult to wrap○ Often types have natural Python analogs

● Done for many built in types e.g. strings● Can set automatic converters

○ From C++ return values to python values○ From Python arguments to C++ arguments

● Consider performance● See http://misspent.wordpress.

com/2009/09/27/how-to-write-boost-python-converters/

Conversion Examples

Worked well:● python datetime to boost::posix_time ● Lots of utilities in python for datetime● boost::posix_time is super hard to wrap

Failed:● Python set to C++ set● Python sets are hashsets, C++ sets are

trees● Different semantics (ordering), subtle errors● Expensive to convert

Use the preprocessor

● The C preprocessor is fast

● gccxml (which powers py++) is slow

● Use it to aggregate all the headers you need into one header file (all.hpp)

● Makes things way faster

What is the C preprocessor?

It the thing that interprets statements like these:

#include "fileA.hpp"

or#ifndef FILEA#def FILE A...#endif

It is run on your C++ source files before compilation.

Custom wrapper functions

● Sometimes functions just don't wrap nicely○ e.g. when they take a vector<Something>

and you want to pass a regular python list to them

● Write some custom code that does the marshalling between the types you want to work with in python and the types in C++

● Inject this into the wrappers (py++ allows you to do this)

● !!!Don't do this by hand on the generated files● Can make wrapped objects more pythonic

Custom wrapper exampleTo wrap a method: int my_sum_method(vector<int> numbers)

to take a python list.

int my_sum_method(bp::list& pylist_numbers) {

::std::vector<int> vector_numbers;

for (int i = 0; i < len(pylist_numbers); ++i) {

int number = bp::extract<int>(pylist_numbers[i]);

vector_numbers.push_back(number);

}

return my_sum_method(vector_number);

}

Python list as a parameter

Same name as underlying function, uses overloading

Extract contents of python list and place it in vector

Call original method

Call policies

● Methods can return objects that have to be treated in different ways.○ Objects by value○ Raw pointers○ References

● Sometimes Py++ can figure out what to do, sometimes you need to help it.

● You can set the call policy for a method in py++, e.g.:

myObj.member_functions("get_child").call_policies = call_policies.return_internal_reference()

The GIL

● The Global Interpreter Lock ensures only one python instruction runs at one time.

● "the thing in CPython that prevents multiple threads from actually running in your Python code in parallel." -- PyPy Blog

● But it treats a call out to a C/C++ routine as a single atomic operation

● Bad if your methods are long running. Locks up all other threads.

Releasing the GIL

● You can release the GIL to allow other threads to run.

● But then you have to aquire it when your method ends

● Don't screw this up○ Think about

exceptions● Useful Macros

Your C++ code

Py_BEGIN_ALLOW_THREADS

Py_END_ALLOW_THREADS

Python code

Python code

Your long running C++ method

Write your code with wrapping in mind

● Sometimes you have to change the way you write code

● Should try to avoid this but BE PRAGMATIC● Some constructs do not translate easily● Don't use exotic structures (unions, wierd

memory maps, etc.)● Return types that are easily wrapped or

converted (have natural analogs)● Keep your code simple

Debugging through the layers

● Wrapped code can be hard to debug● You can run python under GDB● Step through the Python VM and eventually

into your own extension module● Takes some setting up but works very nicely● Worth doing!● Checkout the Stripe blog: https://stripe.

com/blog/exploring-python-using-gdb

Automate everything

● Customisation, wrapper generation and compilation should all be automated

● Use a decent build system (Sconstruct, Cmake)

● Py++ won't regenerate files that haven't changed, works well with MAKE

● Don't check generated code into your source control system (git,bzr,hg,svn)○ Make the generation easily reproducible

● Don't let anything slow your team down

The final system

C++ header

files*.hpp

Py++

C++wrapping Source

files*.cpp

Compile

Python Extension

module*.so

Single header

fileall.hpp

Wrapping DSL script

MakeFiles

CMake (Build

System)

The end result

● Single simple configuration file● Completely automated generation and

compilation● Speedy compilation● Easy to update

Using Python from C++ (Embedding)

C++ Computation Engine

Machine learning in scikit-learn

Embedding vs Passing in objects

● Two ways to go about it○ Embed an interpreter in your C++, run a script and

use the results.○ Pass python objects to your C++ (Extension Module)

and do something with them.● If your code is already an extension module

the latter is easier to reason about (IMHO)

boost::python again

● Makes it easy to use python objects in C++, kinda feels like python

● Has high level objects that mimic python objects○ bp::object, bp::list,

etc.● No need to touch

PyObject*

Python

def f(x, y): if (y == 'foo'): x[3:7] = 'bar' else: x.items += 3

return x

C++ using boost::python

object f(object x, object y) { if (y == "foo") x.slice(3,7) = "bar"; else x.attr("items") += 3; return x;}

Calling methods

Simple ways to call methods and get C++ types back.string id_number = "7803705112068";object y = x.attr("get_name")(id_number);string name = extract<string>(y);

or automatically do the type conversion

bp::call_method<string>(x,"get_name",id_number);

Pretty simple hey?

The GIL again

Python code

Python code

Your C++ method

Python code

Your C++ code

Your C++ code

If you run python code from C++ make sure you still have the GIL aquired.

The GIL again fixedPython code

Python code

Your long running C++ method

Python code

Your C++ code

Your C++ code

Py_BEGIN_ALLOW_THREADS

Py_END_ALLOW_THREADS

Py_BEGIN_BLOCK_THREADS

Py_BEGIN_UNBLOCK_THREADS

● More Marcos

Lessons and thoughts

The sliding scale between Python and C++

Python C++

● Performance● Huge amount of

existing code

● Speed of development

● Elegance● joie de vivre

Actually pretty fast Not that bad to use

What's right for you?

If you are gonna do it do it right

● Done wrong, wrappings are a nightmare● Done right, they can be quite manageable● Is the extra performance worth the

development overhead?● If you are writing the C++ start the wrapping

process early

Alternatives

● Just use libraries: Numpy, etc.● PyPy (CPPYY)● Cython● Weave● SWIG● ctypes

Contact

James Saundersjames@businessoptics.biz@james.h.saundershttp://blog.jamessaunders.co.za/

www.businessoptics.biztechblog.businessoptics.biz