CIS192 Python Programming - University of Pennsylvaniacis192/spring2015/files/lec/lec14.pdfCIS192 Python Programming Mixing C with Python/Modules and Packages Eric Kutschera University

CIS192 Python ProgrammingMixing C with Python/Modules and Packages

Eric Kutschera

University of Pennsylvania

April 24, 2015

Eric Kutschera (University of Pennsylvania) CIS 192 April 24, 2015 1 / 24

Final Project

Sign up for a demo slotLink is on the website for week 14Email me if none of the demo slots work for your group


Outline

1 Mixing C with PythonPython/C APIHigher Level Tools

2 Module and Package SystemModules


Why Mix C with Python

Some things are not so easy to do in Pure PythonMaking system calls

Some things are not possibleAccessing CPython interpreter internalsSpecifying exact memory layouts of objects

Some things are Slow in PythonBasically everything

Some problems are already solved with good C codeDon’t reinvent the wheel


Potential Disadvantages of C Extensions

Only work on a specific version of CPythonIntroduce non-Python dependenciesPlatform specific compiled binariesPartially why numpy/scipy can be a pain to install


C Extensions from scratch

The Python/C API allows direct extensions to CPythonVery low level CPython detailsHave to worry about reference counting (garbage collection)“ It is intended primarily for creators of those tools, rather thanbeing a recommended way to create your own C extensions.”-Python docs

Allows you to do things not possible in pure PythonDefine new built-in object typesCall C library functions and system calls

Can be much fasterCPython specific

Not portable across implementations (Jython)Other alternative that are more portable (See later)


Python.h

#include <Python.h> must be the first lineDefines types/functions used to extend the CPython InterpreterAfter that you can include any c librariesDefine Python functions as C functions

Must return a PyObjectHave to parse Python inputs into C types (str → char*)

Must explicitly defineMethod tableModule initialization


Building the .c

Need to compile into a binary and tell Python where to find itpython setup.py build

(sudo)python setup.py install

from distutils.core import setup, Extension

module1 = Extension(’pycapi’,sources=[’pycapimodule.c’],extra_compile_args=[’-std=c99’])

setup(name=’PycapiPackage’, version=’1.0’,description=’Demo of Python/C API’,

ext_modules=[module1])


Outline




ctypes

Allows loading of C librariesfrom ctypes import cdlllibc = cdll.LoadLibrary(’libc.so.6’)

Once a library is loaded you can call its functionslibc.printf(b’%s\n’, b’a string’)

Some Python types are automatically convertedbyte strings: s = b’bytes’ → char *s = ’bytes’;None: None → NULLIntegers: i = 42 → int i = 42;

All other types must be constructed with a ctypes functionctypes.c_bool(True)ctypes.c_uint(3)

Supports Jython and PyPy


cffi

C Foreign Function InterfaceAims to just require separate knowledge of C and Pythonffi = cffi.FFI() Create a Foreign Function Interfaceffi.cdef(’Some C declarations’) Define the interfaceC = ffi.verify(’paste your .c file’) Compiles CC.some_function(*args) Use things in cdefCan load existing C libraries ffi.dlopen(’lib_name’)Pass C objects to functions with ffi.new(’int[50]’)

Has support for PyPy, Jython, and IronPython


Cython

Cython was released 2007 (based on a 2002 project)Write C extensions to Python mostly with PythonSupports:

Pure PythonPython annotated with static typesCalling C code

Cython Extensions must be compiledAllows for static compile time optimizationsCan’t interpret directly with $python cython_file.py

Cython isn’t a perfect implementation of Pythona is b: Identity comparisons are likely to behave strangelyIntrospection of Cython functions not quite right


Cython basic

cythonhello.pyx

print(’Hello Cython’)

setup.py

from distutils.core import setupfrom Cython.Build import cythonize

setup(ext_modules=cythonize(’cythonhello.pyx’))

$ python setup.py build_ext --inplace

>>> import cythonhelloHello Cython


Other Cython Builds

If you just need the boiler plate setupSkip the setup.py

>>> import pyximport; pyximport.install()>>> import cythonhelloHello Cython

If you want to use C librariesMention the library in setup.py

from distutils.extension import Extensionext_modules = [Extension(’modname’,

sources=[’modname.pyx’],libraries=[’c_lib_name’])


Cython Using C functions

Some C libraries are included with CythonSpecial cimport keywordfrom libc.math cimport sinConverts types for you sin(1.5) → 0.997...

Otherwise link from setup.pyimport with cdef keyword

cdef extern from ’math.h’:double sin(double x)


Cython Static Types

Static types annotations int, double, int[10] can be usedThis allows Cython to avoid using a PyObject in the .cTypes can be given to local variables

cdef int n

To give a function input a typedef my_func(int arg)The input will be converted from a python type at runtimeAn error is thrown if the argument can’t be converted

If all variables in a loop are annotatedCompiler can convert the entire loop into C codeAllows the compiler to do lots of fancy tricksHuge win to annotate entire loop


Numba Overview

Numba is an LLVM compiler for Python (Released 2012)Designed with Numpy in mindCompiles Pure Python to fast machine codeEven easier to use then Cython

from numba import jit

@jitdef some_function(args):...


Numba vs Cython

Both Numba and Cython compile Python to machine codeCython goes through C (gcc)Numba goes through LLVM

Cython pre-compiles / Numba Compiles Just in Time (JIT)Cython and Numba give similar performanceModification from Pure Python

Cython requires lots of type annotations for best performanceNumba uses a single decorator and infers types

InstallationCython requires Python and GCC (easy)Numba requires Numpy and LLVM (pain)

Numba doesn’t support calling arbitrary C code


Outline




Imports Explained

import modulename does 2 thingsLooks for the module on the systemBinds that module to a name in the local scope

A module is python file on the system module.pyAny top-level definition is available after importing a module

Modules are organized into a package hierarchy (file-system)Any directory on the file-system becomes a packageA package can contain an __init__.pyWhen the import statement is executed so is __init__.pyPackages can contain sub-packagesimport pkg.subpkg.modPackages are also modules so they can be importedimport pkg


Module Search

In order for import modulename to succeedThe module search must succeed

First sys.modules is checked. It caches imported modulesThen sys.meta_path, a list of module finders, is usedfinders are just functions that can search for modulesThe default finder looks in the file-system starting at sys.pathAnything that matches the module name gets loaded and boundWhen loading a module within a package

import pkg.modulepkg.__path__ will be checked in addition to sys.path


Installing Packages

What does pip install actually do?It downloads the source from PyPi (Python Package Index)Executes some setup code (Compiles or otherwise prepares files)Copies the resulting package to “pythonX.x/site-packages”Since “site-packages” is in sys.path you can then import it


Virtual Enviornments

Python 3.3 introduces a built-in pyvenvIt is a standard library solution similar to virtualenvThe basic idea of both of these tools is to isolate dependencies$ pyvenv venv_dir_name creates a directory

Scripts to “turn on” the environmentLinks to or copies of the Python binariesA place to locally install new packages

$ source venv_dir_name/bin/activate

$ deactivate

“turning on” the venv just changes where python looksModifies the version of Python in $ echo "$PATH"Changes Python specific Environment Variables


Benefits of Virtual Environments

Allows you to use different versions of Python on one systemSpecify exactly which packages are needed for a project

Once everything is install generate a requirements file$ pip freeze > requirements.txtUse that file to install packages somewhere else$ pip install -r requirements.txt

Can install packages without root permissions


Documents

CIS192 Python Programming - University of Pennsylvaniacis192/spring2015/files/lec/lec14.pdfCIS192 Python Programming Mixing C with Python/Modules and Packages Eric Kutschera University