At some point of time many Python developers   at least in computational science   will
face the situation that they want to interface some natively compiled library from Python. For
binding native code to Python by now a larger variety of tools and technologies are available.
This paper focuses on wrapping shared C libraries, using Python's default Ctypes. Particularly
tools to ease the process (by using code generation) and some best practises will be stressed.
The paper will try to tell a  step by step  story of the wrapping and development process,
that should be transferable to similar problems

Kloss, Guy K.

English

Massey Research Online

Res. Lett. Inf. Math. Sci., 2009, Vol. 13, pp. 17Available online at http://iims.massey.ac.nz/research/letters/1Automatic C Library Wrapping Ctypes from the TrenchesGuy K. KlossComputer ScienceInstitute of Information & Mathematical SciencesMassey University at Albany, Auckland, New ZealandEmail: G.Kloss@massey.ac.nzAt some point of time many Python developers  at least in computational science  willface the situation that they want to interface some natively compiled library from Python. Forbinding native code to Python by now a larger variety of tools and technologies are available.This paper focuses on wrapping shared C libraries, using Python's default Ctypes. Particularlytools to ease the process (by using code generation) and some best practises will be stressed.The paper will try to tell a stepbystep story of the wrapping and development process,that should be transferable to similar problems.Keywords: Python, Ctypes, wrapping, automation, code generation.1 IntroductionOne of the grand fundamentals in software engineering is to use the tools that are best suited for ajob, and not to prematurely decide on an implementation. That is often easier said than done, inthe light of some complimentary requirements (e. g. rapid/easy implementation vs. needed speed ofexecution or vs. low level access to hardware). The traditional way [1] of binding native code toPython through extending or embedding is quite tedious and requires lots of manual coding in C.This paper presents an approach using the Ctypes package [2], which is by default part of Pythonsince version 2.5.As an example the creation of a wrapper for the Little CMS colour management library [3]is outlined. The library oﬀers excellent features, and ships with oﬃcial Python bindings (us-ing SWIG [4]), but unfortunately with several shortcomings (incompleteness, un-Pythonic API,complex to use, etc.). So out of need and frustration the initial steps towards alternative Pythonbindings were undertaken.An alternative would be to ﬁx or improve the bindings using SWIG, or to use one of a varietyof binding tools. The ﬁeld has been limited to tools that are widely in use today within thecommunity, and that are promising to be future proof as well as not overly complicated to use.These are the contestants with (very brief) notes for use cases that suit their particular strengths:• Use Ctypes [2], if you want to wrap pure C code very easily.• Use Boost.Python [5,6], if you want to create a more complete API for C++ that also reﬂectsthe object oriented nature of your native code, including inheritance into Python, etc.• Use cython [7], if you want to easily speed up and migrate code from Python to speediernative code (Mixing is possible!).• Use SWIG [4], if you want to wrap your code against several dynamic languages.Automatic C Library Wrapping  Ctypes from the Trenches 2Of course, wrapper code can be written manually, in this case directly using Ctypes. Thispaper does not provide a tutorial on how Ctypes is used. The reader should be familiar withthis package when attempting to undertake serious library wrapping. The Ctypes tutorial andCtypes reference on the project web site [2] are an excellent starting point for this. For extensivelibraries and robustness towards an evolving API, code generation proved to be a good approachover manual editing. Code generators exist for Boost.Python as well as for Ctypes to ease theprocess of wrapping: Py++ [8] (for Boost.Python) and CtypesLib's [2] h2xml.py and xml2py.py.Three main reasons have inﬂuenced the decision to approach this project using ctypes:• Ubiquity of the binding approach, as Ctypes is part of the default distribution.• No compilation of native code to libraries is necessary. Additionally, this relieves one frominstalling a number of development tools, and the library wrapper can be approached in aplatform independent way.• The availability of a code generator to automate large portions of the wrapper implementationprocess for ease and robustness against changes.The next section of this paper will ﬁrst introduce a simple C example. This example is latermigrated to Python code through the various incarnations of the Python wrapper throughout thepaper. Sect. 3 introduces how to facilitate the C library code from Python, in this case through codegeneration. Sect. 4 explains how to reﬁne the generated code to meet the desired functionality ofthe wrapper. The library is anything but Pythonic, so Sect. 5 explains an object oriented FaçadeAPI for the library that features qualities we love.This paper only outlines some interesting fundamentals of the wrapper building process. Pleaserefer to the source code for more precise details [9].2 The ExampleThe sample code (listing in Fig. 1) aims to convert image data from device dependent colourinformation to a standardised colour space. The input proﬁle results from a device speciﬁc char-acterisation of a Hewlett Packard ScanJet (in the ICC proﬁle HPSJTW.ICM). The output is in thestandard conformant sRGB output colour space as it is used for the majority of displays on com-puters. For this a built-in proﬁle from LittleCMS is used.Input and output are characterised through so called ICC proﬁles. For the input proﬁle thecharacterisation is read from a ﬁle (line 8), and a built in output proﬁle is used (line 9). Thetransformation object is set up using the proﬁles (lines 1113), specifying the colour encoding inthe in- and output as well as some further parameters not worth discussing here. In the for loop(lines 1521) the image data is transformed line by line, operating on the number of pixels usedper line (necessary as array rows are often padded).The goal is to provide a suitable and easy to use API to perform the same task in Python.3 Code GenerationWrapping C data types, functions, constants, etc. with Ctypes is not particularly diﬃcult. Thetutorial, project web site and documentation on the wiki introduce this concept quite well. Butin the presence of an existing larger library, manual wrapping can be tedious and error prone, aswell as hard to keep consistent with the library in case of changes. This is especially true when thelibrary is maintained by someone else. Therefore, it is advisable to generate the wrapper code.Thomas Heller, the author of Ctypes has implemented a corresponding project CtypesLib thatincludes tools for code generation. The tool chain consists of two parts, the parser (for headerﬁles) and the code generator.Automatic C Library Wrapping  Ctypes from the Trenches 31 #include "lcms.h"3 int correctColour(void) {4 cmsHPROFILE inProfile, outProfile;5 cmsHTRANSFORM myTransform;6 int i;8 inProfile = cmsOpenProfileFromFile("HPSJTW.ICM", "r");9 outProfile = cmsCreate_sRGBProfile();11 myTransform = cmsCreateTransform(inProfile, TYPE_RGB_8,12 outProfile, TYPE_RGB_8,13 INTENT_PERCEPTUAL, 0);15 for (i = 0; i < scanLines; i++) {16 /* Skipped pointer handling of buffers. */17 cmsDoTransform(myTransform,18 pointerToYourInBuffer,19 pointerToYourOutBuffer,20 numberOfPixelsPerScanLine);21 }23 cmsDeleteTransform(myTransform);24 cmsCloseProfile(inProfile);25 cmsCloseProfile(outProfile);27 return 0;28 }Figure 1: Example in C using the LittleCMS library directly.3.1 Parsing the Header FileThe C header ﬁles are parsed by the tool h2xml. In the background it uses GCCXML, a GCCcompiler that parses the code and generates an XML tree representation. Therefore, usually thesame compiler that builds the binary of the library can be used to analyse the sources for the codegeneration. Alternative parsers often have problems determining a 100% proper interpretation ofthe code. This is particularly true in the case of C code containing pre-processor macros, whichcan commit massively complex things.3.2 Generating the WrapperIn the next stage the parser tree in XML format is taken to generate the binding code in Pythonusing Ctypes. This task is performed by the xml2py tool. The generator can be conﬁgured in itsactions by means of switches passed to it. Of particular interest here are the -k and the -r switches.The former deﬁnes the kind of types to include in the output. In this case the #defines, functions,structure and union deﬁnitions are of interest, yielding -kdfs. Note: Dependencies are resolvedautomatically. The -r switch takes a regular expression the generator uses to identify symbolsto generate code for. The full argument list is shown in the listing in Fig. 2 (lines 1115). Thegenerated code is written to a Python module, in this case _lcms. It is made private by convention(leading underscore) to indicate that it is not to be used or modiﬁed directly.3.3 Automating the GeneratorBoth h2xml and xml2py are Python scrips. Therefore, the generation process can be automatedin a simple generator script. This makes all steps reproducible, documents the used settings, andAutomatic C Library Wrapping  Ctypes from the Trenches 4makes the process robust towards evolutionary (smaller) changes in the C API. A largely simpliﬁedversion is in the listing of Fig. 2.1 # Skipped declaration of paths.2 HEADER_FILE = ’lcms.h’3 header_basename = os.path.splitext(HEADER_FILE)[0]5 h2xml.main([’h2xml.py’, header_path,6 ’-c’,7 ’-o’,8 ’%s.xml’ % header_basename])10 SYMBOLS = [’cms.*’, ’TYPE_.*’, ’PT_.*’, ’ic.*’, ’LPcms.*’, ...]11 xml2py.main([’xml2py.py’, ’-kdfs’,12 ’-l%s’ % library_path,13 ’-o’, module_path,14 ’-r%s’ % ’|’.join(SYMBOLS),15 ’%s.xml’ % header_basename]Figure 2: Essential parts of the code generator script.Generated code should never be edited manually. As some modiﬁcation will be necessary toachieve the desired functionality (see Sect. 4), automation becomes essential to yield reproducibleresults. Due to some shortcomings (see Sect. 4) of the generated code however, some editing wasnecessary. This modiﬁcation has also been integrated into the generator script to fully remove theneed of manual editing.4 Reﬁning the C APIIn the current version of Ctypes in Python 2.5 it is not possible to add e. g. __repr__() or __str__()methods to data types. Also, code for loading the shared library in a platform independent wayneeds to be patched into the generated code. A function in the code generator reads the wholegenerated module _lcms and writes it back to the ﬁle system, and in the course replacing threelines from the beginning of the ﬁle with the code snippet from the listing in Fig. 3.1 from _setup import *2 import _setup4 _libraries = {}5 _libraries[’/usr/lib/liblcms.so.1’] = _setup._init()Figure 3: Lines to be patched into the generated module _lcms._setup (listing in Fig. 4) monkey patches1 the class ctypes.Structure to include a __repr__()method (lines 410) for ease of use when representing wrapped objects for output. Furthermore,the loading of the shared library (DLL in Windows lingo) is abstracted to work in a platformindependent way using the system's default search mechanism (lines 1213).4.1 Creating the Basic WrapperFurther modiﬁcations are less invasive. For this, the C API is reﬁned into a module c_lcms. Thismodule imports everything from the generated._lcms and overrides or adds certain functionality1A monkey patch is a way to extend or modify the runtime code of dynamic languages without altering theoriginal source code: http://en.wikipedia.org/wiki/Monkey_patchAutomatic C Library Wrapping  Ctypes from the Trenches 51 import ctypes2 from ctypes.util import find_library4 class Structure(ctypes.Structure):5 def __repr__(self):6 """Print fields of the object."""7 res = []8 for field in self._fields_:9 res.append(’%s=%s’ % (field[0], repr(getattr(self, field[0]))))10 return ’%s(%s)’ % (self.__class__.__name__, ’, ’.join(res))12 def _init():13 return ctypes.cdll.LoadLibrary(find_library(’lcms’))Figure 4: Extract from module _setup.py.individually (again through monkey patching). These are intended to make the C API a little biteasier to use through some helper functions, but mainly to make the new bindings more compatiblewith and similar to the oﬃcial SWIG bindings (packaged together with LittleCMS ). The wrappedC API can be used from Python (see Sect. 4.2). Although, it still requires closing, freeing or deletingfrom the code after use, and c_lcms objects/structures do not feature methods for operations. Thisshortcoming will be solved later.4.2 c_lcms ExampleThe wrapped raw C API in Python behaves in exactly the same way, it is just implemented inPython syntax (listing in Fig. 5).1 from c_lcms import *3 def correctColour():4 inProfile = cmsOpenProfileFromFile(’HPSJTW.ICM’, ’r’)5 outProfile = cmsCreate_sRGBProfile()7 myTransform = cmsCreateTransform(inProfile, TYPE_RGB_8,8 outProfile, TYPE_RGB_8,9 INTENT_PERCEPTUAL, 0)11 for line in scanLines:12 # Skipped handling of buffers.13 cmsDoTransform(myTransform,14 yourInBuffer,15 yourOutBuffer,16 numberOfPixelsPerScanLine)18 cmsDeleteTransform(myTransform)19 cmsCloseProfile(inProfile)20 cmsCloseProfile(outProfile)Figure 5: Example using the basic API of the c_lcms module.5 A Pythonic APITo create the usual pleasant batteries included feeling when working with code in Python, anothermodule  littlecms  was manually created, implementing the Façade Design Pattern. From hereAutomatic C Library Wrapping  Ctypes from the Trenches 6on we are moving away from the original C-like API. This high level object oriented Façade takescare of the internal handling of tedious and error prone operations. It also performs sanity checkingand automatic detection for certain crucial parameters passed to the C API. This has drasticallyreduced problems with the low level nature of the underlying C library.5.1 littlecms ExampleUsing littlecms the API is now object oriented (listing in Fig. 6) with a doTransform() method onthe myTransform object. But there are a few more interesting beneﬁts of this API:• Automatic disposing of C API instances hidden inside the Profile and Transform classes.• Largely reduced code size with an easily comprehensible structure.• Redundant passing of information (e. g. the in- and output colour spaces) is determinedwithin the Transform constructor from information available in the Profile objects.• Uses NumPy [10] arrays for convenience in the buﬀers, rather than introducing further customtypes. On these data array types and shapes can be automatically matched up.• The number of pixels for each scan line placed in yourInBuffer can usually be detected auto-matically.• Compatible with the often used PIL [11] library.• Several sanity checks prevent clashes of erroneously passed buﬀer sizes, shapes, types, etc.that would otherwise result in a crashed or hanging process.1 from littlecms import Profile, PT_RGB, Transform3 def correctColour():4 inProfile = Profile(’HPSJTW.ICM’)5 outProfile = Profile(colourSpace=PT_RGB)6 myTransform = Transform(inProfile, outProfile)8 for line in scanLines:9 # Skipped handling of buffers.10 myTransform.doTransform(yourNumpyInBuffer, yourNumpyOutBuffer)Figure 6: Example using the object oriented API of the littlecms module.6 ConclusionBinding pure C libraries to Python is not very diﬃcult, and the skills can be mastered in a rathershort time frame. If done right, these bindings can be quite robust even towards certain changes inthe evolving C API without the need of very time consuming manual tracking of all changes. Aswith many projects for this, it is vital to be able to automate the mechanical processes: Beyondthe outlined code generation in this paper, an important role comes to automated code integritytesting (here: using PyUnit [12]) as well as an API documentation (here: using Epydoc [13]).Unfortunately, as CtypesLib is still work in progress, the whole process did not go as smoothlyas described here. It was particularly important to match up working versions properly betweenGCCXML (which in itself is still in development) and CtypesLib. In this case a current GCCXMLin version 0.9.0 (as available in Ubuntu Intrepid Ibex, 8.10) required a branch of CtypesLib thatAutomatic C Library Wrapping  Ctypes from the Trenches 7needed to be checked out through the developer's Subversion repository. Furthermore, it wasnecessary to develop a ﬁx for the code generator as it failed to generate code for #defined ﬂoatingpoint constants. The patch has been reported to the author and is now in the source code repository.Also patching into the generated source code for overriding some features and manipulating thelibrary loading code can be considered as being less than elegant.Library wrapping as described in this paper was performed on version 1.16 of the LittleCMSlibrary. While writing this paper the author has moved to the now stable version 1.17. Adaptingthe Python wrapper to this code base was a matter of about 15 minutes of work. The main taskwas ﬁxing some unit tests due to rounding diﬀerences resulting from an improved numerical modelwithin the library. The author of LittleCMS made a ﬁrst preview of the upcoming version 2.0 (analmost complete rewrite) available recently. Adapting to that version took only about a good dayof modiﬁcations, even though some substantial changes were made to the API. But even for thiscase only very little amounts of new code had to be written.Overall, it is foreseeable that this type of library wrapping in the Python world will becomemore and more ubiquitous, as the tools for it mature. But already at the present time one does nothave to fear the process. The time spent initially setting up the environment will be easily savedover all projects phases and iterations. It will be interesting to see Ctypes evolve to be able tointerface to C++ libraries as well. Currently the developers of Ctypes and Py++ (Thomas Hellerand Roman Yakovenko) are evaluating potential extensions.References[1] Oﬃcial Python Documentation: Extending and Embedding the Python Interpreter, PythonSoftware Foundation.[2] T. Heller, Python Ctypes Project, http://starship.python.net/crew/theller/ctypes/, last ac-cessed December 2008.[3] M. Maria, LittleCMS project, http://littlecms.com/, last accessed January 2009.[4] D. M. Beazley and W. S. Fulton, SWIG Project, http://www.swig.org/, last accessed De-cember 2008.[5] D. Abrahams and R. W. Grosse-Kunstleve, Building Hybrid Systems with Boost.Python,http://www.boostpro.com/writing/bpl.html, March 2003, last accessed December 2008.[6] D. Abrahams, Boost.Python Project, http://www.boost.org/libs/python/, last accessed De-cember 2008.[7] S. Behnel, R. Bradshaw, and G. Ewing, Cython Project, http://cython.org/, last accessedDecember 2008.[8] R. Yakovenko, Py++ Project, http://www.language-binding.net/pyplusplus/pyplusplus.html, last accessed December 2008.[9] G. K. Kloss, Source Code: Automatic C Library Wrapping  Ctypes from the Trenches,The Python Papers Source Codes, vol. 1, pp. , January 2009, [Online available] http://ojs.pythonpapers.org/index.php/tppsc/issue/view/13.[10] T. Oliphant, NumPy Project, http://numpy.scipy.org/, last accessed December 2008.[11] F. Lundh, Python Imaging Library (PIL) Project, http://www.pythonware.com/products/pil/, last accessed December 2008.[12] S. Purcell, PyUnit Project, http://pyunit.sourceforge.net/, last accessed December 2008.[13] E. Loper, Epydoc Project, http://epydoc.sourceforge.net/, last accessed December 2008.

Automatic C library wrapping Ctypes from the trenches

https://mro.massey.ac.nz/bitstream/10179/4501/1/Automatic_C_Library_Wrapping-Ctypes_from_the_Trenches.pdf

Automatic C library wrapping Ctypes from the trenches

Abstract

Similar works

Full text

Available Versions

Massey Research Online