CFFI 1.0 beta 1

Finally! CFFI 1.0 is almost ready. CFFI gives Python developers a convenient way to call external C libraries. Here "Python" == "CPython or PyPy", but this post is mostly about the CPython side of CFFI, as the PyPy version is not ready yet.

On CPython, you can download the version "1.0.0b1" either by looking for the cffi-1.0 branch in the repository, or by saying

pip install "cffi>=1.0.dev0"

(Until 1.0 final is ready, pip install cffi will still give you version 0.9.2.)

The main news: you can now explicitly generate and compile a CPython C extension module from a "build" script. Then in the rest of your program or library, you no longer need to import cffi at all. Instead, you simply say:

from _my_custom_module import ffi, lib

Then you use ffi and lib just like you did in your verify()-based project in CFFI 0.9.2. (The lib is what used to be the result of verify().) The details of how you use them should not have changed at all, so that the rest of your program should not need any update.

Benefits

This is a big step towards standard practices for making and distributing Python packages with C extension modules:

• on the one hand, you need an explicit compilation step, triggered here by running the "build" script;
• on the other hand, what you gain in return is better control over when and why the C compilation occurs, and more standard ways to write distutils- or setuptools-based setup.py files (see below).

Additionally, this completely removes one of the main drawbacks of using CFFI to interface with large C APIs: the start-up time. In some cases it could be extreme on slow machines (cases of 10-20 seconds on ARM boards occur commonly). Now, the import above is instantaneous.

In fact, none of the pure Python cffi package is needed any more at runtime (it needs only an internal extension module from CFFI, which can be installed by doing "pip install cffi-runtime" [*] if you only need that). The ffi object you get by the import above is of a completely different class written entirely in C. The two implementations might get merged in the future; for now they are independent, but give two compatible APIs. The differences are that some methods like cdef() and verify() and set_source() are omitted from the C version, because it is supposed to be a complete FFI already; and other methods like new(), which take as parameter a string describing a C type, are faster now because that string is parsed using a custom small-subset-of-C parser, written in C too.

In practice

CFFI 1.0 beta 1 was tested on CPython 2.7 and 3.3/3.4, on Linux and to some extent on Windows and OS/X. Its PyPy version is not ready yet, and the only docs available so far are those below.

This is beta software, so there might be bugs and details may change. We are interested in hearing any feedback (irc.freenode.net #pypy) or bug reports.

To use the new features, create a source file that is not imported by the rest of your project, in which you place (or move) the code to build the FFI object:

# foo_build.py
import cffi
ffi = cffi.FFI()

ffi.cdef("""
    int printf(const char *format, ...);
""")

ffi.set_source("_foo", """
    #include <stdio.h>
""")   # and other arguments like libraries=[...]

if __name__ == '__main__':
    ffi.compile()

The ffi.set_source() replaces the ffi.verify() of CFFI 0.9.2. Calling it attaches the given source code to the ffi object, but this call doesn't compile or return anything by itself. It may be placed above the ffi.cdef() if you prefer. Its first argument is the name of the C extension module that will be produced.

Actual compilation (including generating the complete C sources) occurs later, in one of two places: either in ffi.compile(), shown above, or indirectly from the setup.py, shown next.

If you directly execute the file foo_build.py above, it will generate a local file _foo.c and compile it to _foo.so (or the appropriate extension, like _foo.pyd on Windows). This is the extension module that can be used in the rest of your program by saying "from _foo import ffi, lib".

Distutils

If you want to distribute your program, you write a setup.py using either distutils or setuptools. Using setuptools is generally recommended nowdays, but using distutils is possible too. We show it first:

# setup.py
from distutils.core import setup
import foo_build

setup(
    name="example",
    version="0.1",
    py_modules=["example"],
    ext_modules=[foo_build.ffi.distutils_extension()],
)

This is similar to the CFFI 0.9.2 way. It only works if cffi was installed previously, because otherwise foo_build cannot be imported. The difference is that you use ffi.distutils_extension() instead of ffi.verifier.get_extension(), because there is no longer any verifier object if you use set_source().

Setuptools

The modern way is to write setup.py files based on setuptools, which can (among lots of other things) handle dependencies. It is what you normally get with pip install, too. Here is how you'd write it:

# setup.py
from setuptools import setup

setup(
    name="example",
    version="0.1",
    py_modules=["example"],
    setup_requires=["cffi>=1.0.dev0"],
    cffi_modules=["foo_build:ffi"],
    install_requires=["cffi-runtime"],    # see [*] below
)

Note that "cffi" is mentioned on three lines here:

• the first time is in setup_requires, which means that cffi will be locally downloaded and used for the setup.
• the second mention is a custom cffi_modules argument. This argument is handled by cffi as soon as it is locally downloaded. It should be a list of "module:ffi" strings, where the ffi part is the name of the global variable in that module.
• the third mention is in install_requires. It means that in order to install this example package, "cffi-runtime" must also be installed. This is (or will be) a PyPI entry that only contains a trimmed down version of CFFI, one that does not include the pure Python "cffi" package and its dependencies. None of it is needed at runtime.

[*] NOTE: The "cffi-runtime" PyPI entry is not ready yet. For now, use "cffi>=1.0.dev0" instead. Considering PyPy, which has got a built-in "_cffi_backend" module, the "cffi-runtime" package could never be upgraded there; but it would still be nice if we were able to upgrade the "cffi" pure Python package on PyPy. This might require some extra care in writing the interaction code. We need to sort it out now...

Thanks

Special thanks go to the PSF (Python Software Foundation) for their financial support, without which this work---er... it might likely have occurred anyway, but at an unknown future date :-)

(For reference, the amount I asked for (and got) is equal to one month of what a Google Summer of Code student gets, for work that will take a bit longer than one month. At least I personally am running mostly on such money, and so I want to thank the PSF again for their contribution to CFFI---and while I'm at it, thanks to all other contributors to PyPy---for making this job more than an unpaid hobby on the side :-)

Armin Rigo