Wednesday, June 8, 2016

PyPy2 v5.3 released - major C-extension support improvements

We have released PyPy2.7 v5.3, about six weeks after PyPy 5.1 and a week after PyPy3.3 v5.2 alpha 1, the first PyPy release targeting 3.3 compatibility. This new PyPy2.7 release includes major improvements for the C-API compatibility layer. In addition to complete support for lxml, we now pass most (more than 95%) of the upstream numpy test suite. We can build and run scipy and matplotlib as well. Most of the failures have to do with (ab) use of the C-API, for instance writing to a read-only pointer obtained from PyString_AsString().

Note that the C-API compatibility layer is significantly slower than CPython, as explained in the blog post about the new strategy for reflection of C objects into the PyPy interpreter.

We updated cffi to version 1.7 (incremental changes which provide a nicer developer experience, documented here). We would encourage developers to move their C-extension modules to cffi, but are willing to help you work through issues with existing code; come to #pypy on IRC and let us know how we can help you help us do better.

You can download the PyPy2 v5.3 release here:
We would like to thank our donors for their continued support of the PyPy project. We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: PyPy and RPython documentation improvements, tweaking popular modules to run on PyPy, or general help with making RPython’s JIT even better.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It’s fast (PyPy and CPython 2.7 performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

This release supports:
  • x86 machines on most common operating systems (Linux 32/64, Mac OS X 64, Windows 32, OpenBSD, FreeBSD)
  • newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux
  • big- and little-endian variants of PPC64 running Linux
  • s390x running Linux

Other Highlights

(since the release of PyPy 5.1 in April, 2016)

  • New features:
    • Merge a major expansion of the C-API support in cpyext, also expand cpyext tests to allow running them after translation as well as untranslated
    • Instead of “GIL not held when a CPython C extension module calls PyXxx”, we now silently acquire/release the GIL. Helps with C extension modules that call some PyXxx() functions without holding the GIL (arguably, they are theoretically buggy).
    • Support command line -v to trace import statements
    • Revive traceviewer, a tool to use pygame to view traces

  • Numpy via our internal _numpypy module:
    • Implement ufunc.outer
    • Move PyPy-specific numpypy headers to a subdirectory (also changed the repo accordingly)
  • Performance improvements:
    • Use bitstrings to compress lists of descriptors that are attached to an EffectInfo
    • Remove most of the _ovf, _zer and _val operations from RPython. Kills quite some code internally, and allows the JIT to do better optimizations: for example, app-level code like x / 2 or x % 2 can now be turned into x >> 1 or x & 1, even if x is possibly negative.
    • Rework the way registers are moved/spilled in before_call()

  • Internal refactorings:
    • Refactor code to better support Python3-compatible syntax
    • Reduce the size of generated C sources during translation by eliminating many many unused struct declarations (Issue #2281)
    • Reduce the size of generated code by using the same function objects in all generated subclasses
    • Share cpyext Py* function wrappers according to the signature, shrinking the translated by about 10% (without the JIT)
Please update, and continue to help us make PyPy better. Cheers
The PyPy Team


Gary Robinson said...

"We can build and run scipy and matplotlib as well."

That's exciting. Are there special instructions needed to build and run spicy and matplotlib with PyPy to see how well it presently works for particular applications? Or is it not even really ready for outsiders to knock it around yet?

mattip said...

No special instructions, just build from source (binaries precompiled for cpython will not work) using "pypy install", and let us know how it goes. The order should be numpy, matplotlib, scipy (we have reports that pygtk works too fwiw).

There have already been some bug reports, so you might want to patch your pymem.h header in pypy/include with this changeset, and if you are on OSX you may need to patch numpy/distutils/fcompiler/ with this patch

Canesin said...

Great work as usual!