PyPy2 v5.4 released - incremental improvements and enhancements

We have released PyPy2.7 v5.4, a little under two months after PyPy2.7 v5.3. This new PyPy2.7 release includes incremental improvements to our C-API compatibility layer (cpyext), enabling us to pass over 99% of the upstream numpy test suite.

We updated built-in cffi support to version 1.8, which now supports the “limited API” mode for c-extensions on CPython >=3.2.

We improved tooling for the PyPy JIT, and expanded VMProf support to OpenBSD and Dragon Fly BSD

As always, this release fixed many issues and bugs raised by the growing community of PyPy users. We strongly recommend updating.

You can download the PyPy2 v5.4 release here:

http://pypy.org/download.html

We would like to thank our donors for their continued support of the PyPy project. We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: PyPy and RPython documentation improvements, testing and adapting popular modules to run on PyPy, or general help with making RPython’s JIT even better.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It’s fast (PyPy and CPython 2.7 performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

This release supports:

• x86 machines on most common operating systems (Linux 32/64, Mac OS X 64, Windows 32, OpenBSD, FreeBSD)
• newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux
• big- and little-endian variants of PPC64 running Linux
• s390x running Linux

What is New?

(since the release of PyPy 5.3 in June, 2016)

There are many incremental improvements to RPython and PyPy, the complete listing is here. Mozilla generously sponsored work toward python 3.5 compatibility, and we are beginning to see some cross-over improvements of RPython and PyPy2.7 as a result.

Please update, and continue to help us make PyPy better. Cheers

The PyPy Team

PyPy Tooling Upgrade: JitViewer and VMProf

We are happy to announce a major JitViewer (JV) update.
JV allows you to inspect RPython's internal compiler representation (the language in which PyPy is implemented) including the generated machine code of your program. It can graphically show you details of the JIT compiled code and helps you pinpoint issues in your program.

VMProf is a statistical CPU profiler for python imposing very little overhead at runtime.

Both VMProf and JitViewer share a common goal: Present useful information for your python program.
The combination of both can reveal more information than either alone.
That is the reason why they are now both packaged together.
We also updated vmprof.com with various bug fixes and changes including an all new interface to JV.

This work was done with the goal of improving tooling and libraries around the Python/PyPy/RPython ecosystem.
Some of the tools we have developed:

• CFFI - Foreign Function Interface that avoids CPyExt (CFFI docs)
• RevDB - A reverse debugger for python (RevDB blog post)

and of course the tools we discuss here:

• VMProf - A statistical CPU profiler (VMProf docs)
• JitViewer - Visualization of the log file produced by RPython (JitLog docs)

A "brand new" JitViewer

JitViewer has two pieces: you create a log file when running your program, and then use a graphic tool to view what happened.

The old logging format was a hard-to-maintain, plain-text-logging facility. Frequent changes often broke internal tools.
Additionally, the logging output of a long running program required a lot of disk space.

Our new binary format encodes data densely, makes use of some compression (gzip), and tries to remove repetition where possible.
It also supports versioning for future proofing and can be extended easily.

And *drumroll* you no longer need to install a tool to view the log yourself
anymore! The whole system moved to vmprof.com and you can use it any time.

Sounds great. But what can you do with it? Here are two examples for a PyPy user:

PyPy crashed? Did you discover a bug?

For some hard to find bugs it is often necessary to look at the compiled code. The old
procedure often required you to upload a plain text file which was hard to parse and to look through.

A better way to share a crash report is to install the ``vmprof`` module from PyPi and execute either of the two commands:

# this program does not crash, but has some weird behaviour
$ pypy -m jitlog --web <your program args>
...
PyPy Jitlog: http://vmprof.com/#/<hash>/traces
# this program segfaults
$ pypy -m jitlog -o /tmp/log <your program args>
...
<Segfault>
$ pypy -m jitlog --upload /tmp/log
PyPy Jitlog: http://vmprof.com/#/<hash>/traces


Providing the link in the bug report allows PyPy developers to browse and identify potential issues.

Speed issues

VMProf is a great tool to find hot spots that consume a lot of time in your program. As soon as you have identified code that runs slowly, you can switch to jitlog and maybe pinpoint certain aspects that do not behave as expected. You will find an overview, and are able to browse the generated code. If you cannot make sense of all that, you can just share the link with us and we can have a look too.

Future direction

We hope that the new release will help both PyPy developers and PyPy users resolve potential issues and easily point them out.

Here are a few ideas what might come in the next few releases:

•  Combination of CPU profiles and the JITLOG (sadly did not make it into the current release).
• Extend vmprof.com to be able to query vmprof/jitlog.
  An example query for vmprof: 'methods.callsites() > 5' and
  for the jitlog would be 'traces.contains('call_assembler').hasbridge('*my_func_name*')'.
• Extend the jitlog to capture the information of the optimization stage.

Richard Plangger (plan_rich) and the PyPy team

PyPy gets funding from Mozilla for Python 3.5 support

"Python 2.x versus Python 3.x": this is by now an old question. In the eyes of some people Python 2 is here to stay, and in the eyes of others Python has long been 3 only.

PyPy's own position is that PyPy will support Python 2.7 forever---the RPython language in which PyPy is written is a subset of 2.7, and we have no plan to upgrade that. But at the same time, we want to support 3.x. This is particularly true now: a relatively recent development is that Python 3.5 seems to attract more and more people. The "switch" to Python 3.x might be starting to happen.

Correspondingly, PyPy has been searching for a while for a way to support a larger-scale development effort. The goal is to support not just any old version of Python 3.x, but Python 3.5, as this seems to be the version that people are switching to. PyPy is close to supporting all of Python 3.3 now; but the list of what is new in Python 3.4 and 3.5 is far, far longer than anyone imagines. The long-term goal is also to get a version of "PyPy3" that is as good as "PyPy2" is, including its performance and its cpyext layer (CPython C API interoperability), for example.

So, the end result: Mozilla recently decided to award $200,000 to Baroque Software to work on PyPy as part of its Mozilla Open Source Support (MOSS) initiative. This money will be used to implement the Python 3.5 features in PyPy. Within the next year, we plan to use the money to pay four core PyPy developers half-time to work on the missing features and on some of the big performance and cpyext issues. This should speed up the progress of catching up with Python 3.x significantly. We are extremely thankful to Mozilla for supporting us in this way, and will keep you updated on the progress via this blog.