PyPy2.7 and PyPy3.5 v5.7 - two in one release

The PyPy team is proud to release both PyPy2.7 v5.7 (an interpreter supporting Python v2.7 syntax), and a beta-quality PyPy3.5 v5.7 (an interpreter for Python v3.5 syntax). The two releases are both based on much the same codebase, thus the dual release. Note that PyPy3.5 only supports Linux 64bit for now.

This new PyPy2.7 release includes the upstream stdlib version 2.7.13, and PyPy3.5 (our first in the 3.5 series) includes the upstream stdlib version 3.5.3.

We continue to make incremental improvements to our C-API compatibility layer (cpyext). PyPy2 can now import and run many C-extension packages, among the most notable are Numpy, Cython, and Pandas. Performance may be slower than CPython, especially for frequently-called short C functions. Please let us know if your use case is slow, we have ideas how to make things faster but need real-world examples (not micro-benchmarks) of problematic code.

Work proceeds at a good pace on the PyPy3.5 version due to a grant from the Mozilla Foundation, hence our first 3.5.3 beta release. Thanks Mozilla !!! While we do not pass all tests yet, asyncio works and as these benchmarks show it already gives a nice speed bump. We also backported the f"" formatting from 3.6 (as an exception; otherwise “PyPy3.5” supports the Python 3.5 language).

CFFI has been updated to 1.10, improving an already great package for interfacing with C.

We now use shadowstack as our default gcrootfinder even on Linux. The alternative, asmgcc, will be deprecated at some future point. While about 3% slower, shadowstack is much more easily maintained and debuggable. Also, the performance of shadowstack has been improved in general: this should close the speed gap between other platforms and Linux.

As always, this release fixed many issues and bugs raised by the growing community of PyPy users. We strongly recommend updating.

You can download the v5.7 release here:

http://pypy.org/download.html

We would like to thank our donors for the continued support of the PyPy project.
We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: PyPy and RPython documentation improvements, tweaking popular modules to run on pypy, or general help with making RPython’s JIT even better.

 

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7 and CPython 3.5. It’s fast (PyPy and CPython 2.7.x performance comparison) due to its integrated tracing JIT compiler.
We also welcome developers of other dynamic languages to see what RPython can do for them.
The PyPy 2.7 release supports:

• x86 machines on most common operating systems (Linux 32/64 bits, Mac OS X 64 bits, Windows 32 bits, OpenBSD, FreeBSD)
• newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux,
• big- and little-endian variants of PPC64 running Linux,
• s390x running Linux

 

What else is new?

(since the releases of PyPy 2.7 and 3.3 at the end of 2016)

There are many incremental improvements to RPython and PyPy, the complete listing is here.

 

Please update, and continue to help us make PyPy better.

Cheers, The PyPy team

Leysin Winter Sprint Summary

Today is the last day of our yearly sprint event in Leysin. We had lots of ideas on how to enhance the current state of PyPy, we went skiing and had interesting discussions around virtual machines, the Python ecosystem, and other real world problems.

 

Why don't you join us next time?

A usual PyPy sprints day goes through the following stages:

1.  Planning Session: Tasks from previous days that have seen progress or are completed are noted in a shared document. Everyone adds new tasks and then assigns themselves to one or more tasks (usually in pairs). As soon as everybody is happy with their task and has a partner to work with, the planning session is concluded and the work can start.
2. Discussions: A sprint is a good occasion to discuss difficult and important topics in person. We usually sit down in a separate area in the sprint room and discuss until a) nobody wants to discuss anymore or b) we found a solution to the problem. The good thing is that usally the outcome is b).
3. Lunch: For lunch we prepare sandwiches and other finger food.
4. Continue working until dinner, which we eat at a random restaurant in Leysin.
5. Goto 1 the next day, if sprint has not ended.

Sprints are open to everybody and help newcomers to get started with PyPy (we usually pair you with a developer familiar with PyPy). They are perfect to discuss and find solutions to problems we currently face. If you are eager to join next year, please don't hesitate to register next year around January.

 

Sprint Summary   

Sprint goals included to work on the following topics:

• Work towards releasing PyPy 3.5 (it will be released soon)
• CPython Extension (CPyExt) modules on PyPy
• Have fun in winter sports (a side goal)

Highlights

• We have spent lots of time debugging and fixing memory issues on CPyExt. In particular, we fixed a serious memory leak where taking a memoryview would prevent numpy arrays from ever being freed. More work is still required to ensure that our GC always releases arrays in a timely manner.
• Fruitful discussions and progress about how to flesh out some details about the unicode representation in PyPy. Our current goal is to use utf-8 as the unicode representation internally and have fast vectorized operations (indexing, check if valid, ...).
• PyPy will participate in GSoC 2017 and we will try to allocate more resources to that than last year.
• Profile and think about some details how to reduce the starting size of the interpreter. The starting point would be to look at the parser and reduce the amount of strings to keep alive.
• Found a topic for a student's master thesis: correctly freeing cpyext reference cycles.
• Run lots of Python3 code on top of PyPy3 and resolve issues we found along the way.
• Initial work on making RPython thread-safe without a GIL.

List of attendees

- Stefan Beyer

- Antonio Cuni

- Maciej Fijalkowski

- Manuel Jacob

- Ronan Lamy

- Remi Meier

- Richard Plangger

- Armin Rigo

- Robert Zaremba

 

 

We would like to thank our donors for the continued support of the PyPy project and we looking forward to next years sprint in Leysin.

The PyPy Team

Async HTTP benchmarks on PyPy3

Hello everyone,

Since Mozilla announced funding, we've been working quite hard on delivering you a working Python 3.5.

 

We are almost ready to release an alpha version of PyPy 3.5. Our goal is to release it shortly after the sprint. Many modules have already been ported and  it can probably run many Python 3 programs already. We are happy to receive any feedback after the next release. 

To show that the heart (asyncio) of Python 3 is already working we have prepared some benchmarks. They are done by Paweł Piotr Przeradowski @squeaky_pl for a HTTP workload on serveral asynchronous IO libraries, namely the relatively new asyncio and curio libraries and the battle-tested tornado, gevent and Twisted libraries. To see the benchmarks check out https://github.com/squeaky-pl/zenchmarks and the instructions for reproducing can be found inside README.md in the repository. Raw results can be obtained from https://github.com/squeaky-pl/zenchmarks/blob/master/results.csv.

The purpose of the presented benchmarks is showing that the upcoming PyPy release is already working with unmodified code that runs on CPython 3.5. PyPy also manages to make them run significantly faster.

The benchmarks consist of HTTP servers implemented on the top of the mentioned libraries. All the servers are single-threaded relying on underlying event loops to provide concurrency. Access logging was disabled to exclude terminal I/O from the results. The view code consists of a lookup in a dictionary mapping ASCII letters to verses from the famous Zen of Python. If a verse is found the view returns it, otherwise a 404 Not Found response is served. The 400 Bad Request and 500 Internal Server Error cases are also handled.

The workload was generated with the wrk HTTP benchmarking tool. It is run with one thread opening up to 100 concurrent connections for 2 seconds and repeated 1010 times to get consecutive measures. There is a Lua script provided that instructs wrk to continuously send 24 different requests that hit different execution paths (200, 404, 400) in the view code. Also it is worth noting that wrk will only count 200 responses as successful so the actual request per second throughput is higher.

For your convenience all the used libraries versions are vendored into the benchmark repository. There is also a precompiled portable version of wrk provided that should run on any reasonably recent (10 year old or newer) Linux x86_64 distribution. The benchmark was performed on a public cloud scaleway x86_64 server launched in a Paris data center. The server was running Ubuntu 16.04.01 LTS and reported Intel(R) Xeon(R) CPU D-1531 @ 2.20GHz CPU. CPython 3.5.2 (shipped by default in Ubuntu) was benchmarked against a pypy-c-jit-90326-88ef793308eb-linux64 snapshot of the 3.5 compatibility branch of PyPy.

 

 

 

 

We want to thank Mozilla for supporting our work!

Cheers,

fijal, squeaky_pl and the PyPy Team