Thursday, August 18, 2011

PyPy 1.6 - kickass panda

We're pleased to announce the 1.6 release of PyPy. This release brings a lot of bugfixes and performance improvements over 1.5, and improves support for Windows 32bit and OS X 64bit. This version fully implements Python 2.7.1 and has beta level support for loading CPython C extensions. You can download it here:

http://pypy.org/download.html

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7.1. It's fast (pypy 1.6 and cpython 2.6.2 performance comparison) due to its integrated tracing JIT compiler.

This release supports x86 machines running Linux 32/64 or Mac OS X. Windows 32 is beta (it roughly works but a lot of small issues have not been fixed so far). Windows 64 is not yet supported.

The main topics of this release are speed and stability: on average on our benchmark suite, PyPy 1.6 is between 20% and 30% faster than PyPy 1.5, which was already much faster than CPython on our set of benchmarks.

The speed improvements have been made possible by optimizing many of the layers which compose PyPy. In particular, we improved: the Garbage Collector, the JIT warmup time, the optimizations performed by the JIT, the quality of the generated machine code and the implementation of our Python interpreter.

Highlights

  • Numerous performance improvements, overall giving considerable speedups:
    • better GC behavior when dealing with very large objects and arrays
    • fast ctypes: now calls to ctypes functions are seen and optimized by the JIT, and they are up to 60 times faster than PyPy 1.5 and 10 times faster than CPython
    • improved generators(1): simple generators now are inlined into the caller loop, making performance up to 3.5 times faster than PyPy 1.5.
    • improved generators(2): thanks to other optimizations, even generators that are not inlined are between 10% and 20% faster than PyPy 1.5.
    • faster warmup time for the JIT
    • JIT support for single floats (e.g., for array('f'))
    • optimized dictionaries: the internal representation of dictionaries is now dynamically selected depending on the type of stored objects, resulting in faster code and smaller memory footprint. For example, dictionaries whose keys are all strings, or all integers. Other dictionaries are also smaller due to bugfixes.
  • JitViewer: this is the first official release which includes the JitViewer, a web-based tool which helps you to see which parts of your Python code have been compiled by the JIT, down until the assembler. The jitviewer 0.1 has already been release and works well with PyPy 1.6.
  • The CPython extension module API has been improved and now supports many more extensions. For information on which one are supported, please refer to our compatibility wiki.
  • Multibyte encoding support: this was of of the last areas in which we were still behind CPython, but now we fully support them.
  • Preliminary support for NumPy: this release includes a preview of a very fast NumPy module integrated with the PyPy JIT. Unfortunately, this does not mean that you can expect to take an existing NumPy program and run it on PyPy, because the module is still unfinished and supports only some of the numpy API. However, barring some details, what works should be blazingly fast :-)
  • Bugfixes: since the 1.5 release we fixed 53 bugs in our bug tracker, not counting the numerous bugs that were found and reported through other channels than the bug tracker.

Cheers,

Hakan Ardo, Carl Friedrich Bolz, Laura Creighton, Antonio Cuni, Maciej Fijalkowski, Amaury Forgeot d'Arc, Alex Gaynor, Armin Rigo and the PyPy team

20 comments:

Anonymous said...

Finally :) I'm really looking forward to test this code out :)

illume said...

Congrats team pypy!

Anonymous said...

I look forward to support Python 3

Anonymous said...

"and has beta level support for loading CPython C extensions"

does this mean that the regular Numpy and Scipy can be used with this.

almir karic said...

no.

"Unfortunately, this does not mean that you can expect to take an existing NumPy program and run it on PyPy"

thanks for the release pypy team!

Anonymous said...

Impressive as always. Thanks for releasing such great software.
Keep up the good work.

Anghel

profu said...

Where is the windows version?

Anonymous said...

I did some benchmark with some simple parameterized SELECT statements, and found that pg8000 on pypy 1.6 is more than one time slower than pg8000 on python 2.7.1, while the later is already more than one time slower than psycopg2 on python 2.7.1.

Anonymous said...

Still can't build and run python-ldap extension... :(
That's a deal-breaker for me.

Maciej Szumocki said...

What kind of problems prevent releasing a Windows 64bit version?

Lenz said...

Congrats !!! Realy amazing job !!

By the way, where can I find more informations about the alredy implemented numpy functions ?

Thanks.

jensck said...

Amazing - PyPy just keeps making leaps and bounds forward for compat. and processing performance. I don't know how you guys keep up such pace, but I dig it!

How is typical memory usage these days? It's been a while since anything was reported on its resource usage vs. CPython. Maybe such a benchmark could be added to the speed site?

said...

PyPy 1.5: 68 seconds
PyPy 1.6: 65 seconds
Python 3.2 (Intel C compiler): 36 seconds

Extrapolation to the future:
PyPy 1.17: 35 seconds ?

Jokes aside, PyPy's compatibility with CPython is good.

Anonymous said...

I'm still most looking forward to the day your jit makes some .pyj files. Initialization time for the jit is a bit high, especially if you use pypy integrated into other scripts where the init time might impact performance, numpy had no dtypes, and laked almost every function, but atleast it's a step in the right direction :)

Memory usage for one of my own test apps (building a one dimensional dict with int keys, and object (sometimes by reference from a second dict) resulted in 76MB to python2.7 and 108MB to pypy 1.6. So memory usage is still a bit behind tho (the pypy runtime was better with around 35% tho).

Anonymous said...

Is there a 32-bit OSX version somewhere? 64-bit seems to eat up memory in my tests...
Impressive stuff, though :-)

Anonymous said...

But man! What's wrong with Windows?

Will the windows version will be dropped?

Version 1.5 does not fully work on Windows and now you release 1.6 you does not provide a windows version...

So, I really want to know if it will be future support for windows.

This will help to decide if pypy will be an option or one just have to find other options to speed up the programs.

Please, clarify this.

Bests,

Anonymous said...

Pypy does support Windows 32bit, with a couple of bugs, the windows support have been improved from 1.5 to 1.6. Perhaps it will be fully working by 1.7.

Anonymous said...

Ok, but where to download PYPY for Win32 ?

Anonymous said...

I believe you got to compile it yourself.

vak said...

just impressive. If you guys could resurrect the numpy operation like:

boolean_array = arr > value

it would be just a dream. This important operation returns not an array, but a value now.