PyPy Leysin Winter Sprint (11-19st January 2014)

The next PyPy sprint will be in Leysin, Switzerland, for the ninth time. This is a fully public sprint: newcomers and topics other than those proposed below are welcome.

Goals and topics of the sprint

• Py3k: work towards supporting Python 3 in PyPy
• NumPyPy: work towards supporting the numpy module in PyPy
• STM: work towards supporting Software Transactional Memory
• And as usual, the main side goal is to have fun in winter sports :-) We can take a day off for ski.

Exact times

For a change, and as an attempt to simplify things, I specified the dates as 11-19 January 2014, where 11 and 19 are travel days. We will work full days between the 12 and the 18. You are of course allowed to show up for a part of that time only, too.

Location & Accomodation

Leysin, Switzerland, "same place as before". Let me refresh your memory: both the sprint venue and the lodging will be in a very spacious pair of chalets built specifically for bed & breakfast: http://www.ermina.ch/. The place has a good ADSL Internet connexion with wireless installed. You can of course arrange your own lodging anywhere (as long as you are in Leysin, you cannot be more than a 15 minutes walk away from the sprint venue), but I definitely recommend lodging there too -- you won't find a better view anywhere else (though you probably won't get much worse ones easily, either :-)

Please confirm that you are coming so that we can adjust the reservations as appropriate. The rate so far has been around 60 CHF a night all included in 2-person rooms, with breakfast. There are larger rooms too (less expensive per person) and maybe the possibility to get a single room if you really want to.

Please register by Mercurial:

https://bitbucket.org/pypy/extradoc/
https://bitbucket.org/pypy/extradoc/raw/extradoc/sprintinfo/leysin-winter-2014

or on the pypy-dev mailing list if you do not yet have check-in rights:

http://mail.python.org/mailman/listinfo/pypy-dev

You need a Swiss-to-(insert country here) power adapter. There will be some Swiss-to-EU adapters around -- bring a EU-format power strip if you have one.

PyPy 2.2.1 - Incrementalism.1

We're pleased to announce PyPy 2.2.1, which targets version 2.7.3 of the Python language. This is a bugfix release over 2.2.

You can download the PyPy 2.2.1 release here:

http://pypy.org/download.html

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It's fast (pypy 2.2 and cpython 2.7.2 performance comparison) due to its integrated tracing JIT compiler.

This release supports x86 machines running Linux 32/64, Mac OS X 64, Windows 32, or ARM (ARMv6 or ARMv7, with VFPv3).

Work on the native Windows 64 is still stalling, we would welcome a volunteer to handle that.

Highlights

This is a bugfix release. The most important bugs fixed are:

• an issue in sockets' reference counting emulation, showing up notably when using the ssl module and calling makefile().
• Tkinter support on Windows.
• If sys.maxunicode==65535 (on Windows and maybe OS/X), the json decoder incorrectly decoded surrogate pairs.
• some FreeBSD fixes.

Note that CFFI 0.8.1 was released. Both versions 0.8 and 0.8.1 are compatible with both PyPy 2.2 and 2.2.1.

Cheers, Armin Rigo & everybody

CFFI 0.8

Hi all,

CFFI 0.8 for CPython (2.6-3.x) has been released.

Quick download: pip install cffi --upgrade
Documentation: https://cffi.readthedocs.org/en/release-0.8/

What's new: a number of small fixes; ffi.getwinerror(); integrated support for C99 variable-sized structures; multi-thread safety.

--- Armin

Update: CFFI 0.8.1, with fixes on Python 3 on OS/X, and some FreeBSD fixes (thanks Tobias).

NumPy status update

Here is what has been happening with NumPy in PyPy in October thanks to the people who donated to the NumPyPy proposal:

The biggest change is that we shifted to using an external fork of numpy rather than a minimal numpypy module. The idea is that we will be able to reuse most of the upstream pure-python numpy components, replacing the C modules with appropriate RPython micronumpy pieces at the correct places in the module namespace.

The numpy fork should work just as well as the old numpypy for functionality that existed previously, and also include much new functionality from the pure-python numpy pieces that simply hadn't been imported yet in numpypy. However, this new functionality will not have been "hand picked" to only include pieces that work, so you may run into functionality that relies on unimplemented components (which should fail with user-level exceptions).

This setup also allows us to run the entire numpy test suite, which will help in directing future compatibility development. The recent PyPy release includes these changes, so download it and let us know how it works! And if you want to live on the edge, the nightly includes even more numpy progress made in November.

To install the fork, download the latest release, and then install numpy either separately with a virtualenv: pip install git+https://bitbucket.org/pypy/numpy.git; or directly: git clone https://bitbucket.org/pypy/numpy.git; cd numpy; pypy setup.py install.

EDIT: if you install numpy as root, you may need to also import it once as root before it works: sudo pypy -c 'import numpy'

Along with this change, progress was made in fixing internal micronumpy bugs and increasing compatibility:

• Fixed a bug with strings in record dtypes
• Fixed a bug where the multiplication of an ndarray with a Python int or float resulted in loss of the array's dtype
• Fixed several segfaults encountered in the numpy test suite (suite should run now without segfaulting)

We also began working on __array_prepare__ and __array_wrap__, which are necessary pieces for a working matplotlib module.

Cheers,
Romain and Brian

PyPy 2.2 - Incrementalism

We're pleased to announce PyPy 2.2, which targets version 2.7.3 of the Python language. This release main highlight is the introduction of the incremental garbage collector, sponsored by the Raspberry Pi Foundation.
This release also contains several bugfixes and performance improvements.
You can download the PyPy 2.2 release here:

http://pypy.org/download.html

We would like to thank our donors for the continued support of the PyPy project. We showed quite a bit of progress on all three projects (see below) and we're slowly running out of funds. Please consider donating more so we can finish those projects! The three projects are:

• Py3k (supporting Python 3.x): the release PyPy3 2.2 is imminent.
• STM (software transactional memory): a preview will be released very soon, as soon as we fix a few bugs
• NumPy: the work done is included in the PyPy 2.2 release. More details below.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It's fast (pypy 2.2 and cpython 2.7.2 performance comparison) due to its integrated tracing JIT compiler.
This release supports x86 machines running Linux 32/64, Mac OS X 64, Windows 32, or ARM (ARMv6 or ARMv7, with VFPv3).
Work on the native Windows 64 is still stalling, we would welcome a volunteer to handle that.

Highlights

• Our Garbage Collector is now "incremental". It should avoid almost all pauses due to a major collection taking place. Previously, it would pause the program (rarely) to walk all live objects, which could take arbitrarily long if your process is using a whole lot of RAM. Now the same work is done in steps. This should make PyPy more responsive, e.g. in games. There are still other pauses, from the GC and the JIT, but they should be on the order of 5 milliseconds each.
• The JIT counters for hot code were never reset, which meant that a process running for long enough would eventually JIT-compile more and more rarely executed code. Not only is it useless to compile such code, but as more compiled code means more memory used, this gives the impression of a memory leak. This has been tentatively fixed by decreasing the counters from time to time.
• NumPy has been split: now PyPy only contains the core module, called _numpypy. The numpy module itself has been moved to https://bitbucket.org/pypy/numpy and numpypy disappeared. You need to install NumPy separately with a virtualenv: pip install git+https://bitbucket.org/pypy/numpy.git; or directly: git clone https://bitbucket.org/pypy/numpy.git; cd numpy; pypy setup.py install.
• non-inlined calls have less overhead
• Things that use sys.set_trace are now JITted (like coverage)
• JSON decoding is now very fast (JSON encoding was already very fast)
• various buffer copying methods experience speedups (like list-of-ints to int[] buffer from cffi)
• We finally wrote (hopefully) all the missing os.xxx() functions, including os.startfile() on Windows and a handful of rare ones on Posix.
• numpy has a rudimentary C API that cooperates with cpyext

Cheers,
Armin Rigo and Maciej Fijalkowski

Py3k status update #12

This is the 12th status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

Here's an update on the recent progress:

• Thank you to everyone who has provided initial feedback on the PyPy3 2.1 beta
  1 release. We've gotten a number of bug reports, most of which have been
  fixed.
• As usual, we're continually keeping up with changes from the default
  branch. Oftentimes these merges come at a cost (conflicts and or
  reintegration of py3k changes) but occasionally we get goodies for free, such
  as the recent JIT optimizations and incremental garbage collection.
• We've been focusing on re-optimizing Python 2 int sized (machine sized)
  integers:

We have a couple of known, notable speed regressions in the PyPy3 beta release
vs regular PyPy. The major one being with Python 2.x int sized (or machine
sized) integers.

Python 3 drops the distinction between int and long types. CPython 3.x
accomplishes this by removing the old int type entirely and renaming the long
type to int. Initially, we've done the same for PyPy3 for the sake of
simplicity and getting everything working.

However PyPy's JIT is capable of heavily optimizing these machine sized integer
operations, so this came with a regression in performance in this area.

We're now in the process of solving this. Part of this work also involves some
house cleaning on these numeric types which also benefits the default branch.

cheers,
Phil