Leysin 2020 Sprint Report

At the end of February ten of us gathered in Leysin, Switzerland to work on
a variety of topics including HPy, PyPy Python 3.7 support and the PyPy
migration to Heptapod.

We had a fun and productive week. The snow was beautiful. There was skiing
and lunch at the top of Berneuse, cooking together, some late nights at
the pub next door, some even later nights coding, and of course the
obligatory cheese fondue outing.

There were a few of us participating in a PyPy sprint for the first time
and a few familiar faces who had attended many sprints. Many different
projects were represented including PyPy, HPy, GraalPython,
Heptapod, and rust-cpython. The atmosphere was relaxed and welcoming, so if
you're thinking of attending the next one -- please do!

Topics worked on:

HPy

HPy is a new project to design and implement a better API for extending
Python in C. If you're unfamiliar with it you can read more about it at
HPy.

A lot of attention was devoted to the Big HPy Design Discussion which
took up two full mornings. So much was decided that this will likely
get its own detailed write-up, but bigger topics included:

• the HPy GetAttr, SetAttr, GetItem and SetItem methods,
• HPy_FromVoidP and HPy_AsVoidP for passing HPy handles to C functions
  that pass void* pointers to callbacks,
• avoiding having va_args as part of the ABI,
• exception handling,
• support for creating custom types.

Quite a few things got worked on too:

• implemented support for writing methods that take keyword arguments with
  HPy_METH_KEYWORDS,
• implemented HPy_GetAttr, HPy_SetAttr, HPy_GetItem, and HPy_SetItem,
• started implementing support for adding custom types,
• started implementing dumping JSON objects in ultrajson-hpy,
• refactored the PyPy GIL to improve the interaction between HPy and
  PyPy's cpyext,
• experimented with adding HPy support to rust-cpython.

And there was some discussion of the next steps of the HPy initiative
including writing documentation, setting up websites and funding, and
possibly organising another HPy gathering later in the year.

PyPy

• Georges gave a presentation on the Heptapod topic and branch workflows
  and showed everyone how to use hg-evolve.
• Work was done on improving the PyPy CI buildbot post the move to
  heptapod, including a light-weight pre-merge CI and restricting
  when the full CI is run to only branch commits.
• A lot of work was done improving the -D tests.

Miscellaneous

• Armin demoed VRSketch and NaN Industries in VR, including an implementation
  of the Game of Life within NaN Industries!
• Skiing!

Aftermath

Immediately after the sprint large parts of Europe and the world were
hit by the COVID-19 epidemic. It was good to spend time together before
travelling ceased to be a sensible idea and many gatherings were cancelled.

Keep safe out there everyone.

The HPy & PyPy Team & Friends

In joke for those who attended the sprint: Please don't replace this blog post
with its Swedish translation (or indeed a translation to any other language :).

PyPy3 2.3.1 - Fulcrum

We're pleased to announce the first stable release of PyPy3. PyPy3
targets Python 3 (3.2.5) compatibility.

We would like to thank all of the people who donated to the py3k proposal
for supporting the work that went into this.

You can download the PyPy3 2.3.1 release here:

http://pypy.org/download.html#pypy3-2-3-1

Highlights

• The first stable release of PyPy3: support for Python 3!
• The stdlib has been updated to Python 3.2.5
• Additional support for the u'unicode' syntax (PEP 414) from Python 3.3
• Updates from the default branch, such as incremental GC and various JIT
  improvements
• Resolved some notable JIT performance regressions from PyPy2:

• Re-enabled the previously disabled collection (list/dict/set) strategies
• Resolved performance of iteration over range objects
• Resolved handling of Python 3's exception __context__ unnecessarily forcing
  frame object overhead

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for
CPython 2.7.6 or 3.2.5. It's fast due to its integrated tracing JIT compiler.

This release supports x86 machines running Linux 32/64, Mac OS X 64, Windows,
and OpenBSD,
as well as newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux.

While we support 32 bit python on Windows, work on the native Windows 64
bit python is still stalling, we would welcome a volunteer
to handle that.

How to use PyPy?

We suggest using PyPy from a virtualenv. Once you have a virtualenv
installed, you can follow instructions from pypy documentation on how
to proceed. This document also covers other installation schemes.

Cheers,
the PyPy team

Py3k status update #13

This is the 13th status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

We're just finishing up a cleanup of int/long types. This work helps the py3k
branch unify these types into the Python 3 int and restore JIT compilation of
machine sized integers.

This cleanup also removes multimethods from these types. PyPy has
historically used a clever implementation of multimethod dispatch for declaring
methods of the __builtin__ types in RPython.

This multimethod scheme provides some convenient features for doing this,
however we've come to the conclusion that it may be more trouble than it's
worth. A major problem of multimethods is that they generate a large amount of
stub methods which burden the already lengthy and memory hungry RPython
translation process. Also, their implementation and behavior can be somewhat
complicated/obscure.

The alternative to multimethods involves doing the work of the type checking
and dispatching rules in a more verbose, manual way. It's a little more work in
the end but less magical.

Recently, Manuel Jacob finished a large cleanup effort of the
unicode/string/bytearray types that also removed their multimethods. This work
also benefits the py3k branch: it'll help with future PEP 393 (or PEP 393
alternative) work. This effort was partly sponsored by Google's Summer of
Code: thanks Manuel and Google!

Now there's only a couple major pieces left in the multimethod removal (the
float/complex types and special marshaling code) and a few minor pieces that
should be relatively easy.

In conclusion, there's been some good progress made on py3k and multimethod
removal this winter, albeit a bit slower than we would have liked.

cheers,
Phil

Py3k status update #12

This is the 12th status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

Here's an update on the recent progress:

• Thank you to everyone who has provided initial feedback on the PyPy3 2.1 beta
  1 release. We've gotten a number of bug reports, most of which have been
  fixed.
• As usual, we're continually keeping up with changes from the default
  branch. Oftentimes these merges come at a cost (conflicts and or
  reintegration of py3k changes) but occasionally we get goodies for free, such
  as the recent JIT optimizations and incremental garbage collection.
• We've been focusing on re-optimizing Python 2 int sized (machine sized)
  integers:

We have a couple of known, notable speed regressions in the PyPy3 beta release
vs regular PyPy. The major one being with Python 2.x int sized (or machine
sized) integers.

Python 3 drops the distinction between int and long types. CPython 3.x
accomplishes this by removing the old int type entirely and renaming the long
type to int. Initially, we've done the same for PyPy3 for the sake of
simplicity and getting everything working.

However PyPy's JIT is capable of heavily optimizing these machine sized integer
operations, so this came with a regression in performance in this area.

We're now in the process of solving this. Part of this work also involves some
house cleaning on these numeric types which also benefits the default branch.

cheers,
Phil

Py3k status update #11

This is the 11th status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

Here's some highlights of the progress made since the previous update:

• PyPy py3k now matches CPython 3's hash code for
  int/float/complex/Decimal/Fraction
• Various outstanding unicode identifier related issues were
  resolved. E.g. test_importlib/pep263/ucn/unicode all now fully pass. Various
  usage of identifiers (in particular type and module names) have been fixed to
  handle non-ascii names -- mostly around display of reprs and exception
  messages.
• The unicodedata database has been upgraded to 6.0.0.
• Windows support has greatly improved, though it could still use some more
  help (but so does the default branch to a certain degree).
• Probably the last of the parsing related bugs/features have been taken care
  of.
• Of course various other smaller miscellaneous fixes

This leaves the branch w/ only about 5 outstanding failures of the stdlib test
suite:

• test_float

  1 failing test about containment of floats in collections.

• test_memoryview

  Various failures: requires some bytes/str changes among other things (Manuel
  Jacob's has some progress on this on the py3k-memoryview branch)

• test_multiprocessing

  1 or more tests deadlock on some platforms

• test_sys and test_threading

  2 failing tests for the New GIL's new API

Probably the biggest feature left to tackle is the New GIL.

We're now pretty close to pushing an initial release. We had planned for one
around PyCon, but having missed that we've put some more effort into the branch
to provide a more fully-fledged initial release.

Thanks to the following for their contributions: Manuel Jacob, Amaury Forgeot
d'Arc, Karl Ramm, Jason Chu and Christian Hudon.

cheers,
Phil

Py3k status update #10

This is the tenth status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

There's been significant progress since the last update: the linux x86-32
buildbot now passes 289 out of approximately 354 modules (with 39 skips) of
CPython's regression test suite.

That means there's only 26 test module failures left! The list of major items
remaining for 3.2 compatibility are now short enough to list here, with their
related tests:

• Tokenizer support for non-ascii identifiers

• test_importlib
• test_pep263

• memoryview (Manuel Jacob's tackling this on the py3k-memoryview branch)

• test_memoryview

• multiprocessing module currently deadlocks

• test_multiprocessing

• Buggy handling of the new extended unpacking syntax by the compiler:

• test_unpack_ex

• The new Global Interpreter Lock and new thread signal handling

• test_threading
• test_threadsignals
• test_sys

• Upgrade unicodedata to 6.0.0 (requires updates to the actual unicodedata
  generation script)

• test_ucn
• test_unicode
• test_unicodedata

• CPyExt

• test_capi (currently crashes)

• Update int's hash code to match to CPython (float's is already updated on the
  py3k-newhash branch. note that PyPy 2.x doesn't even totally match
  CPython's hashing)

• test_decimal
• test_fractions
• test_numeric_tower

• Miscellaneous:

• test_complex
• test_float
• test_peepholer
• test_range
• test_sqlite (a new cffi based version seems to be coming)
• test_ssl
• test_struct
• test_subprocess
• test_sys_settrace
• test_time

Additionally there are still a number of failures in PyPy's internal test
suite. These tests are usually ran against untranslated versions of PyPy during
development. However we've now began running them against a fully translated
version of PyPy on the buildbot too (thanks to Amaury for setting this
up). This further ensures that our tests and implementation are sane.

We're getting closer to producing an initial alpha release. Before that happens
we'd like to see:

• further test fixes
• the results of test runs on other major platforms (e.g. linux x86-64 and osx
  seem to have some additional failures as of now)
• some basic real world testing

Finally I'd like to thank Manuel Jacob for his various contributions over the
past month, including fixing the array and ctypes modules among other things,
and also Amaury Forgeot d'Arc for his ongoing excellent contributions.

cheers,
Phil

Py3k status update #9

This is the ninth status update about our work on the py3k branch, which
we can work on thanks to all of the people who donated to the py3k
proposal.

Just a very short update on December's work: we're now passing about 223 of
approximately 355 modules of CPython's regression test suite, up from passing
194 last month.

Some brief highlights:

• More encoding related issues were addressed. e.g. now most if not all the
  multibytecodec test modules pass.
• Fixed some path handling issues (test_os, test_ntpath and
  test_posixpath now pass)
• We now pass test_class, test_descr and almost test_builtin (among
  other things): these are notable as they are fairly extensive test suites of
  core aspects of the langauge.
• Amaury Forgeot d'Arc continued making progress on CPyExt (thanks again!)

cheers,
Phil

Py3k status update #8

This is the eight status update about our work on the py3k branch, which
we can work on thanks to all of the people who donated to the py3k
proposal.

Just a short update on November's work: we're now passing about 194 of
approximately 355 modules of CPython's regression test suite, up from passing
160 last month. Many test modules only fail a small number of individual tests
now.

We'd like to thank Amaury Forgeot d'Arc for his contributions, in particular he
has made significant progress on updating CPyExt for Python 3 this month.

Some other highlights:

• test_marshal now passes, and there's been significant progress on
  pickling (thanks Kenny Levinsen and Amaury for implementing
  int.{to,from}_bytes)
• We now have a _posixsubprocess module
• More encoding related fixes, which affects many failing tests
• _sre was updated and now test_re almost passes
• Exception behavior is almost complete per the Python 3 specs, what's mostly
  missing now are the new __context__ and __traceback__ attributes (PEP
  3134)
• Fixed some crashes and deadlocks occurring during the regression tests
• We merged the unicode-strategies branch both to default and to py3k: now we
  have versions of lists, dictionaries and sets specialized for unicode
  elements, as we already had for strings.
• However, for string-specialized containers are still faster in some cases
  because there are shortcuts which have not been implemented for unicode yet
  (e.g., constructing a set of strings from a list of strings). The plan is to
  completely kill the shortcuts and improve the JIT to produce the fast
  version automatically for both the string and unicode versions, to have a
  more maintainable codebase without sacrificing the speed. The autoreds
  branch (already merged) was a first step in this direction.

cheers,
Philip&Antonio

Py3k status update #7

This is the seventh status update about our work on the py3k branch, which
we can work on thanks to all of the people who donated to the py3k
proposal.

The biggest news is that this month Philip started to work on py3k in parallel
to Antonio. As such, there was an increased amount of activity.

The py3k buildbots now fully translate the branch every night and run the
Python standard library tests.

We currently pass 160 out of approximately 355 modules of CPython's standard
test suite, fail 144 and skip approximately 51.

Some highlights:

• dictviews (the objects returned by dict.keys/values/items) has been greatly
  improved, and now they full support set operators
• a lot of tests has been fixed wrt complex numbers (and in particular the
  __complex__ method)
• _csv has been fixed and now it correctly handles unicode instead of bytes
• more parser fixes, py3k list comprehension semantics; now you can no longer
  access the list comprehension variable after it finishes
• 2to3'd most of the lib_pypy modules (pypy's custom standard lib
  replacements/additions)
• py3-enabled pyrepl: this means that finally readline works at the command
  prompt, as well as builtins.input(). pdb seems to work, as well as
  fancycompleter to get colorful TAB completions :-)
• py3 round
• further tightening/cleanup of the unicode handling (more usage of
  surrogateescape, surrogatepass among other things)
• as well as keeping up with some big changes happening on the default branch
  and of course various other fixes.

Finally, we would like to thank Amaury Forgeot d'Arc for his significant
contributions.

cheers,
Philip&Antonio

Py3k status update #6

This is the sixth status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

The coolest news is not about what we did in the past weeks, but what we will
do in the next: I am pleased to announce that Philip Jenvey has been
selected by the PyPy communitiy to be funded for his upcoming work on py3k,
thanks to your generous donations. He will start to work on it shortly, and he
will surely help the branch to make faster progress. I am also particularly
happy of this because Philip is the first non-core developer who is getting
paid with donations: he demonstrated over the past months to be able to work
effectively on PyPy, and so we were happy to approve his application for the
job. This means that anyone can potentially be selected in the future, the
only strict requirement is to have a deep interest in working on PyPy and to
prove to be able to do so by contributing to the project.

Back to the status of the branch. Most of the work since the last status
update has been done in the area of, guess what? Unicode strings. As usual,
this is one of the most important changes between Python 2 and Python 3, so
it's not surprising. The biggest news is that now PyPy internally supports
unicode identifiers (such as names of variables, functions, attributes, etc.),
whereas earlier it supported only ASCII bytes strings. The changes is still
barely visible from the outside, because the parser still rejects non-ASCII
identifiers, however you can see it with a bit of creativity:

>>>> def foo(x): pass
>>>> foo(**{'àèìòù': 42})
Traceback (most recent call last):
  File "<console>", line 1, in <module>
TypeError: foo() got an unexpected keyword argument 'àèìòù'

Before the latest changes, you used to get question marks instead of the
proper name for the keyword argument. Although this might seem like a small
detail, it is a big step towards a proper working Python 3 interpreter and it
required a couple of days of headaches. A spin-off of this work is that now
RPython has better built-in support for unicode (also in the default branch):
for example, it now supports unicode string formatting (using the percent
operator) and the methods .encode/.decode('utf-8').

Other than that there is the usual list of smaller issues and bugs that got
fixed, including (but not limited to):

• teach the compiler when to emit the new opcode DELETE_DEREF (and
  implement it!)
• detect when we use spaces and TABs inconsistently in the source code, as
  CPython does
• fix yet another bug related to the new lexically scoped exceptions (this
  is the last one, hopefully)
• port some of the changes that we did to the standard CPython 2.7 tests to
  3.2, to mark those which are implementation details and should not be run on
  PyPy

Finally, I would like to thank Amaury Forgeot d'Arc and Ariel Ben-Yehuda for
their work on the branch; among other things, Amaury recently worked on
cpyext and on the PyPy _cffi_backend, while Ariel submitted a patch to
implement PEP 3138.

Py3k status update #5

This is the fifth status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

Apart from the usual "fix shallow py3k-related bugs" part, most of my work in
this iteration has been to fix the bootstrap logic of the interpreter, in
particular to setup the initial sys.path.

Until few weeks ago, the logic to determine sys.path was written entirely
at app-level in pypy/translator/goal/app_main.py, which is automatically
included inside the executable during translation. The algorithm is more or
less like this:

1. find the absolute path of the executable by looking at sys.argv[0]
   and cycling through all the directories in PATH
2. starting from there, go up in the directory hierarchy until we find a
   directory which contains lib-python and lib_pypy

This works fine for Python 2 where the paths and filenames are represented as
8-bit strings, but it is a problem for Python 3 where we want to use unicode
instead. In particular, whenever we try to encode a 8-bit string into an
unicode, PyPy asks the _codecs built-in module to find the suitable
codec. Then, _codecs tries to import the encodings package, to list
all the available encodings. encodings is a package of the standard
library written in pure Python, so it is located inside
lib-python/3.2. But at this point in time we yet have to add
lib-python/3.2 to sys.path, so the import fails. Bootstrap problem!

The hard part was to find the problem: since it is an error which happens so
early, the interpreter is not even able to display a traceback, because it
cannot yet import traceback.py. The only way to debug it was through some
carefully placed print statement and the help of gdb. Once found the
problem, the solution was as easy as moving part of the logic to RPython,
where we don't have bootstrap problems.

Once the problem was fixed, I was able to finally run all the CPython test
against the compiled PyPy. As expected there are lots of failures, and fixing
them will be the topic of my next months.

Py3k status update #4

This is the fourth status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

For various reasons, less work than usual has been done since the last status
update. However, some interesting things happened anyway.

As readers know, so far we spent most of the effort in fixing all PyPy's own
tests which started to fail for various py2/py3 differences. Most of them
failed for shallow reasons, e.g. syntactic changes or the int/long
unifications. Others failed for subtle differences and needed a bit more care,
for example the fact that unbound methods are gone in Py3k.

The good news is that finally we are seeing the light at the end of the
tunnel. Most of them have been fixed. For sine other tests, we introduced the
concept of "py3k-skipping": some optimizations and modules are indeed failing,
but right now we are concentrating on completing the core language and so we
are not interested in those. When the core language will be done, we will be
able to easily find and work on the py3k-skipped tests. In particular, for
now we disabled the Int and String dict strategies, which are broken
because of the usual int/long unification and str vs bytes. As for modules,
for now _continuation (needed for stackless) and _multiprocessing do
not work yet.

Another non-trivial feature we implemented is the proper cleaning of exception
variables when we exit except blocks. This is a feature which touches
lots of levels of PyPy, starting from astcompiler, down to the bytecode
interpreter. It tooks two days of headache, but at the end we made it :-).

Additionally, Amaury did a lot of improvements to cpyext, which had been
broken since forever on this branch.

As for the next plans, now that things are starting to work and PyPy's own
tests mostly pass, we can finally start to run the compiled PyPy against
CPython's test suite. It is very likely that we will have tons of failures at
the beginning, but once we start to fix them one by one, a Py3k-compatible
PyPy will be closer and closer.

Py3k status update #3

This is the third status update about my work on the py3k branch, which I can work on thanks to all of the people who donated to the py3k proposal.

A lot of work has been done during the last month: as usual, the list of changes is too big to be reported in a detalied way, so this is just a summary of what happened.

One of the most active areas was killing old and deprecated features. In particular, we killed support for the __cmp__ special method and its counsins, the cmp builtin function and keyword argument for list.sort() and sorted(). Killing is easy, but then you have to fix all the places which breaks because of this, including all the types which relied on __cmp__ to be comparable,, fixing all the tests which tried to order objects which are no longer ordeable now, or implementing new behavior like forbidding calling hash() on objects which implement __eq__ but not __hash__.

Among the other features, we killed lots of now-gone functions in the operator module, the builtins apply(), reduce() and buffer, and the os.* functions to deal with temporary files, which has been deprecated in favour of the new tempfile module.

The other topic which can't miss in a py3k status update is, as usual, string-vs-unicode. At this round, we fixed bugs in string formatting (in particular to teach format() to always use unicode strings) and various corner cases about when calling the (possibly overridden) __str__ method on subclasses of str. Believe me, you don't want to know the precise rules :-).

Other features which we worked on and fixed tests include, but are not limited to, marshal, hashlib, zipimport, _socket and itertools, plus the habitual endless lists of tests which fail for shallow reasons such as the syntactic differences, int vs long, range() vs list(range()) etc. As a result, the number of failing tests dropped from 650 to 235: we are beginning to see the light at the end of the tunnel :-)

Benjamin finished implementing Python 3 syntax. Most of it was small cleanups and tweaks to be compatible with CPython such as making True and False keywords and preventing . . . (note spaces between dots) from being parsed as Ellipsis. Larger syntax additions included keyword only arguments and function annotations.

Finally, we did some RPython fixes, so that it is possible again to translate PyPy in the py3k branch. However, the resuling binary is a strange beast which mixes python 2 and python 3 semantics, so it is unusable for anything but showing friends how cool it is.

I would like to underline that I was not alone in doing all this work. In particular, a lot of people joined the PyPy sprint at Pycon and worked on the branch, as you can clearly see in this activity graph. I would like to thank all who helped!

cheers,
Antonio and Benjamin

Py3k status update #2

This is the second status update about my work on the py3k branch, which I can work on thanks to all of the people who donated to the py3k proposal.

Since my previous status update, things have improved a lot: first of all, I fixed the syntax of many more tests, which were failing on the branch because they used constructs which are no longer valid in Python 3, such as u'' strings, the print statement or the old except Exception, e syntax. I have to say that this work is tedious and not very rewarding, but it has to be done anyway, so that the real failures can stand up.

Then, I spent most of the rest of the time by killing features which are present in Python 2 and are gone in Python 3.

Some of them were easy and mechnical: for example, I removed all the function attributes such as func_code and func_closure, which has been renamed to __code__ and __closure__, and then I had to find and fix all the places which still expected the old ones.

Some were trickier: I removed support for the cmp function and the __cmp__ special method, but this also meant that I had to fix a few types which relied on it to be comparable (for example, did you know that the cells contained in __closure__ are comparable?). At the same time, I also removed the old behavior which in Python 2 allows us to compare arbitrary objects with <, > & co.: in Python 3 the only comparisons allowed between incompatible types are == and !=.

Speaking of old special methods, __hex__ and __oct__ are gone as well (and I didn't even know about their existence before removing them :-))

But the most important breakthrough was the removal of the _file module, containing the implementation of the file type in Python 2, which is now gone since in Python 3 files are handled by the _io module. Killing the module was not straightforward, because some of the importing logic was tightly tied to the internal implementation of files, so it needed some refactoring. Finally, I had to fix the marshal module to correctly detect text files vs. byte files.

Among these things, I fixed tons of smaller issues here and there. As a result, there are many fewer failing tests than a few weeks ago. Obviously the number itself does not mean much, because sometimes fixing a single test takes hours, and some other times by changing one line one fixes tens of tests. But at the end, seeing it dropping from 999 to 650 always is nice and rewarding :-).

The road for having a pypy3k is still long, but everything is going fine so far. Stay tuned for more updates!

cheers, Antonio

Py3k status update

Thank to all the people who donated to the py3k proposal, we managed to collect enough money to start to work on the first step. This is a quick summary of what I did since I began working on this.
First of all, many thanks to Amaury Forgeot d'Arc, who started the py3k branch months ago, and already implemented lots of features including e.g. switching to "unicode everywhere" and the int/long unification, making my job considerably easier :-)
I started to work on the branch at the last Leysin sprint together with Romain Guillebert, where we worked on various syntactical changes such as extended tuple unpacking and keyword-only arguments. Working on such features is a good way to learn about a lot of the layers which the PyPy Python interpreter is composed of, because often you have to touch the tokenizer, the parser, the ast builder, the compiler and finally the interpreter.
Then I worked on improving our test machinery in various way, e.g. by optimizing the initialization phase of the object space created by tests, which considerably speeds up small test runs, and adding the possibility to automatically run our tests against CPython 3, to ensure that what we are not trying to fix a test which is meant to fail :-). I also setup our buildbot to run the py3k tests nightly, so that we can have an up to date overview of what is left to do.
Finally I started to look at all the tests in the interpreter/ directory, trying to unmangle the mess of failing tests. Lots of tests were failing because of simple syntax errors (e.g., by using the no longer valid except Exception, e syntax or the old print statement), others for slightly more complex reasons like unicode vs bytes or the now gone int/long distinction. Others were failing simply because they relied on new features, such as the new lexical exception handlers.
To give some numbers, at some point in january we had 1621 failing tests in the branch, while today we are under 1000 (to be exact: 999, and this is why I've waited until today to post the status update :-)).
Before ending this blog post, I would like to thank once again all the people who donated to PyPy, who let me to do this wonderful job. That's all for now, I'll post more updates soon.
cheers, Antonio

Py3k and Numpy First Stage: Thanks to all who Gave

Last year was quite successful for PyPy fundraising through the Software Freedom Conservancy, and Conservancy and PyPy are very excited to announce that enough was raised to begin the first stages on the Py3k and Numpy grant proposals.

As of the end of 2011, 135 different individuals gave to the Py3k campaign, and 114 to the Numpy campaign. We thank each of you who donated to help make this work possible. Meanwhile, if you haven't given to support these projects, we do hope you'll give generously now to help fund their second stages later this year!

We're also particularly excited that a few donors gave particularly large donations to support this work; those big donations really filled in the gap to help us get started!

Specifically, we're pleased to announce that Google donated $35000 towards implementing Python 3 in PyPy. Google's general support of the Python community is well known, and their specific support of our grant proposal is much appreciated.

Meanwhile, Numpy was supported in part by contributions from Nate Lawson, Cantab Capital Partners, and Getco, as well as more than a hundred other contributors.

With these donations combined with many others, we're now starting work on both projects. This week, the Conservancy signed contracts with Antonio Cuni and Benjamin Peterson to work towards the Stage 1.1 goals in Py3k proposal (and is negotiating for another contractor as well), and with Maciej Fijałkowski to work towards the Stage 1 goals in the Numpy proposal.

In 2012, PyPy will continue regular sprint meetings, at which Py3K and Numpy efforts will certainly have a place. We have some limited funds to fund travels of contributors to those meetings.

We're very thankful for all who donated so far to support these efforts, and we hope that now that work has begun, even more donors will come forward to help us finish the job. In the meantime, watch for the commits showing up from these developers and other contributors in the PyPy repositories!

Cheers, The PyPy Team