Mac meets Arm64

Looking for sponsorship

Apple now ships Macs which are running on an arm64 variant machine with the latest version of MacOS, Big Sur M1. We are getting requests for PyPy to support this new architecture. Here is our position on this topic (or at least mine, Armin Rigo's), and how you can help.

Porting PyPy is harder than just re-running the compiler, because PyPy contains a few big architecture-dependent "details", like the JIT compiler and the foreign function interfaces (CFFI and ctypes).

Fixing the JIT compiler should not be too much work: we already support arm64, just the Linux one. But Apple made various details different (like the calling conventions). A few other parts need to be fixed too, notably CFFI and ctypes, again because of the calling conventions.

Fixing that would be a reasonable amount of work. I would do it myself for a small amount of money. However, the story doesn't finish here. Obviously, the start of the story would be to get ssh access to a Big Sur M1 machine. (If at this point you're thinking "sure, I can give you ssh access for three months", then please read on.) The next part of the story is that we need a machine available long term. It can be either a machine provided and maintained by a third party, or alternatively a pot of money big enough to support the acquision of a machine and ongoing work of one of us.

If we go with the provided-machine solution: What we need isn't a lot of resources. Our CI requires maybe 10 GB of disk space, and a few hours of CPU per run. It should fit into 8 GB of RAM. We normally do a run every night but we can certainly lower the frequency a bit if that would help. However, we'd ideally like some kind of assurance that you are invested into maintaining the machine for the next 3-5 years (I guess, see below). We had far too many machines that disappeared after a few months.

If we go with the money-supported solution: it's likely that after 3-5 years the whole Mac base will have switched to arm64, we'll drop x86-64 support for Mac, and we'll be back to the situation of the past where there was only one kind of Mac machine to care about. In the meantime, we are looking at 3-5 years of lightweight extra maintenance. We have someone that has said he would do it, but not for free.

If either of these two solutions occurs, we'll still have, I quote, "probably some changes in distutils-type stuff to make python happy", and then some packaging/deployment changes to support the "universal2" architecture, i.e. including both versions inside a single executable (which will not be just an extra switch to clang, because the two versions need a different JIT backend and so must be translated separately).

So, now all the factors are on the table. We won't do the minimal "just the JIT compiler fixes" if we don't have a plan that goes farther. Either we get sufficient money, and maybe support, and then we can do it quickly; or PyPy will just remain not natively available on M1 hardware for the next 3-5 years. We are looking forward to supporting M1, and view resources contributed by the community as a vote of confidence in assuring the future of PyPy on this hardware. Contact us: pypy-dev@python.org, or our private mailing list pypy-z@python.org.

Thanks for reading!

Armin Rigo

PyPy 7.3.3 triple release: python 3.7, 3.6, and 2.7

 The PyPy team is proud to release the version 7.3.3 of PyPy, which includes three different interpreters:

• PyPy2.7, which is an interpreter supporting the syntax and the features of Python 2.7 including the stdlib for CPython 2.7.18 (updated from the previous version)
• PyPy3.6: which is an interpreter supporting the syntax and the features of Python 3.6, including the stdlib for CPython 3.6.12 (updated from the previous version).
• PyPy3.7 beta: which is our second release of an interpreter supporting the syntax and the features of Python 3.7, including the stdlib for CPython 3.7.9. We call this beta quality software, there may be issues about compatibility with new and changed features in CPython 3.7. Please let us know what is broken or missing. We have not implemented the documented changes in the re module, and a few other pieces are also missing. For more information, see the PyPy 3.7 wiki page

The interpreters are based on much the same codebase, thus the multiple release. This is a micro release, all APIs are compatible with the 7.3 releases, but read on to find out what is new.

Several issues found in the 7.3.2 release were fixed. Many of them came from the great work by conda-forge to ship PyPy binary packages. A big shout out to them for taking this on.

Development of PyPy has moved to https://foss.heptapod.net/pypy/pypy. This was covered more extensively in this blog post. We have seen an increase in the number of drive-by contributors who are able to use gitlab + mercurial to create merge requests.

The CFFI backend has been updated to version 1.14.3. We recommend using CFFI rather than c-extensions to interact with C, and using cppyy for performant wrapping of C++ code for Python.

A new contributor took us up on the challenge to get windows 64-bit support. The work is proceeding on the win64 branch, more help in coding or sponsorship is welcome. In anticipation of merging this large change, we fixed many test failures on windows.

As always, this release fixed several issues and bugs. We strongly recommend updating. Many of the fixes are the direct result of end-user bug reports, so please continue reporting issues as they crop up.

You can find links to download the v7.3.3 releases here:

https://pypy.org/download.html

We would like to thank our donors for the continued support of the PyPy project. If PyPy is not quite good enough for your needs, we are available for direct consulting work.

We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: PyPy and RPython documentation improvements, tweaking popular modules to run on pypy, or general help with making RPython’s JIT even better. Since the previous release, we have accepted contributions from 2 new contributors, thanks for pitching in.

If you are a python library maintainer and use c-extensions, please consider making a cffi / cppyy version of your library that would be performant on PyPy. In any case both cibuildwheel and the multibuild system support building wheels for PyPy.

What is PyPy?

PyPy is a Python interpreter, a drop-in replacement for CPython 2.7, 3.6, and 3.7. It’s fast (PyPy and CPython 3.7.4 performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

This PyPy release supports:

• x86 machines on most common operating systems (Linux 32/64 bits, Mac OS X 64 bits, Windows 32 bits, OpenBSD, FreeBSD)
• big- and little-endian variants of PPC64 running Linux,
• s390x running Linux
• 64-bit ARM machines running Linux.

PyPy does support ARM 32 bit processors, but does not release binaries.

 

What else is new?

For more information about the 7.3.3 release, see the full changelog.

Please update, and continue to help us make PyPy better.

Cheers,
The PyPy team

 

PyPy 7.3.2 triple release: python 2.7, 3.6, and 3.7

 

The PyPy team is proud to release version 7.3.2 of PyPy, which includes three different interpreters:

• PyPy2.7, which is an interpreter supporting the syntax and the features of Python 2.7 including the stdlib for CPython 2.7.13
• PyPy3.6: which is an interpreter supporting the syntax and the features of Python 3.6, including the stdlib for CPython 3.6.9.
• PyPy3.7 alpha: which is our first release of an interpreter supporting the syntax and the features of Python 3.7, including the stdlib for CPython 3.7.9. We call this an alpha release since it is our first. It is based off PyPy 3.6 so issues should be around compatibility and not stability. Please try it out and let us know what is broken or missing. We have not implemented some of the documented changes in the re module, and other pieces are also missing. For more information, see the PyPy 3.7 wiki page

The interpreters are based on much the same codebase, thus the multiple release. This is a micro release, all APIs are compatible with the 7.3.0 (Dec 2019) and 7.3.1 (April 2020) releases, but read on to find out what is new.

Conda Forge now supports PyPy as a python interpreter. The support is quite complete for linux and macOS. This is the result of a lot of hard work and good will on the part of the Conda Forge team. A big shout out to them for taking this on.

Development of PyPy has transitioning to https://foss.heptapod.net/pypy/pypy. This move was covered more extensively in this blog post. We have seen an increase in the number of drive-by contributors who are able to use gitlab + mercurial to create merge requests.

The CFFI backend has been updated to version 1.14.2. We recommend using CFFI rather than c-extensions to interact with C, and using cppyy for performant wrapping of C++ code for Python.

NumPy has begun shipping wheels on PyPI for PyPy, currently for linux 64-bit only. Wheels for PyPy windows will be available from the next NumPy release. Thanks to NumPy for their support.

A new contributor took us up on the challenge to get windows 64-bit support. The work is proceeding on the win64 branch, more help in coding or sponsorship is welcome.

As always, this release fixed several issues and bugs. We strongly recommend updating. Many of the fixes are the direct result of end-user bug reports, so please continue reporting issues as they crop up.

You can find links to download the v7.3.2 releases here:

https://pypy.org/download.html

We would like to thank our donors for the continued support of the PyPy project. Please help support us at Open Collective. If PyPy is not yet good enough for your needs, we are available for direct consulting work.

We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: PyPy and RPython documentation improvements, tweaking popular modules to run on pypy, or general help with making RPython’s JIT even better. Since the previous release, we have accepted contributions from 8 new contributors, thanks for pitching in.

If you are a python library maintainer and use c-extensions, please consider making a cffi / cppyy version of your library that would be performant on PyPy. In any case both cibuildwheel and the multibuild system support building wheels for PyPy.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7, 3.6, and 3.7. It’s fast (PyPy and CPython 2.7.x performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

This PyPy release supports:

• x86 machines on most common operating systems (Linux 32/64 bits, Mac OS X 64 bits, Windows 32 bits, OpenBSD, FreeBSD)
• big- and little-endian variants of PPC64 running Linux,
• s390x running Linux
• 64-bit ARM machines running Linux.

PyPy does support ARM 32 bit processors, but does not release binaries.

What else is new?

For more information about the 7.3.2 release, see the full changelog.

Please update, and continue to help us make PyPy better.

Cheers,
The PyPy team

 

 

PyPy is on Open Collective

Hi all,

PyPy is now a member of Open Collective, a fiscal host. We have been thinking about switching to this organization for a couple of years; we like it for various reasons, like the budget transparency and the lightweight touch. We can now officially announce our membership!

With this, we are now again free to use PyPy for all financial issues, like receiving funds professionally, paying parts of sprint budgets as we like, and so on. We will shortly be reintroducing buttons that link to Open Collective from the PyPy web site.

Although the old donation buttons were removed last year, we believe that there are still a few people that send regularly money to the SFC, the not-for-profit charity we were affiliated with. If you do, please stop doing it now (and, if you like to do so, please set up an equivalent donation to PyPy on Open Collective).

And by the way, sorry for all of you who were getting mixed feelings from the previous blog post (co-written with the SFC). PyPy is committed to continue being Open Source just like before. This was never in question. What these two blog posts mean is only that we switched to a different organization for our internal finances.

We're looking forward to how this new relationship will go!

Armin Rigo, for the PyPy team

A new chapter for PyPy

PyPy winds down its membership in the Software Freedom Conservancy

Conservancy and PyPy's great work together

PyPy joined Conservancy in the second half of 2010, shortly after the release of PyPy 1.2, the first version to contain a fully functional JIT. In 2013, PyPy started supporting ARM, bringing its just-in-time speediness to many more devices and began working toward supporting NumPy to help scientists crunch their numbers faster. Together, PyPy and Conservancy ran successful fundraising drives and facilitated payment and oversight for contractors and code sprints.

Conservancy supported PyPy's impressive growth as it expanded support for different hardware platforms, greatly improved the performance of C extensions, and added support for Python 3 as the language itself evolved.

The road ahead

Conservancy provides a fiscal and organizational home for projects that find the freedoms and guardrails that come along with a charitable home advantageous for their community goals. While this framework was a great fit for the early PyPy community, times change and all good things must come to an end.

PyPy will remain a free and open source project, but the community's structure and organizational underpinnings will be changing and the PyPy community will be exploring options outside of the charitable realm for its next phase of growth ("charitable" in the legal sense -- PyPy will remain a community project).

During the last year PyPy and Conservancy have worked together to properly utilise the generous donations made by stalwart PyPy enthusiats over the years and to wrap up PyPy's remaining charitable obligations. PyPy is grateful for the Conservancy's help in shepherding the project toward its next chapter.

Thank yous

From Conservancy:

"We are happy that Conservancy was able to help PyPy bring important software for the public good during a critical time in its history. We wish the community well and look forward to seeing it develop and succeed in new ways."

— Karen Sandler, Conservancy's Executive Director

From PyPy:

"PyPy would like to thank Conservancy for their decade long support in building the community and wishes Conservancy continued success in their journey promoting, improving, developing and defending free and open source sofware."

— Simon Cross & Carl Friedrich Bolz-Tereick, on behalf of PyPy.

About

PyPy is a multi-layer python interpreter with a built-in JIT compiler that runs Python quickly across different computing environments. Software Freedom Conservancy (Conservancy) is a charity that provides a home to over forty free and open source software projects.

PyPy 7.3.1 released

The PyPy team is proud to release the version 7.3.1 of PyPy, which includes two different interpreters:

• PyPy2.7, which is an interpreter supporting the syntax and the features of Python 2.7 including the stdlib for CPython 2.7.13
• PyPy3.6: which is an interpreter supporting the syntax and the features of Python 3.6, including the stdlib for CPython 3.6.9.

The interpreters are based on much the same codebase, thus the multiple release. This is a micro release, no APIs have changed since the 7.3.0 release in December, but read on to find out what is new.

Conda Forge now supports PyPy as a Python interpreter. The support right now is being built out. After this release, many more c-extension-based packages can be successfully built and uploaded. This is the result of a lot of hard work and good will on the part of the Conda Forge team. A big shout out to them for taking this on.

We have worked with the Python packaging group to support tooling around building third party packages for Python, so this release updates the pip and setuptools installed when executing pypy -mensurepip to pip>=20. This completes the work done to update the PEP 425 python tag from pp373 to mean “PyPy 7.3 running python3” to pp36 meaning “PyPy running Python 3.6” (the format is recommended in the PEP). The tag itself was changed in 7.3.0, but older pip versions build their own tag without querying PyPy. This means that wheels built for the previous tag format will not be discovered by pip from this version, so library authors should update their PyPy-specific wheels on PyPI.

Development of PyPy is transitioning to https://foss.heptapod.net/pypy/pypy. This move was covered more extensively in the blog post from last month.

The CFFI backend has been updated to version 14.0. We recommend using CFFI rather than c-extensions to interact with C, and using cppyy for performant wrapping of C++ code for Python. The cppyy backend has been enabled experimentally for win32, try it out and let use know how it works.

Enabling cppyy requires a more modern C compiler, so win32 is now built with MSVC160 (Visual Studio 2019). This is true for PyPy 3.6 as well as for 2.7.

We have improved warmup time by up to 20%, performance of io.StringIO to match if not be faster than CPython, and improved JIT code generation for generators (and generator expressions in particular) when passing them to functions like sum, map, and map that consume them. Performance of closures has also be improved in certain situations.

As always, this release fixed several issues and bugs raised by the growing community of PyPy users. We strongly recommend updating. Many of the fixes are the direct result of end-user bug reports, so please continue reporting issues as they crop up.
You can find links to download the v7.3.1 releases here:

http://pypy.org/download.html

We would like to thank our donors for the continued support of the PyPy project. If PyPy is not quite good enough for your needs, we are available for direct consulting work.

We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: PyPy and RPython documentation improvements, tweaking popular modules to run on PyPy, or general help with making RPython’s JIT even better. Since the previous release, we have accepted contributions from 13 new contributors, thanks for pitching in.

If you are a Python library maintainer and use c-extensions, please consider making a cffi / cppyy version of your library that would be performant on PyPy. In any case both cibuildwheel and the multibuild system support building wheels for PyPy wheels.

 

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7, 3.6, and soon 3.7. It’s fast (PyPy and CPython 2.7.x performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

This PyPy release supports:

• x86 machines on most common operating systems (Linux 32/64 bits, Mac OS X 64 bits, Windows 32 bits, OpenBSD, FreeBSD)
• big- and little-endian variants of PPC64 running Linux,
• s390x running Linux
• 64-bit ARM machines running Linux.

What else is new?

For more information about the 7.3.1 release, see the full changelog.

Please update, and continue to help us make PyPy better.

Cheers,
The PyPy team

The PyPy Team 

Leysin 2020 Sprint Report

At the end of February ten of us gathered in Leysin, Switzerland to work on
a variety of topics including HPy, PyPy Python 3.7 support and the PyPy
migration to Heptapod.

We had a fun and productive week. The snow was beautiful. There was skiing
and lunch at the top of Berneuse, cooking together, some late nights at
the pub next door, some even later nights coding, and of course the
obligatory cheese fondue outing.

There were a few of us participating in a PyPy sprint for the first time
and a few familiar faces who had attended many sprints. Many different
projects were represented including PyPy, HPy, GraalPython,
Heptapod, and rust-cpython. The atmosphere was relaxed and welcoming, so if
you're thinking of attending the next one -- please do!

Topics worked on:

HPy

HPy is a new project to design and implement a better API for extending
Python in C. If you're unfamiliar with it you can read more about it at
HPy.

A lot of attention was devoted to the Big HPy Design Discussion which
took up two full mornings. So much was decided that this will likely
get its own detailed write-up, but bigger topics included:

• the HPy GetAttr, SetAttr, GetItem and SetItem methods,
• HPy_FromVoidP and HPy_AsVoidP for passing HPy handles to C functions
  that pass void* pointers to callbacks,
• avoiding having va_args as part of the ABI,
• exception handling,
• support for creating custom types.

Quite a few things got worked on too:

• implemented support for writing methods that take keyword arguments with
  HPy_METH_KEYWORDS,
• implemented HPy_GetAttr, HPy_SetAttr, HPy_GetItem, and HPy_SetItem,
• started implementing support for adding custom types,
• started implementing dumping JSON objects in ultrajson-hpy,
• refactored the PyPy GIL to improve the interaction between HPy and
  PyPy's cpyext,
• experimented with adding HPy support to rust-cpython.

And there was some discussion of the next steps of the HPy initiative
including writing documentation, setting up websites and funding, and
possibly organising another HPy gathering later in the year.

PyPy

• Georges gave a presentation on the Heptapod topic and branch workflows
  and showed everyone how to use hg-evolve.
• Work was done on improving the PyPy CI buildbot post the move to
  heptapod, including a light-weight pre-merge CI and restricting
  when the full CI is run to only branch commits.
• A lot of work was done improving the -D tests.

Miscellaneous

• Armin demoed VRSketch and NaN Industries in VR, including an implementation
  of the Game of Life within NaN Industries!
• Skiing!

Aftermath

Immediately after the sprint large parts of Europe and the world were
hit by the COVID-19 epidemic. It was good to spend time together before
travelling ceased to be a sensible idea and many gatherings were cancelled.

Keep safe out there everyone.

The HPy & PyPy Team & Friends

In joke for those who attended the sprint: Please don't replace this blog post
with its Swedish translation (or indeed a translation to any other language :).

PyPy and CFFI have moved to Heptapod

It has been a very busy month, not so much because of deep changes in the JIT of PyPy but more around the development, deployment, and packaging of the project.

 

Hosting

The biggest news is that we have moved the center of our development off Bitbucket and to the new https://foss.heptapod.net/pypy. This is a friendly fork of Gitlab called heptapod that understands Mercurial and is hosted by Clever Cloud. When Atlassian decided to close down Mercurial hosting on bitbucket.org, PyPy debated what to do. Our development model is based on long-lived branches, and we want to keep the ability to immediately see which branch each commit came from. Mercurial has this, git does not (see our FAQ). Octobus, whose business is Mercurial, developed a way to use Mercurial with Gitlab called heptapod. The product is still under development, but quite usable (i.e., it doesn't get in the way). Octobus partnered with Clever Cloud hosting to offer community FOSS projects hosted on Bitbucket who wish to remain with Mercurial a new home. PyPy took them up on the offer, and migrated its repos to https://foss.heptapod.net/pypy. We were very happy with how smooth it was to import the repos to heptapod/GitLab, and are learning the small differences between Bitbucket and GitLab. All the pull requests, issues, and commits kept the same ids, but work is still being done to attribute the issues, pull requests, and comments to the correct users. So from now on, when you want to contribute to PyPy, you do so at the new home.

CFFI, which previously was also hosted on Bitbucket, has joined the PyPy group at https://foss.heptapod.net/pypy/cffi.

 

Website

Secondly, thanks to work by https://baroquesoftware.com/ in leading a redesign and updating the logo, the https://www.pypy.org website has undergone a facelift. It should now be easier to use on small-screen devices. Thanks also to the PSF for hosting the site.

 

Packaging

Also, building PyPy from source takes a fair amount of time. While we provide downloads in the form of tarballs or zipfiles, and some platforms such as debian and Homebrew provide packages, traditionally the downloads have only worked on a specific flavor of operating system. A few years ago squeaky-pl started providing portable builds. We have adopted that build system for our linux offerings, so the nightly downloads and release downloads should now work on any glibc platform that has not gone EndOfLife. So there goes another excuse not to use PyPy. And the "but does it run scipy" excuse also no longer holds, although "does it speed up scipy" still has the wrong answer. For that we are working on HPy, and will be sprinting soon.
The latest versions of pip, wheel, and setuptools, together with the manylinux2010 standard for linux wheels and tools such as multibuild or cibuildwheels (well, from the next version) make it easier for library developers to build binary wheels for PyPy. If you are having problems getting going with this, please reach out.

 

Give it a try

Thanks to all the folks who provide the infrastructure PyPy depends on. We hope the new look will encourage more involvement and engagement. Help prove us right!

The PyPy Team

Leysin Winter sprint 2020: Feb 29 - March 8th

The next PyPy sprint will be in Leysin, Switzerland, for the fourteenth time. This is a fully public sprint: newcomers and topics other than those proposed below are welcome.

Goals and topics of the sprint

The list of topics is open.  For reference, we would like to work at least partially on the following topics:

• HPy
• Python 3.7 support (buildbot status)

As usual, the main side goal is to have fun in winter sports :-) We can take a day off (for ski or anything else).

Times and accomodation

The sprint will occur for one week starting on Saturday, the 29th of February, to Sunday, the 8th of March 2020 (dates were pushed back one day!)  It will occur in Les Airelles, a different bed-and-breakfast place from the traditional one in Leysin.  It is a nice old house at the top of the village.

We have a 4- or 5-people room as well as up to three double-rooms.  Please register early!  These rooms are not booked for the sprint in advance, and might be already taken if you end up announcing yourself late.  We have a big room for up to 7 people with nice view, which might be split in two or three sub-rooms; plus possibly separately-booked double rooms if needed. (But it is of course always possible to book at a different place in Leysin.)

For more information, see our repository or write to me directly at armin.rigo@gmail.com.