Thursday, March 28, 2013

So, you want to try PyPy


During the PyCon trip multiple people asked me how exactly they could run their stuff on PyPy to get the speedups. Now, in an ideal world, you would just swap CPython with PyPy, everything would run tons of times faster and everyone would live happily ever after. However, we don't live in an ideal world and PyPy does not speed up everything you could potentially run. Chances are that you can run your stuff quite a bit faster, but it requires quite a bit more R&D than just that. This blog post is an attempt to explain certain steps that might help. So here we go:

  • Download and install PyPy. 2.0 beta 1 or upcoming 2.0 beta 2 would be a good candidate; it's not called a beta for stability reasons.
  • Run your tests on PyPy. There is absolutely no need for fast software that does not work. There might be some failures. Usually they're harmless (e.g. you forgot to close the file); either fix them or at least inspect them. In short, make sure stuff works.
  • Inspect your stack. In particular, C extensions, while sometimes working, are a potential source of instability and slowness. Fortunately, since the introduction of cffi, the ecosystem of PyPy-compatible software has been growing. Things I know are written with PyPy in mind:
    • the new version of pyOpenSSL will support PyPy via cffi
    • psycopg2cffi is the most actively maintained postgres binding for PyPy, with pg8000 reported working
    • mysql has a ctypes based implementation (although a cffi-based one would be definitely better)
    • PyPy 2.0 beta 2 will come with sqlite-using-cffi
    • lxml-cffi
    • uWSGI, while working, is almost certainly not the best choice. Try tornado, twisted.web,, gunicorn or gevent (note: gevent support for PyPy is not quite finished; will write about it in a separate blog post, but you can't just use the main branch of gevent)
    • consult (and contribute to) pypy compatibility wiki for details (note that it's community maintained, might be out of date)
  • Have benchmarks. If you don't have benchmarks, then performance does not matter for you. Since PyPy's warm-up time is bad (and yes, we know, we're working on it), you should leave ample time for warm-ups. Five to ten seconds of continuous computation should be enough.
  • Try them. If you get lucky, the next step might be to deploy and be happy. If you're unlucky, profile and try to isolate bottlenecks. They might be in a specific library or they might be in your code. The better you can isolate them, the higher your chances of understanding what's going on.
  • Don't take it for granted. PyPy's JIT is very good, but there is a variety of reasons that it might not work how you expect it to. A lot of times it starts off slow, but a little optimization can improve the speed as much as 10x. Since PyPy's runtime is less mature than CPython, there are higher chances of finding an obscure corner of the standard library that might be atrociously slow.
  • Most importantly, if you run out of options and you have a reproducible example, please report it. A pypy-dev email, popping into #pypy on, or getting hold of me on twitter are good ways. You can also contact me directly at fijall at as well. While it's cool if the example is slow, a lot of problems only show up on large and convoluted examples. As long as I can reproduce it on my machine or I can log in somewhere, I am usually happy to help.
  • I typically use a combination of jitviewer, valgrind and lsprofcalltree to try to guess what's going on. These tools are all useful, but use them with care. They usually require quite a bit of understanding before being useful. Also sometimes they're just plain useless and you need to write your own analysis.

I hope this summary of steps to take is useful. We hear a lot of stories of people trying PyPy, most of them positive, but some of them negative. If you just post "PyPy didn't work for me" on your blog, that's cool too, but you're missing an opportunity. The reasons may vary from something serious like "this is a bad pattern for PyPy GC" to something completely hilarious like "oh, I left this sys._getframe() somewhere in my hot loops for debugging" or "I used the logging module which uses sys._getframe() all over the place".


Monday, March 18, 2013

Numpy status update and developer announcement

Hello, some good news!

First the update:

  • dtype support - NumPy on PyPy now supports non-native storage formats. Due to a lack of true support for longdoubles in rpython, we decided to back out the support of longdouble-as-double which was misleading.
  • missing ndarray attributes - work has been made toward supporting the complete set of attributes on ndarrays. We are progressing alphabetically, and have made it to d. Unsupported attributes, and unsupported arguments to attribute calls will raise a NotImplementedError.
  • pickling support for numarray - hasn't started yet, but next on the list
  • There has been some work on exposing FFI routines in numpypy.
  • Brian Kearns has made progress in improving the numpypy namespace. The python numpypy submodules now more closely resemble their numpy counterparts. Also, translated _numpypy submodules are now more properly mapped to the numpy core c-based submodules, furthering the goal of being able to install numpy as a pure-python module with few modifications.

And now the good news:

While our funding drive over 2012 did not reach our goal, we still managed to raise a fair amount of money in donations. So far we only managed to spend around $10 000 of it. We issued a call for additional developers, and are glad to welcome Romain Guillebert and Ronan Lamy to the numpypy team. Hopefully we will be able to report on speedier progress soon.

Matti Picus, Maciej Fijalkowski

Tuesday, March 5, 2013

Py3k status update #10

This is the tenth status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

There's been significant progress since the last update: the linux x86-32
now passes 289 out of approximately 354 modules (with 39 skips) of
CPython's regression test suite.

That means there's only 26 test module failures left! The list of major items
remaining for 3.2 compatibility are now short enough to list here, with their
related tests:

  • Tokenizer support for non-ascii identifiers
  • test_importlib
  • test_pep263
  • test_memoryview
  • multiprocessing module currently deadlocks
  • test_multiprocessing
  • Buggy handling of the new extended unpacking syntax by the compiler:
  • test_unpack_ex
  • The new Global Interpreter Lock and new thread signal handling
  • test_threading
  • test_threadsignals
  • test_sys
  • Upgrade unicodedata to 6.0.0 (requires updates to the actual unicodedata
    generation script)
  • test_ucn
  • test_unicode
  • test_unicodedata
  • test_capi (currently crashes)
  • Update int's hash code to match to CPython (float's is already updated on the
    py3k-newhash branch. note that PyPy 2.x doesn't even totally match
    CPython's hashing)
  • test_decimal
  • test_fractions
  • test_numeric_tower
  • Miscellaneous:
  • test_complex
  • test_float
  • test_peepholer
  • test_range
  • test_sqlite (a new cffi based version seems to be coming)
  • test_ssl
  • test_struct
  • test_subprocess
  • test_sys_settrace
  • test_time

Additionally there are still a number of failures in PyPy's internal test
suite. These tests are usually ran against untranslated versions of PyPy during
development. However we've now began running them against a fully translated
version of PyPy on the buildbot too (thanks to Amaury for setting this
up). This further ensures that our tests and implementation are sane.

We're getting closer to producing an initial alpha release. Before that happens
we'd like to see:

  • further test fixes
  • the results of test runs on other major platforms (e.g. linux x86-64 and osx
    seem to have some additional failures as of now)
  • some basic real world testing

Finally I'd like to thank Manuel Jacob for his various contributions over the
past month, including fixing the array and ctypes modules among other things,
and also Amaury Forgeot d'Arc for his ongoing excellent contributions.