I recently did some benchmarking of twisted on top of PyPy. For the very impatient: PyPy is up to 285% faster than CPython. For more patient people, there is a full explanation of what I did and how I performed measurments, so they can judge themselves.
The benchmarks are living in twisted-benchmarks and were mostly written by Jean Paul Calderone. Even though he called them "initial exploratory investigation into a potential direction for future development resulting in performance oriented metrics guiding the process of optimization and avoidance of complexity regressions", they're still much much better than average benchmarks found out there.
The methodology was to run each benchmark for quite some time (about 1 minute), measuring number of requests each 5s. Then I looked at dump of data and substracted some time it took for JIT-capable interpreters to warm up (up to 15s), averaging everything after that. Averages of requests per second are in the table below (the higher the better):
|names||10930||11940 (9% faster)||15429 (40% faster)|
|pb||1705||2280 (34% faster)||3029 (78% faster)|
|iterations||75569||94554 (25% faster)||291066 (285% faster)|
|accept||2176||2166 (same speed)||2290 (5% faster)|
|web||879||854 (3% slower)||1040 (18% faster)|
|tcp||105M||119M (7% faster)||60M (46% slower)|
To reproduce, run each benchmark with:
benchname.py -n 12 -d 5
WARNING: running tcp-based benchmarks that open new connection for each request (web & accept) can exhaust number of some kernel structures, limit n or wait until next run if you see drops in request per second.
The first obvious thing is that various benchmarks are more or less amenable to speedups by JIT compilation. Accept and tcp getting smallest speedups, if at all. This is understandable, since JIT is mostly about reducing interpretation and frame overhead, which is probably not large when it comes to accepting connections. However, if you actually loop around, doing something, JIT can give you a lot of speedup.
The other obvious thing is that PyPy is the fastest python interpreter here, almost across-the board (Jython and IronPython won't run twisted), except for raw tcp throughput. However, speedups can vary and I expect this to improve after the release, as there are points, where PyPy can be improved. Regarding raw tcp throughput - this can be a problem for some applications and we're looking forward to improve this particular bit.
The main reason to use twisted for this comparison is a lot of support from twisted team and JP Calderone in particular, especially when it comes to providing benchmarks. If some open source project wants to be looked at by PyPy team, please provide a reasonable set of benchmarks and infrastructure.
If, however, you're a closed source project fighting with performance problems of Python, we're providing contracting for investigating opportunities, how PyPy and not only PyPy, can speed up your project.
- names - simple DNS server
- web - simple http hello world server
- pb - perspective broker, RPC mechanism for twisted
- iterations - empty twisted loop
- accept - number of tcp connections accepted per second
- tcp - raw socket transfer throughput
- CPython 2.6.2 - as packaged by ubuntu
- Unladen swallow svn trunk, revision 1109
- PyPy svn trunk, revision 71439
Twisted version used: svn trunk, revision 28580
Machine: unfortunately 32bit virtual-machine under qemu, running ubuntu karmic, on top of Quad core intel Q9550 with 6M cache. Courtesy of Michael Schneider.