Wednesday, March 3, 2010

Introducing speed.pypy.org

Hello.

Some time ago, we introduced our nightly performance graphs. This was a quick hack to allow us to see performance regressions. Thanks to Miquel Torres, we can now introduce http://speed.pypy.org, which is a Django-powered web app sporting a more polished visualisation of our nightly performance runs.

While this website is not finished yet, it's already far better than our previous approach :-)

Details about announcement on pypy-dev are found here.

If you're are interested in having something similar for other benchmark runs, contact Miquel (tobami at gmail).

Quoting Miquel: "I would also like to note, that if other performance-oriented opensource projects are interested, I would be willing to see if we can set-up such a Speed Center for them. There are already people interested in contributing to make it into a framework to be plugged into buildbots, software forges and the like. Stay tuned!"

20 comments:

Reid K said...

Excellent! We really ought to deploy this for unladen, too. Unfortunately, I don't think I'll have the time to get that going. :(

Sarvi Shanmugham said...

In my mind PyPy with its JIT will/should eventually get us close to matching or beating Java performance for the non-dynamic subset of python. Would that be a fair statment? If so is there some bench mark that allows us to compare that. What that be usefull?

Philip Jenvey said...

I would love to see this become a Python implementation shootout, a single place where we could compare the speeds of CPython/PyPy/Unladen/Jython/IronPython

Anonymous said...

This is great! It's excellent to see the fruits of the pypy jit work so clearly.

I'd also like to see this in place for other Python implementations.

matt harrison said...

Awesome work. One more feature request: Track memory usage.

Maciej Fijalkowski said...

@matt

Ok. So I've seen those feature request often enough. These benchmarks are not good for tracking memory usage - they'll simply measure the amount interpreter allocates at the beginning. If you provide better ones, we'll do it.

Cheers,
fijal

Sarvi Shanmugham said...

With the JIT would a script that does not use the dynamic aspects of python be able to match the speed of Java?

tobami said...

@Reid: maybe I can help you out setting it up. You could actually even begin saving results to speed.pypy.org right away with minimal configuration changes (though I understand you may prefer to have your own site and DB).

tobami said...

@Philip, Anonymous

The first features are catering to trunk development, which was the most urgent thing.

But my plan all along was to implement a third tab for comparing implementations (among other things. See mailing list announcement for details).

So your wish should come to pass :-)

Neil said...

Neat! I still like the original graphs though, it's nice to see the history for all the benchmarks together.

I think the 'average' is pretty meaningless - it implies that a simple average of all the benchmarks will correspond to the typical real-world speed up you will get using pypy with your existing python code, which I don't think is true.

tobami said...

@Neil
a view showing all timeline graphs at once is also planned.

About the average, of course you can not take from it that pypy-c-jit is nearly 3 times as fast as cpython. Because it depends on the particular choice of benchmarks, which right now is not at all representative of actual real-world usage.

Regardless, it is there so that a developer gets an overall feeling for how a given revision change has affected performance across all benchmarks.

We can't avoid the risk of people reaching wrong conclusions, but that is always the case with statistics, averages and benchmarks ;-)

Carl Friedrich Bolz-Tereick said...

@sarvi: reaching the speed of Java is a really non-trivial goal, because Sun's JVM has really been highly optimized over many years. I guess it will take us a long time (if at all) to reach such levels of performance.

Sarvi Shanmugham said...

I understand JVM is highly optimized.
And overtime and once yall have more momentum industry funding I am sure your VM will get just as optimized. I am sure Google will pick you guys up soon. I have no doubt about it. Unladen Swallow seems a waste of time once yall get more credibility.

Even then I do expect Dynamic scripting capabilities to perform slower the Java.

I am just hoping that eventually the non-dynamic parts of python will perform on par with Java.

And we can all program in just Python and C. :-))

della said...

Great work! BTW, could it be possible to also have a quick link to the source code of the benchmarks in the website?

tobami said...

@della

yeah, such things are missing right now.

An about page, and possibly an explanation (with links to the code) of each benchmark are probably going to be implemented. Currently there is only tooltip explanations for some.

Luis said...

Another silly question:
AFAIK, the benchmark improvements seen lately are due to the way you measure avergages, by excluding warmup time. Seeing that warmup takes time that may be critical in some situations, I wonder if it's possible to somehow "save" the generated jited code so it can be reused after the first time it's generated.
This way, it would be possible to distribute programs already "warmed up", kind of a compiled version of them. Sorry if this doesn't make sense at all... for a clueless ignorant like me, it does!

Maciej Fijalkowski said...

@Luis

Hey. It's a valid option, but it's however at least hard (if not next to impossible). There is work planned on reducing warmup time, so it won't matter that much instead.

Cheers,
fijal

stuaxo said...
This comment has been removed by the author.
stuaxo said...

It would be nice if the timeline had the date on it (only where the date changes, and the beginning + end).

large file transfer said...

I am sure your VM will get just as optimized. I am sure Google will pick you guys up soon. I have no doubt about it. Unladen Swallow seems a waste of time once yall get more credibility.