Hello.
Some time ago, we introduced our nightly performance graphs. This was a quick hack to allow us to see performance regressions. Thanks to Miquel Torres, we can now introduce http://speed.pypy.org, which is a Django-powered web app sporting a more polished visualisation of our nightly performance runs.
While this website is not finished yet, it's already far better than our previous approach :-)
Details about announcement on pypy-dev are found here.
If you're are interested in having something similar for other benchmark runs, contact Miquel (tobami at gmail).
Quoting Miquel: "I would also like to note, that if other performance-oriented opensource projects are interested, I would be willing to see if we can set-up such a Speed Center for them. There are already people interested in contributing to make it into a framework to be plugged into buildbots, software forges and the like. Stay tuned!"
20 comments:
Excellent! We really ought to deploy this for unladen, too. Unfortunately, I don't think I'll have the time to get that going. :(
In my mind PyPy with its JIT will/should eventually get us close to matching or beating Java performance for the non-dynamic subset of python. Would that be a fair statment? If so is there some bench mark that allows us to compare that. What that be usefull?
I would love to see this become a Python implementation shootout, a single place where we could compare the speeds of CPython/PyPy/Unladen/Jython/IronPython
This is great! It's excellent to see the fruits of the pypy jit work so clearly.
I'd also like to see this in place for other Python implementations.
Awesome work. One more feature request: Track memory usage.
@matt
Ok. So I've seen those feature request often enough. These benchmarks are not good for tracking memory usage - they'll simply measure the amount interpreter allocates at the beginning. If you provide better ones, we'll do it.
Cheers,
fijal
With the JIT would a script that does not use the dynamic aspects of python be able to match the speed of Java?
@Reid: maybe I can help you out setting it up. You could actually even begin saving results to speed.pypy.org right away with minimal configuration changes (though I understand you may prefer to have your own site and DB).
@Philip, Anonymous
The first features are catering to trunk development, which was the most urgent thing.
But my plan all along was to implement a third tab for comparing implementations (among other things. See mailing list announcement for details).
So your wish should come to pass :-)
Neat! I still like the original graphs though, it's nice to see the history for all the benchmarks together.
I think the 'average' is pretty meaningless - it implies that a simple average of all the benchmarks will correspond to the typical real-world speed up you will get using pypy with your existing python code, which I don't think is true.
@Neil
a view showing all timeline graphs at once is also planned.
About the average, of course you can not take from it that pypy-c-jit is nearly 3 times as fast as cpython. Because it depends on the particular choice of benchmarks, which right now is not at all representative of actual real-world usage.
Regardless, it is there so that a developer gets an overall feeling for how a given revision change has affected performance across all benchmarks.
We can't avoid the risk of people reaching wrong conclusions, but that is always the case with statistics, averages and benchmarks ;-)
@sarvi: reaching the speed of Java is a really non-trivial goal, because Sun's JVM has really been highly optimized over many years. I guess it will take us a long time (if at all) to reach such levels of performance.
I understand JVM is highly optimized.
And overtime and once yall have more momentum industry funding I am sure your VM will get just as optimized. I am sure Google will pick you guys up soon. I have no doubt about it. Unladen Swallow seems a waste of time once yall get more credibility.
Even then I do expect Dynamic scripting capabilities to perform slower the Java.
I am just hoping that eventually the non-dynamic parts of python will perform on par with Java.
And we can all program in just Python and C. :-))
Great work! BTW, could it be possible to also have a quick link to the source code of the benchmarks in the website?
@della
yeah, such things are missing right now.
An about page, and possibly an explanation (with links to the code) of each benchmark are probably going to be implemented. Currently there is only tooltip explanations for some.
Another silly question:
AFAIK, the benchmark improvements seen lately are due to the way you measure avergages, by excluding warmup time. Seeing that warmup takes time that may be critical in some situations, I wonder if it's possible to somehow "save" the generated jited code so it can be reused after the first time it's generated.
This way, it would be possible to distribute programs already "warmed up", kind of a compiled version of them. Sorry if this doesn't make sense at all... for a clueless ignorant like me, it does!
@Luis
Hey. It's a valid option, but it's however at least hard (if not next to impossible). There is work planned on reducing warmup time, so it won't matter that much instead.
Cheers,
fijal
It would be nice if the timeline had the date on it (only where the date changes, and the beginning + end).
I am sure your VM will get just as optimized. I am sure Google will pick you guys up soon. I have no doubt about it. Unladen Swallow seems a waste of time once yall get more credibility.
Post a Comment