tag:blogger.com,1999:blog-3971202189709462152.post6674537807334018925..comments2024-03-11T12:50:02.036+01:00Comments on PyPy Status Blog: First results of the JITCarl Friedrich Bolz-Tereickhttp://www.blogger.com/profile/00518922641059511014noreply@blogger.comBlogger41125tag:blogger.com,1999:blog-3971202189709462152.post-31519380669677901712009-10-05T17:48:52.799+02:002009-10-05T17:48:52.799+02:00Ben Scherrey: 64bit support might happen not too f...Ben Scherrey: 64bit support might happen not too far in the future. Not using too much memory is a different problem, that might take a while longer. It has two aspects, one is that the JIT itself uses way too much memory at the moment. We will work on that soon.<br /><br />The other aspect is making sure that your dataset does not take too much heap. It depends a bit which data structures you use, but it's not likely to be that great right now. That might change at some point, I have some ideas in that direction, but not really time to work on the soon.Carl Friedrich Bolz-Tereickhttps://www.blogger.com/profile/00518922641059511014noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-64288919582921960292009-10-05T16:28:18.251+02:002009-10-05T16:28:18.251+02:00Michael: It is written in RPython (a subset of Pyt...Michael: It is written in RPython (a subset of Python) but then translated to C. By convention we therefore call the executable-name pypy-c. If the executable also contains a JIT, we call it pypy-c-jit.Carl Friedrich Bolz-Tereickhttps://www.blogger.com/profile/00518922641059511014noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-81853814081477160892009-10-05T12:56:07.736+02:002009-10-05T12:56:07.736+02:00Is pypy-c-jit written in C or Python or something ...Is pypy-c-jit written in C or Python or something else? I ask because of the "c" in pypy-c-jit.Michael Allmanhttps://www.blogger.com/profile/02974012909747662741noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-11275620722684063082009-10-04T18:50:52.770+02:002009-10-04T18:50:52.770+02:00NickDaly: checked in, please try. Also, please co...NickDaly: checked in, please try. Also, please come to the mailing list instead of posting here if you have further comments to do... http://codespeak.net/mailman/listinfo/pypy-devArmin Rigohttps://www.blogger.com/profile/06300515270104686574noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-91917995886884649382009-10-04T18:19:07.460+02:002009-10-04T18:19:07.460+02:00A correction/clarification to last night's pos...A correction/clarification to last night's post:<br /><br />There isn't a bug in the -Ojit translation process, I was just missing a dependency that I could've sworn I've installed before.<br /><br />The translation process only takes < 1GB memory if done without any options. Attempting to translate with the -Ojit option takes at least 2.5GB of RAM, as I tried last night (with it as the only running process) and it consumed my swapfile and ran out of memory.<br /><br />Is there any documented way to use a translated pypy binary to build other pypy translations? That might help reduce the build requirements, and would also be mighty cool.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-22547746042910219982009-10-04T18:15:55.564+02:002009-10-04T18:15:55.564+02:00The trouble seems to be implement.gcmap and implem...The trouble seems to be implement.gcmap and implement_9.gcmap. These are bothe empty, and trigger the assertion error.<br /><br />Running trackgcroot as the Makefile does, but without those two files, permits compilation to continue, but linking fails with undefined references to various symbols with the prefix 'pypy_g_'.<br /><br />I suspected the changes might have invalidated the old .gcmap files, so I tried removing them, and got this when it tried to generate implement.gcmap:<br /><br />Traceback (most recent call last):<br /> File "/home/chshrcat/build/pypy-trunk/pypy/translator/c/gcc/trackgcroot.py", line 1214, in <module><br /> tracker.process(f, g, filename=fn)<br /> File "/home/chshrcat/build/pypy-trunk/pypy/translator/c/gcc/trackgcroot.py", line 229, in process<br /> lines = self.process_function(lines, entrypoint, filename)<br /> File "/home/chshrcat/build/pypy-trunk/pypy/translator/c/gcc/trackgcroot.py", line 244, in process_function<br /> table = tracker.computegcmaptable(self.verbose)<br /> File "/home/chshrcat/build/pypy-trunk/pypy/translator/c/gcc/trackgcroot.py", line 285, in computegcmaptable<br /> self.parse_instructions()<br /> File "/home/chshrcat/build/pypy-trunk/pypy/translator/c/gcc/trackgcroot.py", line 365, in parse_instructions<br /> insn = meth(line)<br /> File "/home/chshrcat/build/pypy-trunk/pypy/translator/c/gcc/trackgcroot.py", line 741, in visit_jmp<br /> self.conditional_jump(line)<br /> File "/home/chshrcat/build/pypy-trunk/pypy/translator/c/gcc/trackgcroot.py", line 757, in conditional_jump<br /> raise UnrecognizedOperation(line)<br />__main__.UnrecognizedOperation: jmp T.14141Unhelpfulhttps://www.blogger.com/profile/12227174048122478111noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-8071725561580063502009-10-04T18:04:34.436+02:002009-10-04T18:04:34.436+02:00The new traceback is:
Traceback (most recent cal...The new traceback is: <br /><br />Traceback (most recent call last):<br /> File "/home/chshrcat/build/pypy-trunk/pypy/translator/c/gcc/trackgcroot.py", line 1211, in <module><br /> assert fn.endswith('.s')<br />AssertionError<br /><br />Is the position in input tracked? that might help, or I could package my .gcmap files.Unhelpfulhttps://www.blogger.com/profile/12227174048122478111noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-46161069042887234222009-10-04T13:25:30.313+02:002009-10-04T13:25:30.313+02:00Andrew: can you update and try again? If you stil...Andrew: can you update and try again? If you still have the .c files around it is enough to go there and type "make"; otherwise, restart the build. It should still crash, but give us more information about why it does.Armin Rigohttps://www.blogger.com/profile/06300515270104686574noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-39104462403376359542009-10-04T09:31:46.165+02:002009-10-04T09:31:46.165+02:00First off - congratulations and good job on the gr...First off - congratulations and good job on the great progress. I've been watching this project since the 2007 PyCon in DFW and it's great to see these promising results.<br /><br />That said, while I know there's still a lot of work to do and this is very much an in-progress thing, I'm very much looking forward to an excuse to try this stuff out in anger - real practical situations. For me that means some statistical calculation engines (monto-carlo analysis) front ended by web services. In both situations this brings up two constraints: a) must support 64bit (because our data sets rapidly go above 4GB RAM) and b) must not be overly memory hungry (because any significant incremental overhead really hurts when your data sets are already over 4GB RAM).<br /><br />For now we use Psyco for small stuff but have to re-implement in C++ once we hit that 32-bit limit. PyPy is very exciting as a practical alternative to Psyco because of anticipated 64bit support. I wonder if, due to the existence fo Psyco already, that PyPy shouldn't focus first on 64bit instead?<br /><br />Few things would speed up progress than getting PyPy used out in the wild - even if only by those of us who appreciate it's very much in flux but still understand how to benefit from it.<br /><br />I understand you guys have your focus and goals and encourage you to keep up the good work. Just thought I'd throw this out as an idea to consider. I'm sure there are a lot like me anxious to give it a spin.<br /><br /> -- Ben Scherreyproteusguyhttps://www.blogger.com/profile/06205203744072448163noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-48871554883777023342009-10-04T03:40:55.460+02:002009-10-04T03:40:55.460+02:00Interestingly, the translation itself seems to con...Interestingly, the translation itself seems to consume at most about 960MB of ram. It's easy to translate on a system even with only a gig of ram if you stop everything else.<br /><br />Try switching run levels or the like.<br /><br />The -Ojit option seems to cause an error in translation with Revision 68125, when translated using Python 2.5.2 on Debian Lenny.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-54709146919644768262009-10-04T03:13:20.894+02:002009-10-04T03:13:20.894+02:00I was having the same problem as della, and your f...I was having the same problem as della, and your fix seems to work, but it's breaking somewhere else now. I don't think I have a dependency problem, I can build a working pypy-c without jit. Running make manually produces heaps of warnings about incompatible pointers, some probably harmless (int* vs long int*, these should be the same on x86-32), but others worry me more, like struct stat* vs struct stat64*, or struct SSL* vs char**. I put <a href="http://pastie.org/640911" rel="nofollow">the complete output of a manual run of make</a> online.Unhelpfulhttps://www.blogger.com/profile/12227174048122478111noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-26300401603823509832009-10-02T03:27:20.515+02:002009-10-02T03:27:20.515+02:00This is very exciting! Please, try to post updates...This is very exciting! Please, try to post updates to these figures... thanks!Luishttps://www.blogger.com/profile/01147433030878927988noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-13912730653767980442009-10-01T20:25:29.319+02:002009-10-01T20:25:29.319+02:00Very nice. Do you expect to get faster than psyco?...Very nice. Do you expect to get faster than psyco?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-42511963347106778532009-10-01T18:58:57.727+02:002009-10-01T18:58:57.727+02:00Since the blog post, here are the updated numbers:...Since the blog post, here are the updated numbers: we run richards.py in 2.10 seconds (almost 4x faster than CPython), and only spend 0.916 seconds actually running the assembler (almost 9x faster than CPython).Armin Rigohttps://www.blogger.com/profile/06300515270104686574noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-24824004783003473692009-10-01T18:56:20.914+02:002009-10-01T18:56:20.914+02:00Thanks for the report, della. Fixed, if you want ...Thanks for the report, della. Fixed, if you want to try again. Parsing gcc output is a bit delicate as the exact set of operations used depends on the specific version and command-line options passed to gcc.Armin Rigohttps://www.blogger.com/profile/06300515270104686574noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-12514030964228693792009-10-01T11:31:42.429+02:002009-10-01T11:31:42.429+02:00I can't seem to compile (32-bit Ubuntu 9.10 ch...I can't seem to compile (32-bit Ubuntu 9.10 chroot), by manually executing the Makefile in /tmp/usession-0/testing_1 I get this traceback:<br /><br /> File "/home/della/pkg/pypy/trunk/pypy/translator/c/gcc/trackgcroot.py", line 1210, in (module)<br /> tracker.process(f, g, filename=fn)<br /> File "/home/della/pkg/pypy/trunk/pypy/translator/c/gcc/trackgcroot.py", line 229, in process<br /> lines = self.process_function(lines, entrypoint, filename)<br /> File "/home/della/pkg/pypy/trunk/pypy/translator/c/gcc/trackgcroot.py", line 244, in process_function<br /> table = tracker.computegcmaptable(self.verbose)<br /> File "/home/della/pkg/pypy/trunk/pypy/translator/c/gcc/trackgcroot.py", line 285, in computegcmaptable<br /> self.parse_instructions()<br /> File "/home/della/pkg/pypy/trunk/pypy/translator/c/gcc/trackgcroot.py", line 364, in parse_instructions<br /> meth = self.find_missing_visit_method(opname)<br /> File "/home/della/pkg/pypy/trunk/pypy/translator/c/gcc/trackgcroot.py", line 390, in find_missing_visit_method<br /> raise UnrecognizedOperation(opname)<br />__main__.UnrecognizedOperation: jc<br /><br />there are some type warnings also for pointers, I don't know if they could be any useful. Maybe you can help me?dellahttps://www.blogger.com/profile/16873331731877660793noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-46233294416318559452009-09-30T12:32:43.830+02:002009-09-30T12:32:43.830+02:00Thank you! For many of us, the translation-time sw...Thank you! For many of us, the translation-time switch will be just as good.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-91170115427322411942009-09-30T10:42:05.278+02:002009-09-30T10:42:05.278+02:00Dear anonymous.
you can compile ./translate.py -O...Dear anonymous.<br /><br />you can compile ./translate.py -Ojit --jit-debug=profile<br /><br />There is no runtime switch unfortunately, so far.<br /><br />Cheers,<br />fijalMaciej Fijalkowskihttps://www.blogger.com/profile/11410841070239382771noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-25802155211340587312009-09-29T15:47:21.687+02:002009-09-29T15:47:21.687+02:00Dear Pypy developers, is it possible to switch off...Dear Pypy developers, is it possible to switch off the very agressive JIT logging in pypy-c? First, this could make pypy-c a drop-in replacement for cpython. (Many more beta-testers.) Second, the logging itself seems to be somewhat resource-intensive.<br /><br />Very cool Mandelbrot ascii art, by the way.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-40325409691419365672009-09-29T15:09:47.461+02:002009-09-29T15:09:47.461+02:00nice!
so what is your guess at the moment? how fa...nice!<br /><br />so what is your guess at the moment? how fast can pypy get if you further optimize the jit?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-19397639764567873112009-09-29T10:06:43.046+02:002009-09-29T10:06:43.046+02:00Hi,
this is so unbelievably awesome, it's goi...Hi,<br /><br />this is so unbelievably awesome, it's going to take me a while to recover from all the awesomness.<br /><br />CONGRATS!<br /><br />ps. a nice improvement for users is to get your ./configure script to find dependencies and report the ones missing, and ones used (s/configure/setup.py/g).René Dudfieldhttps://www.blogger.com/profile/17762358075557755436noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-49218702235867882272009-09-29T09:47:40.458+02:002009-09-29T09:47:40.458+02:00Thanks for the missing dependencies; added to the ...Thanks for the missing dependencies; added to the development version of the page. Thanks also for the numbers you report. The next obvious thing we miss is float support (coming soon!), which shows in some of your benchmarks.Armin Rigohttps://www.blogger.com/profile/06300515270104686574noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-83741446437197089492009-09-28T23:28:24.943+02:002009-09-28T23:28:24.943+02:00̉You are probably missing a dependency. See http:/...̉<i>You are probably missing a dependency. See http://codespeak.net/pypy/dist/pypy/doc/getting-started-python.html#translating-the-pypy-python-interpreter</i><br /><br />Dear Armin, it seem like this document should mention libexpat1-dev and libssl-dev as dependencies, too. Anyway, I managed to build pypy-c, and here are the result for some small benchmarks I wrote. (Is there a way here at blogger.com to not break the formatting?)<br /><br /> python 2.5 psyco pypy-c<br />richards 14.9 2.9 3.9<br />mergesort 27.6 4.8 26.3<br />convexhull 9.4 5.6 6.3<br />bigcityskyline 46.9 3.1 7.6<br />fft 14.1 15.4 25.0<br /><br />Thank you all for your efforts.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-74425452151333558082009-09-28T22:17:36.072+02:002009-09-28T22:17:36.072+02:00Here are prebuilt C sources (in which "tracin...Here are prebuilt C sources (in which "tracing" time was reduced by 20-30% since the blog post):<br /><br />http://wyvern.cs.uni-duesseldorf.de/~arigo/chain.tar.bz2<br /><br />Linux x86-32 only. You still need a svn checkout of PyPy, and you still need to compile them with gcc -- but it does not take too long: edit the first entry of the Makefile to point to your checkout of PyPy and type "make". This still assumes that all dependencies have been installed first. Don't complain if the #includes are at the wrong location for your distribution; you would get them right if you translated the whole thing yourself. In fact, don't complain if it does not compile for any reason, please :-) C sources like that are not really supposed to be portable, because they are just intermediates in the translation process.Armin Rigohttps://www.blogger.com/profile/06300515270104686574noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-38845612991434259542009-09-28T17:16:36.135+02:002009-09-28T17:16:36.135+02:00@Luis: yes, it's only first time.
Well, depend...@Luis: yes, it's only first time.<br />Well, depends how you count, but it<br />can be considered 8x speedup...Maciej Fijalkowskihttps://www.blogger.com/profile/11410841070239382771noreply@blogger.com