tag:blogger.com,1999:blog-3971202189709462152.post8684276541915333814..comments2024-03-11T12:50:02.036+01:00Comments on PyPy Status Blog: PyPy-STM: first "interesting" releaseCarl Friedrich Bolz-Tereickhttp://www.blogger.com/profile/00518922641059511014noreply@blogger.comBlogger22125tag:blogger.com,1999:blog-3971202189709462152.post-90389570331102351062014-08-04T15:44:51.838+02:002014-08-04T15:44:51.838+02:00awesome! thanks a lot armin. :Dawesome! thanks a lot armin. :Disomorphhttps://www.blogger.com/profile/11359946465217889610noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-54558376961082076482014-08-04T09:04:45.086+02:002014-08-04T09:04:45.086+02:00Having a JIT in the JVM is very different from hav...Having a JIT in the JVM is very different from having a JIT that can understand Python. For proof, the best (and only) implementation of Python on the JVM, Jython, is running at around CPython speed (generally a bit slower). I suspect that STM is similarly not designed for the purposes to which Jython would put it and would thus perform poorly. The only part that would probably work out of the box would be the GC. A more subtle argument against starting from the JVM is that of semantic mismatch. See for example http://www.stups.uni-duesseldorf.de/mediawiki/images/5/51/Pypy.pdfArmin Rigohttps://www.blogger.com/profile/06300515270104686574noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-45697429655032349672014-07-31T06:46:05.945+02:002014-07-31T06:46:05.945+02:00this is a question for the guys developing PyPy......this is a question for the guys developing PyPy... i am completely new to Python so please bear with me.<br /><br />here is what i don't understand: it seems to me that you are reinventing the wheel because doesn't the Oracle or Azul Systems JVM already provide a super performant GC and JIT? even STM is becoming available. and since Jython can run on the JVM, why do PyPy at all? <br /><br />wouldn't a JVM compliant implementation of Python be more performant than PyPy or CPython?<br /><br />or am i missing something here?<br /><br />any pointers greatly appreciated. thanks.isomorphhttps://www.blogger.com/profile/11359946465217889610noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-2321821510944078312014-07-20T12:26:47.202+02:002014-07-20T12:26:47.202+02:00r2 still doesn't work for me (ubuntu 14.04, in...r2 still doesn't work for me (ubuntu 14.04, intel Core2 CPU T7400)<br />bash: ./pypy: cannot execute binary file: Exec format errorAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-17504801593879335412014-07-19T16:40:20.786+02:002014-07-19T16:40:20.786+02:00If you guys did a facelift on the website like you...If you guys did a facelift on the website like yours HippyVM I believe the project would gain a lot of momentum, it is unfortunate but true that most company managers would visit it and think it is not industrial quality if an employ comes saying that they should sponsor developing something in PyPy.Canesinhttps://www.blogger.com/profile/05359201128746760200noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-9719768688413621482014-07-16T09:16:14.024+02:002014-07-16T09:16:14.024+02:00This is exciting news! I think pypy is the future ...This is exciting news! I think pypy is the future of python.geerknoreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-3310889641025856522014-07-12T02:03:37.538+02:002014-07-12T02:03:37.538+02:00@Armin, cool! I've found that the thread pool ...@Armin, cool! I've found that the thread pool version can be sped up ~2-3x by wrapping the contents of check_prime with 'atomic' too.<br /><br />One more observation: with the atomic context manager, on PyPy-STM the queue implementation will beat the thread pool implementation (slightly), which isn't the case for CPython or ordinary PyPy.Anonymoushttps://www.blogger.com/profile/12455815708286717299noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-2126159620922064532014-07-10T10:31:00.877+02:002014-07-10T10:31:00.877+02:00Added an answer to the question "what about P...Added an answer to the question "what about PyPy3?": https://pypy.readthedocs.org/en/latest/stm.html#python-3Armin Rigohttps://www.blogger.com/profile/06300515270104686574noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-29347996916029939622014-07-10T09:44:09.417+02:002014-07-10T09:44:09.417+02:00Ooooof. Ok, I found out what is wrong in bench-qu...Ooooof. Ok, I found out what is wrong in bench-queue. The issue is pretty technical, but basically if you add "with __pypy__.thread.atomic:" in the main top-level loop in worker(), then it gets vastly faster. On my machine it beats the real-time speed of a regular pypy. See http://bpaste.net/show/450553/<br /><br />It clearly needs to be fixed...Armin Rigohttps://www.blogger.com/profile/06300515270104686574noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-66857579428552374862014-07-08T22:04:52.111+02:002014-07-08T22:04:52.111+02:00@Armin I've pushed a version of bench-queue wi...@Armin I've pushed a version of bench-queue with a tweakable batch size and concurrency level. Doing the work in batches of, say, 1000 does indeed make it go faster with all implementations.<br /><br />I've noticed pypy-stm runs have a large variance. It's not like I'm doing scientific measurements here, but for the queue test I'm getting runtimes from ~15 sec to ~27 sec, whereas for example ordinary PyPy is in the range 4.6 sec - 4.9 sec, and CPython ~22.5 - ~24.7, again, relatively close. Again, this is just something I noticed along the way and not the result of serious benchmarking in isolation.Anonymoushttps://www.blogger.com/profile/12455815708286717299noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-6556232083123164262014-07-08T12:17:09.373+02:002014-07-08T12:17:09.373+02:00@Tin: ...no, I tried too and it doesn't seem t...@Tin: ...no, I tried too and it doesn't seem to help. We'll need to look into this in more details....Armin Rigohttps://www.blogger.com/profile/06300515270104686574noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-52098115948652017072014-07-08T11:47:46.399+02:002014-07-08T11:47:46.399+02:00@Tin: I would tweak bench-queue.py to avoid a mill...@Tin: I would tweak bench-queue.py to avoid a million inter-thread communications via the queue. For example, run 1000 check_primes instead of just 1 for every number received from the queue.Armin Rigohttps://www.blogger.com/profile/06300515270104686574noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-21122830818675797972014-07-07T22:40:44.993+02:002014-07-07T22:40:44.993+02:00This is good news. For many of my applications, a...This is good news. For many of my applications, an important feature in the next phase will be the optimization for <i>[..] the built-in dictionary type, for which we would like accesses and writes using independent keys to be truly independent [..]</i>. My applications are mostly server applications (Twisted-based and others) that store state information on sessions/transactions in a small number of dictionaries that can have hundreds or thousands of entries concurrently, and would be accessed constantly.<br /><br />I'm glad I donated and plan do so again in the future :-)Pimnoreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-35184982819079781242014-07-07T22:35:52.786+02:002014-07-07T22:35:52.786+02:00Hi again,
just to play around a little I've p...Hi again,<br /><br />just to play around a little I've put together https://github.com/Tinche/stm-playground for myself.<br /><br />I picked a generic CPU-bound problem (primality testing) and tried comparing multithreaded implementations in CPython 2.7, ordinary PyPy and PyPy-STM.<br /><br />I figured this would be easily parallelizable (low conflicts) but it doesn't seem to be the case - I don't get all my cores pegged using the STM.<br /><br />bench-threadpool.py, on my machine, gives about the same time for CPython and PyPy-STM, while ordinary PyPy totally smokes them both (even with the GIL :), one order of magnitude difference (20 sec vs 2 sec).<br /><br />bench-threadpool-naive will crash the STM interpreter on my system. :)<br /><br />Getting away from threads, CPython will actually beat PyPy in a multi-process scenario by a factor of 2, which I found surprising. CPython does indeed use up all my cores 100% while dealing with a process pool, while PyPy has won't even come close.<br /><br />For the same workload, PyPy is actually faster running multithreaded with the GIL than multi-process, and fastest running with only 1 thread (expected, with the GIL only being overhead in this scenario).Anonymoushttps://www.blogger.com/profile/12455815708286717299noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-91809224397100982832014-07-07T21:00:41.630+02:002014-07-07T21:00:41.630+02:00This is exciting! One minor bug in the actual post...This is exciting! One minor bug in the actual post: you can describe slowdown / speedup in two different ways, with <i>total time</i> as a percentage of original time, or with <i>time difference</i> as a percentage of original time. You mention a 20% slowdown (clearly using the latter standard) and then a 300% slowdown, which you describe as 3x (suggesting that you use the former standard). To be consistent , you should either describe them as 120% and 300%, respectively (using the former standard), or 20% and 200%, respectively (using the latter standard).<br /><br />Thanks!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-3164660719492635422014-07-07T13:37:10.300+02:002014-07-07T13:37:10.300+02:00Thanks to the author of the now-deleted comments, ...Thanks to the author of the now-deleted comments, we could track and fix a bug that only shows up on some Linux systems. If pypy-stm systematically segfaults at start-up for you too, try the "2.3-r2" release (see update in the post itself).Armin Rigohttps://www.blogger.com/profile/06300515270104686574noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-74177584478883720772014-07-06T09:48:31.767+02:002014-07-06T09:48:31.767+02:00@Ernst: sorry, it works fine for me as well. I tr...@Ernst: sorry, it works fine for me as well. I tried the pypy-stm provided here, both on a Ubuntu 12.04 and a Ubuntu 14.04 machine. Maybe you have a too old virtualenv? Does it work with regular PyPy?Armin Rigohttps://www.blogger.com/profile/06300515270104686574noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-17182429615767779732014-07-06T00:40:37.080+02:002014-07-06T00:40:37.080+02:00If I try virtualenv I get:
virtualenv stmtest -p P...If I try virtualenv I get:<br />virtualenv stmtest -p Projekt/pypy-stm-2.3-linux64/bin/pypy <br />Running virtualenv with interpreter Projekt/pypy-stm-2.3-linux64/bin/pypy<br />[forking: for now, this operation can take some time]<br />[forking: for now, this operation can take some time]<br />New pypy executable in stmtest/bin/pypy<br />[forking: for now, this operation can take some time]<br />ERROR: The executable stmtest/bin/pypy is not functioning<br />ERROR: It thinks sys.prefix is u'/home/ernst' (should be u'/home/ernst/stmtest')<br />ERROR: virtualenv is not compatible with this system or executableErnst Sjöstrandhttps://www.blogger.com/profile/17837850227679012927noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-61293734481667190292014-07-05T21:13:29.397+02:002014-07-05T21:13:29.397+02:00Yes. Sorry, it doesn't make sense to me. You...Yes. Sorry, it doesn't make sense to me. You need to debug with gdb, probably with an executable that has got the debugging symbols. You need to either build it yourself, or recompile the pregenerated sources from: http://cobra.cs.uni-duesseldorf.de/~buildmaster/misc/pypy-c-r72356-stm-jit-SOURCE.txzArmin Rigohttps://www.blogger.com/profile/06300515270104686574noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-79839621562253362422014-07-05T19:23:58.233+02:002014-07-05T19:23:58.233+02:00This comment has been removed by the author.Anonymoushttps://www.blogger.com/profile/12455815708286717299noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-8473048890416416022014-07-05T17:22:48.480+02:002014-07-05T17:22:48.480+02:00You're just extracting and running the "b...You're just extracting and running the "bin/pypy"? It works for me on a very close configuration, Ubuntu 14.04 too...Armin Rigohttps://www.blogger.com/profile/06300515270104686574noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-60199986261491552772014-07-05T16:43:15.405+02:002014-07-05T16:43:15.405+02:00This comment has been removed by the author.Anonymoushttps://www.blogger.com/profile/12455815708286717299noreply@blogger.com