tag:blogger.com,1999:blog-3971202189709462152.post6174120095428192954..comments2024-03-11T12:50:02.036+01:00Comments on PyPy Status Blog: GC improvementsCarl Friedrich Bolz-Tereickhttp://www.blogger.com/profile/00518922641059511014noreply@blogger.comBlogger31125tag:blogger.com,1999:blog-3971202189709462152.post-25773860699742575782011-09-15T11:37:52.832+02:002011-09-15T11:37:52.832+02:00And in assembly it will be even faster and smaller...And in assembly it will be even faster and smaller. <br /><br />Python has many lovely attributes, but efficiency is not its primary virtue. That said, making it more efficient is still a plus, which this work is doingAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-8455099133983611972011-05-07T03:34:21.508+02:002011-05-07T03:34:21.508+02:00This program in C# takes 589 miliseconds, and 52 M...This program in C# takes 589 miliseconds, and 52 MB RAM. 17x faster, 4.75x less RAM.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-58056351836726056352010-09-22T20:30:10.535+02:002010-09-22T20:30:10.535+02:00@PJE PyPy offers collections.identity_dict, or som...@PJE PyPy offers collections.identity_dict, or something similar which would have the effect how you like (but internally doesn't use id operation, just the object identity).Maciej Fijalkowskihttps://www.blogger.com/profile/11410841070239382771noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-82641934201346668822010-09-22T20:22:16.358+02:002010-09-22T20:22:16.358+02:00I can think of one place where I use a lot of id()...I can think of one place where I use a lot of id() calls, and that's in PEAK-Rules' generic function implementation, for indexing "is" tests.<br /><br />For example, if you have a bunch of methods that test if "x is Something" (for different values of Something), then a dictionary of id()'s is used to identify which of these tests went off. While the total number of Somethings isn't likely to be high, the weakref dict in PyPy means that every 'x' the function is called with will end up burning memory and speed to hold an id forever.<br /><br />While it's perhaps the case that I could avoid this by using a linear search (ugh) in cases where the number of Somethings is small, it's an example of a place where id() makes an operation neat, fast, and simple in regular Python.<br /><br />Of course, if there were another way to put arbitrary (i.e possibly-unhashable, comparable only by identity) objects in a dictionary, and then determine whether a given object was one of them, that'd certainly be a suitable substitute.<br /><br />Or, if PyPI offered a temp_id() that would simply let you *check* identity, without forcing the object to hold onto it, that'd work fine too. Say, if there was a has_id() that told you if an id() is outstanding for the object already, or a get_id() that returned None for an object whose id() had never been taken.<br /><br />With an API like that, I could prevent memory/speed blowup by not having each call of the function adding more objects to PyPy's id() dict.<br /><br />(Heck, perhaps such an API should be added across Python versions, i.e., to CPython and Jython as well.)PJEhttps://www.blogger.com/profile/04688223805457202941noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-5544112414181852762010-04-14T16:14:58.203+02:002010-04-14T16:14:58.203+02:00I'm astonished a bit by your need to pack vtab...I'm astonished a bit by your need to pack vtables together within 256KB. How many bits do you need for mark-and-sweep marking or similar stuff? The usual solution I've seen for this is to use the low two bits of the vtable pointer for flags, usually, and mask them off when reading the vtable pointer. Would it work here?<br /><br />If that isn't enough, then you have to pack vtables together as you do (maybe in a bigger space if you can use more bits).Anonymoushttps://www.blogger.com/profile/04485097839438234853noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-55277154392876125682009-11-02T22:51:34.988+01:002009-11-02T22:51:34.988+01:00Wohoo, nice performanceWohoo, nice performanceomul cu 6233https://www.blogger.com/profile/03065040375317980599noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-15584373267069838812009-10-21T15:34:36.659+02:002009-10-21T15:34:36.659+02:00Yes, I've been thinking about that too.
But t...Yes, I've been thinking about that too.<br /><br />But that can be patched - the weak key dict could still be used for those objects that haven't been collected yet. Since addition of the id would most likely happen in the nursery, or the first generation at most (big assumption), I don't think the dict would grow very big even under heavy id() usage.klaussfreirehttps://www.blogger.com/profile/05255172019231560487noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-81223382876482049232009-10-21T10:11:17.195+02:002009-10-21T10:11:17.195+02:00Claudio: this doesn't work (unless I misunders...Claudio: this doesn't work (unless I misunderstood). You cannot add a field like the hash field at any point in time, but only during collections when the object moves.Armin Rigohttps://www.blogger.com/profile/06300515270104686574noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-56510595780761470392009-10-19T18:38:45.155+02:002009-10-19T18:38:45.155+02:00Wouldn't it free up the GC from all that burde...Wouldn't it free up the GC from all that burden if only a set of live ids were kept? (ie: no weak dict)<br /><br />So, when you get an id() call, you check the object to see if there's a cached id (much like the hash hack) - if not, you generate a random (or sequential) unused id and store it both in the "live ids" set and in the object's structure, as done with hash values.<br /><br />So, successive calls to id() would be as fast as in CPython, and garbage collection would be fast too (only an extra set deletion per object whose id was obtained).<br /><br />In fact, this set could be implemented as a bit array with "free lists", which could be very very efficient, given that its size will be bound by the number of live objects.klaussfreirehttps://www.blogger.com/profile/05255172019231560487noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-57528537027628012022009-10-19T12:30:44.013+02:002009-10-19T12:30:44.013+02:00Marius: as I said, feel free to :-) but the curre...Marius: as I said, feel free to :-) but the current situation is, no, not as far as I know.Armin Rigohttps://www.blogger.com/profile/06300515270104686574noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-85922580307878256262009-10-18T20:49:07.618+02:002009-10-18T20:49:07.618+02:00@Anonymous:
Try deleting all your .pyc files and ...@Anonymous:<br /><br />Try deleting all your .pyc files and see what happens.Maciej Fijalkowskihttps://www.blogger.com/profile/11410841070239382771noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-87443165414469850682009-10-18T11:14:18.091+02:002009-10-18T11:14:18.091+02:00Is there any possibility to translate pypy under O...Is there any possibility to translate pypy under OSX 10.6 as 32bit? Translation works but I get an "ValueError: bad marshal data" when running pypy-c. I assume that is due to the fact that I got a 64bit binary.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-87525527908819086762009-10-18T00:20:00.881+02:002009-10-18T00:20:00.881+02:00When I was writing objgraph I saw no way of traver...When I was writing <a href="http://mg.pov.lt/objgraph/" rel="nofollow">objgraph</a> I saw no way of traversing arbitrary object graphs without using id().<br /><br />collections.identitydict sounds like a nice idea. Has anyone written a PEP for it?Marius Gedminashttps://www.blogger.com/profile/15155998626202067226noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-51767034142006776882009-10-17T18:32:47.862+02:002009-10-17T18:32:47.862+02:00Too bad for the current implementation of pickle a...Too bad for the current implementation of pickle and deepcopy. The fault in that case is CPython's general view that id() is cheap, despite repeated attempts to convince them otherwise. These attempts have been done notably by guys from Jython, even before PyPy time; indeed id() is a mess for any implementation apart from CPython's simple non-moving GC).<br /><br />A suitable replacement would be e.g. a 'collections.identitydict' type, if someone feels like going Yet Another Time to python-dev with this issue.Armin Rigohttps://www.blogger.com/profile/06300515270104686574noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-5027983261868271382009-10-17T12:08:55.752+02:002009-10-17T12:08:55.752+02:00what about pickle - as far as i remember its memo ...what about pickle - as far as i remember its memo code for dealing with object cycles is using id, tooRonnyPfannschmidthttps://www.blogger.com/profile/06295961645811499445noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-78654522859250581952009-10-17T11:57:25.801+02:002009-10-17T11:57:25.801+02:00Doesn't deepcopy use id() a lot? I remember on...Doesn't deepcopy use id() a lot? I remember once using deepcopy on a complicated structure, resulting in thousands of id() calls.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-47297609521063516322009-10-17T03:17:19.144+02:002009-10-17T03:17:19.144+02:00@Lucian:
ctypes and JIT works just fine together....@Lucian:<br /><br />ctypes and JIT works just fine together.Maciej Fijalkowskihttps://www.blogger.com/profile/11410841070239382771noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-87474174246633245702009-10-17T00:19:35.827+02:002009-10-17T00:19:35.827+02:00Yay, I remember talking about removing the gc type...Yay, I remember talking about removing the gc type pointer, oh, about 3.5 years ago :) Cool that it got done, sounds like a neat pair of hacks.Michael Hudson-Doylehttps://www.blogger.com/profile/08800334898826186482noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-9393388164080699722009-10-16T22:53:36.538+02:002009-10-16T22:53:36.538+02:00Carl, Shahms: I couldn't agree more about id()...Carl, Shahms: I couldn't agree more about id() not being important.<br /><br />Probably Guido should have refrained from making it available in CPython at the time. I suppose it was just easy to add it to the language with the memory allocation model of CPython. The fact is that I don't really see any use for id() once you have the "is" operator and the hash() method...Skandalfohttps://www.blogger.com/profile/03657421926790640934noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-9352705098234633142009-10-16T22:44:32.836+02:002009-10-16T22:44:32.836+02:00Carl, no doubt you're right. I know that I ca...Carl, no doubt you're right. I know that I can probably count the number of times I've needed to use id() on one hand and I'm pretty sure the vast majority of those cases was sticking an-hashable object in a dict.Shahmshttps://www.blogger.com/profile/00799196834273346407noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-47435194388100384052009-10-16T22:38:01.426+02:002009-10-16T22:38:01.426+02:00Shahms: I confess I don't understand your prop...Shahms: I confess I don't understand your proposal. Do you mean you can have at most as many live objects as the available address space divided by the object alignment?<br /><br />When I talked about id space I wasn't referring to the memory required to store the per-object id value, but the fact that if you assign the id values using sequential values, and those values are, for instance, 64 bit integers, you could theoretically create and destroy a lot of objects in a long lived process and the sequence would wrap around.<br /><br />About making hash/id the same, I've just checked that CPython does indeed use the id() value as the value returned by the default hash() implementation.<br /><br />You could just do the same, and use the id value as the "master" one. For hash() you would just call id(). This allows you to use just one value attached to the objects for both functions.<br /><br />The cost of that approach would be having to assign an id immediately (having to put it into the temporary map, then having to look it up in the map until the next time the object is moved) for the call to hash() (with no call to id()) too.<br /><br />The good thing compared to the weak key dict, is that the temporary map doesn't need to be garbage collected at all. The entries are removed when objects are moved (or collected).Skandalfohttps://www.blogger.com/profile/03657421926790640934noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-28939855942645390382009-10-16T22:37:26.620+02:002009-10-16T22:37:26.620+02:00Skandalof, Shahms: I guess there are possible ways...Skandalof, Shahms: I guess there are possible ways to make id a bit faster than what we have now. What we have now is well-tested and works reasonably enough. I assume anyway that there is not too much Python code whose performance depends critically on having an extremely efficient implementation of id (and if there is, I am prepared to ask the author to rewrite the code instead :-) ).Carl Friedrich Bolz-Tereickhttps://www.blogger.com/profile/00518922641059511014noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-3077204891985820342009-10-16T22:19:17.390+02:002009-10-16T22:19:17.390+02:00Thanks, Carl. Following up what Skandalfo said, (...Thanks, Carl. Following up what Skandalfo said, (although this is probably a poor forum for such discussions), it seems like you could reuse the hash field for id as well. Given that the minimum size for a Python object is > 1 byte, you should have at least that much space for offsetting the hash/id. As the GC/allocator has to store information about addresses and blocks anyway it should be a relatively simple matter of building and maintaining a bloom filter of offsets in use for a particular base address.<br /><br />Of course, this also constraints the addresses at which Python objects may be allocated and the lower bits in the address may already be used for other purposes...Shahmshttps://www.blogger.com/profile/00799196834273346407noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-50865602341543541722009-10-16T22:05:14.192+02:002009-10-16T22:05:14.192+02:00Wow! You guys that you are my computing heroes.
W...Wow! You guys that you are my computing heroes.<br /><br />Whenever I talk to other people about your project, I always state you are the best example I can imagine of REAL innovation in computer languages.<br /><br />That said, I gather the only thing making id() different from hash() is that you need to guarantee that the values for live objects are always unique.<br /><br />You could just use the same strategy as with the hash, sticking the id value along the object the next time the object is moved by the GC.<br /><br />Meanwhile, from the time id() is called to the time the object is moved, you can just temporarily store an {address: id} mapping somewhere. Entries would be removed from the map once the objects get moved. From then on the id would be attached to the object.<br /><br />If GC cycles are frequent, the map doesn't have to grow too large.<br /><br />I don't know if the need for id reuse after the id space gets exhausted is important or not. Once you get to the end of the space, you would have to scan the map and heap to find a convenient "hole" to reuse, I suppose.Skandalfohttps://www.blogger.com/profile/03657421926790640934noreply@blogger.comtag:blogger.com,1999:blog-3971202189709462152.post-35211397357449201542009-10-16T21:57:30.076+02:002009-10-16T21:57:30.076+02:00Alex: you are right. We use exactly CPython's ...Alex: you are right. We use exactly CPython's algorithm for implementing dicts, so having bad hash functions is not a big problem. However, if you really have hash value collisions (e.g. everything hashes to 1) your dict still degenerates to essentially a linear search.Carl Friedrich Bolz-Tereickhttps://www.blogger.com/profile/00518922641059511014noreply@blogger.com