Garbage Collection

Scripting Issues
1

A while back I promised I would write a new page for the Panda3D manual to cover garbage collection, because it is such an important topic and it has so little coverage so far. I’m way, way past the deadline I set for myself on this, but I haven’t forgotten about it. I’ve been collecting and compiling what knowledge I could about garbage collection since then. I’ve put that knowledge to work in the Panda3D Beginner’s Guide that Packt Publishing has commissioned me to write, and I thought I would post a segment from that book here. This is not necessarily a complete guide to garbage collection, but it does cover all of the steps necessary for the game created over the course of the book.

According to my contract I have the right to reuse segments of the book for other writings, so long as they do not constitute more than 10% of the book and I credit the book as the source. Therefore, I have the right to reuse this material for the Panda3D manual, but the manual page must note that this writing comes from Panda3D: Beginner’s Guide - How to Build a Complete Game from Packt Publishing.

I leave it up to those who manage the manual as to whether or not that sort of psuedo advertisement is acceptable to put in the manual.

If anything in this writing is in error, please let me know.

Python will automatically garbage collect a custom class instance when all the references to that instance are removed. In theory, this makes garbage collection as simple as cleaning up those references, but because there are so many different places and reasons for these references garbage collection can quickly grow complicated. Following these steps will help to ensure that a custom class instance is properly garbage collected.

  1. Call removeNode on all NodePaths in the scene graph – The first step is to clear out the NodePaths that the custom class has added to the scene graph. If this step isn’t accomplished, it won’t necessarily prevent the custom class instance from being garbage collected, but it could. Even if the custom class instance is still garbage collected the scene graph itself will retain references to the NodePaths that haven’t been cleared out and they will remain in the scene graph. There is one exception to this rule: when a parent NodePath has removeNode called on it that ultimately result in the removal of its child NodePaths, so long as nothing else retains a reference to them. However, relying on this behaviour is an easy way to make mistakes so it’s better to manually remove all of the NodePaths a custom class adds to the scene graph.

  2. Call delete on all Actors – Just calling removeNode on an actor isn’t enough. Calling delete will remove ties to animations, exposed joints, and so on to ensure that all the extra components of the Actor are removed from memory as well.

  3. Set all Intervals, Sequences, and Parallels equal to None – It’s very common for Intervals, Sequences, and Parallels to retain references to something in the class and prevent the class instance from being cleaned up. To be safe, it’s best to remove the references to these Intervals so that they get cleaned up themselves and any references they have to the class are removed.

  4. Detach all 3D sounds connected to class NodePaths – 3D sounds won’t actually retain references to the custom class, but if the NodePaths they are attached to are removed with removeNode and the sounds aren’t detached, they’ll generate an error and crash the program when they try to access the removed NodePaths. Play it safe and detach the sounds.

  5. End all tasks running in the class – The task manager will retain a reference to the class instance so long as the class instance has a task running, so set up all of the tasks in the custom class to end themselves with return task.done. This is the most reliable way to stop them and clear the reference to the custom class in the task manager.

  6. If the custom class inherits from DirectObject, call self.ignoreAll() – Panda3D’s message system will also retain a reference to the custom class if it is set up to receive messages. To be on the safe side, every class that inherits from DirectObject and will be deleted during run time should call self.ignoreAll() to tell the message system that the class is no longer listening to messages. That will remove the reference.

  7. Remove all direct references to the custom class instance – Naturally, the custom class instance won’t get cleaned up if something is referencing it directly, either through a circular self reference, or because it was created as a “child” of another class and that other class has a reference to it stored as a variable. All of these references need to be removed. This also includes references to the custom class instance placed in PythonTags.

The del method is a good way to test if a custom class is being garbage collected. The del method is similar to the init method in that we don’t call it ourselves; it gets called when something happens. init is called when a new instance of the class is created; del is called when an instance of the class is garbage collected. It’s a pretty common thought to want to put some important clean up steps in the del method itself, but this isn’t wise. In fact, it’s best not to have a del method in any of our classes in the final product because the del method can actually hinder proper garbage collection. A better usage is to put a simple print statement in the del method that will serve as a notifier that Python has garbage collected the custom class instance. For example:

def __del__(self):
  print("Instance of Custom Class Alpha Removed")

Once we’ve confirmed that our custom class is being garbage collected properly, we can remove the del method.

2

As a panda3d user I would like a page like this in the manual. When I first started with the engine I coyldn’t find a place to learn all this. Altough I made a forum topic and got most of the questions answered, I think there should be something like this so people wont make duplicate topics asking the same things.

3

Lifetime of objects is certainly one of the most complex topics. Especially for Panda3D. I hope you don’t mind if I add a few comments.

Why only ‘custom’ class instances? What is a ‘custom’ class, by the way?
Python uses reference counting for all Python objects, no matter if the are ‘custom’ or not. Even unique objects like “None” have a reference count.

I think it would be good to distinguish better between Node and NodePath. removeNode removes the Node from the scene graph. The NodePath is just a handle to tell Panda3D what Node to remove.

I think it would be good to point readers at the layered approach of Panda3D. There is the Python layer on top, and the C++ layer below. Lifetime of Python objects and lifetime of the corresponding C++ objects is per se not correlated.

Take a look at this code snippet:

  def foo(self):
    node = PandaNode('foo')
    np = render.attachNewNode(node)
    del node
    del np

A function which might be called at any time during the game. The first line creates a new instance of the Python class ‘PandaNode’. This class is not defined in Python code, but within a C++ extension to Python (by the way, this is C++ extension is auto-generated by the interrogate tool). The init code of this Python class does something important: it creates an instance of a C++ class ‘PandaNode’. A completely different class; pure C++. So at the end of the first line we have:

  • a Python object with reference count 1, and
  • a C++ object which does not even know about Python reference counting.

The second line just calls a function where the Python ‘PandaNode’ instance is passed as a parameter. Depending on how interrogate-generated method is implemented the reference count might change (up or down), or it might not change. In this case it does not change.
I have put this line in because I wanted to introduce a second reference counting mechanism: some C++ objects do reference counting for themself - those C++ objects which inherit from the C++ class ReferenceCount. The C++ class ‘PandaNode’ is such a class. And even though calling the attachNewNode function does not change the Python reference count it does change the C++ reference count. So we have now:

  • a Python object with Python reference count 1, and
  • a C++ object with ReferenceCount count of 2 (one reference hold by the Python wrapper, and one by the scene graph)

The third line deletes the one reference to the Python ‘PandaNode’ instance. This line is not required, since this instance would be deleted at the end of the function ‘foo’ anyway, but I put it there to make it more explicit.
The situation is the following now: The Python ‘PandaNode’ instance has a reference count of 0, and it will get deallocated as soon as the garbage collector find time to do so. The C++ ‘PandaNode’ object is still alive, since it still has a C++ reference count of 1.

The only way to get rid of the C++ PandaNode instance is to destroy it on the C++ level. This can be done by getting a handle to the node (a NodePath), and call ‘removeNode’ on this handle. This will remove the C++ reference to the node hold by the scene graph. Getting a handle could be done by searching the scene graph:

 def bar(self):
  np = render.find("**/foo")
  np.removeNode()

Let’s consider a slightly modified version of this method:

 def bar(self):
  np = render.find("**/foo")
  node = np.node()

The Python ‘NodePath’ object is back. But not really - on the C++ level it is the same C++ NodePath instance, but on the Python level it is a different object which has been created just at this point of the code.

Maybe I am diving too deep into the complexity of Python/Interrogate/C++, so I try to bring it to the point:

  • Most Panda3D objects are C++ objects, and just have a Python wrapper around them.
  • Lifetime of C++ objects is not always the same as the lifetime of the corresponding Python objects.
  • Some C++ objects do reference counting, like Python objects do. Those C++ objects inherit from ReferenceCount. Others don’t.

Hope this didn’t cause too much confusion. The below script is nice for playing a little bit. Just keep in mind that sys.getrefcount returns one more than you might expect, since getrefcount creates one additional temporary reference.

import sys
import direct.directbase.DirectStart
from direct.showbase.DirectObject import DirectObject
from panda3d.core import PandaNode

class World(DirectObject):
  def __init__(self):
    self.accept('escape', self.doExit)
    self.accept('a', self.doA)
    self.accept('b', self.doB)
    self.accept('c', self.doC)
    self.accept('d', self.doD)

  def doExit(self):
    sys.exit(1)

  def doA(self):
    node = PandaNode('foo')
    np = render.attachNewNode(node)
    print 'A - node:', sys.getrefcount(node), id(node)
    print 'A - np:', sys.getrefcount(np)

  def doB(self):
    self.tmp = PandaNode('something else...')

  def doC(self):
    np = render.find("**/foo")
    node = np.node() 
    #np.removeNode()
    print 'C - node:', sys.getrefcount(node), id(node)
    print 'C - np:', sys.getrefcount(np)

  def doD(self):
    np = render.find("**/foo")
    np.removeNode()
    print 'D - np:', sys.getrefcount(np)
    print 'D - node:', np.node() # assert empty raised!

world = World()
run()
4

Remember, the book is a Beginner’s guide. :stuck_out_tongue:

5

I don’t see the manual as a “beginners guide”. I see it as the place for all documentation thats not in the language reference. If not in the manual, where would advanced garbage collection details go? That kind of thing needs to be documented somewhere. Besides, subclassing C++ classes and losing the python part of it is a confusing and somewhat common mistake (and already documented in the manual by the way for the case of NodePaths). Its not something to just leave out.

6

Right, this has been too much in depth. I agree it is best not to explain how everything works, but to give a list of hints how to clean up (what resources to release or remove, break up cyclic references and so on).

7

piratePanda, will you ever add this info to the manual?

8

This time I had completely forgotten about it. I’ve added it to the manual now, after the section on Collision Detection.

Hopefully more folks, such as enn0x, will add to the new garbage collection section.

9

Nice.
I don’t think the page title is very accurate, you are not merely talking about removing “custom” class instances, you talk about removeNode(), destroy(), taskMgr.remove(), etc.
And I think there are some things missing, like the destroy() method for some Python wrapper classes, and cleanup() for Particle Systems, also a mention of other Python objects and Pythons “del” statement.
I’d add it, but I don’t know much about them myself.

10

The page is talking about removing the “custom” class instances, and the steps necessary to do so. That involves a lot of the information you’re talking about, but the blanket topic covering it all is the removal of those “custom” class instances.

11

Hm, well I guess you documented those functions when used with a custom class instance. Documenting them again for general use in a separate page is not a bad idea then.