Possible Proof of Memory Leaking in P3D

[markjacksonguy]

Hi.

I’ve been trying to decide rather or not to use this game engine. I’ve been playing around with it some and it seems ok.

I wanted to test the ability to unload resources with this Engine and the only thing I could find is methods posted in the Manual.

I created a little program that randomly creates objects and those objects will destroy after a short period of time.

The only problem is, memory is never freed!

I let this program run for several days on my extra PC and when I finally checked it again, the application memory in windows task manager was reporting over 1gb of memory used (and rising!)!

Pstats also showed higher memory usage.

This program started out with very little memory, so that proves the memory is not being freed. So go ahead and run my application. You can even view the code and make changes.

If anyone can figure out how to truly release the memory, please let me know.

Thanks,

Mark

4shared.com/file/Fuw_cETu/P3DmemLeak.html

[drwr]

That’s quite a complex program. I find it difficult to follow through everything it is doing. For the purposes of testing for leaks, wouldn’t it generally be better to have a much simpler program, for instance, one that runs a simple loop of loading an object (or 100 objects) and then freeing the previous object (or 100 objects)?

In your case, I tracked down the memory leak to the fact that your SpEfs_BldSplsh class stores a pointer to a task that it spawns, as self.DwnDriftBld. This task, of course, in turn keeps a pointer to the class method, self.DrifDwnCtrl, which in turn keeps a pointer to self, causing a circular reference, and therefore creating an object that can never be freed. The solution is to add “del self.DrifDwnCtrl” in the destroy() method for this class.

Note, by the way, that all of your objects inherit from ShowBase. You should never do that. There should always be exactly one instance of ShowBase in your application, never more than one. Each time you instantiate a new ShowBase, you’re reinitializing many low-level things within Panda, and no doubt causing all sorts of terrible things to happen. In fact, importing DirectStart is sufficient to create a single, global ShowBase. Since you are already importing DirectStart, there’s no need to inherit from ShowBase at all. Inherit from DirectObject instead; that’s probably what you meant to do.

Still, since you never upcall to the base class init() method, you never actually instantiate any of these other ShowBases, so that’s probably not causing you any real harm in this particular example.

David

[markjacksonguy]

That’s quite a complex program. I find it difficult to follow through everything it is doing. For the purposes of testing for leaks, wouldn’t it generally be better to have a much simpler program, for instance, one that runs a simple loop of loading an object (or 100 objects) and then freeing the previous object (or 100 objects)?

In your case, I tracked down the memory leak to the fact that your SpEfs_BldSplsh class stores a pointer to a task that it spawns, as self.DwnDriftBld. This task, of course, in turn keeps a pointer to the class method, self.DrifDwnCtrl, which in turn keeps a pointer to self, causing a circular reference, and therefore creating an object that can never be freed. The solution is to add “del self.DrifDwnCtrl” in the destroy() method for this class.

Note, by the way, that all of your objects inherit from ShowBase. You should never do that. There should always be exactly one instance of ShowBase in your application, never more than one. Each time you instantiate a new ShowBase, you’re reinitializing many low-level things within Panda, and no doubt causing all sorts of terrible things to happen. In fact, importing DirectStart is sufficient to create a single, global ShowBase. Since you are already importing DirectStart, there’s no need to inherit from ShowBase at all. Inherit from DirectObject instead; that’s probably what you meant to do.

Still, since you never upcall to the base class init() method, you never actually instantiate any of these other ShowBases, so that’s probably not causing you any real harm in this particular example.

David

Don’t know if I followed you correctly or not, but I received the error DirectObject not defined. Did I fail to import something?

#Created by Mark - this program creates instances of an obj class and cleans them up after a short time, but Panda3D DOES NOT free the memory!!!

#from pandac.PandaModules import loadPrcFileData
#loadPrcFileData("", "want-directtools #t")
#loadPrcFileData("", "want-tk #t")



import direct.directbase.DirectStart
#from direct.showbase.ShowBase import ShowBase
from direct.showbase import Audio3DManager
from pandac.PandaModules import *
from direct.actor.Actor import Actor
from direct.gui.OnscreenText import OnscreenText
from direct.gui.OnscreenImage import OnscreenImage
from direct.filter.CommonFilters import CommonFilters
from direct.filter.FilterManager import *


from math import floor, ceil, pi, sin, cos
import sys
from os.path import isfile, isdir, join, basename, splitext, abspath
from os import mkdir, listdir, rmdir, rename, remove, system
from os import name as os_name
from random import *


PStatClient.connect();

#ffFPS = 60 
#globalClock = ClockObject.getGlobalClock() 
#globalClock.setMode(ClockObject.MLimited) 
#globalClock.setFrameRate(ffFPS)
#---


class SeffectsMgr(DirectObject):
    def __init__(self):
        self.LiveEffs = [];
        self.DeadEffs = [];

    def KillEffs(self, Marker):
        Marker.destroy();
        del Marker;

SplEffsMgr = SeffectsMgr();



class SpEfs_BldSplsh(DirectObject):
    def __init__(self):
        #Vars================
        self.AtchPnt = None;
        self.WldPos = Vec3(0,0,0);
        self.DrftTime = 30;
        #end vars===========
        #Tasks===================
        self.SetPntSp = taskMgr.add(self.SetSplPnt, "SpPntSet");
        #end tasks=================

        self.BlSplash = loader.loadModel("RandObj");
        self.BlSplash.flattenStrong();
        self.BlSplash.setShaderAuto();
        self.BlSplash.lookAt(base.cam);
        self.BlSplash.setHpr(0,45-floor(random()*45),0);


    def SetSplPnt(self, task):
        if self.AtchPnt != None:
            self.BlSplash.reparentTo(self.AtchPnt);
            self.BlSplash.setPos(self.AtchPnt, 0, 2.2, 0);
            self.WldPos = self.BlSplash.getPos(render);
            self.BlSplash.reparentTo(render);
            self.BlSplash.setPos(self.WldPos);
            self.DwnDriftBld = taskMgr.add(self.DrifDwnCtrl, "DriftBldsp");
            return task.done;
        else:
            return task.cont;

    def DrifDwnCtrl(self, task):
        if self.DrftTime > 0:
            self.BlSplash.setPos(-10+floor(random()*20), -10+floor(random()*20), self.BlSplash.getZ()-0.1);
            self.DrftTime-=1;
            return task.cont;
        elif self.DrftTime <= 0:
            self.BlSplash.detachNode();
            SplEffsMgr.KillEffs(self);
            return task.done;

    def destroy(self):
        self.ignoreAll();
        try:
            taskMgr.remove(self.SetPntSp);
        except:
            pass;
        try:
            taskMgr.remove(self.DwnDriftBld);
        except:
            pass;

        del self.DrifDwnCtrl;
        del self.SetSplPnt;

        del self.AtchPnt;
        del self.WldPos;
        del self.DrftTime;
        del self.SetPntSp;

        self.BlSplash.removeNode(); del self.BlSplash;



class MemDumper(DirectObject):
    def __init__(self):
        self.accept("shift", self.KleanTest);

    def KleanTest(self):
        TexturePool.garbageCollect();
        ModelPool.garbageCollect();
        base.win.getGsg().releaseAll();
DebugMem = MemDumper();




class MakeObje(DirectObject):
    def __init__(self):
        self.PNTobj = render.attachNewNode("SpotNodey");
        self.PNTobj.setPos(0,0,0);
        base.cam.setPos(self.PNTobj, 0, -40, 0);
        base.cam.lookAt(self.PNTobj);

        self.accept("escape", self.EnDdGame);
        self.Trigger = taskMgr.add(self.ConstantCreate, "MakeObjs");
        
    def ConstantCreate(self, task):
        SpEfs_BldSplsh().AtchPnt = self.PNTobj;
        return task.cont;

    def EnDdGame(self):
        base.exitfunc();
        sys.exit();
GrandMaker = MakeObje();




        

render.setAntialias(AntialiasAttrib.MFaster+AntialiasAttrib.MAuto);

run();

[DangerOnTheRanger]

Adding this at the start of your script:

from panda3d.core import DirectObject

should solve your problem.

[drwr]

Actually, it’s:

from direct.showbase.DirectObject import DirectObject

David

[markjacksonguy]

Adding this at the start of your script:

from panda3d.core import DirectObject

should solve your problem.

New Error Message:

SpEFfs_BldSplsh instance has no attribute “DrifDwnCtrl”

[drwr]

Oh, and my apologies, I misspoke: you should have “del self.DwnDriftBld”, not “del self.DrifDwnCtrl”, in destroy(). The thing is to delete the task pointer, not the method.

David

[DangerOnTheRanger]

Actually, it’s:

from direct.showbase.DirectObject import DirectObject

David

Ouch! Apologies for not catching that

[markjacksonguy]

Oh, and my apologies, I misspoke: you should have “del self.DwnDriftBld”, not “del self.DrifDwnCtrl”, in destroy(). The thing is to delete the task pointer, not the method.

David

Wow! Thank you very Much! The memory does not spike now.

That’s a little scary though. If you leave just one Pointer behind, that’s a memory leak. Wow.

One question,

Since I’m using DirectStart, is there ever a need to use ShowBase?

[DangerOnTheRanger]

One question,

Since I’m using DirectStart, is there ever a need to use ShowBase?

No, there isn’t a need to use ShowBase if you use DirectStart. However, you’ll find a large percentage of the Panda community (myself included) prefer to use ShowBase instead of DirectStart, since ShowBase encourages you to pass your ShowBase instance around manually (as a parameter), instead of using the global “base” variable DirectStart creates. This leads to cleaner, more robust, and more readable code.

[drwr]

That’s a little scary though. If you leave just one Pointer behind, that’s a memory leak. Wow.

Yeah, it’s a nuisance. It’s really a problem with Python more than Panda, though, but that doesn’t make it less of a problem.

Still, it’s not just any pointer, it’s only circular pointers, and you learn to watch out for things like that. Storing a pointer to your own task is a circular pointer. Using self.accept() to call your own class method is a circular pointer. But most other things aren’t. So if you create any circular pointers, you have to delete them explicitly in your destroy() method; you don’t have to delete everything else, though.

And there are tools to help you track these things down. Python has the gc module, and things that it can’t delete are supports to be placed in gc.garbage, for you to inspect later and discover them. Also, Panda has the MemoryUsage class which can tell you what Panda objects are accumulating, which can also give you a clue (but it’s not usually as helpful as knowing the Python objects that are holding them, which gc provides).

Since I’m using DirectStart, is there ever a need to use ShowBase?

No. From time to time you will make calls into the global ShowBase instance, which is called base. For instance, you might want to reference base.win to do something with the default window. But you will never need to invoke the ShowBase class by name, and there’s no reason to import ShowBase.

David

[markjacksonguy]

Is there a way to count the number of instances in render by code? Something like, render.instCount(Object);

[drwr]

I don’t know what you mean. Instances of what? You can use render.analyze() to count up the number of Geoms, nodes, vertices, and whatnot.

David

[markjacksonguy]

I don’t know what you mean. Instances of what? You can use render.analyze() to count up the number of Geoms, nodes, vertices, and whatnot.

David

Like in the program, where you solved the memory leak. If I wanted to count the number of all those objects being created, within so many seconds, to get a total (for that amount of time passed), would that be possible with the P3D engine?

[drwr]

As I said, you can use the Python gc module to find some kinds of memory leaks. Use gc.collect(), and then check gc.garbage. If Python has detected any leaks, they will be listed in gc.garbage.

That doesn’t work for all kinds of memory leaks, though. I tracked yours down with the MemoryUsage class. This is a bit tricky to use, but the gist of it is: set “track-memory-usage 1” in your Config.prc file (this will slow things down a whole lot). Load up your app with python -i, then break into it with control-C. Type MemoryUsage.freeze() to remove all previously-allocated objects from the list and show only currently-allocated objects from this point on. Type run() to let the application run for a while. From time to time, break into it again with control-C and type MemoryUsage.showCurrentTypes() to list the currently-allocated objects since the call to freeze(), by type. Look for upward-increasing trends. In your case, I saw that PythonTask was increasing without limit, which told me that something that had a pointer to a task object was leaking. (MemoryUsage doesn’t track the Python objects, only the Panda objects, so it couldn’t tell me that it was a SpEfs_BldSplsh object that was holding the task object–I had to figure that out myself).

With a bit more effort, you can find out more details about the leaked Panda objects. For instance, I was able to look at each of the task objects and see that they were all named “DriftBldsp”, which is what pointed me at the SpEfs_BldSplsh class. To do this, you need to call MemoryUsage.getPointersOfType(). I leave the rest of this interface as an exercise; it’s documented in the API ref.

Also note that simply adding “track-memory-usage 1” to your Config.prc will allow Panda to report the distribution of allocated objects by type in PStats, when you drill into the system memory category. That may be an easier way to visually see leaks by object type.

David

[markjacksonguy]

Another Question -

If you create a pointer by starting a Task like:

self.ATask = taskMgr.add(self.Mytask, "Thetask");

If that task were to end with task.done, would that automatically delete the pointer?

[drwr]

No, it stops the task from running, but the task object you have created persists until you explicitly “del self.ATask”, or you simply reassign it like “self.ATask = None”.

It has to persist until you reassign it or del it, because that’s the way Python works. You have assigned self.ATask to a task object. That member will remain assigned to a task object until you assign it to something else.

David

[markjacksonguy]

I have another questions that’s off of this topic, but I didn’t want to create another New Topic just for this one question.

I was able to get collision checking to work while using a task to constantly check for collisions.

For some reason, I can’t get the same code to work while using a function that will check for collisions only once then end.

In the Task version of my code, I am using an Actor object with a collision ray. In the Function version, I am using a Model with a collision ray.

Does ground detection using a collsion ray only work for a collision node attached to an Actor or something?

Or does it only work in a continuing task and not a one time function call?

[drwr]

Nothing about collisions inherently requires either Actors or continuously running tasks, so I have no idea why your second function isn’t working.

But in general, don’t be afraid to start a new thread when you have a new topic of discussion.

David

[markjacksonguy]

Nothing about collisions inherently requires either Actors or continuously running tasks, so I have no idea why your second function isn’t working.

But in general, don’t be afraid to start a new thread when you have a new topic of discussion.

David

So this is fine then?

class BoxObject(DirectObject):
    def __init__(self):
        self.CretMeBspl = loader.loadModel("images/commons/objs/toybox");
        self.CretMeBspl.flattenStrong();
        self.CretMeBspl.setShaderOff();
            
        self.CretMeMask = self.CretMeBspl.attachNewNode(CollisionNode('GrndFallEff'));
        self.CretMeMask.node().addSolid(CollisionRay(0, 0, 0.1, 0, 0, -1));
        self.CretMeMask.node().setIntoCollideMask(0);
        self.CretMeMaskColH = CollisionHandlerQueue();
        #self.CretMeMask.show();

        self.CretMeMskTrav = CollisionTraverser("BXOnGrDTrav");
        base.cgTrav = self.CretMeMskTrav;
        self.CretMeMskTrav.addCollider(self.CretMeMask, self.CretMeMaskColH);
        #self.CretMeMskTrav.showCollisions(render);

    def PutToGrnd(self):
        base.cgTrav.traverse(render);

        if MainObj.Mode == "Freelance":
            MyList= [];
            for dex in range(self.CretMeMaskColH.getNumEntries()):
                CurDex = self.CretMeMaskColH.getEntry(dex);
                MyList.append(CurDex);

            if MyList != []:
                SFES = MyList[0];
                ReOrdera = (str(SFES.getIntoNodePath().getName()));
                if ReOrdera == "GroundC":
                    MyList.reverse();

            if MyList!= []:
                TT=0;
                TreatCol = "";
                for IdeX in MyList:
                    CCR = MyList[TT];
                    RTT = CCR.getSurfacePoint(render).getZ();
                    InName = (str(CCR.getIntoNodePath().getName()));

                    if InName == "GroundA" and TreatCol == "":
                        self.RELLPOINT = render.getRelativePoint(self.CretMeBspl, (0, 1, 0));
                        self.NorAng = CCR.getSurfaceNormal(render);
                        self.CretMeBspl.headsUp(self.RELLPOINT, self.NorAng);
                        TreatCol = "GroundA";
                        self.CretMeBspl.setZ(RTT+0.1);

                    elif InName == "GroundC" and TreatCol == "":
                        self.RELLPOINT = render.getRelativePoint(self.CretMeBspl, (0, 1, 0));
                        self.NorAng = CCR.getSurfaceNormal(render);
                        self.CretMeBspl.headsUp(self.RELLPOINT, self.NorAng);
                        TreatCol = "GroundC";
                        self.CretMeBspl.setZ(RTT+0.1);
                    TT+=1;

---

I figured out what was wrong.