Finding a memory leak

I’ve just discovered a memory leak in my project, and I’m not sure of what might be causing it.

To be specific, over multiple reloads of the same level I’m seeing increases in both system- and graphics- memory usage.

Looking at PStats, I’m seeing increases in these two collectors:

  • SystemMemory-->Heap-->Array-->VertexDataBuffer

  • GraphicsMemory-->context1-->Inactive-->texture

I’m also seeing a jump in “Vertex Data” and “Render States”–but not in “Vertices”, “Nodes”, “Geoms”, or “Transform States”!

Furthermore, the “Frame Time”, both standard and from the graphics card, both return to somewhere around their normal values after a load.

See this screenshot, showing some of the above after two loads:

It looks like I’m perhaps not releasing some piece of textured geometry–but I don’t see where that might be. Any suggestions?

This is the first time I have seen you create a Scripting Issue topic. You know mostly everything. Anyway, can you show the code?

Hahah, I know far from everything! There are very much those topics in which I am unlearned–procedural generation of geometry, for example. Nevertheless, I do thank you for the kind words. :slight_smile:

As to the code… it’s huge. This is a big project. And I really don’t know where the problem is coming from, I’m afraid.

I’ve tried to create a simple program that demonstrates the problem, but have thus far failed–in part, I daresay, because I’m currently only guessing at what that problem might be.

What are you trying to make in the first place?

It’s this work-in-progress game:

Hmm, there are ways to clear all those individual caches, but for this to occur there must be some references remaining to some models. This could be in your Python code (try gc.collect()), or in the ModelPool (if you haven’t disabled RAM caching; there’s a way to clear it), or in a scene graph that is still attached to something.

It is a little strange, though, that you say that it occurs upon a reload of the same level. Panda’s caching should make sure that the same resources aren’t duplicated unnecessarily. Especially texture memory shouldn’t increase unless you are explicitly disabling Panda’s RAM caching.

I’m pretty sure that I’m not intentionally disabling RAM caching, at least. Could something be disabling it that I might not be aware of? Something that does so as a side-effect, or a setting that’s been changed…?

Based on a quick test, placing the given line in my level-class’s “destroy” method (and with a print-statement to double-check that said method is being reached), this doesn’t seem to help, I’m afraid. :/

I presume that this is done via “ModelPool.garbageCollect()”? If so, then I’m afraid that this doesn’t seem to help, either. :/

This is the obvious thought, indeed–but I’m not sure of what it might be, especially as I’ve previously hunted memory leaks of this sort.

Indeed, that’s part of why this is particularly distressing for me right now: I thought that I had this issue fixed! :/

ModelPool.garbageCollect() will remove unreferenced models; in your case, you apparently have models that are still referenced, so, you would need ModelPool.releaseAllModels() to clear it.

However, I’d personally try to find out which objects are still alive and where. There may be tools out there that can help graph out remaining references on the Python end.

As for RenderState, you can get all the entries in that cache using RenderState.getStates(), and print those out to see which states it concerns, and use RenderState.clearCache() to clear it.

1 Like

That said, “releaseAllModels” does at least provide a means of testing whether the problem does lie with unreleased model-data.

And… It doesn’t seem to be. Adding “ModelPool.releaseAllModels()” at the end of my level-destruction function doesn’t seem to remove the problem. :/

In all fairness, I haven’t been carefully comparing the amounts of memory leaked–it’s possible that some of these measures have had some degree of effect, without solving the problem entire.

Hmm… Interestingly, printing out these states seems to suggest that my level-model isn’t being given up: I’m seeing listings for materials specific to the level in question, in a RenderState-printout located between the destruction of the level and its reloading… This may bear investigation.

[edit]
But if it’s the level-model, then I find myself wondering why the memory-usage doesn’t go up more. Perhaps it’s just the materials, somehow…?

Ah, but this calls for more careful investigation, perhaps.
[/edit]

I may look into that–thank you!

That is most probably the same issue I adressed on 20 july 2020, see:

The only way I have to able to circumvent that is to, after a simulation with a certain scenery, close and restart the complete rendering engine automatically.

Thank you for mentioning that!

Hmm… it is a concerning possibility.

Conversely however, if I’m reading this correctly you were seeing no increase in System Memory, while I am seeing an increase there. For now, I think that I’m going to focus on looking for something on my end that isn’t being properly freed up…

Got it! :smiley:

It looks like I wasn’t properly cleaning up one of my cell-culling structures, a grid that’s used instead of portal-culling in some places. And said grid held references to level-geometry…

How this slipped the previous memory-leak hunt I’m not sure–perhaps because I was focussing on a different level, one that didn’t use the grid to any great degree.

There’s still a small leak in one level, but I’m guessing that this is related to something specific to that level, and which should thus be fairly easy to track down…

Glad you found the source of the issue!

Also, maybe this goes without saying, but make sure you are using Panda3D 1.10.8, which fixed some memory leaks in Panda.

1 Like

I am indeed using 1.10.8. Nevertheless, thank you for noting it!

(I actually tried loading a few older versions while investigating this, thinking that it might have been engine-induced. However, I’m pretty sure that I loaded 1.10.8 again thereafter!)

In fact, I just double-checked via “pip3 show” to be on the safe side, which confirms a version number of 1.10.8. :slight_smile: