Experiment comparing memory usage when reusing procedurally-generated models

[KennedyRichard]

As most may know, loaded .egg files are cached in the model pool (panda3d.core.ModelPool), which means subsequent loads reuse the same geometry, thus saving memory. However, the same doesn’t happen for generated geometry (either using the Geom or EggData approaches).

At first I was worried that this fact would discourage people from using procedurally-generated models, fearing there was no way for such geometry to be cached. However, after some discussion on Discord, it was pointed out to me that as long as I reuse the Geom, there would be no significant extra memory usage for subsequent replicas.

That was encouraging and I decided to do a small experiment in order to observe this.

This repository represents such experiment, including discussion and results/conclusions and explains how to run each of the included scenarios: GitHub - KennedyRichard/procgenmemtest: Small Panda3D experiment comparing memory usage when reusing procedurally-generated models

The code is in the public domain and is all plain Python code that relies solely on Panda3D and the standard library to generate UV sphere geometry for the different scenarios in the experiment.

Scenarios A, A1, B and B1 are for testing memory usage of different methods (manually, by checking your own task manager, etc.), and scenario X, depicted in the GIF at the top of this post, is just a playground where users can define and display a UV sphere with given arguments (optionally saving it as an .egg file as well).

This discussions was created just to leave a permanent record for people interested in this kind of thing and to allow people to share feedback/observations, if desired. The code might be useful to some as well. The geometry generation in particular was made with the goal of striking a good balance between memory efficiency, avoiding needless calculations and good Pythonic design/code.

Anyway, whenever a good discussion pops up on Discord, I feel compelled to sooner or later record it in a more perennial format, like this forum, so that it is not lost to the ephemerality of the Discord timeline. Discord is great for chatting and real-time support, but horrible for organizing/indexing knowledge and supporting long deep discussions (although that doesn’t stop us from engaging in long deep discussions there as well from time to time ).

[serega-kkz]

This is useful until you want to make changes to the VertexPool of a specific model or something else, for example: change the material If changes have occurred in the VertexPool, a deep copy of all the vertices will be created automatically.

As a rule, the game does not need a monotonous geometry, for the purity of the experiment, you can try using random materials, textures, and so on.

[KennedyRichard]

Thank you for the insights. As I said in the repo’s README, I’m not much experienced with Panda3D, so I’m always eager for feedback, corrections and additional context/insights like the one your provide.

I also said in the repo that it was meant as a one-off experiment, not something that I intend to maintain indefinitely, but of course such important insights are not to be overlooked. I’ll find some time to include this piece of information in the README as well, and better yet, see if I can include a scenario demonstrating that.

Now, that I think of it, I should’ve also included my intended use for this kind of feature (the reuse of procgen-ed models), so that its need and details was more obvious (I’ll find time to include that in the README of the repo, too).

This intended use is more or less something like this: think about a game like The Sims, but one where users can customize furniture using sliders and other widgets, etc. If one of such objects is a chair, for instance, it makes sense for the user to customize the object and then get a few instances of it, after all, we usually have multiple chairs around a table. That’s why it is important not only to be able to generate the models, but also be able to duplicate them in a memory efficient manner.

For this particular intended usage, I think not touching the vertex pool anymore is not a problem.

In the scenario where the user ever wants to change the chair’s geometry or materials, I don’t mind that the new model is a different object in memory. As long as I can duplicate this new geometry in a memory-efficient way, it is okay for this final new model to represent an entirely new deep copy. Someone (Dylan) on Discord suggested me to store the generated Geoms in a dict, and I was even considering using the different arguments used to generate the model (including not only the geometry, but materials, etc.) as the keys, so that even reused combination of arguments would get access to the cached generated model.

As long as whichever generated geometry can be reused in a memory-efficient way, I’m satisfied. And we don’t even need to go as far as considering a game like The Sims. Even more simple games or games from other genres sometimes benefit from some form of customization of objects that have to be generated on the spot. For instance, allowing the user to customize their base of operations/headquarters, weapons, etc.

Again, your comment is still very relevant and not only something that I didn’t know, but something that could easily come back to bite one in the future, hence why I’ll take some time in the future to update the repo, if not with actual code demonstrating it, at least with mention of this insight you provided, so thank you again. Further feedback is always welcome, so please don’t hesitate to share more if something else comes to mind.

I did read most of the manual section on the GeomVertexData approach to generating geometry, but as it was not something that I intended to do for now, or at least not my intended usage as explained above, I didn’t read the subsections on rewriting/editing the geometry.

[KennedyRichard]

To everyone reading this, here’s a bit more info/context that I didn’t talk about in the repo. Stuff that I will probably not include there because it is tangential to the repo’s/experiment’s purpose, but still interesting nonetheless.

Funnily, I didn’t mean to use UV spheres originally, but instead wanted to reproduce the “lathe/lathing” feature present in some modelling software, where a 2D cross-section of a solid is rotated to generate the full solid (usually to produce cup, bowl or vase-like objects, but not necessarily limited to that).

Here’s an example of this feature in action in just a few seconds of this Youtube short: https://www.youtube.com/shorts/rlNcoG_kK3I

However, before I could write the required math to “fold” the geometry inwards by creating the respective inner faces (also for the sake of the solid to have some thickness to it), I realized that if the cross-section was a semicircle and the resulting shape was closed at the top, I’d get a UV sphere.

I still intend to finish the lathing generation utility. It will probably not be included in this repo, but perhaps as a gist on GitHub, I’m not sure. The reason I want to do it is because the math required is very interesting and fun as well, and the feature itself might be useful in my projects (perhaps for custom vases the player can break, lol).

The UV sphere generation code, both employing the Geom (modelgen/uvspheregeom.py) and the EggData (modelgen/uvsphereegg.py) approaches, was very fun to me. Each of the modules has more or less 300 lines, evenly split into blank space (for increased readability), comments (because I love commenting/discussing stuff) and actual code.

Both approaches do use 02 functions from another local module, but the functions are relatively small and were kept in their own module for the sake of organization and because they are useful in other contexts as well (one of the functions just generates 2d points forming a circle and the other is just another version of it that allow more control over the generated points). For instance, I also use these functions to generate circular paths for the camera to spin around an object or many objects and also to generate a large circular positional distribution of spots wherein to position the multiple copies generated in the stress scenarios.

I also didn’t mean to write my own generative code actually, but the stuff I found online employed needless stuff, like creating custom classes. In this case I deemed such stuff needless, as both the Geom and EggData approaches already offer all API needed, turning the approaches that make use of custom classes to access the Geom or EggData interfaces into needless cognitive burden on the user (again, IMHO).

In my own solution, I tried to strike a balance between memory efficiency, avoiding needless calculations and achieving Pythonic code/design. I also got to use collections.deque, one of my favorites data structures due to its particularities (used/use it a lot for 2D animation in pygame-ce projects as well).