Trying to optimize scene with lots of nodes

Hi! I’m creating something that is close to a city simulator and I’m trying to solve a performance issue, probably related to the number of nodes on the scene. I have read a lot of the user manual and it helped me to get to where I am now, but I’m still not sure on how to get to what I need.

I’m importing roads from a database and generating meshes programmatically for them and then adding each one to the scene graph. The meshes are very low poly, but there are some thousands of them (think all roads in a big city), so there is a performance impact on rendering them. I have included a screenshot just so you can have a better idea of what I’m doing.

I was able to get some serious improvements by calling flatten_strong() on the parent node of the roads. If I could just leave at that, it would be perfect already, but I need to be able to demolish some roads and create others, so I need to access individual nodes on this graph, delete some of them and create some new ones. When I use flatten, I can’t access individual nodes anymore.

I was wondering if there is some feature of Panda3d that can help me there. I unfortunately could never find examples of someone doing something similar on Panda3d.

One way that I’m thinking about is to separate the map in chunks and flatten each chunk separately, but then, if I need to change something, I just destroy the changed chunk entirely and then recreate it. Chunks would need to be small enough that it would not take too long to regenerate, but big enough that it would still significantly lower the number of nodes. It should work, but I feel that the Panda3d API may offer something more appropriate for this situation (?), that’s why I’m posting this here.

Also, is there a way to do something like flatten, but keeping a copy of the original graph in memory, in a way that later I could “deflatten” it?

Thanks

Hmm… You could try the “Rigid Body Combiner”.

Do note the point made on that manual-page about the call to the combiner’s “collect” method being costly. If that proves problematic, perhaps you could experiment with dividing your city into chunks, each under its own combiner, to limit the cost of a change.

I kind of feel bad that I said

and then you just linked something that seems to be described exactly as the solution to my problem, right from the manual. Thank you very much, I’ll take a look and give feedback after trying it out.

Don’t worry about it–that manual is pretty big! (I’m pretty sure that I haven’t read all of it by any means.)

Good luck with the Rigid Body Combiner–I hope that it works out for you. :slight_smile:

Since you are already generating your geometry procedurally, I can show you how to add and remove specific parts of a single model used for all of the roads.

The following code example creates a grid-like road setup (just simple rectangles) and allows you to randomly add and remove roads using the + and - keys, respectively:

from panda3d.core import *
from direct.showbase.ShowBase import ShowBase
import array
import random


class MyApp(ShowBase):

    def __init__(self):

        ShowBase.__init__(self)

        # set up a light source
        p_light = PointLight("point_light")
        p_light.set_color((1., 1., 1., 1.))
        self.light = self.camera.attach_new_node(p_light)
        self.light.set_pos(5., -10., 7.)
        self.render.set_light(self.light)

        # set up the road geometry
        vertex_format = GeomVertexFormat.get_v3n3t2()  # position, normal and uvs
        vertex_data = GeomVertexData("vertex_data", vertex_format, Geom.UH_static)
        tris_prim = GeomTriangles(Geom.UH_static)
        tris_prim.set_index_type(Geom.NT_uint32)
        geom = Geom(vertex_data)
        geom.add_primitive(tris_prim)
        geom_node = GeomNode("roads")
        geom_node.add_geom(geom)
        self.roads = self.render.attach_new_node(geom_node)

        # define the road dimensions
        length = 5.
        width = 2.

        self.roads.set_pos(-4. * length, 70., -10.)
        self.roads.set_p(30.)
#        tex = self.loader.load_texture("my_texture.png")
#        self.roads.set_texture(tex)

        # initialize road data
        self.road_count = 0
        self.road_ids = []

        # generate the roads
        for i in range(10):
            for j in range(10):
                pos1 = Point3((i - 1) * length, (j - 1) * length, 0.)
                pos2 = Point3((i - 1) * length, j * length, 0.)
                self.add_road(pos1, pos2, width)
                pos1 = Point3((i - 1) * length, (j - 1) * length, 0.)
                pos2 = Point3(i * length, (j - 1) * length, 0.)
                self.add_road(pos1, pos2, width)

        # enable randomly adding and removing roads at runtime
        self.accept("+", self.__add_random_road)
        self.accept("-", self.__remove_random_road)

    def add_road(self, pos1, pos2, width):

        values = array.array("f", [])
        tri_data = array.array("I", [])
        length_vec = pos2 - pos1
        normal = Vec3(0., 0., 1.)
        width_vec = length_vec.normalized().cross(normal) * width * .5
        points = (pos1 - width_vec, pos1 + width_vec,
                  pos2 - width_vec, pos2 + width_vec)
        uvs = ((0., 0.), (1., 0.), (0., 1.), (1., 1.))

        for point, uv in zip(points, uvs):
            values.extend(point)    # coordinates
            values.extend(normal)   # normal
            values.extend(uv)       # texture coordinates

        geom = self.roads.node().modify_geom(0)
        vertex_data = geom.modify_vertex_data()
        old_vert_count = vertex_data.get_num_rows()
        new_vert_count = old_vert_count + 4
        vertex_data.set_num_rows(new_vert_count)
        data_array = vertex_data.modify_array(0)
        memview = memoryview(data_array).cast("B").cast("f")
        # start the memoryview slice at the previous size, which equals
        # the old number of vertex rows multiplied by the number of float
        # values in each row (8: 3 for position, 3 for the normal and
        # another 2 for the texture coordinates)
        memview[old_vert_count * 8:] = values

        tris_prim = geom.modify_primitive(0)
        tris_array = tris_prim.modify_vertices()
        old_count = tris_array.get_num_rows()
        new_count = old_count + 6  # 2 triangles make up the road rectangle
        tris_array.set_num_rows(new_count)
        tri_data.extend((old_vert_count, old_vert_count + 1, old_vert_count + 3))
        tri_data.extend((old_vert_count, old_vert_count + 3, old_vert_count + 2))
        memview = memoryview(tris_array).cast("B").cast("I")
        memview[old_count:] = tri_data

        self.road_count += 1
        self.road_ids.append("road_{:d}".format(self.road_count))

    def remove_road(self, road_id):

        index = self.road_ids.index(road_id)
        self.road_ids.remove(road_id)

        geom = self.roads.node().modify_geom(0)
        vertex_data = geom.modify_vertex_data()
        start = index * 32  # 4 vertex rows per road; 8 float values per row
        end = start + 32
        old_vert_count = vertex_data.get_num_rows()
        new_vert_count = old_vert_count - 4
        data_array = vertex_data.modify_array(0)
        memview = memoryview(data_array).cast("B").cast("f")
        memview[start:-32] = memview[end:]
        vertex_data.set_num_rows(new_vert_count)

        tris_prim = geom.modify_primitive(0)
        start = index * 6  # 6 index rows per road; 1 int value per row
        end = start + 6
        old_count = tris_prim.get_num_vertices()
        new_count = old_count - 6
        tris_prim.offset_vertices(-4, end, old_count)
        tris_array = tris_prim.modify_vertices()
        memview = memoryview(tris_array).cast("B").cast("I")
        memview[start:-6] = memview[end:]
        tris_array.set_num_rows(new_count)

    def __add_random_road(self):

        values = array.array("f", [])
        r = random.random
        pos1 = Point3(r() * 50., r() * 50., r() * 2.)
        pos2 = Point3(r() * 50., r() * 50., r() * 2.)
        self.add_road(pos1, pos2, 2.)

    def __remove_random_road(self):

        road_id = self.road_ids[random.randint(0, len(self.road_ids) - 1)]
        self.remove_road(road_id)


app = MyApp()
app.run()

Instead of creating multiple nodes, a unique ID is generated for each road and stored in a list. Based on the index of the ID in the list, the start and end of the associated vertex data is calculated and this can be used to remove the geometry representing a particular road from the single model that includes all of the created roads.

Adding and removing geometry is done using memoryviews, so it should be quite efficient.
It’s a rather advanced way of dealing with geometry, but if you’re really into procedural geometry creation, it might be worth looking into.

2 Likes

Thank you! This looks great for this use case. It looks like the rigid body combiner will be the best option for adding cars and other moving things, but for the roads, I really liked this approach. But there are some questions.

In your code, I don’t know if I got it right, but it looks like every time you remove a road, you have to make a big copy operation to move the rest of the array so that it fills the erased position. Isn’t this too expensive when this array is huge?

And also, when you get the data_array with vertex_data.get_array(), is the array copied in memory? I would guess not, initially, but when you call

vertex_data.set_num_rows(new_vert_count)

you would be removing a part of the array, but you can still access its last positions with old_view[end:]. This is confusing to me.

Argh, you’re right; I used code from another project where I needed to copy part of one memoryview into a different memoryview, while I should just use one and the same memoryview in this case. Sorry about that!

The code has been edited to use a single memoryview for removing vertex data (and the same for removing vertex indices from the GeomTriangles).
Thanks for pointing out the mistake!

Hopefully the code will serve you well :slight_smile: .

Even with that copy operation that i mentioned involved, it seems like your method will work nicely. I made some tests, measuring the time it takes to remove a road with n roads and I got something close to:

road segments    time(s)
          100    0.0002
        10000    0.008
      1000000    0.25

It’s very acceptable to start with and maybe, in the future, I can manage vertices in a way that requires less and smaller copy operations.

Thank you very much for your help, both @Epihaius and @Thaumaturge. I would not be able to find these solutions by myself.

It’s my pleasure for my part; I’m glad that you seem to have found tools that work for you. :slight_smile: