Performance impact on thousands of Nodes

That’s true–but if they’re present as nodes in the scene-graph, then they may well yet have an impact, I believe.

Of course, if you devise some method of dynamically removing and adding nodes in the scene-graph, then you might still get it to work. (This calls for some means of determining which blocks are visible, I imagine.) However, you may also find that doing so calls for going to the C++ side of development, for the sake of execution speed.

It is possible, but doing so dynamically may still prove to impact performance. Plus, you then have to be careful that you don’t lose access to the individual blocks in merging them.

I suspect that the approach that Epihaius posted above is likely a good one: don’t create a huge set of nodes, but instead create a smaller number of nodes whose models are updated dynamically.

Can you explain what do you mean by “create a smaller number of nodes whose models are updated dynamically”? So each node would consist a few models?

In short, each node would have one “model”, which would depict multiple cubes–likely quite a lot of them, I imagine. This model would then be dynamically updated to represent changes in the state of those cubes.

I think I understand something now, so 1 node, 1 model, but the model contains multiple blocks and would dynamically update the “model” when breaking/adding blocks. We achieve that by rigid body combiner correct?

1 Like

Hmm… You could try using combiners, but I’m not sure that you wouldn’t see significant lag whenever the contents of a given combiner have to be altered.

Again, I’m inclined to think that the method suggested by Epihaius, above, might be more likely to perform well.

Still, you can try using rigid body combiners–they do provide a simpler approach, and if it turns out that doing so works for your application, then well and good!

What exactly are those block models that you are loading? Are they simple cubes, or something much more complex (e.g. more detailed geometry, with chamfered or rounded corners, etc.)? If they’re just simple cubes, there’s no need to model them in an external application (e.g. Blender) and then load them into Panda. Instead, you can easily create them procedurally in Panda at runtime.

Below is a code sample that allows you to create, add and remove cubes (with different UV coordinates for each side and correct normals); they all become part of one and the same model node:

from panda3d.core import *
from direct.showbase.ShowBase import ShowBase
import array
import random

def create_cube():

    vertex_count = 0
    values = array.array("f", [])
    indices = array.array("I", [])
    # use an offset along the U-axis to give each side of the cube different
    # texture coordinates, such that each side shows a different part of a
    # texture applied to the cube
    u_offset = 0.

    for direction in (-1, 1):

        for i in range(3):

            normal = VBase3()
            normal[i] = direction

            for a, b in ((-1., -1.), (-1., 1.), (1., 1.), (1., -1.)):

                pos = Point3()
                pos[i] = direction
                pos[(i + direction) % 3] = a
                pos[(i + direction * 2) % 3] = b
                u, v = [pos[j] for j in range(3) if j != i]
                u *= (-1. if i == 1 else 1.) * direction
                uv = (max(0., u) / 6. + u_offset, max(0., v))

                values.extend(pos)
                values.extend(normal)
                values.extend(uv)

            u_offset += 1. / 6.
            vertex_count += 4

            indices.extend((vertex_count - 2, vertex_count - 3, vertex_count - 4))
            indices.extend((vertex_count - 4, vertex_count - 1, vertex_count - 2))

    vertex_format = GeomVertexFormat.get_v3n3t2()

    vertex_data = GeomVertexData("cube_data", vertex_format, Geom.UH_static)
    vertex_data.unclean_set_num_rows(vertex_count)
    data_array = vertex_data.modify_array(0)
    memview = memoryview(data_array).cast("B").cast("f")
    memview[:] = values

    tris_prim = GeomTriangles(Geom.UH_static)
    tris_prim.set_index_type(Geom.NT_uint32)
    tris_array = tris_prim.modify_vertices()
    tris_array.unclean_set_num_rows(len(indices))
    memview = memoryview(tris_array).cast("B").cast("I")
    memview[:] = indices

    geom = Geom(vertex_data)
    geom.add_primitive(tris_prim)
    node = GeomNode("cube")
    node.add_geom(geom)

    return node


class MyApp(ShowBase):

    def __init__(self):

        ShowBase.__init__(self)

        # set up a light source
        p_light = PointLight("point_light")
        p_light.set_color((1., 1., 1., 1.))
        self.light = self.camera.attach_new_node(p_light)
        self.light.set_pos(5., -100., 7.)
        self.render.set_light(self.light)

        # create an initially empty model parented to the scene root
        vertex_format = GeomVertexFormat.get_v3n3t2()
        vertex_data = GeomVertexData("model_data", vertex_format, Geom.UH_dynamic)
        tris_prim = GeomTriangles(Geom.UH_dynamic)
        # allow adding more than 65536 vertices
        tris_prim.set_index_type(Geom.NT_uint32)
        model_geom = Geom(vertex_data)
        model_geom.add_primitive(tris_prim)
        model_node = GeomNode("model")
        model_node.add_geom(model_geom)
        self.model = self.render.attach_new_node(model_node)

        # the number of vertices in each cube
        self.cube_vert_count = 24  # 6 sides, 8 vertices per side

        # the number of vertex indices in the GeomTriangles primitive of each cube
        self.cube_prim_vert_count = 36  # 6 sides, 6 index rows per side

        # the number of float values used for the data of a single vertex
        self.data_stride = 8  # 3 coordinates + 3 normal components + 2 UVs

        # define the possible positions for the cubes
        # (they form a pyramid shape in this example)
        self.positions = positions = []
        for i in range(-2, 3):
            pos = (i * 2, -4, 0)
            positions.append(pos)
        for i in range(-2, 3):
            pos = (i * 2, 4, 0)
            positions.append(pos)
        for i in range(-1, 2):
            pos = (-4, i * 2, 0)
            positions.append(pos)
        for i in range(-1, 2):
            pos = (4, i * 2, 0)
            positions.append(pos)
        for i in range(-1, 2):
            pos = (i * 2, -2, 2)
            positions.append(pos)
        for i in range(-1, 2):
            pos = (i * 2, 2, 2)
            positions.append(pos)
        positions.append((-2, 0, 2))
        positions.append((2, 0, 2))
        positions.append((0, 0, 4))
        # keep track of which positions are currently occupied
        self.free_positions = list(range(len(positions)))
        self.used_positions = []

        # the cubes that have been separated and are falling down
        self.falling_cubes = []

        # start a task that makes separated cubes fall down
        self.task_mgr.add(self.__drop_cubes, "drop_cubes")

        # enable randomly adding and removing cubes at runtime
        self.accept("+", self.__add_random_cube)
        self.accept("-", self.__remove_random_cube)

    def add_cube(self, pos):

        model_geom = self.model.node().modify_geom(0)
        model_array = model_geom.modify_vertex_data().modify_array(0)
        model_vert_count = model_array.get_num_rows()
        model_array.set_num_rows(model_vert_count + self.cube_vert_count)
        model_view = memoryview(model_array).cast("B").cast("f")
        prim_array = model_geom.modify_primitive(0).modify_vertices()
        model_prim_vert_count = prim_array.get_num_rows()
        prim_array.set_num_rows(model_prim_vert_count + self.cube_prim_vert_count)
        model_prim_view = memoryview(prim_array).cast("B").cast("I")

        cube_node = create_cube()
        cube_geom = cube_node.modify_geom(0)
        vertex_data = cube_geom.modify_vertex_data()
        mat = Mat4.translate_mat(*pos)
        vertex_data.transform_vertices(mat)
        vert_array_data = vertex_data.get_array(0)
        vert_array_view = memoryview(vert_array_data).cast("B").cast("f")
        model_view[model_vert_count * self.data_stride:] = vert_array_view
        prim = cube_geom.modify_primitive(0)
        offset = self.cube_vert_count * model_prim_vert_count // self.cube_prim_vert_count
        prim.offset_vertices(offset, 0, self.cube_prim_vert_count)
        cube_prim_array = prim.get_vertices()
        cube_prim_view = memoryview(cube_prim_array).cast("B").cast("I")
        model_prim_view[model_prim_vert_count:] = cube_prim_view

    def prepare_separated_cube(self):

        vertex_format = GeomVertexFormat.get_v3n3t2()
        vertex_data = GeomVertexData("cube_data", vertex_format, Geom.UH_static)
        cube_array = vertex_data.modify_array(0)
        cube_array.unclean_set_num_rows(self.cube_vert_count)
        cube_view = memoryview(cube_array).cast("B").cast("f")
        tris_prim = GeomTriangles(Geom.UH_static)
        tris_prim.set_index_type(Geom.NT_uint32)
        cube_geom = Geom(vertex_data)
        cube_geom.add_primitive(tris_prim)
        cube_node = GeomNode("cube")
        cube_node.add_geom(cube_geom)
        separated_cube = self.render.attach_new_node(cube_node)

        # give the separated cube a red color to easily identify it as a
        # falling cube
        separated_cube.set_color(1., 0., 0.)

        cube_prim_array = tris_prim.modify_vertices()
        # the call to `GeomPrimitive.unclean_set_num_rows` fills the primitive
        # with random integers, which will cause an AssertionError when an
        # integer bigger than the number of vertex data rows is encountered
        # while adding the primitive to the geometry, so only call this method
        # after the primitive, its geom and its node have been added to the
        # scenegraph
        cube_prim_array.unclean_set_num_rows(self.cube_prim_vert_count)
        cube_prim_view = memoryview(cube_prim_array).cast("B").cast("I")

        # the separated cube will start to fall
        self.falling_cubes.append(separated_cube)

        # return the created memoryviews for further manipulation
        return cube_view, cube_prim_view

    def remove_cube(self, index, separate=True):

        model_geom = self.model.node().modify_geom(0)
        model_array = model_geom.modify_vertex_data().modify_array(0)
        model_vert_count = model_array.get_num_rows()
        model_view = memoryview(model_array).cast("B").cast("f")
        cube_data_size = self.cube_vert_count * self.data_stride
        start = index * cube_data_size
        end = start + cube_data_size

        if separate:
            cube_view, cube_prim_view = self.prepare_separated_cube()
            cube_view[:] = model_view[start:end]

        model_view[start:-cube_data_size] = model_view[end:]
        model_array.set_num_rows(model_vert_count - self.cube_vert_count)

        prim = model_geom.modify_primitive(0)
        prim_array = prim.modify_vertices()
        start = index * self.cube_prim_vert_count
        end = start + self.cube_prim_vert_count
        model_prim_vert_count = prim_array.get_num_rows()
        prim.offset_vertices(-self.cube_vert_count, end, model_prim_vert_count)
        model_prim_view = memoryview(prim_array).cast("B").cast("I")

        if separate:
            cube_prim_view[:] = model_prim_view[:self.cube_prim_vert_count]

        model_prim_view[start:-self.cube_prim_vert_count] = model_prim_view[end:]
        prim_array.set_num_rows(model_prim_vert_count - self.cube_prim_vert_count)

    def __add_random_cube(self):

        if not self.free_positions:
            return

        index = random.choice(self.free_positions)
        pos = self.positions[index]
        self.free_positions.remove(index)
        self.used_positions.append(index)
        self.add_cube(pos)

    def __remove_random_cube(self):

        if not self.used_positions:
            return

        index = random.randint(0, len(self.used_positions) - 1)
        self.remove_cube(index)
        pos_index = self.used_positions[index]
        self.used_positions.remove(pos_index)
        self.free_positions.append(pos_index)

    def __drop_cubes(self, task):

        for cube in self.falling_cubes[:]:
                
            z = cube.get_z()

            if z < -10.:
                self.falling_cubes.remove(cube)
                cube.detach_node()
            else:
                cube.set_z(z - .1)

        return task.cont


app = MyApp()
app.run()

When you run the code, press + on the keyboard to add a cube and - to remove a random cube.

Each cube will be placed at a different position. There is a limited number of available positions, which form a pyramid-like shape, so it is easier to see which cubes are added or removed.
The geometry of each cube is associated with a specific position index, and the code keeps track of which positions are used and which are still free.
The relationship between a cube and its position is determined like this:
when a cube is created, its geometry is added to the model node and the index of its position is added to the list of used positions.
So when a cube is removed, it is done by index; this index is used to locate its vertex data in the model geometry, and it is also used to find the position index in the list of used positions.

As for the generation and manipulation of the geometry itself, memoryviews are used, which is more efficient than the use of GeomVertexReaders/Writers, flattenStrong operations or rigid body combiners (which just call flattenStrong under the hood anyway).

It is indeed not simple, but the complexity of what you’re trying to do should not be underestimated either, so I can strongly recommend that you work as much as possible with the low-level methods like I’ve shown in the sample.

Even if the models that you use are quite complex, it is still possible to read their geometry data and copy it into a single model node, as I’ve done with the simple cube in the sample.

If you need more help, just ask.

3 Likes

Thank you very much. For the most part, it is cubes, but with different textures. But with some exception, though. Also, for my curiosity, what is the difference between your approach and the flattenStrong operation, that made your’s quicker(if I understood correctly)?

Can you point towards a tutorial/manual for memoryview?


1 Like

I have just tried it and it does work, child nodes can be removed even if the parent is flattened (node is also removed from render successfully).

In that case, and presuming that you used “flattenStrong” (as opposed to “flattenMedium” or “flattenLight”, that is), I’d recommend checking that the nodes were actually flattened properly: as you may recall, “flattenStrong” won’t operate past a “ModelNode”, and thus may leave elements uncombined.

Ah, that might require adding those textures to one single “texture atlas”, since one model can only have a single texture (well actually it can have up to something like 8 eight different textures if you use multitexturing, but I’m not sure if multitexturing would help in this case).

Perhaps it’s not that much faster than flattenStrong, but looking at the source code, I see that a SceneGraphReducer is responsible for the entire process and the methods used seem to be quite complex.

So what I basically mean by “more efficient” is, that it’s likely better to start with geometry that is already in a state suitable to be manipulated (no unnecessary ModelNodes, the primitive index type already set to accommodate many vertices, no specific TransformState or RenderState), instead of a model created with a modelling program which may contain those things, which would then have to be removed/combined/converted and so on.

At the very least, with the code I posted you have total control over what happens with the geometry, while it’s not always clear exactly what flattenStrong does or does not do (for example, it won’t remove duplicate vertices, even though they have the exact same position, normal, UVs, etc.).

Don’t get me wrong, flattenStrong certainly is very useful, but in this particular case I would go with low-level methods, as it leads to code that clearly details every step of the process, and it also gives more control. (At least, that’s my personal opinion :sweat_smile:.)

1 Like

I also forgot, is it possible to separate them into single models again? Because some blocks will need to be able to fall down.

Sorry to interfere, but if we are talking about the style of minecraft physics of the blocks in this game does not exist. This is just a vertex-level animation, so you don’t need to divide the vertices into separate models to manage them, you can recalculate them directly in memory.

I’m not sure if I understood your statement correctly. I think the falling block animation would be more efficient if it is it’s own separate model, instead of updating the entire model containing multiple blocks multiple times for whatever frames it takes the block to fall down.

Hi,

If you’re intent on using the flattenStrong method and things like that, you could have 2 copies: one would be the collection of blocks in your “chunk” when they aren’t flattened, the other would be your flattened model. When nothing is being modified, show your flattened model and hide your collection of blocks. When something is being modified, swap to showing the collection of single blocks and after the modification is done, destroy your old flattened model and create a new one by re-flattening your collection of blocks in their updated state. Then after the modification is done, hide your block-collection again and show the flattened-model. Swapping back and forth like that could work for things like simulating animations…but obviously it might make things sluggish if you’re dealing with thousands of blocks.

This is just an idea, but the best way to implement a minecraft-like world with many blocks would be to deal directly with low-level geometry as others have suggested on this thread.

1 Like

Hi,

I’m a bit curious about whether this increase in efficiency involves significant speed increases? Does using memoryviews lead to faster geometry-manipulation as compared to using GeomVertexReaders/Writers? If so, is it by a significant margin?

Yes, it is very much possible. In fact, this might be another good reason to not use flattenStrong, as it isn’t obvious what indices the vertices will end up having within the flattened geometry (might depend on the order of the children of the flattened node, but I haven’t checked).

Anyway, the solution is to associate the start of the block’s vertex data with an index in some list (in my code sample it’s the index of its position in a list of predetermined positions). Then you copy that vertex data to a new Geom (and GeomPrimitive) in a new GeomNode.

The code sample has been edited to include a prepare_separated_cube method that sets up these new structures, which are then filled with the separated cube’s vertex data. There’s also a __drop_cubes task now, which lowers the z-value of the separated cube(s) every frame by a constant amount until a certain value is reached, upon which the node is detached from the scenegraph.
A bit simplistic, but it’s just an example.

The separated cubes have been given a red color to differentiate them from the others.

That question is best answered in this blog post, I guess :slight_smile: .
It does take a little getting used to (memoryviews are quite rigid, in that you can’t change their size, so you need to make sure that the data you create a view of already has the correct size), but I do find it worth the effort to learn how to use them.

And using GeomVertexArrayDataHandle and its methods is an interface that is being discouraged in favor of memoryview usage as well.
Let me know if you need any help with that :wink: .

Well, if the advantages of using them far outweigh those of using whatever has been written in the manual and the api as well, I think those into procedural geometry creation and manipulation would benefit from a tutorial that covers:

  • The best way to create geometry procedurally using memoryviews.
  • The best way to extend the existing geometry of a model using memoryviews.
  • The best way to delete the existing geometry of a model using memoryviews.

If a clear and concise tutorial were written by someone competent and experienced with memoryviews that covers those 3 things, then I think we would all benefit from it. Not to put all that on your shoulders of course… :sweat_smile:

My attempt at doing so can be found here :wink: :

instead of making use of GeomVertexWriters, my version uses memoryviews to access vertex data.
To make things more interesting, I’ve made it possible to add and delete any of the cube’s sides at runtime by direct vertex data manipulation.

Perhaps it could still be improved; when I have some time I should have a look at it again.