Faster geomVertexDataArray requires numpy?

Scripting Issues
1

I’m trying to port one of my Dark Basic Professional projects to Panda, and I’m running into some speed issues. I need to create quickly various geometries loaded from text format. The process I have works nicely, but is a bit too slow. Searching the forums, I’ve come across a couple of threads that seem to point toward some faster approaches for using geomVertexData to create geometry in-game.

[url][SOLVED]Help with asynchronous creation of vertex data bufrs]

[url]GeomVertexDataArray from Numpy array]

format=GeomVertexFormat.getV3n3t2() #vertex 3 normal 3 texture 2 (v3n3t2)
      vdata=GeomVertexData('geometry', format, Geom.UHStatic)
      tri=GeomTriangles(Geom.UHStatic)

       geomVertexNumpyData = np.empty((vertListLength,8),dtype=np.float32)
       #add some data to the array and tri

       data = geomVertexNumpyData.tostring()
       geomVertexDataArray.modifyArray(0).modifyHandle().setData(data)

      geomChunk=Geom(vdata)
      geomChunk.addPrimitive(tri)

There seems to be a better way to do what I need to do, then, based on these threads and a couple of other leads. Unfortunately, every indication I can find on the forums suggests that you need to acquire properly formatted array data from numpy, to do this in Python. I would prefer to avoid any numpy dependency in my code, if I can avoid it.

So I’ve been trying to get the same result using various combinations of ctypes, struct, and Python’s own array format. Most of these tests give me corrupted geometry. A couple of ctypes efforts gave me a correct geometry, but they were slower than using pure Python.

So I am asking here, is there a way to get out of plain Python data which is equivalent to what numpy would produce, to allow me to use the approach noted above? :question: I’ve come across several undocumented tricks in Panda since I began using it, so I’m hoping this is a case where there’s a secret method. If there is, this seems to be a problem I can’t tackle on my own. Need a little help. Someone, haaaaalp! :laughing:

Attaching my testing code. Move the little box around using the arrow keys. When he hits the edge of the room, a room change occurs, involving a geometry swap. That change is too slow, resulting in a noticeable pause. I would prefer to avoid writing out the geometries to egg or bam or another format, if possible.

Image1.jpg805×627 28.7 KB


adventure_slow.zip (12.8 KB)

2

Check out the latest blog post and make sure you’re using a recent development build. I’ve checked in this feature for the same purpose that you have; to be able to quickly generate or load new geometry data from disk without the overhead of the geometry writers. It results in very significant speedups.

The idea is to create an array of floats, containing the data in the right order, and then load it via copyDataFrom. It should be no different whether you use array.array or a numpy array (both support the buffer protocol), there is no numpy requirement.

modifyHandle().copyDataFrom(array)

This avoids the need for copying to string first and then back to array, which is an unnecessarily expensive operation. (You can also use copySubdataFrom to copy a specific range or into a specific range.) You might want to make sure that the Panda buffer is the right size first, though.

3

Thanks, rdb. I read that blog entry twice, yesterday, and I downloaded but have not yet installed the latest development build. Your link in the blog leads to documentation of Python’s memoryview class, and every example I could find for using that suggested that I needed Python 3 to use it. Having encountered contradictory and/or incomplete information, I grew confused and began exploring other options. :laughing: I will go ahead and test it.

I do have some questions about this overall process. The array data which goes into the buffer needs to be formatted a specific way, but I can’t quite sort out how the overall process needs to work. I was only able to get this process to work (slowly) when I still sent data on a per-polygon level to a poly/tri compiling routine. It only worked when I submitted nothing but the vertices, in a flattened array, all in single group. But the examples I see online (the two above plus two more, links below) suggest that all of the vertexdata for my format should be submitted in one array. If I need to compile vertices, normals, texcoords, and vertex colors into a single flattened array, what form does this need to take? Should this be compiled on a per-polygon basis, or all at once for the entire geometry? Either way, how should the tris be compiled? My experiments didn’t really clarify any of this for me, and these methods are both undocumented and not fully clarified by any existing example code. I need a roadmap to understand what I’m doing here, because once the idea of a “buffer” is invoked, I’m… just a little bit lost. :laughing:

Here are the additional leads I found for the process I note in the top post:

http://web.mit.edu/~pbatt/www/scenesim-doc/_modules/scenesim/objects/pso.html

http://nullege.com/codes/show/src%40m%40e%40meshtool-0.3%40meshtool%40filters%40panda_filters%40pandacore.py/220/panda3d.core.GeomVertexArrayData.modifyHandle/python

4

Well, this is interesting. I get huge geometry-building and rendering speed gains by simply changing the loop structure when I create my geometries.

This is slow:

    def build(self):
        """Create the geometry from the submitted arrays"""
        verts = self.object.verts
        polys = self.object.polys
        if self.object.geomnode.getNumGeoms():
            self.object.geomnode.removeAllGeoms()
        self.object.color_lookup = []    
        for poly_index in range(len(polys)):
            p = polys[poly_index]
            uvs = self.object.uvs[poly_index]       
            norm = self.object.vnorms[poly_index]
            face = self.addPolygon([verts[i] for i in p], uvs, norm)
            self.object.geomnode.addGeom(face)

def addPolygonGOOD(self, face, uvs, norm):
        """        
        """        
        format=GeomVertexFormat.getV3n3cpt2()        
        vdata=GeomVertexData('ngon', format, Geom.UHStatic)

        vertex = GeomVertexWriter(vdata, 'vertex')
        normal = GeomVertexWriter(vdata, 'normal')
        color = GeomVertexWriter(vdata, 'color')
        texcoord = GeomVertexWriter(vdata, 'texcoord')
        
        vertex.addData3f(face[0][0], face[0][1], face[0][2])
        vertex.addData3f(face[1][0], face[1][1], face[1][2])
        vertex.addData3f(face[2][0], face[2][1], face[2][2])
        vertex.addData3f(face[3][0], face[3][1], face[3][2])
        
        normal.addData3f(norm[0], norm[1], norm[2])
        normal.addData3f(norm[0], norm[1], norm[2])
        normal.addData3f(norm[0], norm[1], norm[2])
        normal.addData3f(norm[0], norm[1], norm[2])
        
        color.addData4f(self.color[0], self.color[1], self.color[2], 1.0)
        color.addData4f(self.color[0], self.color[1], self.color[2], 1.0)
        color.addData4f(self.color[0], self.color[1], self.color[2], 1.0)
        color.addData4f(self.color[0], self.color[1], self.color[2], 1.0)

        texcoord.addData2f(uvs[0][0], uvs[0][1])
        texcoord.addData2f(uvs[1][0], uvs[1][1])
        texcoord.addData2f(uvs[2][0], uvs[2][1])
        texcoord.addData2f(uvs[3][0], uvs[3][1])        

        tri=GeomTriangles(Geom.UHStatic)

        tri.addVertices(0,1,3)
        tri.addVertices(1,2,3)
        tri.closePrimitive()
        
        square=Geom(vdata)
        square.addPrimitive(tri)
        return square

This is… fast! :open_mouth:

def build(self):
        """Create the geometry from the submitted arrays"""
        verts = self.object.verts
        polys = self.object.polys
        if self.object.geomnode.getNumGeoms():
            self.object.geomnode.removeAllGeoms()

        format=GeomVertexFormat.getV3n3cpt2()        
        vdata=GeomVertexData('geom', format, Geom.UHStatic)

        vertex = GeomVertexWriter(vdata, 'vertex')
        normal = GeomVertexWriter(vdata, 'normal')
        color = GeomVertexWriter(vdata, 'color')
        texcoord = GeomVertexWriter(vdata, 'texcoord')
        tri=GeomTriangles(Geom.UHStatic)
           
        for poly_index in range(len(polys)):
            p = polys[poly_index]
            uvs = self.object.uvs[poly_index]       
            norm = self.object.vnorms[poly_index]
            face = [verts[i] for i in p]                
            
            vertex.addData3f(face[0][0], face[0][1], face[0][2])
            vertex.addData3f(face[1][0], face[1][1], face[1][2])
            vertex.addData3f(face[2][0], face[2][1], face[2][2])
            vertex.addData3f(face[3][0], face[3][1], face[3][2])
            
            normal.addData3f(norm[0], norm[1], norm[2])
            normal.addData3f(norm[0], norm[1], norm[2])
            normal.addData3f(norm[0], norm[1], norm[2])
            normal.addData3f(norm[0], norm[1], norm[2])
            
            color.addData4f(self.color[0], self.color[1], self.color[2], 1.0)
            color.addData4f(self.color[0], self.color[1], self.color[2], 1.0)
            color.addData4f(self.color[0], self.color[1], self.color[2], 1.0)
            color.addData4f(self.color[0], self.color[1], self.color[2], 1.0)

            texcoord.addData2f(uvs[0][0], uvs[0][1])
            texcoord.addData2f(uvs[1][0], uvs[1][1])
            texcoord.addData2f(uvs[2][0], uvs[2][1])
            texcoord.addData2f(uvs[3][0], uvs[3][1]) 

            tri.addVertices(p[0],p[1],p[3])
            tri.addVertices(p[1],p[2],p[3])

            if poly_index == len(polys)-1:
                tri.closePrimitive()            
                geom=Geom(vdata)
                geom.addPrimitive(tri)
                self.object.geomnode.addGeom(geom)

Whereas I was previously getting frame rates well below 100, with 400 to 1500 geometries being created per geomnode, now I get 500 to 700 FPS with only one geom per geomnode. Much better. Almost frighteningly so. :open_mouth: :laughing:

I guess I don’t need any elaborate approaches to building the geometry, then. It would be really nice if this sort of thing were documented, so beginners like me don’t have to spend days or weeks struggling to find a good approach. :neutral_face:

Now I guess I need to see how the whole thing handles with collisions. That’s another problem. The ocquadtreefy module looks promising, but that’s something else which is absolutely not documented. :laughing:

5

I’m no great expert in this, so take my thoughts with a grain of salt, but I suspect that the speed gain is a result of method calls being somewhat slow in Python, and your slow version calling your “addPolygonGOOD” method quite a few times per iteration.

6

That would certainly make sense. I was following the example of Panda’s Procedural Cube sample script, when I set it up that way. That seems to be a common problem when a new user begins working with vertexdata, at least based on the examples I’ve seen online. That Procedural Cube example is right there, accessible, obvious, and presenting itself as though it has all the right answers. And then it leads us astray. :laughing: To find a better approach for working with vertexdata, you really have to search far and wide, puzzle over the limited formal documentation, study often baffling code from old projects posted online, and piece hints together from diverse forum posts. The better approach should be the obvious one. Beginners need some useful vertexdata examples which won’t lead them in the wrong directions. I’ve been trying to help with that, but I’m learning as I go and my beginner’s missteps could be misleading or confusing in themselves. :blush:

I think I could still use just a slight speed gain, for the final project. I’ll have to set up the dev build and try rdb’s new approach.

7

I’m glad, then, that you’re posting all of this–it could be very useful for future users looking to do similar things. :slight_smile:

As to a further speed gain, I’m not sure of what to suggest, I’m afraid; good luck with the dev build.

8

I’ve used a couple of other 3D game creation systems, and the only area in which they top Panda is documentation. Panda has more options and better options, generally, but they’re not always obvious. That’s frustrating, and it makes things more difficult than they need to be. Sadly the Panda forums seem to be almost abandoned nowadays, so we can’t necessarily come here for a quick answer to a question. What are there, maybe ten people posting regularly in the last month or so? :frowning:

I got the collisions working, and I’m still getting frame rates in the 700+ range most of the time, even with collisions. No octree needed, thank goodness. :laughing: The octree setup for each room really slowed down the room changes, and those room transitions are the only area where I’m seeing any (slight) speed issues. The specifics of this game are such that I can just place a CollisionBox at any grid square where I have level geometry, then I can use a simple Pusher collider for everything. Next I need to work out how to do sliding collisions with this setup. I was under the impression that sliding should happen normally with the pusher, but apparently not.

Attached is the current form of the script, including over 300 rooms, if anyone ever wants to dig around in it and see what’s happening there. It’s not necessarily a good example of how to do things, but there might be some useful bits in there for someone.
adventure_faster_collisions.zip (179 KB)

9

memoryview is available in Python 2.6 and above.

I highly advise creating a small test case for yourself that uses the geometry code and then running the Python profiler on that (run “python -m cProfile -s tottime test.py”). For me, it has been an invaluable tool in seeing how my run time gets divided up.

Keep in mind that when you call one method many times, some things that usually aren’t a problem can suddenly become a big deal, such as dictionary lookups. To give a slight example, this loop:

        for poly_index in range(len(polys)):
            p = polys[poly_index]
            uvs = self.object.uvs[poly_index]       
            norm = self.object.vnorms[poly_index]
            face = self.addPolygon([verts[i] for i in p], uvs, norm)
            self.object.geomnode.addGeom(face)

That can be optimised as such to avoid unnecessary dictionary lookups:

        uvs = self.object.uvs
        vnorms = self.object.vnorms
        addPolygon = self.addPolygon
        addGeom = self.object.geomnode.addGeom

        for poly_index, p in enumerate(polys):
            uv = uvs[poly_index]
            norm = vnorms[poly_index]
            face = addPolygon([verts[i] for i in p], uv, norm)
            addGeom(face)

This becomes crucial when one method suddenly gets called fifty million times, which isn’t unthinkable when it comes to geometry generation.

10

Ooh. Optimization advice is good stuff. Thank you. :slight_smile:

I’m carrying around a lot of bad Python-coding habits, due to my heavy use of Poser Python. Poser 5 uses Python 2.1 (IIRC), which lacks (for instance) enumerate. Backwards compatibility is considered very important in Poserdom. So even as a Poser Pro 2014 user today (or even when coding for Panda, not Poser :laughing: ), I still reflexively use approaches which would be compatible with Poser 5. It’s a problem. :laughing:

There are probably dozens of little things like that which I’m doing wrong. Particularly if I’m trying to post helpful examples for future beginners, I should try to fix such errors in my publicly posted code. So please let me know if you note anything glaringly awful. I may be confused by any criticism which is over my head and too smart for me, but I’ll try not to panic or lose my temper. :laughing:

Can you explain this? I do pretty much everything with IDLE, and I only fire up the console when I need to invoke one of the Panda binaries or load a new Python library from its download distribution. Thus, I’m not sure what you mean. :question:

11

It might be worthwhile (for general forum reference, as well as my current project) to discuss methods for handling collisions with procedurally-generated vertexdata geometries. I spent much of yesterday trying different things with the collisions. I need a fast setup for the collisions and geometry, but I also need collisions that handle well. In this current case, I need to be able to slide against walls without slipping through cracks or getting caught at corners.

For setup speed, I found that simply adding a CollisionBox at each map grid location where I’m adding room geometry was best. Unfortunately, CollisionBox tends to cause problems with wall sliding, as the player’s From collider can end up being tested against two or more Into box normals simultaneously.

The best solution I found was to modify the box normals:

if collision:
            # Monkey with the normals, to enable sliding collision with sphere to box; otherwise we stick against box corners
            b = CollisionBox(Point3(center[0], center[1], center[2]), scale*1.25, scale*1.25, scale*1.25) 
            #self.object.colnode.addSolid(CollisionBox(Point3(center[0], center[1], center[2]), scale*1.25, scale*1.25, scale*1.25))
            n = Vec3(0.0,0.0,0.0)
            for side in range(6):
                if sides[side]:
                    ns = self.norms[side]
                    n += Vec3(ns[0], ns[1], ns[2])
            n /= sides.count(1)
            b.setEffectiveNormal(n)
            b.setRespectEffectiveNormal(True)
            self.object.colnode.addSolid(b)

This generally works to restore wall sliding, but it completely breaks the collisions in cases involving a thin wall which is only one layer deep. I need to come up with a better general solution, but perhaps this trick could be useful in some cases. The boxes here are scaled up slightly as well, to prevent any problems with slipping through the cracks between adjacent boxes. I found that collision boxes worked far better than collision polygons, for this implementation, and I found that involving octree in any way was too slow.

I also noted that one of the documented uses of CollisionBox is broken, with Python in 1.8.1. You can’t use

box = CollisionBox(center[0], center[1], center[2], scale[0], scale[1], scale[2])

The center data has to be formatted as Point3, when presenting it as an argument for CollisionBox.

12

If your gameplay is essentially two-dimensional, then I’ve had good results from CollisionTubes laid out along the bases of the walls, as I recall. For one thing, the sphere that caps each end of a tube allows the player to slip around convex corners, and adds a little extra collision solid at concave corners, which should help to reduce the problem of slipping into them.

If not, perhaps consider placing vertical tubes at your convex corners, and extending your boxes slightly into concave corners.

Either way, it might be worth investing some setup time in detecting “runs” of walls and then generating single collision objects for those, rather than individual objects for each cell; this should greatly reduce the problem of slipping through cracks.

13

Those are good suggestions. Thank you. :slight_smile: I’ll think about how I might integrate them into my process.

My gameplay is very much two dimensional. I’m adapting a very simple 2D game into a complicated 3D game. :laughing: This is my fifth time adapting Atari 2600 Adventure, so you might think I’d be a little bit better at some of this by now. The first adaptation burned down and sank into the swamp. :laughing: Actually, that first version was 2D, with Game Maker, and I did it without using even a single “for” loop or any arrays. Didn’t know how to use them yet. The fourth one was in 3D, for Dark Basic Professional, and that’s what I’m trying to port to Panda right now.

Just for fun, my previous versions are posted here:
Game Maker version 3 (old and 2D):
http://www.morphography.uk.vu/~cagepage/adventure/adventure.html
DBP version in 3D:
http://www.morphography.uk.vu/~cagepage/DBP_Adventure/dbpadventure.html
I just got my website set up on a new server this week, so maybe I’m a little excited and I’m being goofy by posting these links. :laughing: :blush:

Simply using enumerate in a few places, as in rdb’s suggestion, seems to speed things up a lot. Good stuff. Thank you. :smiley: