Writing Arbitrary Components to custom GeomVertexArrayData

[jarata]

Hi all!

So I’m trying to write an arbitrary length array of data to each vertex in a model. The idea is: if you can give a vertex a color with 3 or 4 components, I’m looking to give a vertex a color with any number of components ( a color spectrum of wavelengths ). Is this possible?

Here’s the segment of python code I’ve tried:

# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #

# Create custom GeomVertexArrayFormat
spectrum = GeomVertexArrayFormat( )
spectrum.addColumn( "vertex", 3, Geom.NTFloat32, Geom.CPoint )
spectrum.addColumn( "texcoord", 2, Geom.NTFloat32, Geom.CTexcoord)

# num_components is variable
# Geom.C_other?? Maybe a different Geom.___?
spectrum.addColumn( "spectrum", num_components, Geom.NTFloat32, Geom.C_other )

spectrum_format = GeomVertexFormat( )
spectrum_format.add_array( spectrum )
spectrum_format = GeomVertexFormat.registerFormat( spectrum_format )

# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #

def fill_custom_vertices( num_rows ):

  # Create GeomVertexData with custom format and set up writers
  custom_vdata = GeomVertexData( 'custom_vdata', spectrum_format, Geom.UHStatic )
  custom_vdata.setNumRows( num_rows )

  wvertex = GeomVertexWriter( custom_vdata, 'vertex' )
  wtexcoord = GeomVertexWriter( custom_vdata, 'texcoord' )
  wspectrum = GeomVertexWriter( custom_vdata, 'spectrum' )

  # Probably not the most efficient way to type this out
  vertex_arr = np.random.randint( 2, size = 3 )
  vertex_vec = LVecBase3f( vertex_arr[ 0 ], vertex_arr[ 1 ], vertex_arr[ 2 ] )

  texcoord_arr =  np.random.randint( 2, size = 2 )
  texcoord_vec = LVecBase2f( texcoord_arr[ 0 ], texcoord_arr[ 1 ] )

  for row in range( num_rows ):

    wvertex.addData3( vertex_vec )
    wtexcoord.addData2( texcoord_vec )

  # If num_components == 4, this code works as there's a built in LVecBase4f class and the
  # ability to write 4 component data with GeomVertexWriter.addData4
  spec_arr =  np.random.randint( 2, size = 4 )
  spec_vec = LVecBase4f( spec_arr[0], spec_arr[1], spec_arr[2], spec_arr[3] )

  for row in ( num_rows ):
    wspectrum.addData4( spec_vec )

  # Is there a way to do something similar for an arbitrarily long array (arbitrary number of 
  # components?) I can create a GeomVertexArrayFormat with a column with an arbitrary 
  # number of components, I'm just not sure how to write data with the same number of
  # components to it. For example, I figured it would be something like:

  spec_arr = np.random.randint( 2, size = 7 )
  spec_vec = LVecBase4f( spec_arr[0], spec_arr[1], spec_arr[2], spec_arr[3], spec_arr[4], spec_arr[5], spec_arr[6] )
  wspectrum.addData7( spec_vec )

# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #

I’m sure there’s a better way to write data to a vertex, as well as build the LVecBase#f objects from numpy arrays, but I’m more wondering about write that arbitrary component data. Is this the way to do it? Is it possible? Is there a better way that I’m missing? I’m pretty new to working with Panda3D, so I appreciate any help you may have to offer.

Thanks and all the best!

[Epihaius]

Hi and welcome to the Panda3D community !

Although I’m not sure whether a data column containing more than 4 components might cause some unforeseen problems or what limit there is to the number of components it can have, at least it seems to be possible to add more than 4 using a memoryview (which is more efficient than using GeomVertexWriters anyway):

import array

# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #

# Create custom GeomVertexArrayFormat
spectrum = GeomVertexArrayFormat( )
spectrum.addColumn( "vertex", 3, Geom.NTFloat32, Geom.CPoint )
spectrum.addColumn( "texcoord", 2, Geom.NTFloat32, Geom.CTexcoord)

# num_components is variable
num_components = 7
spectrum.addColumn( "spectrum", num_components, Geom.NTFloat32, Geom.C_other )

spectrum_format = GeomVertexFormat( )
spectrum_format.add_array( spectrum )
spectrum_format = GeomVertexFormat.registerFormat( spectrum_format )

# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #

def fill_custom_vertices( num_rows ):

    custom_vdata = GeomVertexData( 'custom_vdata', spectrum_format, Geom.UHStatic )
    custom_vdata.setNumRows( num_rows )
    view = memoryview(custom_vdata.modify_array(0)).cast("B").cast("f")
    values = array.array("f", [])  # a numpy array should also work

    vertex_arr = np.random.randint( 2, size = 3 )
    texcoord_arr = np.random.randint( 2, size = 2 )
    spec_arr = np.random.randint( 2, size = 7 )

    for row in range( num_rows ):

        values.extend(vertex_arr)
        values.extend(texcoord_arr)
        values.extend(spec_arr)

    view[:] = values

# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #

As I don’t use numpy myself, I cannot guarantee that the above code will work correctly (I tested with Python’s array and random modules instead).

[jarata]

Hey Epihaius!

Thanks for getting back to me and for the memoryview tip, that definitely helped. I figured out a way to fill the vertex data using NumPy arrays, but the method I had got to be pretty slow with more and more rows in the GeomVertexData. I found this forum post from MaXL130 from last year and tried out their method for filling data from NumPy arrays

Sample: Directly filling a GeomVertexData instance using NumPy arrays

Today I tried to fill a GeomVertexData directly, and above all FAST, using a numpy array. Using a GeomVertexWriter requires me to iterate over my data (medium sized pointclouds), which is very slow in python. Instead I directly filled the buffer using the following. For this, I created a “custom” vertexDataFormat, so there are two GeomVertexArrays that I can fill separately later on. import panda3d.core as core import numpy as np # Create the vertex array data format positionColFormat = core.…

It seems like they manage to speed things up with the NumPy arrays by altering the format of the GeomVertexData such that it has one GeomVertexArrayData for each different type of data you want to hold in a vertex, (then each array has only one column). Before the format I was using had one GeomVertexArrayData with multiple columns, where each column represented the different types of data (i.e. vertex, texcoord, spectrum, etc.)

Anyways, I got that to work and compared them for efficiency, and it looks like the altered GeomVertexData format is about 30x faster on average. Pretty sweet! Thanks for the memoryview tip either way, that definitely cleared the roadblock I had with this in the first place!

Here’s the script I have to test the speed of the 2 methods. It’s using 501 rows, 501 components to the ‘spectrum’ data, and is averaging over 1000 loops

# -*- coding: UTF-8 -*-

# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #
import sys
import os
import array
import numpy as np

from direct.gui.DirectGui import *
from panda3d.core import TextNode, GeomVertexReader, GeomVertexWriter, GeomVertexArrayFormat, GeomVertexArrayData, GeomVertexData, GeomVertexFormat, Geom, LVecBase4f, LVecBase3f, LVecBase2f

import time

# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #

num_components = 501
num_rows = 501

# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #

# Create custom GeomVertexArrayFormat with multiple columns
spectrum = GeomVertexArrayFormat( )
spectrum.addColumn( "vertex", 3, Geom.NTFloat32, Geom.CPoint )
spectrum.addColumn( "texcoord", 2, Geom.NTFloat32, Geom.CTexcoord)
spectrum.addColumn( "spectrum", num_components, Geom.NTFloat32, Geom.CPoint )

# Create custom GeomVertexArray - 1 array with multiple columns
spectrum_format = GeomVertexFormat( )
spectrum_format.add_array( spectrum )
spectrum_format = GeomVertexFormat.registerFormat( spectrum_format )

# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #

# Create custom array formats for each set of data individually, with 1 column each
vertex_col_format = GeomVertexArrayFormat( "vertex", 3, Geom.NTFloat32, Geom.C_point )
texcoord_col_format = GeomVertexArrayFormat( "texcoord", 2, Geom.NTFloat32, Geom.C_texcoord )
spectrum_col_format = GeomVertexArrayFormat( "spectrum", num_components, Geom.NTFloat32, Geom.C_point )

# Add these to a custom vertex format and register it, give a GeomVertexData with multiple arrays, each having 1 column
custom_vertex_format = GeomVertexFormat()
custom_vertex_format.addArray( vertex_col_format )
custom_vertex_format.addArray( texcoord_col_format )
custom_vertex_format.addArray( spectrum_col_format )
custom_vertex_format = GeomVertexFormat.registerFormat( custom_vertex_format )

# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #

def fill_custom_vertices( num_rows ):

  custom_vdata = GeomVertexData( 'custom_vdata', spectrum_format, Geom.UHStatic )

  custom_vdata.setNumRows( num_rows )
  view = memoryview(custom_vdata.modify_array(0)).cast("B").cast("f")

  vertex_arr = np.arange( 3 )
  texcoord_arr = np.arange( 2 )
  spec_arr = np.arange( num_components )

  values = array.array("f", [])  # a numpy array should also work

  for row in range( num_rows ):
    values.extend(vertex_arr)
    values.extend(texcoord_arr)
    values.extend(spec_arr)

  view[:] = values

  view_obj = view.obj

  # print( view_obj )

# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #

def fill_custom_vertices3( num_rows ):

  # Create GeomVertexData with custom format and set number of rows
  custom_vdata = GeomVertexData( 'custom_vdata', custom_vertex_format, Geom.UHStatic )

  custom_vdata.setNumRows( num_rows )

  # NumPy arrays to copy to GeomVertexData
  vertex_arr = np.ones( ( num_rows, 3 ) )
  texcoord_arr = np.zeros( ( num_rows, 2 ) )
  spec_arr = 2*np.ones( ( num_rows, num_components ) )

  arrayHandle0: GeomVertexArrayData = custom_vdata.modifyArray( 0 )
  arrayHandle1: GeomVertexArrayData = custom_vdata.modifyArray( 1 )
  arrayHandle2: GeomVertexArrayData = custom_vdata.modifyArray( 2 )

  # Make sure the data is copied using the right type
  arrayHandle0.modifyHandle().copyDataFrom( vertex_arr.astype( np.float32 ) )
  arrayHandle1.modifyHandle().copyDataFrom( texcoord_arr.astype( np.float32 ) )
  arrayHandle2.modifyHandle().copyDataFrom( spec_arr.astype( np.float32 ) )

  # print( custom_vdata )

# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ #

if __name__ == "__main__":

  time_loops = 1000

  # -------------------------------------------------------------------------- #

  start_time = time.time()
  for t in range( time_loops ):
    fill_custom_vertices( num_rows = num_rows )
  elapsed_time = time.time( ) - start_time
  avg_time = elapsed_time / time_loops

  print( elapsed_time )
  print( f"Average time for memory view custom vertices: {avg_time} seconds" )

  # -------------------------------------------------------------------------- #

  start_time = time.time()
  for t in range( time_loops ):
    fill_custom_vertices3( num_rows = num_rows )
  elapsed_time = time.time( ) - start_time
  avg_time = elapsed_time / time_loops

  print( elapsed_time )
  print( f"Average time for alt numpy array custom vertices: {avg_time} seconds" )

[Epihaius]

Glad to hear you got it to work !

Although it won’t make much difference in terms of speed, I’d like to point out that you can also use memoryviews in your fill_custom_vertices3 function instead of using GeomVertexArrayDataHandle.copyDataFrom:

  view0: memoryview = memoryview(custom_vdata.modifyArray( 0 )).cast("B").cast("f")
  view1: memoryview = memoryview(custom_vdata.modifyArray( 1 )).cast("B").cast("f")
  view2: memoryview = memoryview(custom_vdata.modifyArray( 2 )).cast("B").cast("f")

  # Make sure the data is copied using the right type
  view0[:] = vertex_arr.astype( np.float32 )
  view1[:] = texcoord_arr.astype( np.float32 )
  view2[:] = spec_arr.astype( np.float32 )

[jarata]

Oh awesome, that’s great to know/have both options, thanks again for the help here!