Does anybody have any hints or ideas how to solve the following problem:
I have a
Showbase envrironment in which I dynamically load different actor models. Let’s say I load two actors:
actors1 = Actor(‘models/actor_model1’)
actors2 = Actor(‘models/actor_model2’)
taskMgr.add(update_environment) I am able to update the position of actors dynamically inside the function
update_environment(). Additionally, every time I am inside
update_environment(), I extract the image (as a numpy array) of the current window of the ShowBase environment (I basically get the same image I see in the ShowBase window). To do this I initialize a texture object
self.texture = Texture()
self.win.addRenderTexture(self.texture, GraphicsOutput.RTM_copy_ram, GraphicsOutput.RTP_color)
in the constructor and every time I am inside the
update_environment() function, I call
image = np.frombuffer(data, np.uint8)
to obtain my image (I skipped some of the post-processing steps).
Now to the question: Is there a way to obtain the information about which pixels of the
image a occupied by which actor. So additionally to the RGB
image I would also like to get information about pixels occupied by
actor2. I need this to solve a classification task where every actor as well as the empty space belongs to a different class.
Thanks in advance!
You could render the scene a second time, with the scenery rendered in black and the Actors rendered in different colours, solid and unshaded. The resulting image could then be queried, with pixel-colours indicating specific Actors.
What @Thaumaturge suggested could be accomplished by making use of the tag state key system.
Basically, a second camera would render the actors in a flat color using alternative render states associated with specific tags that are set on the actors.
Here’s a very basic example:
from direct.showbase.ShowBase import ShowBase
from panda3d.core import *
dir_light = DirectionalLight("light")
dir_light.set_color((1., 1., 1., .5))
light_np = self.render.attach_new_node(dir_light)
light_np.set_hpr(-20., -20., 0.)
mask = BitMask32.bit(2)
state_cam = self.make_camera(self.win, camName="state_cam", mask=mask,
displayRegion=(.75, 1., 0., .25))
state_np = NodePath("state")
state_np.set_color((1., 0., 0., 1.), 1)
red_state = state_np.get_state()
state_np.set_color((0., 1., 0., 1.), 1)
green_state = state_np.get_state()
state_cam_node = state_cam.node()
state_cam_node.tag_state_key = "flat_color"
state_cam_node.tag_states["red"] = red_state
state_cam_node.tag_states["green"] = green_state
model1 = self.loader.load_model("smiley")
model1.tags["flat_color"] = "red"
model2 = self.loader.load_model("frowney")
model2.tags["flat_color"] = "green"
app = App()
When you move the camera around, you should see a smiley and a frowney model, textured and lit in the main display region, but with flat colors in a secondary, small display region at the bottom right hand corner of the window.
Instead of this secondary display region (which I only use to quickly show what is being rendered), you will likely need to set up an off-screen texture buffer for that second camera to render into.
@Thaumaturge, @Epihaius Thank you very much for your suggestions! And especially for the provided example (@Epihaius ), I will try it out!