Does anybody have any hints or ideas how to solve the following problem:
I have a Showbase
envrironment in which I dynamically load different actor models. Let’s say I load two actors:
actors1 = Actor(‘models/actor_model1’)
actors2 = Actor(‘models/actor_model2’)
actor1.reparentTo(self.render)
actor2.reparentTo(self.render)
By using taskMgr.add(update_environment)
I am able to update the position of actors dynamically inside the function update_environment()
. Additionally, every time I am inside update_environment()
, I extract the image (as a numpy array) of the current window of the ShowBase environment (I basically get the same image I see in the ShowBase window). To do this I initialize a texture object
self.texture = Texture()
self.win.addRenderTexture(self.texture, GraphicsOutput.RTM_copy_ram, GraphicsOutput.RTP_color)
in the constructor and every time I am inside the update_environment()
function, I call
self.texture.getRamImageAs(‘RGBA’)
image = np.frombuffer(data, np.uint8)
to obtain my image (I skipped some of the post-processing steps).
Now to the question: Is there a way to obtain the information about which pixels of the image
a occupied by which actor. So additionally to the RGB image
I would also like to get information about pixels occupied by actor1
and actor2
. I need this to solve a classification task where every actor as well as the empty space belongs to a different class.
Thanks in advance!
You could render the scene a second time, with the scenery rendered in black and the Actors rendered in different colours, solid and unshaded. The resulting image could then be queried, with pixel-colours indicating specific Actors.
1 Like
What @Thaumaturge suggested could be accomplished by making use of the tag state key system.
Basically, a second camera would render the actors in a flat color using alternative render states associated with specific tags that are set on the actors.
Here’s a very basic example:
from direct.showbase.ShowBase import ShowBase
from panda3d.core import *
class App(ShowBase):
def __init__(self):
ShowBase.__init__(self)
dir_light = DirectionalLight("light")
dir_light.set_color((1., 1., 1., .5))
light_np = self.render.attach_new_node(dir_light)
light_np.set_hpr(-20., -20., 0.)
self.render.set_light(light_np)
mask = BitMask32.bit(2)
state_cam = self.make_camera(self.win, camName="state_cam", mask=mask,
displayRegion=(.75, 1., 0., .25))
state_np = NodePath("state")
state_np.set_light_off(1)
state_np.set_texture_off(1)
state_np.set_material_off(1)
state_np.set_color((1., 0., 0., 1.), 1)
red_state = state_np.get_state()
state_np.set_color((0., 1., 0., 1.), 1)
green_state = state_np.get_state()
state_cam_node = state_cam.node()
state_cam_node.tag_state_key = "flat_color"
state_cam_node.tag_states["red"] = red_state
state_cam_node.tag_states["green"] = green_state
model1 = self.loader.load_model("smiley")
model1.reparent_to(self.render)
model1.set_x(-2.)
model1.tags["flat_color"] = "red"
model2 = self.loader.load_model("frowney")
model2.reparent_to(self.render)
model2.set_x(2.)
model2.tags["flat_color"] = "green"
app = App()
app.run()
When you move the camera around, you should see a smiley and a frowney model, textured and lit in the main display region, but with flat colors in a secondary, small display region at the bottom right hand corner of the window.
Instead of this secondary display region (which I only use to quickly show what is being rendered), you will likely need to set up an off-screen texture buffer for that second camera to render into.
2 Likes
@Thaumaturge, @Epihaius Thank you very much for your suggestions! And especially for the provided example (@Epihaius ), I will try it out!
2 Likes