Extract pixel indices occupied by different actors/models

Scripting Issues
#1

Does anybody have any hints or ideas how to solve the following problem:

I have a Showbase envrironment in which I dynamically load different actor models. Let’s say I load two actors:

actors1 = Actor(‘models/actor_model1’)
actors2 = Actor(‘models/actor_model2’)
actor1.reparentTo(self.render)
actor2.reparentTo(self.render)

By using taskMgr.add(update_environment) I am able to update the position of actors dynamically inside the function update_environment(). Additionally, every time I am inside update_environment(), I extract the image (as a numpy array) of the current window of the ShowBase environment (I basically get the same image I see in the ShowBase window). To do this I initialize a texture object

self.texture = Texture()
self.win.addRenderTexture(self.texture, GraphicsOutput.RTM_copy_ram, GraphicsOutput.RTP_color)

in the constructor and every time I am inside the update_environment() function, I call

self.texture.getRamImageAs(‘RGBA’)
image = np.frombuffer(data, np.uint8)

to obtain my image (I skipped some of the post-processing steps).

Now to the question: Is there a way to obtain the information about which pixels of the image a occupied by which actor. So additionally to the RGB image I would also like to get information about pixels occupied by actor1 and actor2. I need this to solve a classification task where every actor as well as the empty space belongs to a different class.

Thanks in advance!

#2

You could render the scene a second time, with the scenery rendered in black and the Actors rendered in different colours, solid and unshaded. The resulting image could then be queried, with pixel-colours indicating specific Actors.

1 Like
#3

What @Thaumaturge suggested could be accomplished by making use of the tag state key system.
Basically, a second camera would render the actors in a flat color using alternative render states associated with specific tags that are set on the actors.

Here’s a very basic example:

from direct.showbase.ShowBase import ShowBase
from panda3d.core import *

class App(ShowBase):

    def __init__(self):

        ShowBase.__init__(self)

        dir_light = DirectionalLight("light")
        dir_light.set_color((1., 1., 1., .5))
        light_np = self.render.attach_new_node(dir_light)
        light_np.set_hpr(-20., -20., 0.)
        self.render.set_light(light_np)

        mask = BitMask32.bit(2)
        state_cam = self.make_camera(self.win, camName="state_cam", mask=mask,
            displayRegion=(.75, 1., 0., .25))
        state_np = NodePath("state")
        state_np.set_light_off(1)
        state_np.set_texture_off(1)
        state_np.set_material_off(1)
        state_np.set_color((1., 0., 0., 1.), 1)
        red_state = state_np.get_state()
        state_np.set_color((0., 1., 0., 1.), 1)
        green_state = state_np.get_state()
        state_cam_node = state_cam.node()
        state_cam_node.tag_state_key = "flat_color"
        state_cam_node.tag_states["red"] = red_state
        state_cam_node.tag_states["green"] = green_state

        model1 = self.loader.load_model("smiley")
        model1.reparent_to(self.render)
        model1.set_x(-2.)
        model1.tags["flat_color"] = "red"
        model2 = self.loader.load_model("frowney")
        model2.reparent_to(self.render)
        model2.set_x(2.)
        model2.tags["flat_color"] = "green"

app = App()
app.run()

When you move the camera around, you should see a smiley and a frowney model, textured and lit in the main display region, but with flat colors in a secondary, small display region at the bottom right hand corner of the window.
Instead of this secondary display region (which I only use to quickly show what is being rendered), you will likely need to set up an off-screen texture buffer for that second camera to render into.

2 Likes
#4

@Thaumaturge, @Epihaius Thank you very much for your suggestions! And especially for the provided example (@Epihaius ), I will try it out!

2 Likes