Get camera image to OpenCV::Mat

[Pypaut]

Greetings,

As stated in the title, I would like to get the current displayed image in my Panda3D application, to any type of image data that I could work on using OpenCV (ideally convertible to Mat type).

I could use the ShowBase.screenshot() method to save image to disk and then load id back using cv.imread(), but you can guess it is not the best in terms of efficiency (60 → 20 FPS).

Reading through the documentation, I came up with self.camera.GetTexture().getRamImage(), but the texture apparently has to be set before calling GetTexture().

I am fairly new to Panda3D, and would greatly appreciate any hints.

Thank you by advance!

Edit

All the mentioned code is called from a method added to the task manager, thus called every frame

Edit 2

I found this topic which is actually exactly my question : Getting camera image directly, not sure why I didn’t find this in the first place. Here is the code I tried:

tex = Texture()
base.win.addRenderTexture(
            tex,
            mode=GraphicsOutput.RTM_copy_ram,
            bitplane=GraphicsOutput.RTP_color,
)
image = tex.getRamImage()

The code runs, but did not check the resulting image. The FPS drop is about the same as writing + loading to/from disk, so I might just to this anyways.

Thank you!

[Simulan]

These threads seem related to your issue:

OpenCV VideoCapture to Panda3D Texture?

The issue is with USB cameras I don’t think you can access it by two separate processess at the same time. My OpenCV code already needs to access the same camera for machine vision tasks, so I don’t think I can access it simultaneously with Panda and instead need to access the frame data already available to me by my OpenCV code. Correct me if I’m wrong. And my issue is not the fps of the texture but the whole Panda window dropping below 8 fps.

Reading from Webcam to display as a background

Any body has working solution ? None of the above worked . Recent one gave me this error glxGraphicsPipe (all display modules loaded.) :device(error): Error adding inotify watch on /dev/input: No space left on device TRYING WITH WebcamVideo :vision(error): Failed to map buffer! :vision(error): Failed to map buffer! :vision(error): Failed to map buffer! :vision(error): Failed to map buffer! WebcamVideo based texure infos: -> 2d_texture HD WebCam: HD WebCam 2-d, 1280 x 720 pixels, each 3 by…

[Pypaut]

Yes indeed, they are related, but they do the exact opposite of what I need : I need to get the image from Panda3D, while they have their image and want to integrate it to Panda3D.

Thank you nonetheless, I’ll keep reading the threads just in case.

[rdb]

You are using the right approach with RTM_copy_ram. Did you end up getting it to work?

I’m afraid it is inherently going to be slow because the GPU is heavily pipelined, and getting results back to the CPU means waiting for the GPU to finish all its work, which means work on the next frame can’t start yet.