[SOLVED] Access image from C++ memory location within Python

C++ coding using Panda3D
1

I’m using faceAPI within C++ to retrieve face tracking data. (Right now it is a separate Win32 / C++ client that sends data over OSC back to Python - created by forum member August.)

Besides tracking data, faceAPI also outputs image frames into memory. I want to access this memory location within Python / P3D, load the image frames as a video with the goal of overlaying 3D models on top.

  1. What is the best approach to access the image frames from my separate C++ exe?

  2. Once I have the memory location how do I load in the images to create a smooth video?

Thanks so much,
R

2

You can use Texture::set_ram_mipmap_pointer() to tell the texture to take its source data from the pointer you specify. Note that it’s dependent on you to specify the size and format of the texture data properly.

See the Panda3D blog post on this subject entitled “Pointer Textures”.

David

3

Thanks David.
I’ve been looking at the Pointer Textures blog post, but didn’t fully understand the code so I wasn’t sure if I could use it.

The post says that the data goes directly to the video card, can I overlay models on top of it still?

How would I get the location of my C++ pointer when it is launched as a separate exe? I am transmitting data over OSC from C++ to Python.

Thanks,
R

4

Yes; the data is loaded as a texture. That means it is drawn wherever you apply that texture. You can certainly apply the texture to a card in the background and place models in front of it.

What is OSC? Open Sound Control? Anyway, I think the answer to this question is between you and OSC. Note that Python does not really have a void pointer object. Are you writing Python code to control Panda3D? If so, you won’t have access to the set_ram_mipmap_pointer() function, because of the lack of void pointer issue; but you can use set_ram_mipmap_pointer_from_int() if you can get a Python integer that corresponds to your void pointer. It all depends on what form your data arrives in from OSC.

David

5

Yes, Open Sound Control. Yes, my code is in Python.

Right now the data is coming through OSC as an array, but I believe it could be a single integer.

So I can convert my image pointer to int in C++ and then pass it through OSC to Python? Then I would be able to load that same memory location with the passed int using set_ram_mipmap_pointer_from_int()?

If this is the case, is there a way to set the memory location in C++ to static so I only need to pass the integer once? (I believe faceAPI will set a new memory allocation every time for every frame, but I’m not 100% sure of this)

Thanks this is very helpful!

6

I doubt you would be able to pass the integer pointer through OSC to a separate process–different processes have different memory space. A pointer in one process is not valid in another process. Unless you allocated the array in shared memory; but then you would pass the shared memory handle, and convert it to a pointer at the other end.

If by “array” you mean you have the data as a packed character string, then you can also use Texture.setRamImage() each frame to reload the texture data from your packed character string. This will copy the image data into the texture, which is a tiny bit less efficient, but probably safer given the complexity of your situation.

The mechanics of getting your data through OSC is really entirely outside of Panda’s domain. But once you have the image data in the Panda process, you can load it into a texture via one of these methods.

David

7

My only intent would be to pass the pointer (or shared memory location) through OSC.

I would not send the image data through as I don’t think my framerate would be high enough.

I’m assuming that in C++ I setup the shared memory and then in Python I read from it via the pointer passed through OSC? Do you have a preferred library on the C++ side for shared memory and also a module on the Python side for shared memory access?

I’ve seen the libraries Boost and Poco for C++ that have a shared memory function.

Thanks again!

8

Panda doesn’t provide any shared memory functions. You’re on your own for that. :slight_smile:

David

9

Hi drwr,

I’ve setup named shared memory in C++ and I have used mmap in Python to successfully transfer data from C++ to Python. Both applications are separate processes launched on their own.

In C++ the image is stored as the first element of array plane_addr and then to get it into shared memory I copy it like this:

CopyMemory((PVOID)pBuf, video_frame_image_info.plane_addr[0], BUF_SIZE);

In Python I access the data via:

m = mmap.mmap(0, 307221, "Global\\MyFileMappingObject", mmap.ACCESS_READ)

With m.read(1024) I can read the image as a data stream of chars, but obviously not wanted.

I’m not sure how to get the pointer as long int to use with set_ram_mipmap_pointer_from_int() from my shared memory as m prints as: “<mmap.mmap object at 0x00B42688>”

I’m able to read/save the image using PIL, but I’d like to be able to set this direct to the ramImage

import mmap
from PIL import Image
m = mmap.mmap(0, 307221, "Global\\MyFileMappingObject", mmap.ACCESS_READ)
size = 640, 480
im = Image.frombuffer("L", size, m, "raw", "L", 0, 1)
im.save("test.jpg", "JPEG")

Do you have any thoughts how I can make it so I can get the pointer to load it in via set_ram_mipmap_pointer_from_int() or secondly how to take my data stream and load that directly into setRamImage()?

Thanks!

10

I don’t know if there are any Python-level API’s that can help with this, since Python doesn’t really know what a void pointer is. You might need to write a custom C function to map the shared memory and pass its pointer to the Texture object.

David

11

And you don’t think setRamImage can take the image in the format that C++ is sending it back?

I get “Texture.setRamImage() argument 1 must be ConstPointerToArrayUnsignedChar, not mmap”

Anyway to convert this to ConstPointerToArray?

12

I’ve made it a bit farther with setRamImage, but I’m getting an error.

m = mmap.mmap(0, 307221, "Global\\MyFileMappingObject", mmap.ACCESS_READ)
x = PTAUchar.emptyArray(0) 
x.setData(m)
texture.setRamImage(CPTAUchar(x),MovieTexture.CMOff)

Assertion failed: compression != CM_off || image.size() == do_get_expected_ram_image_size() at line 830 of c:\panda3d-1.7.0\panda\src\gobj\texture.cxx

13

This means that the string you passed to setData() wasn’t the right size for the texture properties you specified.

Can you really treat a Python mmap object as a string? You appear to be doing that here, and expecting its string value to be the value of the shared memory buffer. That might be the way that mmap works (I haven’t used it myself), and if so, then you only need to check that the texture properties are being set up correctly. But if mmap doesn’t work that way, then you will need to call mmap.read() or whatever in order to return a string object.

David

14

I’ve been trying to match up the sizes of all my items, but I’ve never been 100% sure that my sizes are correct.

  1. In C++ I can’t figure out the allocated memory of the image, but when I open the string in a text editor it is 307221 bytes. This is close to 640x480 = 307200.

In P3D I’m setting up the following:

texture = Texture('movie')
texture.setup2dTexture(640, 480, Texture.CMOff, Texture.FRgba8)

In my task I get the same error if I run this:

m = mmap.mmap(0, 307221, "Global\\MyFileMappingObject", mmap.ACCESS_READ)
im = Image.frombuffer("L", (640,480), m, "raw", "L", 0, 1)  #PIL    
x = PTAUchar.emptyArray(0) 
x.setData(im.tostring())
texture.setRamImage(CPTAUchar(x),MovieTexture.CMOff)

or by just using the mmap output like this:

m = mmap.mmap(0, 307221, "Global\\MyFileMappingObject", mmap.ACCESS_READ)
x.setData(m)
texture.setRamImage(CPTAUchar(x),MovieTexture.CMOff)

Do all my sizes look correct?

15

Well, 307221 can’t be the correct size for a texture buffer. As you say, it is “close to” 640 * 480, but you need something that is precisely right, not merely close.

Perhaps there are an extra 21 bytes that your library is appending to the beginning or end of your texture data? If so, then you need to chop those bytes off before you pass it to the Texture. Even still, though, if your image is 640x480, 307220 is only the correct size if you have precisely one byte per pixel, which might be the case if you have a grayscale image. In this case, you need to advise the texture that it’s a grayscale image, by using:

tex.setup2dTexture(640, 480, Texture.TUnsignedByte, Texture.FLuminance)

David

16

Great news, with your help I got this working! Thanks so much.

I was able to use the mmap object and load it straight into PTAUchar.setdata().

The frame rate is as fast as using webcamvideo directly in Panda3D. My image source is greyscale so that might be helping to keep the frame rate up.

Here is the final Python code

from pandac.PandaModules import *

loadPrcFileData("", "auto-flip 1") #usualy the drawn texture lags a bit behind the calculted positions. this is a try to reduce the lag.
loadPrcFileData("", "win-size 640 480")

from direct.directbase.DirectStart import *
import mmap

def update(task):
    m = mmap.mmap(0, 307200, "Global\\MyFileMappingObject", mmap.ACCESS_READ) #640x480 = 307200
    x = PTAUchar.emptyArray(0) 
    x.setData(m)
    texture.setRamImage(CPTAUchar(x),MovieTexture.CMOff)
    return task.cont

texture = Texture('movie')
texture.setup2dTexture(640, 480, Texture.TUnsignedByte, Texture.FLuminance) #image is greyscale

cardMaker = CardMaker('cardMaker')
cardMaker.setFrame(4/3.0,-4/3.0,1,-1)
card = render.attachNewNode(cardMaker.generate())
card.setTexture(texture)
card.setTwoSided(True)
card.setY(5)
card.setScale(1.72)

taskMgr.add(update, "update")
       
run()

The C++ code is pretty much exactly what I found here.

Thanks again!