Comparing Texture to Image Using Shader

Scripting Issues
1

I’ve set up a scene with multiple buffers that get the result of render-to-texture.

I use the following code to set up my texture/buffer/camera

# config is a dumb container that stores properties
# and behaves exactly how you'd expect

film_size = config.intrinsics.get_size()
window_props = p3dc.WindowProperties.size(*film_size)
buffer = self.graphics_engine.make_output(
  self.pipe,
  f'Buffer[{name}]',
  config.render_order,
  config.frame_buffer_properties,
  window_props,
  p3dc.GraphicsPipe.BFRefuseWindow,  # don't open a window,
  self.win.getGsg(),
  self.win
)
texture = p3dc.Texture()
buffer.add_render_texture(texture, p3dc.GraphicsOutput.RTMCopyRam)
buffer.set_clear_color_active(True)
buffer.set_clear_color(config.clear_color)

lens = p3dc.PerspectiveLens()
lens.set_film_size(*film_size)
lens.set_fov(*np.rad2deg(config.intrinsics.get_fov()))
camera = self.make_camera(buffer, lens=lens, camName=f'Camera[{name}]')
camera.reparent_to(self.render)
camera.set_pos(*pos)
camera.set_hpr(*np.rad2deg(hpr))

camera.node().set_initial_state(config.render_state)

Now I can retrieve the results of rendering from the buffer using something like

texture = buffer.get_texture()
data = texture.get_ram_image()
frame = np.frombuffer(data, self._dtype)

What I really want to do (without losing access to the existing buffer/texture that I have, because sometimes I do want to examine the image), is take this texture and compare it to an image I already have.

For example, I want to be able to load an image into a second texture, and then produce a third buffer/texture that gives the difference between the textures at each pixel. I believe I should be using a fragment shader (?) to do this, but am unclear on how to set up the mechanics in Panda3D.

I think I know how to load the image in as the second texture:

second_texture = loader.load_texture("myImage.png")

but am unclear how to set up the third texture. Should I be using filters for this (even though I don’t want anything to appear in the window)?

2

I realized maybe I wasn’t very clear in the above.

All I’m really looking to do, is calculate the difference between two textures and store the result in a third texture. For example, say I have tex1 and tex2, what’s the easiest (fast) way to compute

tex3 = tex1 - tex2

and get the result to the CPU (I assume through a buffer)?

I would really appreciate any suggestions.

3

Writing a shader to do perform that operation shouldn’t be too difficult in and of itself. What you might then do is use render-to-texture (see this manual page; I imagine that you would be using the FilterManager) to send the result to a texture that you can access in your code.

Whether that’s the fastest way of doing it I don’t know, I’m afraid.

4

A compute shader might be a good alternative that doesn’t require any buffer setup.

1 Like
5

I’ve set up render-to-texture, using the following code:

import numpy as np
import panda3d.core as p3dc
from direct.showbase.ShowBase import ShowBase
import cv2
import math

base = ShowBase()

# put a box in the scene
box = base.loader.load_model('models/box.egg')
box.reparent_to(base.render)
box.set_pos(-0.5, -0.5, 0)

fb_props = p3dc.FrameBufferProperties()
fb_props.set_float_color(True)
fb_props.set_rgba_bits(8, 8, 8, 8)

# set up offscreen buffer to render camera into
buffer = base.graphics_engine.make_output(
    base.pipe,
    'Buffer',
    -2,                                 # render order
    fb_props,
    p3dc.WindowProperties.size((512, 512)),
    p3dc.GraphicsPipe.BFRefuseWindow,   # don't open a window,
    base.win.getGsg(),
    base.win
)

# create texture and set texture to render to buffer
texture = p3dc.Texture()
buffer.add_render_texture(texture, p3dc.GraphicsOutput.RTMCopyRam)
buffer.set_clear_color_active(True)
buffer.set_clear_color((1., 1., 1., 0.))

# set up camera to render into buffer
lens = p3dc.PerspectiveLens()
lens.set_film_size((512, 512))
lens.set_fov((30, 30))
camera = base.make_camera(buffer, lens=lens, camName=f'Camera')
camera.reparent_to(base.render)
camera.set_pos((6, 0, 4))
camera.set_hpr((90, -30.256, 0))

# render frame
base.graphics_engine.render_frame()

# get rendering and display
data = texture.get_ram_image()
frame = np.frombuffer(data, np.float32)
frame.shape = (512, 512, 4)
frame = np.flipud(frame)
cv2.imshow("Rendering", frame)
cv2.waitKey()

which renders a box, like this:

rendering
rendering512×512 43.5 KB

@rdb I’m trying to figure out compute shaders (I’ve ordered a new graphics cards with OpenGL 4.3) and am trying to figure out how to use them in a pipeline. For example, can I render the scene from multiple cameras into textures (like the above example), and then take those textures and apply compute shaders to those.

Hopefully this diagram makes sense, where camn would be the nth rendering, and for example, swapn has the R and G channels of these renderings swapped.

       ----- cam1 ----- swap1 ----- ...
      /
scene ------ cam2 ----- swap2 ----- ...
      \
       ----- cam3 ----- swap3 ----- ...

Is there a way to set this up using buffers with appropriate render orders (something like below), so I can just call base.graphics_engine.render_frame() and then pull the final texture? If not, is there a better way to sequence these post-processing compute shader applications?

ll_buffer = base.graphics_engine.make_output(
    base.pipe,
    'Buffer',
    -1,                                 # render order
    fb_props,
    p3dc.WindowProperties.size((1920, 1080)),
    p3dc.GraphicsPipe.BFRefuseWindow,   # don't open a window,
    base.win.getGsg(),
    base.win
)

ll_texture = p3dc.Texture()
ll_buffer.add_render_texture(ll_texture, p3dc.GraphicsOutput.RTMCopyRam)
ll_buffer.set_clear_color_active(True)
ll_buffer.set_clear_color((0., 0., 0., 0.))

shader_code = """#version 430

// Set the number of invocations in the work group.
// In this case, we operate on the image in 16x16 pixel tiles.
layout (local_size_x = 16, local_size_y = 16) in;

// Declare the texture inputs
uniform readonly image2D fromTex;
uniform writeonly image2D toTex;

void main() {
  // Acquire the coordinates to the texel we are to process.
  ivec2 texelCoords = ivec2(gl_GlobalInvocationID.xy);

  // Read the pixel from the first texture.
  vec4 pixel = imageLoad(fromTex, texelCoords);

  // Swap the red and green channels.
  pixel.rg = pixel.gr;

  // Now write the modified pixel to the second texture.
  imageStore(toTex, texelCoords, pixel);
}
"""

node = p3dc.ComputeNode("compute")
node.add_dispatch(math.ceil(512 / 16), math.ceil(512 / 16), 1)
node_path = base.render.attach_new_node(node)
shader = p3dc.Shader.make_compute(p3dc.Shader.SL_GLSL, shader_code)
node_path.set_shader(shader)
node_path.set_shader_input("fromTex", texture)
node_path.set_shader_input("toTex", ll_texture)

# render frame
base.graphics_engine.render_frame()

# get rendering and display
data = ll_texture.get_ram_image()
frame = np.frombuffer(data, np.float32)
frame.shape = (512, 512, 4)
frame = np.flipud(frame)
cv2.imshow("Rendering", frame)
cv2.waitKey()

I appreciate all the help!