Hey guys, I’m new to Panda3D and I’m building a software that models a building. For this I need the 3D coordinate of my mouse, how can I get it?
If feasible, the simplest approach might be to use a ray-cast, and to look for the first hit.
However, this requires that either we have collision geometry for the ray to test against, or that we allow the ray to test against visible geometry–the latter of which may be slow.
There is also the “extrude” method of the Lens class–see this API entry, and the one below it, for more. However, this requires some means of determining the depth of the given point.
If you just want the mouse-position on a single plane, then this easy enough–Panda includes a method for determining the relevant intersection. For something more complex otherwise, however, this becomes more complex. (If you can query your depth-buffer, then that might do it.)
1 Like
The first approach suggested by Thaumaturge is explained on this manual page, I believe.
There is another way, which involves creating a second camera that renders (a small part of) the scene a second time–to an offscreen buffer–where each model is rendered with its own unique, flat color. Then it’s a matter of reading the color of the rendered pixels (actually only one pixel) to find out which model is under the mouse cursor.
If you’re interested, I could give you some code to do this.
Then there is also the possibility of using a shader, although probably only worth trying if you want more advanced types of selection, like rectangular region-selection, lasso-selection, etc.
2 Likes
With regards to the rendering-to-flat-colour approach mentioned above, if you additionally want the 3D coordinates of the mouse-click (i.e. not just the selected object), you might be able to do so in a shader by rendering not only to a flat colour, but also to a colour that encodes position-data.
1 Like
I say that you use the base.mouseWatcherNode().getMouse() method. Assign this to a variable. Then you can use var_name.getX() and var_name.getY() methods. Then when you are using the method to pass 3D coordinares, just pass the mouseX to the xpos, 0 (or whatever suits you) to ypos, and mouseY to zpos (yes, that’s right).
I’m afraid that this would only really work for an isometric camera-lens, and even then only when looking directly down the y-axis. In addition, it wouldn’t allow one to determine the appropriate y-coordinate, as far as I see–after all, the y-coordinate would presumably be related to the distance to the specific piece of geometry under the mouse-cursor.
I’d like to see the code you mentioned, if it not bothers you. 
I want to find the 3D point in this white plane I drew, the code below gives me a ray and if the mouse is touching any object, nothing else.
class Distance:
def __init__(self, showbase):
self.app = showbase
showbase.taskMgr.doMethodLater(0.05, self.cursorWatch, "cursorWatch", appendTask=True)
def cursorWatch(self, task):
if self.app.mouseWatcherNode.hasMouse():
myTraverser = CollisionTraverser()
myHandler = CollisionHandlerQueue()
pickerNode = CollisionNode('mouseRay')
pickerNP = self.app.cam.attachNewNode(pickerNode)
pickerNode.setFromCollideMask(GeomNode.getDefaultCollideMask())
pickerRay = CollisionRay()
pickerNode.addSolid(pickerRay)
myTraverser.addCollider(pickerNP, myHandler)
mpos = self.app.mouseWatcherNode.getMouse()
pickerRay.setFromLens(self.app.camNode, mpos.getX(), mpos.getY())
myTraverser.traverse(self.app.render)
# Assume for simplicity's sake that myHandler is a CollisionHandlerQueue.
if myHandler.getNumEntries() > 0:
print("Collision Registered")
print(pickerNode.getSolid(0).direction.normalized())
print(pickerNode.getSolid(0).origin)
return task.again
Oh, if you just want a point on the plane, then this is quite easy with Panda–the “Plane” class provides a neat convenience method for determining the intersection of the plane with a line. See this post for more:
The code that you have is more generalised: it should pick up any collision-object that you might have in the scene. (Presuming that you don’t use bitmasks or other tricks to prevent this.)
If that’s what you want, then you might look at the manual entry for CollisionHandlerQueue for a partial example and some of the methods involved in getting the first hit of the ray.
As a side-note, let me point out that you don’t have to construct your traverser, handler, ray, etc. every time the task is run–it would be a little more efficient to do so only once, before the task is used, I imagine!
1 Like
The suggestion made by Thaumaturge should indeed be the most efficient solution to this particular problem.
To be honest, the alternatives I suggested are more for selecting objects than for finding exact coordinates, but it might still be useful to you when you get to implementing interactive object selection.
So here is the promised code:
from panda3d.core import *
from direct.showbase.ShowBase import ShowBase
from direct.showbase.DirectObject import DirectObject
class PickingCamera:
def __init__(self, showbase):
self._mouse_watcher = showbase.mouseWatcherNode
self._main_cam_lens = showbase.camLens
win = showbase.win
self._pixel_color = VBase4()
self._tex = Texture("picking_texture")
self._tex_peeker = None
# self._pos_tex = Texture("pos_texture")
# self._pos_tex_peeker = None
props = FrameBufferProperties()
props.float_color = True
props.set_rgba_bits(32, 32, 32, 8)
props.depth_bits = 16
# props.aux_rgba = 1
self._buffer = bfr = win.make_texture_buffer("picking_buffer",
1, 1,
self._tex,
to_ram=True,
fbp=props)
# bfr.add_render_texture(self._pos_tex, GraphicsOutput.RTM_bind_or_copy,
# GraphicsOutput.RTP_aux_rgba_0)
bfr.clear_color = (0., 0., 0., 0.)
bfr.set_clear_color_active(True)
self._np = showbase.make_camera(bfr)
self.node = node = self._np.node()
lens = node.get_lens()
lens.fov = 1.5 # render only a small portion of the scene near the mouse cursor
node.tag_state_key = "picking_color"
showbase.task_mgr.add(self.__get_pixel_under_mouse, ("get_pixel_under_mouse"))
def __get_pixel_under_mouse(self, task=None):
if not self._mouse_watcher.has_mouse():
return task.cont if task else None
screen_pos = self._mouse_watcher.get_mouse()
far_point = Point3()
self._main_cam_lens.extrude(screen_pos, Point3(), far_point)
self._np.look_at(far_point)
if self._tex_peeker:
self._tex_peeker.lookup(self._pixel_color, .5, .5)
else:
self._tex_peeker = self._tex.peek()
return task.cont if task else None
@property
def pixel_under_mouse(self):
return VBase4(self._pixel_color)
class MyApp(ShowBase):
def __init__(self):
ShowBase.__init__(self)
# set up a light source
p_light = PointLight("point_light")
p_light.set_color((1., 1., 1., 1.))
self.light = self.camera.attach_new_node(p_light)
self.light.set_pos(5., -100., 7.)
self.render.set_light(self.light)
self.picking_cam = PickingCamera(self)
state_np = NodePath("state_np")
state_np.set_texture_off(1)
state_np.set_material_off(1)
state_np.set_light_off(1)
state_np.set_color_scale_off(1)
state_np.set_transparency(TransparencyAttrib.M_none, 1)
packed_color = 1
self.models = {}
for i in range(10):
smiley = self.loader.load_model("smiley")
smiley.reparent_to(self.render)
color_id = f"color{i}"
smiley.set_tag("picking_color", color_id)
smiley.set_x(2. * i - 10.)
self.models[packed_color] = smiley
color = self.get_color_vec(packed_color)
packed_color += 1
state_np.set_color(color, 1)
state = state_np.get_state()
self.picking_cam.node.set_tag_state(color_id, state)
self.accept("mouse1", self.get_picked_model)
def get_color_vec(self, packed_color):
r = packed_color >> 16
g = (packed_color ^ (r << 16)) >> 8
b = packed_color ^ (r << 16) ^ (g << 8)
return LColorf(r, g, b, 255.) / 255.
def get_packed_color(self, color_vec):
r, g, b, a = [int(round(c * 255.)) for c in color_vec]
return r << 16 | g << 8 | b
def get_picked_model(self):
color_vec = self.picking_cam.pixel_under_mouse
packed_color = self.get_packed_color(color_vec)
if packed_color > 0:
print(f"Picked model {packed_color}!")
return self.models[packed_color]
else:
print("No model picked!")
app = MyApp()
app.run()
The code loads multiple smiley models and associates each of them with a unique color. The special “picking camera” then renders each smiley with its associated color, using the “tag state key” technique: when the camera encounters a model node that has a tag whose key equals “picking_color” (the key assigned to Camera.tag_state_key), it check if a state matching its value (e.g. “color1”) is defined for that camera (using set_tag_state) and if so, renders that node with that state.
The picking camera is updated each frame such that it looks in the direction of the mouse cursor. That way, it only needs to render a very small portion of the scene to a single-pixel texture. The color of that pixel is retrieved when left-clicking; if the color is not black, it can be used to identify the model associated with it.
The colors themselves are stored as single integers (“packed colors”), so it is easy to define a new color by incrementing the current packed_color value.
Currently, the code does not provide a way to get the 3D coordinates of the clicked point.
@Thaumaturge Do you have an idea of how to make a fragment shader render to an auxiliary render target? Honestly I have too little experience with shaders to figure it out at the moment.
For example, if i were to add a render target texture like this:
bfr.add_render_texture(self._pos_tex, GraphicsOutput.RTM_bind_or_copy,
GraphicsOutput.RTP_aux_rgba_0)
do you know how I would make the shader render to it?
Thanks!
Let’s see… While I don’t know offhand–it’s not something that I’m familiar with either, I fear!–I think that I recall seeing something about this previously…
Alas, I don’t seem to be finding the reference, but I think that it was something to do with rendering not to a single “color” output, but to an array of outputs…
Perhaps @rdb knows?
That’s OK, at least I know now that the fragment shader can output to an array, thanks for the info!
Sadly I still couldn’t get it to work. Oh well, here’s a revised code that no longer correctly detects the model that was clicked, but it does return the coordinates of the clicked point, so at least it does what was originally requested 
:
EDIT:
OK, I think I figured it out:
from panda3d.core import *
from direct.showbase.ShowBase import ShowBase
from direct.showbase.DirectObject import DirectObject
VERT_SHADER = """
#version 130
uniform mat4 p3d_ModelViewProjectionMatrix;
uniform mat4 p3d_ModelMatrix;
in vec4 p3d_Vertex;
out vec4 pos;
void main() {
gl_Position = p3d_ModelViewProjectionMatrix * p3d_Vertex;
pos = p3d_ModelMatrix * p3d_Vertex;
}
"""
FRAG_SHADER = """
#version 130
uniform vec4 p3d_Color;
in vec4 pos;
out vec4 p3d_fragData[2];
void main() {
// output the vertex coordinates as a color value
p3d_fragData[0] = pos;
p3d_fragData[1] = p3d_Color;
}
"""
class PickingCamera:
def __init__(self, showbase):
self._mouse_watcher = showbase.mouseWatcherNode
self._main_cam_lens = showbase.camLens
self._win = showbase.win
self._pixel = (VBase4(), VBase4())
self._col_tex = Texture("picking_texture")
self._col_peeker = None
self._pos_tex = Texture("pos_texture")
self._pos_peeker = None
props = FrameBufferProperties()
props.float_color = True
props.set_rgba_bits(32, 32, 32, 8)
props.depth_bits = 16
props.aux_rgba = 1
self._buffer = bfr = self._win.make_texture_buffer("picking_buffer",
1, 1,
self._pos_tex,
to_ram=True,
fbp=props)
bfr.add_render_texture(self._col_tex, GraphicsOutput.RTM_copy_ram,
GraphicsOutput.RTP_aux_rgba_0)
bfr.clear_color = (0., 0., 0., 0.)
bfr.set_clear_color_active(True)
bfr.set_clear_active(3, True)
self._np = showbase.make_camera(bfr)
self.node = node = self._np.node()
self._lens = lens = node.get_lens()
lens.fov = 1.5 # render only a small portion of the scene near the mouse cursor
node.tag_state_key = "picking_color"
showbase.task_mgr.add(self.__get_pixel_under_mouse, ("get_pixel_under_mouse"))
def __get_pixel_under_mouse(self, task=None):
if not self._mouse_watcher.has_mouse():
return task.cont if task else None
screen_pos = self._mouse_watcher.get_mouse()
far_point = Point3()
self._main_cam_lens.extrude(screen_pos, Point3(), far_point)
self._np.look_at(far_point)
if self._col_peeker:
self._col_peeker.lookup(self._pixel[0], .5, .5)
else:
self._col_peeker = self._col_tex.peek()
if self._pos_peeker:
self._pos_peeker.lookup(self._pixel[1], .5, .5)
else:
self._pos_peeker = self._pos_tex.peek()
return task.cont if task else None
@property
def pixel_under_mouse(self):
x, y, z, _ = self._pixel[1]
return (LColorf(self._pixel[0]), Point3(x, y, z))
class MyApp(ShowBase):
def __init__(self):
ShowBase.__init__(self)
# set up a light source
p_light = PointLight("point_light")
p_light.set_color((1., 1., 1., 1.))
self.light = self.camera.attach_new_node(p_light)
self.light.set_pos(5., -100., 7.)
self.render.set_light(self.light)
self.picking_cam = PickingCamera(self)
shader = Shader.make(Shader.SL_GLSL, VERT_SHADER, FRAG_SHADER)
state_np = NodePath("state_np")
state_np.set_shader(shader, 1)
state_np.set_texture_off(1)
state_np.set_material_off(1)
state_np.set_light_off(1)
state_np.set_color_scale_off(1)
state_np.set_transparency(TransparencyAttrib.M_none, 1)
packed_color = 1
self.models = {}
for i in range(10):
smiley = self.loader.load_model("smiley")
smiley.reparent_to(self.render)
color_id = f"color{i}"
smiley.set_tag("picking_color", color_id)
smiley.set_x(2. * i - 10.)
self.models[packed_color] = smiley
color = self.get_color_vec(packed_color)
packed_color += 1
state_np.set_color(color, 1)
state = state_np.get_state()
self.picking_cam.node.set_tag_state(color_id, state)
self.accept("mouse1", self.get_picked_model)
def get_color_vec(self, packed_color):
r = packed_color >> 16
g = (packed_color ^ (r << 16)) >> 8
b = packed_color ^ (r << 16) ^ (g << 8)
return LColorf(r, g, b, 255.) / 255.
def get_packed_color(self, color_vec):
r, g, b, a = [int(round(c * 255.)) for c in color_vec]
return r << 16 | g << 8 | b
def get_picked_model(self):
pixel = self.picking_cam.pixel_under_mouse
color_vec = pixel[0]
pos = pixel[1]
packed_color = self.get_packed_color(color_vec)
if packed_color > 0:
print(f"Point {pos} on model {packed_color}")
return self.models[packed_color], pos
else:
print("No model picked!")
app = MyApp()
app.run()
It seems that FrameBufferProperties.aux_rgba defines the number of auxiliary bitplanes added to the buffer, so it needs to be set to 1 in this case.
As I couldn’t find a way to set the type of the GraphicsOutput.RTP_aux_rgba_0 values to float, the coordinates are rendered to the regular color bitplane instead, while the model color is rendered to the auxiliary bitplane.
Anyway, the code now allows getting the color ID of the picked model, as well as the coordinates of the clicked point 
.
1 Like