Help with a basic problem of 3D point in world space to 2D pixel in image space

I have been confused for this problem for a long time.
How do I get exact pixel position in window client area which is projected from a 3D point in a model loaded by Panda3D with OrthographicLens?

Here is my model loading and camera setting code. Now I have a 3D point (-0.2, 0.21, 0.5). What is its exact coordinates of pixel on a screen window in OrthographicLens?

class PandaShotMachine(ShowBase):

def __init__(self):
    ShowBase.__init__(self)

    self.windowhalfedge=768
    self.openDefaultWindow(size=[self.windowhalfedge,self.windowhalfedge]);
    base.disableMouse()
    
    self.running=True

def LoadAndConfigModel(self, filepath: str):
    self.setBackgroundColor(0.0,0.0,0.0,1.0)

    winfile=filepath
    pandafile = Filename.fromOsSpecific(winfile)
    
    # Load the environment model.
    self.scene = self.loader.loadModel(pandafile, noCache=True)        

    self.SetCameraAndLight(0.0, False, False);

    # Reparent the model to render.
    self.scene.reparentTo(self.render)

def SetCameraAndLight(self, centerrotateangle: float, isconceptual: bool, perspective: bool):
    c=self.scene.getBounds().getCenter()
    r=self.scene.getBounds().getRadius()
    edge=self.scene.getTightBounds()[1][0]-self.scene.getTightBounds()[0][0]
    if edge<self.scene.getTightBounds()[1][2]-self.scene.getTightBounds()[0][2]:
        edge=self.scene.getTightBounds()[1][2]-self.scene.getTightBounds()[0][2]
    height=self.scene.getTightBounds()[1][1]-self.scene.getTightBounds()[0][1]
    focus=c
    eyepos=LVector3f(c[0]+r,c[1]+r,c[2]+r)
    filmedge=r
    lightmag=0.1
    #print([edge, height])
    if edge*2<height:
        if perspective:
            focus=LVector3f(c[0],self.scene.getTightBounds()[1][1]-edge,c[2])
            filmedge=height-edge/1.414
        er=log(height/edge/2)+1
        eyepos=LVector3f(focus[0]+edge*er,focus[1]+edge*er,focus[2]+edge*er)
        lightmag=0.2

    cr=eyepos[0]-focus[0]
    eyepos=LVector3f(focus[0]+cr*cos(centerrotateangle*pi/180), focus[1]+cr, focus[2]+cr*sin(centerrotateangle*pi/180))

    #self.scene.showTightBounds()
    #self.scene.showBounds()

    if perspective:
        lens = PerspectiveLens()
        lens.setFov(60)
        lens.setViewVector(0.0,0.0,0.0,0.0,1.0,0.0)
        base.cam.node().setLens(lens)
    else:
        lens = OrthographicLens()
        lens.setFar(1.732*2*r)
        lens.setNear(0.0)
        lens.setViewVector(0.0,0.0,0.0,0.0,1.0,0.0)
        lens.setFilmSize(2*filmedge, 2*filmedge)  # Or whatever is appropriate for your scene            
        base.cam.node().setLens(lens)

    self.camera.setPos(eyepos)
    self.camera.lookAt(focus, LVector3f(0.0,1.0,0.0))
    hpr=self.camera.getHpr()#ļ¼ļ¼ļ¼ļ¼
    self.projectfocus=focus
    self.projectcameraplanecenter=eyepos
    self.projecthalfedge=filmedge

if name == ā€˜mainā€™:
try:
winfile = uā€™D:/MultiWheel_HATS.objā€™

    app = PandaShotMachine()
    app.LoadAndConfigModel(winfile)
    app.SetCameraAndLight(60.0, False, False)

    while app.running:
        if not app.win.getProperties().getOpen():
            app.running=False
            break
        app.taskMgr.step()

    app.destroy()
except Exception as e:
    print(e)

Greetings, and welcome to the forum! I hope that you find your time here to be positive! :slight_smile:

As to your question: I think that you might want the project method of the Lens class, followed by a conversion of the result into the relevant space.

Specifically, the ā€œprojectā€ method will return the 2D point corresponding to a 3D point, and does so in the range (-1, 1).

This corresponds well to the space of ā€œrender2dā€ā€“but thatā€™s not quite what youā€™re looking for.

However, Panda3D also has a 2D root-node called ā€œpixel2dā€, which operates, well, in the space of window-pixels. Thus, if we convert the point from the space of ā€œrender2dā€ to the space of ā€œpixel2dā€, we should get the sort of output that you want!

Something like the following:

# Given a point in 3D space named "myPoint"...

lens = base.camNode.getLens()
resultPoint = Point2()
success = lens.project(myPoint, resultPoint)
if success:
    finalPoint = base.pixel2d.getRelativePoint(base.render2d, Point3(resultPoint.x, 0, resultPoint.y))
    # ("getRelativePoint" expects a Point3, not a Point2)

You may also want to convert the z-coordinate of the resultā€“which will start with 0 at the top of the screen and run to -screenHeight at the bottom of the screen, I believeā€“into a positive range.

Thank you very much for your kind reply! You do let me know something important.

But there are still problems.

My camera is intended to rotate around the object. The window looks like this.

So I think the point should be transferred into camera space first. Or the output will be fixed into one result, though the camera is rotated. Hence, I wrote this, in which A is the point (e.g., -0.200000002980,0.209999993443,0.000000000000)

    cameraToWorld=self.camera.getMat()
    ctw_inverse=LMatrix4f()
    ctw_inverse.invertFrom(cameraToWorld)
    pp=LVector4f(A[0],A[1],A[2],1);
    pCamera=LVector4f()
    for i in range(4):
        pCamera[i]=pp.dot(ctw_inverse.getCol(i))

    pCamera=LPoint3f(pCamera[0],pCamera[1],pCamera[2])
    lens = base.camNode.getLens()
    resultPoint = LPoint3f()
    success = lens.project(pCamera, resultPoint)
    if success:
        finalPoint = base.pixel2d.getRelativePoint(base.render2d, LPoint3f(resultPoint.x, 0, resultPoint.z))
        print(finalPoint[0],finalPoint[1], finalPoint[2])

I really donā€™t know whether such operation leads to adverse consequence. It works on no points but the focus point (the program output (384,-384) for the focus point the camera looking at). Let me use an image to explain more clear.

The red circle indicates the location of my dear A point. The above program outputs (364,-384) as below figure shown

image

Itā€™s apparently wrong, since the window client has 768*768 size. The coordinate should be around (291, 388).

Please help me with that. Itā€™s really wired.

Oh wow, there are much simpler ways to do this, I daresay! ^^;

If I may suggest it, you could just add a new node at the desired position of rotation, attach the camera to that node, and then rotate the node, I believe.

Something like this:

# Given a point (x, y, z) around which you want to rotate...

# Create a node, attach it to "render", and set its position:
self.rotationNodePath = NodePath(PandaNode("rotation node"))
self.rotationNodePath.reparentTo(render)
self.rotationNodePath.setPos(x, y, z)
# (Those first two lines could be condensed to:
#  self.rotationNodePath = render.attachNewNode(PandaNode("rotation node"))

# Now, attach the camera. We're using "wrtReparentTo" in order
# to preserve the camera's relative transformation as we do so
self.camera.wrtReparentTo(self.rotationNodePath)

# Now rotate the camera. This would presumably be more animated
# in actual usage, but for the purposes of demonstration I'm
# keeping it simple
self.rotationNodePath.setH(20)

Iā€™mā€¦ not sure that I follow what youā€™re saying here. Could you clarify, please?

I will be honest: it may be that Iā€™m tired (which I am), but Iā€™m not managing to follow what that program is intended to do. Could you walk me through the logic, please?

Thanks for the suggestion. Maybe I was not so clear about what my program do with Panda3D. I list my code at the last paragragh.

My program first load a model (.obj file with texture), and then create a camera looking at its geographic center (or exactly its bounding sphere center). The window it create with Panda3D can accept R key to set new position of the camera around the model (the camera is always looking at its bounding sphere center, just like the figure shown. I rotate three times). The model does not move.




I indeed add a function ProjectToCamera to print the 2D pixel coordinate projected from any 3D point in the model, after each time rotating the camera. For the point A (-0.200000002980,0.209999993443,0.000000000000), it print the following results.

image

A is located in the screenshot with red circle. It is actually the toppest point of the protrusion on the vehicle. That is how I find it in the screenshots (^-^). I want to know how to obtain its corresponding pixel coordinate in the screenshots.

The following is my complete code,. It can run directly in Python. Donā€™t laugh at me. My Pyhton skill sucks. If I remove the first piece of code in ProjectToCamera by setting A to pCamera, the results are shown as the following figure.

image

from direct.showbase.ShowBase import ShowBase
from panda3d.core import Filename
from panda3d.core import LVector3f, LVector4f, LMatrix4f, LVector2d, LPoint3f, LPoint2f, LPlanef
from panda3d.core import PerspectiveLens, OrthographicLens
from panda3d.core import PointLight, AmbientLight, DirectionalLight, AntialiasAttrib, Material
from direct.task import Task
from math import sin, cos, pi, log, floor

class PandaShotMachine(ShowBase):

def __init__(self):
    ShowBase.__init__(self)

    self.windowhalfedge=768
    self.openDefaultWindow(size=[self.windowhalfedge,self.windowhalfedge]);
    base.disableMouse()

    self.accept('r-up', self.Orbit)
    
    # Exit on pressing the escape button.
    self.accept('escape', self.StopRunning)
    self.running=True
    self.orbitangle=0.0

def StopRunning(self):
    self.running=False

def LoadAndConfigModel(self, filepath: str):
    self.setBackgroundColor(0.0,0.0,0.0,1.0)

    winfile=filepath
    pandafile = Filename.fromOsSpecific(winfile)
    
    # Load the environment model.
    self.scene = self.loader.loadModel(pandafile, noCache=True)
    
    # Apply scale and position transforms on the model.
    #self.scene.setScale(0.25, 0.25, 0.25)
    #self.scene.setPos(0, 0, 0)

    self.SetCameraAndLight(0.0, False, False);

    # Reparent the model to render.
    self.scene.reparentTo(self.render)

def SetCameraAndLight(self, centerrotateangle: float, isconceptual: bool, perspective: bool):
    c=self.scene.getBounds().getCenter()
    r=self.scene.getBounds().getRadius()
    edge=self.scene.getTightBounds()[1][0]-self.scene.getTightBounds()[0][0]
    if edge<self.scene.getTightBounds()[1][2]-self.scene.getTightBounds()[0][2]:
        edge=self.scene.getTightBounds()[1][2]-self.scene.getTightBounds()[0][2]
    height=self.scene.getTightBounds()[1][1]-self.scene.getTightBounds()[0][1]
    focus=c
    eyepos=LVector3f(c[0]+r,c[1]+r,c[2]+r)
    filmedge=r
    lightmag=0.1
    #print([edge, height])
    if edge*2<height:
        if perspective:
            focus=LVector3f(c[0],self.scene.getTightBounds()[1][1]-edge,c[2])
            filmedge=height-edge/1.414
        er=log(height/edge/2)+1
        eyepos=LVector3f(focus[0]+edge*er,focus[1]+edge*er,focus[2]+edge*er)
        lightmag=0.2

    **# Rotate camera**
    cr=eyepos[0]-focus[0]
    eyepos=LVector3f(focus[0]+cr*cos(centerrotateangle*pi/180), focus[1]+cr, focus[2]+cr*sin(centerrotateangle*pi/180))

    if perspective:
        lens = PerspectiveLens()
        lens.setFov(60)
        lens.setViewVector(0.0,0.0,0.0,0.0,1.0,0.0)
        base.cam.node().setLens(lens)
    else:
        lens = OrthographicLens()
        lens.setFar(1.732*2*r)
        lens.setNear(0.0)
        lens.setViewVector(0.0,0.0,0.0,0.0,1.0,0.0)
        lens.setFilmSize(2*filmedge, 2*filmedge)  # Or whatever is appropriate for your scene            
        base.cam.node().setLens(lens)

    self.camera.setPos(eyepos)
    self.camera.lookAt(focus, LVector3f(0.0,1.0,0.0))
    hpr=self.camera.getHpr()#ļ¼ļ¼ļ¼ļ¼
    self.projectfocus=focus
    self.projectcameraplanecenter=eyepos
    self.projecthalfedge=filmedge

    #光ęŗ
    self.render.clearLight()
    dlight_direction = LVector3f(hpr[0], hpr[1], hpr[2]) if isconceptual else LVector3f(0.0,180.0,0.0) #LVector3f(-hpr[0], hpr[1], -hpr[2])
    plight_position = LVector3f(focus[0]*2-eyepos[0],eyepos[1],focus[2]*2-eyepos[2]) if isconceptual else eyepos #LVector3f(eyepos[0]+r,eyepos[1]+r,eyepos[2]+r)
                    
    dlight = DirectionalLight('dlight')
    dlight.setColor((1.0, 1.0, 1.0, 1) if not isconceptual else (0.2,0.2,0.2,1.0))
    dlnp = self.render.attachNewNode(dlight)
    #print(dlight_direction)
    dlnp.setHpr(dlight_direction)
    self.render.setLight(dlnp)

    #dlight = DirectionalLight('dlight2')
    #dlight.setColor((0.2, 0.2, 0.2, 1))
    #dlnp = self.scene.attachNewNode(dlight)
    ##dlnp.setHpr(-hpr[0], hpr[1], -hpr[2])
    #dlnp.setHpr(hpr)
    #self.scene.setLight(dlnp)

    plight = PointLight('plight')
    plight.setColor((lightmag, lightmag, lightmag, 1))
    plight.attenuation=(0.1, 0.0, 0)
    plight.setMaxDistance(1.732*2*r)
    plnp = self.render.attachNewNode(plight)
    plnp.setPos(plight_position)
    self.render.setLight(plnp)

    plight = PointLight('plight2')
    plight.setColor((0.6*lightmag, 0.6*lightmag, 0.6*lightmag, 1))
    plight.attenuation=(0.1, 0.0, 0)
    plight.setMaxDistance(1.732*2*r)
    plnp = self.render.attachNewNode(plight)
    plnp.setPos(focus[0]*2-eyepos[0],eyepos[1],focus[2]*2-eyepos[2])
    self.render.setLight(plnp)

    alight = AmbientLight('alight')
    alight.setColor((0.2, 0.2, 0.2, 1))
    alnp = self.render.attachNewNode(alight)
    self.render.setLight(alnp)

def Orbit(self):
    self.orbitangle+=15.0
    self.SetCameraAndLight(self.orbitangle, False, False)
    res=self.ProjectToCamera(LPoint3f(-0.200000002980, 0.209999993443, 0))
    self.ProjectToCamera(LPoint3f(0.0, 1.30704, 0))

def ProjectToCamera(self, A: LPoint3f):
    cameraToWorld=self.camera.getMat()
    #print(len.getFov())
    ctw_inverse=LMatrix4f()
    ctw_inverse.invertFrom(cameraToWorld)
    pp=LVector4f(A[0],A[1],A[2],1);
    pCamera=LVector4f()
    for i in range(4):
        pCamera[i]=pp.dot(ctw_inverse.getCol(i))

    pCamera=LPoint3f(pCamera[0],pCamera[1],pCamera[2])
    lens = base.camNode.getLens()
    resultPoint = LPoint3f()
    success = lens.project(pCamera, resultPoint)
    if success:
        finalPoint = base.pixel2d.getRelativePoint(base.render2d, LPoint3f(resultPoint.x, 0, resultPoint.z))
        print(finalPoint[0],finalPoint[1], finalPoint[2])

if name == ā€˜mainā€™:
try:
winfile = uā€™path to any obj.objā€™

    app = PandaShotMachine()
    app.LoadAndConfigModel(winfile)
    app.SetCameraAndLight(0.0, False, False)

    while app.running:
        if not app.win.getProperties().getOpen():
            app.running=False
            break
        app.taskMgr.step()

    app.destroy()
except Exception as e:
    print(e)

Thank you for the explanation! :slight_smile:

Hmmā€¦ In that case, I stand by my original suggestion: A separate node, placed at the desired ā€œtarget-pointā€, to which the camera is attached, and which is then rotated to rotate the camera around that point.

However, it looks like you have the feature working, so well and good! :slight_smile:

Regarding the position of a given point on the screen, is that goal separate from the goal of rotating the camera around the object?

Looking at what you have, do I take it correctly that the value of ā€œAā€, as provided to the ā€œProjectToCameraā€ method, is a point in world-space? That is, that it doesnā€™t change at all?

If so, then the code that I gave in my first response should, I think, largely workā€¦

ā€¦ Except for, looking at some code of my own, one thing that I seem to keep forgetting: the point has to be transformed into the space of the camera, first.

Iā€™m guessing that this is what the matrix-related code in your ā€œProjectToCameraā€ method is intended to do.

Iā€™m not sure of why that matrix-related code isnā€™t workingā€“Iā€™m not all that familiar with that side of thingsā€“but it can be done much more simply, I believe, via another call to ā€œgetRelativePointā€.

Iā€™ve thus modified your ā€œProjectToCameraā€ method, like so:

	def ProjectToCamera(self, A: LPoint3f):
		lens = base.cam.node().getLens()
		resultPoint = LPoint2f()
		relativePoint = base.cam.getRelativePoint(render, A) # <--Note!
		success = lens.project(relativePoint, resultPoint)
		if success:
		    # Note that I also have this call below using x and y, not z
		    finalPoint = base.pixel2d.getRelativePoint(base.render2d, LPoint3f(resultPoint.x, 0, resultPoint.y))
		    print(finalPoint[0],finalPoint[1], finalPoint[2])

Try it out, and see whether that produces output thatā€™s more accurate!

Tanks for your patient.

I have tried your code. It really works as my matrix trick code does. Thank you. It is much simpler.

I have found that I gave the wrong coordinates of A to the program. It should be ((0.0, 2.31, 1.05)). Sorry for that. However, though I use the correct data now, the results are still partly wrong.

Here are the results. The X coordinates are relatively correct. But the Y coordinates are quite confusing.

image

The right X-Y pairs should be about
238 283
246 320
264 350
297 374
336 390
384 395
434 387
471 370

Is that something more we can do?

Here is my ProjectToCamera function. It is the same to that you wrote.

def ProjectToCamera(self, A: LPoint3f):
pCamera=A
lens = self.camNode.getLens()
relativePoint=base.camera.getRelativePoint(self.render, pCamera)
resultPoint = LPoint3f()
success = lens.project(relativePoint, resultPoint)

    if success:
        finalPoint = base.pixel2d.getRelativePoint(base.render2d, LPoint3f(resultPoint.x, resultPoint.y, resultPoint.z))
        print(finalPoint[0], finalPoint[2])
1 Like

:smiley:
Finally, I have figured out how the problem is solved, though there is some trivial calculation errors.

I used very old school way, ie., 3D analytic geometry and the LPanef.project function. In detail, I project my dearest point A to camera plane, and calculate the distances between projected A and camera plane center (aka. window client area center) from classic 2D arises. By adding window size as offsets, I obtain the exact coordinates of pixel corresponding to A.

Thank you very much for your patience and kindness. I got to know many things that I have never learned before.

The code of function ProjectToCamera is as follows.

def ProjectToCamera(self, A: LPoint3f):       

    S=self.projectfocus # The point self.camera looking at
    C=self.projectcameraplanecenter # self.camera.getPos
    focusline=S-C
    cplane= LPlanef(focusline,C)
    pA=cplane.project(A)

    #real=LPoint3f()
    lens= base.cam.node().getLens()
    #lens.project(pA, real)
    r=lens.getFilmSize()[0]/2
    edge=self.windowhalfedge
    real=pA

    focusline=LVector3f(focusline)
    focusline=focusline.normalized()
    alpha=focusline.angleDeg(LVector3f(0,1.0,0))-90
    YY=-(real[1]-C[1])/cos(alpha*pi/180)

    (x,y,z)=pA
    (xC,yC,zC)=C
    XX=(x-xC)**2 + (y-yC)**2 + (z-zC)**2-YY**2
    if abs(XX)<=0.000001:
        XX=0
    elif XX<0:
        return None
    XX=XX**0.5

    Xaris=focusline.cross(LVector3f(0,1.0,0))
    Xaris=Xaris.normalized()
    pCA=LVector3f(x-xC,0,z-zC)
    pCA=pCA.normalized()
    if Xaris.angleDeg(pCA)>90:
        XX=-XX

    print([edge/2+XX*edge/r/2, edge/2+YY*edge/r/2])
1 Like

Iā€™m glad that you got it working, and for my part, itā€™s my pleasure! :slight_smile:

Looking at your code, your version isnā€™t quite the same as mine, and perhaps the discrepancy lies there.

To be specific, in your version, the calculation of ā€œfinalPointā€ looks like this:

finalPoint = base.pixel2d.getRelativePoint(base.render2d, LPoint3f(resultPoint.x, resultPoint.y, resultPoint.z))

Whereas in my version, it looks like this:

finalPoint = base.pixel2d.getRelativePoint(base.render2d, LPoint3f(resultPoint.x, 0, resultPoint.y))

That is, in my version Iā€™m not using the z-coordinate from the result of ā€œlens.projectā€, and instead am using the y-coordinate as the z-coordinate in the point being passed to ā€œgetRelativePointā€.

I tried your code. It works!!! Thank you!!

Have a nice day!

image

1 Like

Itā€™s very much my pleasure! And you have a good day, too! :slight_smile: