Hi,
I’m aiming to get Panda3D to duplicated my OpenCV/computer vision style camera parameterisation (the result of calibrating a real-world camera) so that Panda can be used for rendering things onto images captured by the camera. Basically what is done with augmented reality, but more for the purposes of debugging and visualising the results of various computer vision algorithms.
In computer vision, we commonly represent the projection of a point from a world coordinate system to the image by a transformation such as:
pi = P.L.pw
where:
pw: point in world space
L : transformation that takes point from world space to camera space
P : 3x4 projection matrix [K|0], where K is a 3x3 “camera calibration” matrix and the right hand column of P is all 0s.
K normally looks like this:
f s cx
0 a.f cy
0 0 1
and has f (focal length), s (skew, often 0), (cx,cy) principle point (point in the image where the optical axis intersects the image plane), and a (aspect ratio, often assumed to be 1.0 )
Generally, the image coordinate system is top-left origin, x positive to the right, and y positive down. The camera coordinate system is the same as the image, with z positive in the direction the camera looks (“into the screen”), making a right handed coordinate system. The world coordinate system is again right handed, and rarely standardised but often the z=0 plane would be the ground.
Allowing for homogeneous division, P projects directly from 3D to the 2D image.
Given that I can get suitable calibration for my cameras using OpenCV (calibration provides P and L), I want to now be able to make a camera in Panda that will duplicate my real-world cameras.
I can do this in OpenGL by specifying the projection matrix as:
2*K(0,0)/w, 2*K(0,1)/w, ( w - 2*K(0,2))/w , 0,
0, 2*K(1,1)/h, ( h - 2*K(1,2))/h , 0,
0, 0, (-far - near)/(far-near) , -2*far*near/(far - near),
0, 0, -1, 0 ;
(where w and h are the image width and height, far and near are opengl clipping planes, K is the K matrix I described above)
And ensuring that there is a 180 rotation around the x-axis before projection (openGL having a camera coordinate system that is right-handed, y-up, z-negative i.e. 180 rotation from my calibrated camera system).
Can anyone advise me what I need to do to enable me to do this with Panda? (I’m relatively new to Panda, having used it a few years ago. I’ve been just using OpenGL directly for a while, but would now like to avoid being so low level for new visualisations and tools).
I suspect that what I need to know is what the projection matrix should be trying to achieve. Does it project directly to the image, or, like OpenGL, does it project to some “normalised device coordinate” system first?