Integrating Panda3d in PySide2/Qt5


#1

I’ve been working on embedding a Panda3D window inside a PySide2/Qt5 widget. There’s a previous effort here that’s a bit aged:

However since that time, Qt4 has changed to Qt5, the Qt OpenGL interface class has changed to QOpenGLWidget and the previous solution deprecated, and Panda3D has moved into preferring the ShowBase class. So time for an update. I tried working from the snippets posted above but ended up more relying on WxPandaWindow.py and trying to infer the differences required in PySide2.

The main issues that I’ve not solved yet:

  1. Cannot resize. I try setting WindowProperties as follows:

     wp = WindowProperties()
     pos = self.pos()
     wp.setOrigin(pos.x(), pos.y())
     wp.setSize(width, height)
     self.base.win.requestProperties(wp)
    

It appears to change for one frame which is only partially rendered, and then back to the
original size.
2. Not sure how to propagate mouse events to Panda3D in a nice manner.
3. Any required cleanup on application exit?
4. Sometimes I get janky frames where it looks to me that the frame buffer is being swapped unexpectedly.

The provided snippet here is a port of the samples/shader_terrain module, so you will have to put the file in that directory to allow it to find the required assets. The other requirement is pyside2, which can be installed via pip install pyside2 :

from typing import Tuple
import logging
logging.basicConfig(level=logging.DEBUG, datefmt='%H:%M')
log = logging.getLogger('p3d_qt')

import os, sys, platform
from functools import partial

from PySide2 import QtGui as qg, QtCore as qc, QtWidgets as qw
from PySide2.QtCore import Qt

import OpenGL  # install as `pyopengl`
OpenGL.ERROR_CHECKING = True

# Panda imports
import panda3d.core as p3c
from panda3d.core import Thread
from panda3d.core import WindowProperties, CallbackGraphicsWindow, Texture

# Set up Panda environment
from direct.showbase.ShowBase import ShowBase
from panda3d.core import ShaderTerrainMesh, Shader, load_prc_file_data
from panda3d.core import SamplerState
from panda3d.core import KeyboardButton, MouseButton

class MainWindow(qw.QMainWindow):

    def __init__(self):
        qw.QMainWindow.__init__(self)

        self.setupUi()

    def setupUi(self):
        splitter = qw.QSplitter(parent=self)
        self.setCentralWidget(splitter)

        # Sidebar
        # -------
        sidebar = qw.QWidget()
        sidebar.setFixedWidth(200)
        splitter.addWidget(sidebar)

        self.sideForm = qw.QFormLayout(parent=sidebar)
        sidebar.setLayout(self.sideForm)

        cbWireframe = qw.QCheckBox('Wireframe', parent=sidebar)
        self.sideForm.addRow('', cbWireframe)

        # Panda3D
        # -------
        self.view3d = ShaderTerrainWidget(parent=self)
        splitter.addWidget(self.view3d)

        # Menubar
        # -------
        menubar = qw.QMenuBar(self)
        self.setMenuBar(menubar)

        menuFile = qw.QMenu(menubar)
        menuFile.setTitle('File')
        actionQuit = qw.QAction(self)
        actionQuit.setText('Quit')
        actionQuit.triggered.connect(self.quitApp)
        menuFile.addAction(actionQuit)

        menuHelp = qw.QMenu(menubar)
        menuHelp.setTitle('Help')

        menubar.addAction(menuFile.menuAction())
        menubar.addAction(menuHelp.menuAction())

        # Status Bar
        # ==========
        self.statusbar = qw.QStatusBar(parent=self)
        self.setStatusBar(self.statusbar)

        statusWidget = qw.QWidget(parent=self)
        statusLayout = qw.QHBoxLayout(statusWidget)
        statusLayout.setContentsMargins(5, 0, 5, 3)
        self.statusbar.addPermanentWidget(statusWidget)

        self.progressbar  = qw.QProgressBar(parent=self)
        self.progressbar.setGeometry(30, 40, 100, 25)
        self.progressbar.setFormat('%p %')
        self.progressbar.reset()
        statusLayout.addStretch()
        statusLayout.addWidget(self.progressbar)

        # Connect signals
        # ===============
        cbWireframe.toggled.connect(self.view3d.base.toggleWireframe)

        self.resize(1920, 1080)
        self.setWindowTitle('Panda3D embedded in PySide2/Qt5')

    def quitApp(self):
        log.info('Shutting down...')
        del self.view3d
        Q_APP.exit() 


class ShaderTerrainWidget(qw.QOpenGLWidget):
    """
    Here we extend the `panda3d/samples/shader_terrain/main.py` module to be 
    embedded as a `QtWidgets.QOpenGLWidget` inside a Qt5/PySide2 application.

    An effort was made very long ago to embed Panda3D in PyQt4:

        https://discourse.panda3d.org/t/panda-in-pyqt/3964

    However, since then both the API for Panda3D has changed to use `ShowBase`
    and the API for Qt5 has changed to use `QOpenGLWidget` (instead of the 
    deprecated `QGLWidget`). 
    
    Therefore this effort is mostly from scratch. The model is the 
    `direct/wxwidgets` module, but there are substantial differences in the 
    programming model of Wx and Qt5. The general desire, however, is to disable 
    the Panda3D event loop and use the Qt main GUI thread event loop to trigger 
    rendering and buffer swaps.

    The relevant Qt classes are:

    QtWidget.QOpenGLWidget:
        https://doc-snapshots.qt.io/qtforpython/PySide2/QtWidgets/QOpenGLWidget.html
    QtGui.QOpenGLContext:
        https://doc-snapshots.qt.io/qt5-5.11/qopenglcontext.html
    QtGui.QSurface:
        https://doc-snapshots.qt.io/qt5-5.11/qsurface.html

    Here I'm using the Qt5 C++ docs instead of the PySide2 docs as they are 
    considerably more complete for the OpenGL side.

    In terms of Panda3D classes, I recommend looking at the source as the API 
    docs are not illuminating.

    direct.wxwidgets.WxPandaWindow:
        https://github.com/panda3d/panda3d/blob/master/direct/src/wxwidgets/WxPandaWindow.py

    Main issues:

    1. Cannot resize. Or it does but then the frame buffer changes back again afterward?
    2. Not sure how to propagate mouse events to Panda3D in a nice manner.
    3. TODO: any required cleanup?
    4. Collision detection. See for example `direct/src/controls/ObserverWalker.py`
    """

    def __init__(self, parent=None):
        qw.QOpenGLWidget.__init__(self, parent=parent)

        # Load some configuration variables, its important for this to happen
        # before the ShowBase is initialized
        load_prc_file_data("", """
            window-type offscreen
            load-display pandagl
            gl-coordinate-system default
            show-frame-rate-meter #t
            gl-debug #t
            """)

        self.visible = False
        self.base = ShowBase()

        # Initialize OpenGL in Qt
        self.initializeGL()

        # Can't share the GSG when a new wxgl.GLContext is created automatically.
        gsg = None

        if not self.base.pipe:
            self.base.makeDefaultPipe()
        pipe = self.base.pipe
        if pipe.getInterfaceName() != 'OpenGL':
            self.base.makeAllPipes()
            for pipe in self.base.pipeList:
                if pipe.getInterfaceName() == 'OpenGL':
                    break
        if pipe.getInterfaceName() != 'OpenGL':
            raise Exception("Couldn't get an OpenGL pipe.")

        callbackWindowDict = {
                'Events' : self._eventsCallback,
                'Properties' : self._propertiesCallback,
                'Render' : self._renderCallback,
            }
        self.base.win = self.base.openWindow(callbackWindowDict=callbackWindowDict, 
                pipe=pipe, gsg=gsg, type='onscreen')
        self.hasCapture = False
        self.inputDevice = None
        if hasattr(self.base.win, 'getInputDevice'):
            self.inputDevice = self.base.win.getInputDevice(0)
        else:
            log.info(f'WARNING: no `getInputDevice` in {self.base.win}')

        self.base.setFrameRateMeter(True)
        # Increase camera FOV as well as the far plane
        self.base.camLens.set_fov(110)
        self.base.camLens.set_near_far(0.1, 50000)

        # Construct the terrain
        self.base.terrain_node = ShaderTerrainMesh()

        # Set a heightfield, the heightfield should be a 16-bit png and
        # have a quadratic size of a power of two.
        # TODO generate with `pyfastnoisesimd`
        heightfield = self.base.loader.loadTexture("heightfield.png")
        heightfield.wrap_u = SamplerState.WM_clamp
        heightfield.wrap_v = SamplerState.WM_clamp
        self.base.terrain_node.heightfield = heightfield

        # Set the target triangle width. For a value of 10.0 for example,
        # the terrain will attempt to make every triangle 10 pixels wide on screen.
        self.base.terrain_node.target_triangle_width = 10.0

        # Generate the terrain
        self.base.terrain_node.generate()

        # Attach the terrain to the main scene and set its scale. With no scale
        # set, the terrain ranges from (0, 0, 0) to (1, 1, 1)
        self.base.terrain = self.base.render.attach_new_node(self.base.terrain_node)
        self.base.terrain.set_scale(1024, 1024, 100)
        self.base.terrain.set_pos(-512, -512, -70.0)

        # Set a shader on the terrain. The ShaderTerrainMesh only works with
        # an applied shader. You can use the shaders used here in your own application
        terrain_shader = Shader.load(Shader.SL_GLSL, 'terrain.vert.glsl', 'terrain.frag.glsl')
        self.base.terrain.set_shader(terrain_shader)
        self.base.terrain.set_shader_input('camera', self.base.camera)

        # Set some texture on the terrain
        grass_tex = self.base.loader.loadTexture('textures/grass.png')
        grass_tex.set_minfilter(SamplerState.FT_linear_mipmap_linear)
        grass_tex.set_anisotropic_degree(4)
        self.base.terrain.set_texture(grass_tex)

        # Load a skybox
        skybox = self.base.loader.loadModel('models/skybox.bam')
        skybox.reparent_to(self.base.render)
        skybox.set_scale(20000)

        skybox_texture = self.base.loader.loadTexture('textures/skybox.jpg')
        skybox_texture.set_minfilter(SamplerState.FT_linear)
        skybox_texture.set_magfilter(SamplerState.FT_linear)
        skybox_texture.set_wrap_u(SamplerState.WM_repeat)
        skybox_texture.set_wrap_v(SamplerState.WM_mirror)
        skybox_texture.set_anisotropic_degree(16)
        skybox.set_texture(skybox_texture)

        skybox_shader = Shader.load(Shader.SL_GLSL, 'skybox.vert.glsl', 'skybox.frag.glsl')
        skybox.set_shader(skybox_shader)

        self._key_map = {
            Qt.Key_Up        : partial(self.shiftCamera, ( 0.0,  2.5,  0.0)),
            Qt.Key_Down      : partial(self.shiftCamera, ( 0.0, -2.5,  0.0)),
            Qt.Key_Left      : partial(self.shiftCamera, (-2.5,  0.0,  0.0)),
            Qt.Key_Right     : partial(self.shiftCamera, ( 2.5,  0.0,  0.0)),
            Qt.Key_PageDown  : partial(self.shiftCamera, ( 0.0,  0.0, -2.5)),
            Qt.Key_PageUp    : partial(self.shiftCamera, ( 0.0,  0.0,  2.5)),
            Qt.Key_F3        : self.base.toggleWireframe,
        }

        # Setup a timer in Qt that runs taskMgr.step() to simulate Panda's own main loop
        # Looking at ShowBase.spawnWxLoop() here for guidance
        self.pandaTimer = qc.QTimer(self)
        self.pandaTimer.timeout.connect(self.paintGL)
        self.pandaTimer.start(1)


    def __del__(self):
        """ 
        Parent windows should call cleanup() to clean up the
        wxPandaWindow explicitly (since we can't catch EVT_CLOSE
        directly). 
        """
        log.info('TODO: cleanup the hidden Panda3D window?')

    def initializeGL(self):
        log.info('TODO: what does tasks does Panda3D need done before start?')
        self.visible = True

    def resizeGL(self, width: int, height: int):
        """
        The Qt event loop calls this function whenever it wants to resize the 
        OpenGL frame.
        """
        if self.base.win:
            # Taken from `WxPandaWindow.py` but does not appear to actually work...
            # Or it seems to change for one frame and then change back...
            wp = WindowProperties()
            pos = self.pos()
            wp.setOrigin(pos.x(), pos.y())
            wp.setSize(width, height)

    # log.info(f'Resize to extent: {pos.x(), pos.y(), width, height}')
    # log.info(f'Qt5 QSurfaceFormat: {self.format()}')
    
    self.base.win.requestProperties(wp)
        else:
            log.error('base.win is None')

    def paintGL(self):
        """
        The Qt event loops calls this function whenever it needs to repaint the 
        OpenGL frame.
        """
        self.makeCurrent()
        self.visible = True
        if Thread.getCurrentThread().getCurrentTask():
            # This happens when the QTimer expires while igLoop is
            # rendering. Ignore it.
            return
        self.base.taskMgr.step()
        self.doneCurrent()

    # TODO: try to push keys/mouse to the Panda3D instance.
    def mousePressEvent(self, event: qg.QMouseEvent) -> None:
        button = event.button()
        log.info(f'Pressed button {button}')
        self.setFocus(qc.Qt.MouseFocusReason)

    def mouseReleaseEvent(self, event: qg.QMouseEvent) -> None:
        button = event.button()
        log.info(f'Released button {button}')
        
    def mouseEnterEvent(self, event: qg.QMouseEvent) -> None:
        log.info('Enter event')

    def mouseMoveEvent(self, event: qg.QMouseEvent):
        self.setFocus(qc.Qt.MouseFocusReason)

    def wheelEvent(self, event: qg.QMouseEvent) -> None:
        if event.delta() > 0:
            self.shiftCamera((0.0, 0.0, -2.5))
        else:
            self.shiftCamera((0.0, 0.0, 2.5))

    def keyPressEvent(self, event: qg.QKeyEvent) -> None:
        # Focus doesn't automatically come to the window so generally it has 
        # to be managed in the mouse events.
        key = event.key()
        # There are some more sophisticated camera controls in the 
        # `direct/src/controls` directory
        if key in self._key_map:
            self._key_map[key]()
        else:
            log.info(f'Received unhandled key press: {key}')

    def _eventsCallback(self, data):
        data.upcall()

    def _propertiesCallback(self, data):
        data.upcall()

    def paintEvent(self, event) -> None:
        """
        Some confusion here on whether this should be a do-nothing. It seems 
        like if this is not empty then it paints gray over the Panda3D buffer.

        From Qt docs:

            Drawing directly to the QOpenGLWidget's framebuffer outside the 
            GUI/main thread is possible by reimplementing paintEvent() to do 
            nothing. The context's thread affinity has to be changed via 
            QObject::moveToThread(). After that, makeCurrent() and 
            doneCurrent() are usable on the worker thread. Be careful to move 
            the context back to the GUI/main thread afterwards.

            Unlike QGLWidget, triggering a buffer swap just for the 
            QOpenGLWidget is not possible since there is no real, onscreen 
            native surface for it. Instead, it is up to the widget stack to 
            manage composition and buffer swaps on the gui thread. When a 
            thread is done updating the framebuffer, call update() on the 
            GUI/main thread to schedule composition.
        """
        return

    def _renderCallback(self, data: 'p3c.RenderCallbackData'):
        """
        It looks to me that we get two of these, one of the RCTBeginFrame and 
        then the RCTEndFlip when we should swap the buffer. Except Qt does the 
        buffer swap in `self.paintGL`.
        """
        fb = self.defaultFramebufferObject()
        cbType = data.getCallbackType()
        if cbType == CallbackGraphicsWindow.RCTBeginFrame:
            if not self.visible:
                # TODO: any nicer way than having some sentinel flag to prevent 
                # the _renderCallback from executing?
                data.setRenderFlag(False)
                return
            # makeCurrent is called in paintGL
            # Don't upcall() in this case.
            return

        elif cbType == CallbackGraphicsWindow.RCTEndFlip:
            # Old WxPython means of swapping frame buffer:
            # self.SwapBuffers() 
            #
            # According to Qt docs, we should just call `self.update()` here
            # and it will swap the buffers and call `self.doneCurrent().
            self.update()
            self.visible = False
        data.upcall()

    def shiftCamera(self, delta: Tuple[float]) -> None:
        """
        Moves the camera. Designed to be connected via Qt slots using `partial`.

        Parameters
        ----------
        delta
            a (x,y,z) tuple giving the desired relative shift from the current position.
        """
        camera = self.base.camera
        camera.setPos(camera.getX() + delta[0], camera.getY() + delta[1], camera.getZ() + delta[2])


if __name__ == '__main__':
    Q_APP = qw.QApplication(sys.argv)
    mainWindow = MainWindow()
    mainWindow.show()
    Q_APP.exec_()

    del mainWindow
    del Q_APP

    sys.exit(0)

If anyone has any advice, particularly on the resizing, that would be appreciated.


Developers: Favorite GUI framework?
#2

I actually asked how to do this exact thing on StackExchange with no luck… so, I’m not much help, but I’ll be following this.