Offscreen rendering on EC2 machine

ekiefl · February 24, 2023, 9:45pm

My collaborator and I have been working on an offscreen renderer that saves many panda3d scenes as a series of images. The goal is to create hundreds of thousands of these images as a large training dataset for some machine learning models.

Everything is working on our local machines, and we’re ready to “scale up” and start running on the cloud. As our playground we’ve chosen a weak AWS EC2 machine to start, but we’re running into graphics troubles.

Here was our troubleshooting breadcrumbs, and we were wondering if anyone could spot the potential problem. Our hypothesis is that our machine doesn’t have a graphics card, and hence why we’re running into these troubles?

Running our example script, we first ran into this error:

:display(warning): Unable to load libpandagl.so: libGL.so.1: cannot open shared object file: No such file or directory
:display(warning): Unable to load libp3headlessgl.so: libGL.so.1: cannot open shared object file: No such file or directory
Known pipe types:
(all display modules loaded.)
Traceback (most recent call last):
  File "sandbox/simple.py", line 66, in <module>
    interface = pt.ShotViewer()
  File "/home/ec2-user/pooltool/pooltool/ani/animate.py", line 227, in __init__
    Interface.__init__(self, config=config)
  File "/home/ec2-user/pooltool/pooltool/ani/animate.py", line 116, in __init__
    super().__init__(self, windowType=config.window_type)
  File "/home/ec2-user/miniconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 341, in __init__
    self.openDefaultWindow(startDirect = False, props=props)
  File "/home/ec2-user/miniconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 1026, in openDefaultWindow
    self.openMainWindow(*args, **kw)
  File "/home/ec2-user/miniconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 1061, in openMainWindow
    self.openWindow(*args, **kw)
  File "/home/ec2-user/miniconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 771, in openWindow
    win = func()
  File "/home/ec2-user/miniconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 751, in <lambda>
    func = lambda : self._doOpenWindow(
  File "/home/ec2-user/miniconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 823, in _doOpenWindow
    self.makeDefaultPipe()
  File "/home/ec2-user/miniconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 652, in makeDefaultPipe
    self.notify.error(
  File "/home/ec2-user/miniconda3/envs/pooltool/lib/python3.8/site-packages/direct/directnotify/Notifier.py", line 130, in error
    raise exception(errorString)
Exception: No graphics pipe is available!
Your Config.prc file must name at least one valid panda display
library via load-display or aux-display.

We were able to “solve” this error with the following command:

sudo yum install libglvnd-glx

Here is the installation:

Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Resolving Dependencies
--> Running transaction check
---> Package libglvnd-glx.x86_64 1:1.0.1-0.1.git5baa1e5.amzn2.0.1 will be installed
--> Processing Dependency: libglvnd(x86-64) = 1:1.0.1-0.1.git5baa1e5.amzn2.0.1 for package: 1:libglvnd-glx-1.0.1-0.1.git5baa1e5.amzn2.0.1.x86_64
--> Processing Dependency: mesa-libGL(x86-64) >= 13.0.4-1 for package: 1:libglvnd-glx-1.0.1-0.1.git5baa1e5.amzn2.0.1.x86_64
--> Processing Dependency: libXext.so.6()(64bit) for package: 1:libglvnd-glx-1.0.1-0.1.git5baa1e5.amzn2.0.1.x86_64
--> Processing Dependency: libX11.so.6()(64bit) for package: 1:libglvnd-glx-1.0.1-0.1.git5baa1e5.amzn2.0.1.x86_64
--> Processing Dependency: libGLdispatch.so.0()(64bit) for package: 1:libglvnd-glx-1.0.1-0.1.git5baa1e5.amzn2.0.1.x86_64
--> Running transaction check
---> Package libX11.x86_64 0:1.6.7-3.amzn2.0.2 will be installed
--> Processing Dependency: libX11-common >= 1.6.7-3.amzn2.0.2 for package: libX11-1.6.7-3.amzn2.0.2.x86_64
--> Processing Dependency: libxcb.so.1()(64bit) for package: libX11-1.6.7-3.amzn2.0.2.x86_64
---> Package libXext.x86_64 0:1.3.3-3.amzn2.0.2 will be installed
---> Package libglvnd.x86_64 1:1.0.1-0.1.git5baa1e5.amzn2.0.1 will be installed
---> Package mesa-libGL.x86_64 0:18.3.4-5.amzn2.0.1 will be installed
--> Processing Dependency: mesa-libglapi = 18.3.4-5.amzn2.0.1 for package: mesa-libGL-18.3.4-5.amzn2.0.1.x86_64
--> Processing Dependency: libxshmfence.so.1()(64bit) for package: mesa-libGL-18.3.4-5.amzn2.0.1.x86_64
--> Processing Dependency: libglapi.so.0()(64bit) for package: mesa-libGL-18.3.4-5.amzn2.0.1.x86_64
--> Processing Dependency: libXxf86vm.so.1()(64bit) for package: mesa-libGL-18.3.4-5.amzn2.0.1.x86_64
--> Processing Dependency: libXfixes.so.3()(64bit) for package: mesa-libGL-18.3.4-5.amzn2.0.1.x86_64
--> Processing Dependency: libXdamage.so.1()(64bit) for package: mesa-libGL-18.3.4-5.amzn2.0.1.x86_64
--> Running transaction check
---> Package libX11-common.noarch 0:1.6.7-3.amzn2.0.2 will be installed
---> Package libXdamage.x86_64 0:1.1.4-4.1.amzn2.0.2 will be installed
---> Package libXfixes.x86_64 0:5.0.3-1.amzn2.0.2 will be installed
---> Package libXxf86vm.x86_64 0:1.1.4-1.amzn2.0.2 will be installed
---> Package libxcb.x86_64 0:1.12-1.amzn2.0.2 will be installed
--> Processing Dependency: libXau.so.6()(64bit) for package: libxcb-1.12-1.amzn2.0.2.x86_64
---> Package libxshmfence.x86_64 0:1.2-1.amzn2.0.2 will be installed
---> Package mesa-libglapi.x86_64 0:18.3.4-5.amzn2.0.1 will be installed
--> Running transaction check
---> Package libXau.x86_64 0:1.0.8-2.1.amzn2.0.2 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================
 Package                            Arch                        Version                                                 Repository                       Size
==============================================================================================================================================================
Installing:
 libglvnd-glx                       x86_64                      1:1.0.1-0.1.git5baa1e5.amzn2.0.1                        amzn2-core                      125 k
Installing for dependencies:
 libX11                             x86_64                      1.6.7-3.amzn2.0.2                                       amzn2-core                      606 k
 libX11-common                      noarch                      1.6.7-3.amzn2.0.2                                       amzn2-core                      165 k
 libXau                             x86_64                      1.0.8-2.1.amzn2.0.2                                     amzn2-core                       29 k
 libXdamage                         x86_64                      1.1.4-4.1.amzn2.0.2                                     amzn2-core                       20 k
 libXext                            x86_64                      1.3.3-3.amzn2.0.2                                       amzn2-core                       39 k
 libXfixes                          x86_64                      5.0.3-1.amzn2.0.2                                       amzn2-core                       18 k
 libXxf86vm                         x86_64                      1.1.4-1.amzn2.0.2                                       amzn2-core                       17 k
 libglvnd                           x86_64                      1:1.0.1-0.1.git5baa1e5.amzn2.0.1                        amzn2-core                       89 k
 libxcb                             x86_64                      1.12-1.amzn2.0.2                                        amzn2-core                      216 k
 libxshmfence                       x86_64                      1.2-1.amzn2.0.2                                         amzn2-core                      7.2 k
 mesa-libGL                         x86_64                      18.3.4-5.amzn2.0.1                                      amzn2-core                      162 k
 mesa-libglapi                      x86_64                      18.3.4-5.amzn2.0.1                                      amzn2-core                       45 k

Transaction Summary
==============================================================================================================================================================
Install  1 Package (+12 Dependent packages)

Total download size: 1.5 M
Installed size: 5.6 M
Is this ok [y/d/N]: y
Downloading packages:
(1/13): libX11-common-1.6.7-3.amzn2.0.2.noarch.rpm                                                                                     | 165 kB  00:00:00
(2/13): libX11-1.6.7-3.amzn2.0.2.x86_64.rpm                                                                                            | 606 kB  00:00:00
(3/13): libXau-1.0.8-2.1.amzn2.0.2.x86_64.rpm                                                                                          |  29 kB  00:00:00
(4/13): libXdamage-1.1.4-4.1.amzn2.0.2.x86_64.rpm                                                                                      |  20 kB  00:00:00
(5/13): libXext-1.3.3-3.amzn2.0.2.x86_64.rpm                                                                                           |  39 kB  00:00:00
(6/13): libXfixes-5.0.3-1.amzn2.0.2.x86_64.rpm                                                                                         |  18 kB  00:00:00
(7/13): libXxf86vm-1.1.4-1.amzn2.0.2.x86_64.rpm                                                                                        |  17 kB  00:00:00
(8/13): libglvnd-1.0.1-0.1.git5baa1e5.amzn2.0.1.x86_64.rpm                                                                             |  89 kB  00:00:00
(9/13): libglvnd-glx-1.0.1-0.1.git5baa1e5.amzn2.0.1.x86_64.rpm                                                                         | 125 kB  00:00:00
(10/13): libxcb-1.12-1.amzn2.0.2.x86_64.rpm                                                                                            | 216 kB  00:00:00
(11/13): libxshmfence-1.2-1.amzn2.0.2.x86_64.rpm                                                                                       | 7.2 kB  00:00:00
(12/13): mesa-libGL-18.3.4-5.amzn2.0.1.x86_64.rpm                                                                                      | 162 kB  00:00:00
(13/13): mesa-libglapi-18.3.4-5.amzn2.0.1.x86_64.rpm                                                                                   |  45 kB  00:00:00
--------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                         6.4 MB/s | 1.5 MB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libXau-1.0.8-2.1.amzn2.0.2.x86_64                                                                                                         1/13
  Installing : libxcb-1.12-1.amzn2.0.2.x86_64                                                                                                            2/13
  Installing : libX11-common-1.6.7-3.amzn2.0.2.noarch                                                                                                    3/13
  Installing : libX11-1.6.7-3.amzn2.0.2.x86_64                                                                                                           4/13
  Installing : libXext-1.3.3-3.amzn2.0.2.x86_64                                                                                                          5/13
  Installing : libXfixes-5.0.3-1.amzn2.0.2.x86_64                                                                                                        6/13
  Installing : libXdamage-1.1.4-4.1.amzn2.0.2.x86_64                                                                                                     7/13
  Installing : libXxf86vm-1.1.4-1.amzn2.0.2.x86_64                                                                                                       8/13
  Installing : 1:libglvnd-1.0.1-0.1.git5baa1e5.amzn2.0.1.x86_64                                                                                          9/13
  Installing : libxshmfence-1.2-1.amzn2.0.2.x86_64                                                                                                      10/13
  Installing : mesa-libglapi-18.3.4-5.amzn2.0.1.x86_64                                                                                                  11/13
  Installing : 1:libglvnd-glx-1.0.1-0.1.git5baa1e5.amzn2.0.1.x86_64                                                                                     12/13
  Installing : mesa-libGL-18.3.4-5.amzn2.0.1.x86_64                                                                                                     13/13
  Verifying  : mesa-libGL-18.3.4-5.amzn2.0.1.x86_64                                                                                                      1/13
  Verifying  : libXext-1.3.3-3.amzn2.0.2.x86_64                                                                                                          2/13
  Verifying  : libX11-1.6.7-3.amzn2.0.2.x86_64                                                                                                           3/13
  Verifying  : mesa-libglapi-18.3.4-5.amzn2.0.1.x86_64                                                                                                   4/13
  Verifying  : libxshmfence-1.2-1.amzn2.0.2.x86_64                                                                                                       5/13
  Verifying  : 1:libglvnd-1.0.1-0.1.git5baa1e5.amzn2.0.1.x86_64                                                                                          6/13
  Verifying  : libXxf86vm-1.1.4-1.amzn2.0.2.x86_64                                                                                                       7/13
  Verifying  : libX11-common-1.6.7-3.amzn2.0.2.noarch                                                                                                    8/13
  Verifying  : 1:libglvnd-glx-1.0.1-0.1.git5baa1e5.amzn2.0.1.x86_64                                                                                      9/13
  Verifying  : libXdamage-1.1.4-4.1.amzn2.0.2.x86_64                                                                                                    10/13
  Verifying  : libXfixes-5.0.3-1.amzn2.0.2.x86_64                                                                                                       11/13
  Verifying  : libXau-1.0.8-2.1.amzn2.0.2.x86_64                                                                                                        12/13
  Verifying  : libxcb-1.12-1.amzn2.0.2.x86_64                                                                                                           13/13

Installed:
  libglvnd-glx.x86_64 1:1.0.1-0.1.git5baa1e5.amzn2.0.1

Dependency Installed:
  libX11.x86_64 0:1.6.7-3.amzn2.0.2               libX11-common.noarch 0:1.6.7-3.amzn2.0.2                 libXau.x86_64 0:1.0.8-2.1.amzn2.0.2
  libXdamage.x86_64 0:1.1.4-4.1.amzn2.0.2         libXext.x86_64 0:1.3.3-3.amzn2.0.2                       libXfixes.x86_64 0:5.0.3-1.amzn2.0.2
  libXxf86vm.x86_64 0:1.1.4-1.amzn2.0.2           libglvnd.x86_64 1:1.0.1-0.1.git5baa1e5.amzn2.0.1         libxcb.x86_64 0:1.12-1.amzn2.0.2
  libxshmfence.x86_64 0:1.2-1.amzn2.0.2           mesa-libGL.x86_64 0:18.3.4-5.amzn2.0.1                   mesa-libglapi.x86_64 0:18.3.4-5.amzn2.0.1

Complete!

Running again, we got this error:

Known pipe types:
  glxGraphicsPipe
(1 aux display modules not yet loaded.)
:display:x11display(error): Could not open display ":0.0".
:display(warning): Unable to load libp3headlessgl.so: libEGL.so.1: cannot open shared object file: No such file or directory
:ShowBase(warning): Unable to open 'offscreen' window.
Traceback (most recent call last):
  File "sandbox/offscreen.py", line 90, in <module>
    main(args)
  File "sandbox/offscreen.py", line 14, in main
    interface = pt.ImageSaver()
  File "/home/ec2-user/pooltool/pooltool/ani/animate.py", line 324, in __init__
    Interface.__init__(self, config=config)
  File "/home/ec2-user/pooltool/pooltool/ani/animate.py", line 116, in __init__
    super().__init__(self, windowType=config.window_type)
  File "/home/ec2-user/miniconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 341, in __init__
    self.openDefaultWindow(startDirect = False, props=props)
  File "/home/ec2-user/miniconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 1026, in openDefaultWindow
    self.openMainWindow(*args, **kw)
  File "/home/ec2-user/miniconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 1061, in openMainWindow
    self.openWindow(*args, **kw)
  File "/home/ec2-user/miniconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 806, in openWindow
    raise Exception('Could not open window.')
Exception: Could not open window.

We were able to “solve” this error with this command:

sudo yum install mesa-libEGL

Which ultimately led to this error:

Known pipe types:
  glxGraphicsPipe
(1 aux display modules not yet loaded.)
:display:x11display(error): Could not open display ":0.0".
:display:egldisplay(warning): Couldn't initialize the default EGL display: EGL_NOT_INITIALIZED
:display:egldisplay(error): Failed to find or initialize a suitable EGL display connection.
:ShowBase(warning): Unable to open 'offscreen' window.
Traceback (most recent call last):
  File "sandbox/offscreen.py", line 90, in <module>
    main(args)
  File "sandbox/offscreen.py", line 14, in main
    interface = pt.ImageSaver()
  File "/home/ec2-user/pooltool/pooltool/ani/animate.py", line 324, in __init__
    Interface.__init__(self, config=config)
  File "/home/ec2-user/pooltool/pooltool/ani/animate.py", line 116, in __init__
    super().__init__(self, windowType=config.window_type)
  File "/home/ec2-user/miniconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 341, in __init__
    self.openDefaultWindow(startDirect = False, props=props)
  File "/home/ec2-user/miniconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 1026, in openDefaultWindow
    self.openMainWindow(*args, **kw)
  File "/home/ec2-user/miniconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 1061, in openMainWindow
    self.openWindow(*args, **kw)
  File "/home/ec2-user/miniconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 806, in openWindow
    raise Exception('Could not open window.')
Exception: Could not open window.

Based on this panda3d topic, our problem might be that our EC2 has no graphics card, but we aren’t sure and would like some clarification.

Getting some clarification on this would be super helpful. Thanks in advance!

rdb · February 24, 2023, 9:56pm

What’s the instance type? I would assume you need a G type instance in order to have a working OpenGL implementation.

As an alternative you can use a software OpenGL implementation or set load-display p3tinydisplay in Config.prc to use Panda’s built-in software renderer, but performance and graphics quality will be quite poor.

ekiefl · February 24, 2023, 10:19pm

What’s the instance type? I would assume you need a G type instance in order to have a working OpenGL implementation.

Ok, that’s got to be the problem. We’re currently using a T2 instance, which upon reading the specs, makes a lot of sense why it wasn’t working! We’ll try using a GPU-based instance next. Thanks @rdb

As an alternative you can use a software OpenGL implementation or set load-display p3tinydisplay in Config.prc to use Panda’s built-in software renderer, but performance and graphics quality will be quite poor.

Ahh it seems p3tinydisplay has its own set of issues, some of them resulting from my codebase, rather than the environment / computing system.

Known pipe types:
  TinyXGraphicsPipe
  TinyOffscreenGraphicsPipe
(1 aux display modules not yet loaded.)
:display:x11display(error): Could not open display ":0.0".
AL lib: (WW) pulse_load: Failed to load libpulse.so.0
AL lib: (WW) alc_initconfig: Failed to initialize backend "pulse"
AL lib: (WW) alsa_load: Failed to load libasound.so.2
AL lib: (WW) alc_initconfig: Failed to initialize backend "alsa"
AL lib: (EE) ALCplaybackOSS_open: Could not open /dev/dsp: No such file or directory
AL lib: (WW) alcSetError: Error generated on device (nil), code 0xa004
AL lib: (EE) ALCplaybackOSS_open: Could not open /dev/dsp: No such file or directory
AL lib: (WW) alcSetError: Error generated on device (nil), code 0xa004
:audio(error): Couldn't open default OpenAL device
:audio(error): OpenALAudioManager: No open device or context
:audio(error):   OpenALAudioManager is not valid, will use NullAudioManager
AL lib: (EE) ALCplaybackOSS_open: Could not open /dev/dsp: No such file or directory
AL lib: (WW) alcSetError: Error generated on device (nil), code 0xa004
AL lib: (EE) ALCplaybackOSS_open: Could not open /dev/dsp: No such file or directory
AL lib: (WW) alcSetError: Error generated on device (nil), code 0xa004
:audio(error): Couldn't open default OpenAL device
:audio(error): OpenALAudioManager: No open device or context
:audio(error):   OpenALAudioManager is not valid, will use NullAudioManager
:prc(warning): value queried before default value set for gl-version.
:display(error): The 'textures_power_2' configuration is set to 'none', meaning
that non-power-of-two texture support is required, but the video
driver I'm trying to use does not support non-power-of-two textures.
:display(error): The 'none' did not come from the config file.  In other words,
the variable 'textures_power_2' was altered procedurally.
Assertion failed: Cannot resize buffer unless it is created with BF_resizeable flag at line 60 of panda/src/display/graphicsBuffer.cxx
Traceback (most recent call last):
  File "sandbox/offscreen.py", line 90, in <module>
    main(args)
  File "sandbox/offscreen.py", line 48, in main
    interface.save(
  File "/home/ec2-user/pooltool/pooltool/ani/animate.py", line 419, in save
    self._resize_window(size)
  File "/home/ec2-user/pooltool/pooltool/ani/animate.py", line 351, in _resize_window
    Global.base.win.setSize(*[int(dim) for dim in size])
AssertionError: Cannot resize buffer unless it is created with BF_resizeable flag at line 60 of panda/src/display/graphicsBuffer.cxx

ekiefl · March 11, 2023, 9:35pm

Hey @rdb, we’ve encountered some more downstream issues. One of us is a scientist, the other an engineer, so we’re a bit out of our element here. We appreciate your patience with us.

We’re now using a Sagemaker GPU instance. This machine is preconfigured for computer vision tasks, and it has OpenGL preconfigured. These are the specs (it’s a g4dn.xlarge).

For a proper diagnosis, we’ve got the the following Config.prc:

# Displays frame right in top right
show-frame-rate-meter #t

win-size 1400 875

# Full screen
fullscreen #f

# If frame rate exceeds the monitor refresh rate, set frame rate to monitor refresh rate
sync-video #f

gl-debug #t
gl-check-errors #t
audio-library-name null

When we run our test script, we get a BFResizeable error:

Known pipe types:
  glxGraphicsPipe
(1 aux display modules not yet loaded.)
:display:x11display(error): Could not open display ":0.0".
Assertion failed: Cannot resize buffer unless it is created with BF_resizeable flag at line 60 of panda/src/display/graphicsBuffer.cxx
Traceback (most recent call last):
  File "sandbox/offscreen/offscreen.py", line 104, in <module>
    main(args)
  File "sandbox/offscreen/offscreen.py", line 55, in main
    datapack = interface.gen_datapack(
  File "/home/ec2-user/SageMaker/pooltool/pooltool/ani/image/interface.py", line 106, in gen_datapack
    _resize_window(size)
  File "/home/ec2-user/SageMaker/pooltool/pooltool/ani/image/interface.py", line 32, in _resize_window
    Global.base.win.setSize(*[int(dim) for dim in size])
AssertionError: Cannot resize buffer unless it is created with BF_resizeable flag at line 60 of panda/src/display/graphicsBuffer.cxx

I don’t think it’s possible to explicitly set this flag when funneling through ShowBase.openMainWindow (which is how I’ve built my application) so as a diagnostic monkey patch I modified this line

github.com

panda3d/panda3d/blob/4bb6d8f6d8be69b0c9a4e8d48e061f95e3f57a46/direct/src/showbase/ShowBase.py#L955


      
          if name is None:
              name = 'window%s' % (self.nextWindowIndex)
              self.nextWindowIndex += 1
          
          win = None
          
          flags = GraphicsPipe.BFFbPropsOptional
          if type == 'onscreen':
              flags = flags | GraphicsPipe.BFRequireWindow
          elif type == 'offscreen':
              flags = flags | GraphicsPipe.BFRefuseWindow
          
          if callbackWindowDict:
              flags = flags | GraphicsPipe.BFRequireCallbackWindow
          
          if host:
              assert host.isValid()
              win = self.graphicsEngine.makeOutput(pipe, name, 0, fbprops,
                                                   props, flags, host.getGsg(), host)
          elif gsg:
              win = self.graphicsEngine.makeOutput(pipe, name, 0, fbprops,

To be this:

    flags = flags | GraphicsPipe.BFRefuseWindow | GraphicsPipe.BFResizeable

But unfortunately, the issue is not as simple as adding a missing flag. Here is the error with the unsuccessful monkey patch:

Known pipe types:
  glxGraphicsPipe
(1 aux display modules not yet loaded.)
:display:x11display(error): Could not open display ":0.0".
:ShowBase(warning): Unable to open 'offscreen' window.
Traceback (most recent call last):
  File "sandbox/offscreen/offscreen.py", line 104, in <module>
    main(args)
  File "sandbox/offscreen/offscreen.py", line 15, in main
    interface = pt.ImageSaver()
  File "/home/ec2-user/SageMaker/pooltool/pooltool/ani/image/interface.py", line 44, in __init__
    Interface.__init__(self, config=config)
  File "/home/ec2-user/SageMaker/pooltool/pooltool/ani/animate.py", line 108, in __init__
    super().__init__(self, windowType=config.window_type)
  File "/home/ec2-user/anaconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 341, in __init__
    self.openDefaultWindow(startDirect = False, props=props)
  File "/home/ec2-user/anaconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 1026, in openDefaultWindow
    self.openMainWindow(*args, **kw)
  File "/home/ec2-user/anaconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 1061, in openMainWindow
    self.openWindow(*args, **kw)
  File "/home/ec2-user/anaconda3/envs/pooltool/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 806, in openWindow
    raise Exception('Could not open window.')
Exception: Could not open window.

Although resizeable offscreen windows are a requirement for us, I decided to comment out all of the window resizing code to see if everything else worked:

Known pipe types:
  glxGraphicsPipe
(1 aux display modules not yet loaded.)
:display:x11display(error): Could not open display ":0.0".
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): at 9053 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): at 9053 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): at 9053 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): at 9053 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated. The required buffer is missing.
:display:gsg:glgsg(error): at 9143 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): at 9053 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): at 9053 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): at 9053 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): at 9053 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated. The required buffer is missing.
:display:gsg:glgsg(error): at 9143 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): at 9053 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): at 9053 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): at 9053 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): at 9053 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated. The required buffer is missing.
:display:gsg:glgsg(error): at 9143 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): at 9053 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): at 9053 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): at 9053 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): at 9053 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated. The required buffer is missing.
:display:gsg:glgsg(error): at 9143 of panda/src/glstuff/glGraphicsStateGuardian_src.cxx : invalid operation
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display(error): Deactivating eglGraphicsStateGuardian.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated. The required buffer is missing.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated. The required buffer is missing.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated. The required buffer is missing.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated. The required buffer is missing.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated.
:display:gsg:glgsg(error): GL_INVALID_OPERATION error generated. The required buffer is missing.

The “required buffer is missing” error repeats thousands of times, but interestingly our script runs to completion.

Our script generates image frames of a pool break, and the first few frames look great:

But all subsequent frames look the same as the 5th image (above). Relating this back to the error message, my guess is that as soon as the eglGraphicsStateGuardian is deactivated, the frame buffer can no longer be read from.

My lack of knowledge is making it difficult to ask the right question to you @rdb, but I guess the bottom line is this: do you have any recommendations for how to properly set up our GPU instance?

Currently this is how we’re doing:

#!/bin/bash
export PATH="/home/ec2-user/anaconda3/condabin/:$PATH"

source /home/ec2-user/anaconda3/etc/profile.d/conda.sh
conda init zsh 
source activate

conda deactivate
conda env remove --name pooltool
conda create -y -n pooltool python=3.8.10
conda activate pooltool

cd /home/ec2-user/SageMaker/pooltool
pip install -r requirements.txt

whoami
sudo -i -u ec2-user bash << EOF
echo "In"
whoami
conda init zsh 
source activate
echo 'export PYTHONPATH=$PYTHONPATH:/home/ec2-user/SageMaker/pooltool' >> ~/.zshrc
echo 'source activate' >> ~/.zshrc
echo 'conda activate pooltool' >> ~/.zshrc
echo 'cd /home/ec2-user/SageMaker/pooltool' >> ~/.zshrc
echo ~ec2-user
EOF
echo "Out"
whoami

panda3d is installed with pip in that requirements.txt.

Thanks for taking a look.

Cheers,
Evan

rdb · March 13, 2023, 12:48pm

Can you set load-display p3headlessgl in Config.prc? It’s loading X11, which it’s not supposed to. I wonder if all the GL_INVALID_OPERATION errors stem from that.

There’s probably a bug going on that causes that GL_INVALID_OPERATION, but it’s hard to know exactly what it is, since there’s no gl-debug information. Can you share your precise Panda3D version? Maybe I can do something with the line number.

ekiefl · March 26, 2023, 2:41am

Hey @rdb thanks for the response. Sorry for halting the momentum on this, I was swamped with work (but I’ve since quit my job so now I have all the time in the world lol).

Ok, added load-display p3headlessgl. No more X11 warning, but unfortunately it didn’t fix the problem.

Panda version:

>>> import panda3d
>>> panda3d.__version__
'1.10.13'

rdb · March 27, 2023, 12:46pm

I think I need more debug information, could you put notify-level-display debug and notify-level-glgsg debug? Setting them to spam will be more verbose, but if you capture it in a log file, that would also be helpful. An “apitrace” log would be even more helpful if you can gather that.

I think there’s something about the framebuffer configuration that’s not right, maybe adjusting the framebuffer flags would help.

ekiefl · March 28, 2023, 5:12am

Ok here is the output with “spam” verbosity.

debug.txt.zip (70.1 KB)

Unfortunately I couldn’t get apitrace on the GPU instance. Part of the problem is definitely my lack of Linux experience.

rdb · March 28, 2023, 9:04am

Okay, the good news is that that seems like very capable hardware and drivers, supporting all OpenGL features.

I’ll experiment a bit on my end; knowing a bit more about your engine set-up might help me. It looks like you’re creating an offscreen buffer (pbuffer), immediately closing it and its context, and then creating a new pbuffer with its context?

I do note that both buffers are double-buffered, which may not be necessary for your use case, and may be related to the error at hand. Can you try putting this in Config.prc?

back-buffers 0

If we can’t figure it out, I’ll provision one of those instances for myself and get it to work there—I promise you we’ll figure this out.

rdb · March 28, 2023, 9:28am

By the way, an eglGraphicsBuffer (pbuffer) isn’t resizeable (that’s why you got the “Couldn’t open window” error). If you want a resizeable buffer, you need an FBO. FBOs have more features than pbuffers, so generally are what you want to have. To get that, you need to create a new buffer with the initial buffer you created passed in as host window. You can just leave the original pbuffer open, setting it to non-active.

ekiefl · March 28, 2023, 7:24pm

First of all, thanks so much!

Well, in truth I developed this with a “yay it worked” approach without any understanding about buffers/contexts, so whatever buffer/context creation/destruction logic you’re spotting is really just a byproduct of getting things to work (on my mac), rather than based on intentional design.

I don’t think it would be very time efficient for you to wade/suffer through my codebase to understand what I’m doing, so today I’m going to try and make a script that mimics what I do, in far fewer lines. Maybe we can clean up the design, remove unnecessary buffer gymnastics, and fix this problem all in one go.

Since the script runs on my mac without error, does this mean my local default graphics buffer is an FBO?

ekiefl · March 28, 2023, 7:33pm

The error persists, but let me know if you would like the output.

ekiefl · March 29, 2023, 12:42am

Ok, I’ve mimicked what I think my engine is doing with this simple script. Hopefully it captures the buffer and context events you’re observing in the log:

import simplepbr
from direct.showbase.ShowBase import ShowBase
from direct.showbase import ShowBaseGlobal
from panda3d.core import (
    FrameBufferProperties,
)

import argparse


class Interface(ShowBase):
    def __init__(self, window_type, fb_prop=None):
        super().__init__(self, windowType=window_type)
        self.openMainWindow(fbprops=fb_prop)
        simplepbr.init(enable_shadows=True, max_lights=13)


class Game(Interface):
    """This is the game interface with interactive control"""

    def __init__(self):
        Interface.__init__(self, window_type="onscreen")


OFFSCREEN_FBP = FrameBufferProperties()
OFFSCREEN_FBP.setRgbColor(True)
OFFSCREEN_FBP.setRgbaBits(8, 8, 8, 0)
OFFSCREEN_FBP.setDepthBits(24)


class FrameStepper(Interface):
    """This is a more programmatic interface, enabling frame-by-frame analysis"""

    def __init__(self):
        Interface.__init__(self, window_type="offscreen", fb_prop=OFFSCREEN_FBP)


if __name__ == "__main__":
    ap = argparse.ArgumentParser(
        description=(
            "This is a toy representation of pooltool's current design. There is a "
            "game interface, where you can take shots using an onscreen window "
            "and there is a more programmatic offscreen interface where you can pass "
            "a pool shot as an argument and analyze it frame by frame. In the current "
            "design, these two distinct routines are managed by two respective "
            "subclasses of ShowBase, since that's the only way I could manage to "
            "create an onscreen _and_ and offscreen window. The downside of this "
            "approach is that you can only initialize one ShowBase instance, so you "
            "must decide at the start of a script's life whether it is for onscreen or "
            "offscreen rendering. It would be great if I could use one class and "
            "properly manage buffers and contexts to switch between on and offscreen "
            "rendering."
        )
    )
    ap.add_argument(
        "--mode", choices=("game", "analysis"), required=True
    )
    args = ap.parse_args()

    if args.mode == "game":
        game = Game()
        game.run()
    elif args.mode == "analysis":
        frame_stepper = FrameStepper()
        for _ in range(20):
            ShowBaseGlobal.base.task_mgr.step()
            # do something

rdb · March 29, 2023, 2:23pm

The good news is that I can reproduce the problem with your code on my Linux laptop, if I turn off my X11 display server. I’m looking into it.

rdb · March 29, 2023, 4:55pm

Okay, according to the EGL spec, section 2.2.2:

EGL, OpenGL, and OpenGL ES support two rendering models: back buffered and single buffered.
Back buffered rendering is used by window and pbuffer surfaces. Memory for the color buffer used during rendering is allocated and owned by EGL. When the client is finished drawing a frame, the back buffer may be copied to a visible window using eglSwapBuffers. Pbuffer surfaces have a back buffer but no associated window, so the back buffer need not be copied.

So, EGL pbuffers have a back buffer, but no front buffer. And Panda is trying to select the front buffer for rendering, because that’s the default state in OpenGL and what you normally render to when there’s only a single buffer.

This is kind of a weird design choice, and I don’t know why it’s only causing an error with certain implementations, but I’ve just checked in a change that makes Panda always select the back buffer instead of the front buffer with an EGL pbuffer. It shouldn’t really make a difference because the buffers are never swapped anyway.

You can grab these builds which contain the fix:
https://buildbot.panda3d.org/downloads/21d7a83bfc1215c38827b003ac53f1abdf68f76f/

Let me know if you still have any trouble!

Yes, because on macOS it’s possible to create a context that doesn’t have an associated surface (window or offscreen pbuffer), so we can create an FBO right away. (This is also possible in EGL via an extension, but it’s not currently implemented in Panda.)

ekiefl · March 31, 2023, 3:36pm

Can’t wait to try this, just trying to find a chunk of time

ekiefl · April 7, 2023, 9:30pm

Hey @rdb it worked! Thanks very much.

To get resizing, I’ll have to ensure I’m building an FBO. Do you know how I could modify the small example I posted above to ensure an FBO is created? With that I should be able to translate it into the true application.

rdb · April 8, 2023, 8:23am

The main window cannot be an FBO (at least not without us adding support for surfaceless contexts), it must be a pbuffer, and pbuffers aren’t resizeable. So you’d need to create a new window and use that as the target for simplepbr to composite into.

However, upon further reflection, the limitation that pbuffers aren’t resizeable is kind of silly. FBOs aren’t really resizeable either, Panda just recreates it when it needs to, and we could do the same with pbuffers. Are you able to wait for me to add this and create a new build for you or do you need a solution now?

rdb · April 8, 2023, 8:38am

I just implemented resizeability for EGL pbuffers. Buildbot builds will appear here when they’re done:
https://buildbot.panda3d.org/downloads/f5d5340ad314cf36ed4426292beeffb4aa53cf73/

ekiefl · April 8, 2023, 9:12pm

This is awesome! It seems to be working like a charm. Thanks for everything, I think that was the last bottleneck and now I have offscreen rendering on a powerful compute node