Running Panda3D on Linux over SSH

seyuboglu · February 8, 2021, 11:40pm

I’m using Panda3D to render a large number of short videos. When I try to run my script on a remote Linux machine (CentOS release 7.6.1810), I get the following x11 display error.

  glxGraphicsPipe
(all display modules loaded.)
:display:x11display(error): Could not open display ":0.0".
:ShowBase(warning): Unable to open 'onscreen' window.
Traceback (most recent call last):
  File "win/render/samples/ball-in-maze/dup.py", line 358, in <module>
    demo = BallInMazeDemo()
  File "win/render/samples/ball-in-maze/dup.py", line 53, in __init__
    ShowBase.__init__(self)
  File "/home/users/eyuboglu/miniconda3/envs/win/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 339, in __init__
    self.openDefaultWindow(startDirect = False, props=props)
  File "/home/users/eyuboglu/miniconda3/envs/win/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 1021, in openDefaultWindow
    self.openMainWindow(*args, **kw)
  File "/home/users/eyuboglu/miniconda3/envs/win/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 1056, in openMainWindow
    self.openWindow(*args, **kw)
  File "/home/users/eyuboglu/miniconda3/envs/win/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 801, in openWindow
    raise Exception('Could not open window.')
Exception: Could not open window.

I have added the two following lines to the default Config.prc:

window-type offscreen
audio-library-name null

and the Exception is raised after calling ShowBase.__init__(self).

Would appreciate any guidance on how to use Panda3D on a remote machine over SSH.

Thanks!

rdb · February 8, 2021, 11:43pm

Hi, welcome to the community!

By default, Panda is compiled to use GLX. This requires an X11 server to be running. From your description, I am guessing this is a headless server?

If so, you can recompile Panda3D without X11 support (using the --no-x11 flag to makepanda), and (assuming you do have a GPU on the server) it will use EGL instead, which is able to run headless.

Also see this issue:

seyuboglu · February 9, 2021, 12:30am

Thanks for the quick reply! I’m indeed on a headless server.

I’m compiling Panda3D without X11 support using

python makepanda/makepanda.py --everything --installer --no-x11

but am running into

built/lib/libpandaegg.so: undefined reference to `PointerTo<EggTexture>::PointerTo()'
collect2: error: ld returned 1 exit status

I tried implementing the workaround described here, but it looks like pointerTo.h has since been changed

Any ideas?

(I’m using: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36) and Python 3.8.5)

rdb · February 9, 2021, 11:01am

That looks like a compiler bug—it’s a pretty old compiler, are you able to update it at all?

Which version of the Panda sources are you building? We already have changed it to mark the PointerTo constructor as ALWAYS_INLINE.

You could try to edit panda/src/egg/pt_EggTexture.h and comment out the EXPORT_TEMPLATE_CLASS lines. Perhaps you’ll also need to comment out the template class lines in the .cxx file, not sure.

seyuboglu · February 14, 2021, 3:31am

Updating GCC did the trick (now using 10.1.0)

I’m running a SLURM cluster without access to rpmbuild, dpkg-deb, or patchelf. Is there a way to get the build installed in my Python environment without these packages? Maybe by using the installpanda.py script?

Thanks for the help!

rdb · February 14, 2021, 11:11am

You can build Panda with the --wheel flag and build a .whl, and install that with pip. Or you can indeed use installpanda (though it’s hard to uninstall it after you do that).

seyuboglu · February 15, 2021, 6:11pm

I don’t have patchelf on this machine, so I replaced subprocess.call(["patchelf", "--force-rpath", "--set-rpath", "$ORIGIN", temp.name]) in makepanda/makewheel.py with

import lief
elf = lief.parse(temp.name)
elf += lief.ELF.DynamicEntryRunPath("$ORIGIN")
elf.write(temp.name)

This seems to successfully build a wheel, but then after pip installing it, I’m unable to import panda3d.core. I can import direct and import panda3d though.

rdb · February 15, 2021, 9:16pm

What error message do you get when trying to import panda3d.core?

seyuboglu · February 15, 2021, 9:19pm

I get ModuleNotFoundError: No module named 'panda3d.core'

seyuboglu · February 16, 2021, 1:54am

I was able to successfully build Panda3d with the lief substitution described above using:

python ./makepanda/makepanda.py --everything --no-x11   --python-incdir /path/to/conda/envs/win/include   --python-libdir /path/to/conda/envs/win/lib --wheel

With this build, I’m able to import panda3d.core. However, I still am running into the same issue Exception: Could not open window. I set the window-type to be “offscreen” with

ConfigVariableString("window-type", "offscreen").setValue("offscreen")

Is there anything else I should be doing to get things running in a headless environment?
Thanks!

rdb · February 16, 2021, 11:07am

Did makepanda warn you about the build missing EGL, and do you have a libpandagl.so or libpandagles2.so produced as part of the build?

Are there any other warnings or error messages?

Does the machine have access to the GPU?

seyuboglu · February 16, 2021, 10:18pm

Did makepanda warn you about the build missing EGL, and do you have a libpandagl.so or libpandagles2.so produced as part of the build?

Yes I have both under built/lib. I don’t see any such warnings in the logs. I’ve shared the full build log as an attachment if that would be helpful.

Are there any other warnings or messages?

When I run the test script here., I get

Known pipe types:
  eglGraphicsPipe
(all display modules loaded.)
:display:egldisplay(error): Couldn't initialize the EGL display: EGL_NOT_INITIALIZED
:ShowBase(warning): Unable to open 'offscreen' window.

before the “Could not open window exception”.

Does the machine have access to the GPU?

Yes it does.

Thanks so much for the help! build_log.txt.zip (9.1 KB)

rdb · February 19, 2021, 9:50pm

Well, the EGL display is failing to open, which could be a problem with the driver configuration. What kind of GPU is in there? Are you able to update your graphics drivers?

One user has noted that replacing the EGL initialisation code with the use of the EGL_EXT_platform_device extension was necessary to get it to work:

Note that if you run makepanda with --threads 4, the build will go considerably faster.

seyuboglu · February 19, 2021, 11:27pm

The machine is using GeForce RTX 2080 Ti. I’m unable to update the drivers myself, but confirmed that the cluster carries the latest NVIDIA drivers.

Could you point me to a commit with the changes needed to use the EGL_PLATFORM_DEVICE_EXT extension. I’m trying to implement the changes, but I think I’m missing some context. For example, what exactly is EGL_PLATFORM_DEVICE_EXT and where an I include it from?

Thanks for the help!

rdb · February 24, 2021, 2:03pm

I just checked in this change to the Git repository:

You should be able to just pull the latest changes and rebuild. Let me know whether it works for you. If not, please set notify-level-display debug in Config.prc and post the output.

rdb · February 24, 2021, 5:11pm

Upon further reflection, what’s happening in your case is that the default EGL display connection is insufficient, which will cause it to never hit the enumeration code I have checked in, so you additionally need this commit, which I’ve just pushed to Git (you can just do a pull):

Now, Panda will try initialising every available device until it finds one that succeeds.

seyuboglu · March 1, 2021, 9:11pm

Thanks for pushing this change! I pulled and rebuilt (with a few small modifications needed to run the build on my machine, see attached git_diff.txt git_diff.txt.zip (1.5 KB) ).

from direct.showbase.ShowBase import ShowBase


class MyApp(ShowBase):

    def __init__(self):
        ShowBase.__init__(self, windowType="offscreen")


app = MyApp()
app.run()

When running the test script, I’m still getting

Known pipe types:
  eglGraphicsPipe
(all display modules loaded.)
:display:egldisplay(warning): Couldn't initialize the default EGL display: EGL_NOT_INITIALIZED
:display:egldisplay(error): Failed to find or initialize a suitable EGL display connection.
:ShowBase(warning): Unable to open 'offscreen' window.
Traceback (most recent call last):
  File "test.py", line 10, in <module>
    app = MyApp()
  File "test.py", line 7, in __init__
    ShowBase.__init__(self, windowType="offscreen")
  File "/home/users/eyuboglu/miniconda3/envs/win/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 335, in __init__
    self.openDefaultWindow(startDirect = False, props=props)
  File "/home/users/eyuboglu/miniconda3/envs/win/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 1015, in openDefaultWindow
    self.openMainWindow(*args, **kw)
  File "/home/users/eyuboglu/miniconda3/envs/win/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 1050, in openMainWindow
    self.openWindow(*args, **kw)
  File "/home/users/eyuboglu/miniconda3/envs/win/lib/python3.8/site-packages/direct/showbase/ShowBase.py", line 795, in openWindow
    raise Exception('Could not open window.')
Exception: Could not open window.

Could the changes I introduced have broken anything? I’ve also attached the build log. gpu_build_log.txt.zip (9.0 KB)

rdb · March 1, 2021, 10:33pm

As far as I can tell (without debug-level output), your EGL implementation is just not showing any supported devices. You could set notify-level-display debug in Config.prc to get more debug info output, which might be helpful. But, I suspect at this point that the problem doesn’t lie on Panda’s end.

It may be helpful at this point to try to run eglinfo or other OpenGL applications to see whether EGL and OpenGL work at all on your system, or whether there’s still a driver configuration issue to sort out.