Deadlock! All threads blocked, while loading models

Ubuntu 9.4 64 bit, running cvs head.

When loading models now I first get a

Then I get

Here is the trace back:

0  0x00007f024a3b1fb5 in raise () from /lib/libc.so.6
#1  0x00007f024a3b3bc3 in abort () from /lib/libc.so.6
#2  0x00007f0247e2f811 in ThreadSimpleManager::choose_next_context () from /usr/lib64/panda3d/libpanda.so
#3  0x00007f0247e35fa7 in save_thread_context () from /usr/lib64/panda3d/libpanda.so
#4  0x00007f0247e2a427 in ThreadSimpleManager::next_context () from /usr/lib64/panda3d/libpanda.so
#5  0x00007f0247e3024e in MutexSimpleImpl::do_acquire () from /usr/lib64/panda3d/libpanda.so
#6  0x00007f0247f24180 in TexturePool::ns_load_texture () from /usr/lib64/panda3d/libpanda.so
#7  0x00007f0247f316f7 in Texture::make_from_bam () from /usr/lib64/panda3d/libpanda.so
#8  0x00007f02483187de in FactoryBase::make_instance_more_general () from /usr/lib64/panda3d/libpanda.so
#9  0x00007f0248326898 in BamReader::p_read_object () from /usr/lib64/panda3d/libpanda.so
#10 0x00007f0248326f58 in BamReader::read_object () from /usr/lib64/panda3d/libpanda.so
#11 0x00007f02483277ec in BamCache::do_read_record () from /usr/lib64/panda3d/libpanda.so
#12 0x00007f0248328563 in BamCache::read_record () from /usr/lib64/panda3d/libpanda.so
#13 0x00007f0248328a0f in BamCache::find_and_read_record () from /usr/lib64/panda3d/libpanda.so
#14 0x00007f024832aa6c in BamCache::lookup () from /usr/lib64/panda3d/libpanda.so
#15 0x00007f0247a2f394 in Loader::try_load_file () from /usr/lib64/panda3d/libpanda.so
#16 0x00007f0247a2fe6d in Loader::load_file () from /usr/lib64/panda3d/libpanda.so
#17 0x00007f0247a30733 in ModelPool::ns_load_model () from /usr/lib64/panda3d/libpanda.so
#18 0x00007f0247a2eeaa in Loader::try_load_file () from /usr/lib64/panda3d/libpanda.so
#19 0x00007f0247a300a9 in Loader::load_file () from /usr/lib64/panda3d/libpanda.so
#20 0x00007f0247baab26 in Dtool_Loader_load_sync_1210 () from /usr/lib64/panda3d/libpanda.so
#21 0x00000000004a2b03 in PyEval_EvalFrameEx ()
#22 0x00000000004a4649 in PyEval_EvalCodeEx ()
#23 0x00000000004a2cb0 in PyEval_EvalFrameEx ()
#24 0x00000000004a4649 in PyEval_EvalCodeEx ()
#25 0x00000000005329ad in ?? ()
#26 0x000000000041d3bd in PyObject_Call ()
#27 0x0000000000424f48 in ?? ()
#28 0x000000000041d3bd in PyObject_Call ()
#29 0x000000000049cd46 in PyEval_CallObjectWithKeywords ()
#30 0x0000000000427d16 in PyInstance_New ()
#31 0x000000000041d3bd in PyObject_Call ()
#32 0x00000000004a000f in PyEval_EvalFrameEx ()
#33 0x00000000004a3dae in PyEval_EvalFrameEx ()
#34 0x00000000004a3dae in PyEval_EvalFrameEx ()
#35 0x00000000004a3dae in PyEval_EvalFrameEx ()
#36 0x00000000004a4649 in PyEval_EvalCodeEx ()
#37 0x0000000000532ab4 in ?? ()
#38 0x000000000041d3bd in PyObject_Call ()
#39 0x00000000004a1460 in PyEval_EvalFrameEx ()
#40 0x00000000004a4649 in PyEval_EvalCodeEx ()
#41 0x00000000005329ad in ?? ()
#42 0x000000000041d3bd in PyObject_Call ()
#43 0x0000000000424f48 in ?? ()
#44 0x000000000041d3bd in PyObject_Call ()
#45 0x000000000047519c in ?? ()
#46 0x000000000041d3bd in PyObject_Call ()
#47 0x00000000004a000f in PyEval_EvalFrameEx ()
#48 0x00000000004a4649 in PyEval_EvalCodeEx ()
#49 0x0000000000532ab4 in ?? ()
#50 0x000000000041d3bd in PyObject_Call ()
#51 0x00000000004a1460 in PyEval_EvalFrameEx ()

Code that breaks for me:

import direct.directbase.DirectStart
model = loader.loadModel("data/units/sword/sword.bam")

What can I do?

Running pview also crashes:

pview sword.bam 
Known pipe types:
  glxGraphicsPipe
(all display modules loaded.)
Loading sword.bam
Assertion failed: (_flags & F_lock_count) != 0 at line 68 of built/include/mutexSimpleImpl.I
:thread(error): Deadlock!  All threads blocked.
Aborted

here is the evil bam file:
aff2aw.com/affdata/panda3d/sword.bam

Hmm, I’m not able to reproduce this. Perhaps something related to the 64-bit build? I’d like to see the stack from the first assertion. Please add:

assert-abort 1

to your Config.prc file and re-run (within gdb), and show me the stack trace. Thanks!

David

i had the same problem on win 7 64, but dont remember whats wrong and how i fixed :frowning:
try converting to egg

i get this when trieng to convert

Loading sword.bam
:util:bam(error): Bam file is version 6.22.
:util:bam(error): This program can only load version 6.14 through 6.19 bams.
:loader(error): Couldn't load file sword.bam: invalid.
Unable to load sword.bam

I though this will go away after I installed the most recent version of ubuntu 9.10 64 bit from the nightly builder: But it looks like its still there as my unit tests say:

======================================================================
FAIL: testModelLoad (client.tests.clienttest.ShowSimTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/p/2awr/client/tests/clienttest.py", line 32, in testModelLoad
    model = loader.loadModel("data/units/breeze/breeze-clean.egg")
  File "/usr/share/panda3d/direct/showbase/Loader.py", line 159, in loadModel
    node = self.loader.loadSync(Filename(modelPath), loaderOptions)
AssertionError: (_flags & F_lock_count) != 0 at line 68 of built/include/mutexSimpleImpl.I

Eraslt, i get this problem with egg or bam. Does not matter.

me@centurion:/p/2awr/test$ gdb python
GNU gdb (GDB) 7.0-ubuntu
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/python...(no debugging symbols found)...done.
(gdb) run mutexbug.py
Starting program: /usr/bin/python mutexbug.py
[Thread debugging using libthread_db enabled]
DirectStart: Starting the game.
Known pipe types:
  glxGraphicsPipe
(all display modules loaded.)
:audio(error):   load_dso(libp3openal_audio.so) failed, will use NullAudioManager
:audio(error):     libopenal.so.1: cannot open shared object file: No such file or directory
:loader(debug): Attempt to register loader library pandaegg (egg) when extension is already known.
:loader: loading file type module: p3ptloader
Assertion failed: (_flags & F_lock_count) != 0 at line 68 of built/include/mutexSimpleImpl.I

Program received signal SIGABRT, Aborted.
0x00007ffff6fe54b5 in raise () from /lib/libc.so.6
(gdb) bt
#0  0x00007ffff6fe54b5 in raise () from /lib/libc.so.6
#1  0x00007ffff6fe8f50 in abort () from /lib/libc.so.6
#2  0x00007ffff661f37c in Notify::assert_failure(char const*, int, char const*) () from /usr/lib64/panda3d/libp3dtoolconfig.so
#3  0x00007ffff459b2f0 in MutexSimpleImpl::release() () from /usr/lib64/panda3d/libpanda.so
#4  0x00007ffff4b33c0d in Texture::do_unlock_and_reload_ram_image(bool) () from /usr/lib64/panda3d/libpanda.so
#5  0x00007ffff4b33edd in Texture::do_reload() () from /usr/lib64/panda3d/libpanda.so
#6  0x00007ffff4b36cc0 in Texture::fillin_from(Texture*) () from /usr/lib64/panda3d/libpanda.so
#7  0x00007ffff4b506f6 in Texture::make_from_bam(FactoryParams const&) () from /usr/lib64/panda3d/libpanda.so
#8  0x00007ffff4f21bee in FactoryBase::make_instance_more_general(TypeHandle, FactoryParams const&) () from /usr/lib64/panda3d/libpanda.so
#9  0x00007ffff4f2cd54 in BamReader::p_read_object() () from /usr/lib64/panda3d/libpanda.so
#10 0x00007ffff4f2d428 in BamReader::read_object(TypedWritable*&, ReferenceCount*&) () from /usr/lib64/panda3d/libpanda.so
#11 0x00007ffff4f2dbc2 in BamCache::do_read_record(Filename&, bool) () from /usr/lib64/panda3d/libpanda.so
#12 0x00007ffff4f2de49 in BamCache::read_record(Filename const&, Filename const&, int) () from /usr/lib64/panda3d/libpanda.so
#13 0x00007ffff4f2e2cf in BamCache::find_and_read_record(Filename const&, Filename const&) () from /usr/lib64/panda3d/libpanda.so
#14 0x00007ffff4f333f6 in BamCache::lookup(Filename const&, std::string const&) () from /usr/lib64/panda3d/libpanda.so
#15 0x00007ffff466df3a in Loader::try_load_file(Filename const&, LoaderOptions const&, LoaderFileType*) const () from /usr/lib64/panda3d/libpanda.so
#16 0x00007ffff466e78d in Loader::load_file(Filename const&, LoaderOptions const&) const () from /usr/lib64/panda3d/libpanda.so
#17 0x00007ffff466d8ef in ModelPool::ns_load_model(Filename const&, LoaderOptions const&) () from /usr/lib64/panda3d/libpanda.so
#18 0x00007ffff466dc15 in Loader::try_load_file(Filename const&, LoaderOptions const&, LoaderFileType*) const () from /usr/lib64/panda3d/libpanda.so
#19 0x00007ffff466e619 in Loader::load_file(Filename const&, LoaderOptions const&) const () from /usr/lib64/panda3d/libpanda.so
#20 0x00007ffff481f28f in Loader::load_sync(Filename const&, LoaderOptions const&) const () from /usr/lib64/panda3d/libpanda.so
#21 0x00007ffff47e18f9 in Dtool_Loader_load_sync_1210(_object*, _object*, _object*) () from /usr/lib64/panda3d/libpanda.so
#22 0x00000000004a290d in PyEval_EvalFrameEx ()
#23 0x00000000004a40e0 in PyEval_EvalCodeEx ()
#24 0x00000000004a245f in PyEval_EvalFrameEx ()
#25 0x00000000004a40e0 in PyEval_EvalCodeEx ()
#26 0x00000000004a41b2 in PyEval_EvalCode ()
#27 0x00000000004c33a0 in PyRun_FileExFlags ()
#28 0x00000000004c3564 in PyRun_SimpleFileExFlags ()
#29 0x0000000000418ab7 in Py_Main ()
#30 0x00007ffff6fd0abd in __libc_start_main () from /lib/libc.so.6
#31 0x0000000000417ca9 in _start ()
support-threads #f

Does not help.

The stack trace helps considerably, thanks! I just checked in a fix that I think will address this issue. Please let me know if it works. :slight_smile:

David

Yes I saw you committing it on irc. I tried to figure it out myself but that was a bit hard. I compiled panda3d without threads and that worked. Now I need to get your fix and compile with threads.

I just build and tested that seem to do the trick! Thanks!

I am trying to use packp3d.p3d and it deadlocks too!

me@centurion:/p/2awr/tools/viewer$ panda3d packp3d.p3d -o viewer.p3d
:loader: Reading /p/2awr/tools/viewer/breeze.bam
:gobj: Loading texture /p/2awr/tools/viewer/breeze-rgba.png
:gobj: Reloading texture breeze-rgba
:thread(error): Deadlock!  All threads blocked.
:thread: On blocker 0x2e0d9b8:
:thread:   MainThread Main
me@centurion:/p/2awr/tools/viewer$ gdb panda3d
GNU gdb (GDB) 7.0-ubuntu
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/panda3d...(no debugging symbols found)...done.
(gdb) run packp3d.p3d -o viewer.p3d
Starting program: /usr/bin/panda3d packp3d.p3d -o viewer.p3d
[Thread debugging using libthread_db enabled]
[New Thread 0x7ffff5346910 (LWP 8791)]
:loader: Reading /p/2awr/tools/viewer/breeze.bam
:gobj: Loading texture /p/2awr/tools/viewer/breeze-rgba.png
:gobj: Reloading texture breeze-rgba
:thread(error): Deadlock!  All threads blocked.
:thread: On blocker 0x1186068:
:thread:   MainThread Main
[Thread 0x7ffff5346910 (LWP 8791) exited]

Program received signal SIGPIPE, Broken pipe.
0x00007ffff73e504b in write () from /lib/libpthread.so.0
(gdb) bt
#0  0x00007ffff73e504b in write () from /lib/libpthread.so.0
#1  0x00007ffff5d23d99 in HandleStreamBuf::write_chars(char const*, unsigned long) () from /home/me/.panda3d/coreapi/linux_amd64/p3d_plugin.so
#2  0x00007ffff5d23e3d in HandleStreamBuf::sync() () from /home/me/.panda3d/coreapi/linux_amd64/p3d_plugin.so
#3  0x00007ffff6f5e272 in std::ostream::flush() () from /usr/lib/libstdc++.so.6
#4  0x00007ffff5d230d3 in write_xml(std::ostream&, TiXmlDocument*, std::ostream&) () from /home/me/.panda3d/coreapi/linux_amd64/p3d_plugin.so
#5  0x00007ffff5cd6e12 in P3DSession::drop_pyobj(int) () from /home/me/.panda3d/coreapi/linux_amd64/p3d_plugin.so
#6  0x00007ffff5cf0aa3 in P3DPythonObject::~P3DPythonObject() () from /home/me/.panda3d/coreapi/linux_amd64/p3d_plugin.so
#7  0x00007ffff5cd110e in P3DMainObject::set_pyobj(_P3D_object*) () from /home/me/.panda3d/coreapi/linux_amd64/p3d_plugin.so
#8  0x00007ffff5d0daf1 in P3DInstance::get_request() () from /home/me/.panda3d/coreapi/linux_amd64/p3d_plugin.so
#9  0x00007ffff5d0e02f in P3D_instance_get_request () from /home/me/.panda3d/coreapi/linux_amd64/p3d_plugin.so
#10 0x0000000000418042 in ?? ()
#11 0x0000000000413c04 in ?? ()
#12 0x0000000000415938 in ?? ()
#13 0x00007ffff66d7abd in __libc_start_main () from /lib/libc.so.6
#14 0x00000000004101b9 in ?? ()
#15 0x00007fffffffe3e8 in ?? ()
#16 0x000000000000001c in ?? ()
#17 0x0000000000000004 in ?? ()
#18 0x00007fffffffe6ae in ?? ()
#19 0x00007fffffffe6bf in ?? ()
#20 0x00007fffffffe6cb in ?? ()
#21 0x00007fffffffe6ce in ?? ()
#22 0x0000000000000000 in ?? ()
(gdb) 

That’s because packp3d.p3d uses the rtdist build from runtime.panda3d.org, which is not rebuilt with the fix yet.

ok, I though making my own panda3d runtime would include the fix. I guess not.

You’d have to host your own runtime somewhere, and build it with the URL to that address; and then you’d have to use the packp3d that was built with that URL.

David