Debugging Pipelining Deadlock

So my game runs fine under 1.8.0. With all my shader hacks though. I have a good deal of draw, and app time, so I tried out the pipelining. All pipelined choices behave the same: my map loads properly, a few (maybe 5-10) frames render, then deadlock with the CPU idle.

I disabled all my fancy features so I only have one buffer in use with no aux bitplanes. Same result.

Full text-states output up to deadlock: dpaste.com/704830/

I ran with “notify-level-gobj spam” and got no abnormal errors. The last thing it did was a ton of lines like:

It does not seem to be limited to my project. The Tut-Shadow-Mapping-Basic.py sample also fails with the deadlock (but before rendering anything but a white screen) if I throw a loadPrcFileData(“”, “threading-model /Draw”) at the top.

It does however work for Tut-Carousel, and gives a bit of a frame rate boost.

I haven’t tried anything other than these, I can test more if desired.

I’m on Mac OSX 10.6.8.

Is there any more information I can provide, or things I can try to help debug this?

A minimal version of Tut-Shadow-Mapping-Basic.py that deadlocks:


from panda3d.core import *
loadPrcFileData("", "threading-model /Draw")
import direct.directbase.DirectStart

class World():
    def __init__(self):
        # Load the scene.
        cm=CardMaker('')
        floor = render.attachNewNode(PandaNode("floor"))
        for y in range(12):
            for x in range(12):
                nn = floor.attachNewNode(cm.generate())
                nn.setPos((x-6)*4, (y-6)*4, 0)

        self.light = render.attachNewNode(Spotlight("Spot"))
        self.light.node().setScene(render)
        self.light.node().setShadowCaster(True)
        render.setLight(self.light)

        # Important! Enable the shader generator.
        render.setShaderAuto()

World()
run()

And a sample of it deadlocked. It really looks like a classic dependency cycle type deadlock.
(Sorry for the wrapping of the long lines. Paste into a text editor for reading)
Also, whats “FMOD::SystemI::createSoundInternal” doing in there? I don’t own an FMOD license, and there is no sound in this application! Panda shouldn’t be calling it for no reason.

Sampling process 1862 for 3 seconds with 1 millisecond of run time between samples
Sampling completed, processing symbols...
Analysis of sampling Python (pid 1862) every 1 millisecond
Call graph:
    2343 Thread_220076   DispatchQueue_1: com.apple.main-thread  (serial)
      2343 0x1ea9
        2343 0x1f82
          2343 Py_Main
            2343 PyRun_SimpleFileExFlags
              2343 PyRun_FileExFlags
                2343 PyEval_EvalCode
                  2343 PyEval_EvalCodeEx
                    2343 PyEval_EvalFrameEx
                      2343 PyEval_EvalFrameEx
                        2343 PyEval_EvalCodeEx
                          2343 PyEval_EvalFrameEx
                            2343 PyEval_EvalFrameEx
                              2343 Dtool_AsyncTaskManager_poll_121(_object*, _object*, _object*)
                                2343 AsyncTaskManager::poll()
                                  2343 AsyncTaskChain::do_poll()
                                    2343 AsyncTaskChain::service_one_task(AsyncTaskChain::AsyncTaskChainThread*)
                                      2343 AsyncTask::unlock_and_do_task()
                                        2343 PythonTask::do_task()
                                          2343 PythonTask::do_python_task()
                                            2343 Thread::call_python_func(_object*, _object*)
                                              2343 PyObject_Call
                                                2343 instancemethod_call
                                                  2343 PyObject_Call
                                                    2343 function_call
                                                      2343 PyEval_EvalCodeEx
                                                        2343 PyEval_EvalFrameEx
                                                          2343 Dtool_GraphicsEngine_render_frame_611(_object*, _object*, _object*)
                                                            2343 GraphicsEngine::render_frame()
                                                              2343 pthread_mutex_lock
                                                                2343 semaphore_wait_signal_trap
    2343 Thread_220080   DispatchQueue_2: com.apple.libdispatch-manager  (serial)
      2343 start_wqthread
        2343 _pthread_wqthread
          2343 _dispatch_worker_thread2
            2343 _dispatch_queue_invoke
              2343 _dispatch_mgr_invoke
                2343 kevent
    2343 Thread_220083
      2343 thread_start
        2343 _pthread_start
          2343 ThreadPosixImpl::root_func(void*)
            2343 GraphicsEngine::RenderThread::thread_main()
              2343 GraphicsEngine::WindowRenderer::do_frame(GraphicsEngine*, Thread*)
                2343 GraphicsEngine::draw_bins(ov_set<PointerTo<GraphicsOutput>, IndirectLess<GraphicsOutput> > const&, Thread*)
                  2343 GraphicsEngine::draw_bins(GraphicsOutput*, DisplayRegion*, Thread*)
                    2343 GraphicsEngine::do_draw(CullResult*, SceneSetup*, GraphicsOutput*, DisplayRegion*, Thread*)
                      2343 CullResult::draw(Thread*)
                        2343 CullBinStateSorted::draw(bool, Thread*)
                          2343 GLGraphicsStateGuardian::set_state_and_transform(RenderState const*, TransformState const*)
                            2343 ShaderGenerator::synthesize_shader(RenderState const*)
                              2343 ShaderGenerator::create_shader_attrib(std::string const&)
                                2343 ShaderGenerator::update_shadow_buffer(NodePath)
                                  2343 GraphicsStateGuardian::make_shadow_buffer(NodePath const&, GraphicsOutputBase*)
                                    2343 GraphicsEngine::make_output(GraphicsPipe*, std::string const&, int, FrameBufferProperties const&, WindowProperties const&, int, GraphicsStateGuardian*, GraphicsOutput*)
                                      2343 GraphicsEngine::get_threading_model() const
                                        2343 pthread_mutex_lock
                                          2343 semaphore_wait_signal_trap
    2343 Thread_220091
      2343 thread_start
        2343 _pthread_start
          2343 CAPThread::Entry(CAPThread*)
            2343 HP_IOThread::ThreadEntry(HP_IOThread*)
              2343 HP_IOThread::WorkLoop()
                2209 CAGuard::WaitUntil(unsigned long long)
                  2209 CAGuard::WaitFor(unsigned long long)
                    2208 pthread_cond_timedwait_relative_np
                      2208 _pthread_cond_wait
                        2200 semaphore_timedwait_signal_trap
                        4 _pthread_cond_wait
                        2 _pthread_cond_remove
                          1 __spin_lock
                          1 _pthread_cond_remove
                        1 OSSpinLockLock
                        1 pthread_mutex_lock
                          1 semaphore_wait_trap
                    1 CAGuard::WaitFor(unsigned long long)
                134 HP_IOThread::PerformIO(AudioTimeStamp const&, double)
                  134 IOA_Device::CallIOProcs(AudioTimeStamp const&, AudioTimeStamp const&, AudioTimeStamp const&)
                    129 IOA_SingleDevice::WriteOutputData(AudioTimeStamp const&, unsigned long)
                      129 IOA_HWDevice::WriteOutputData(unsigned long, unsigned long, unsigned long, unsigned long long)
                        129 iokit_user_client_trap
                    5 HP_IOProc::Call(AudioTimeStamp const&, AudioTimeStamp const&, AudioBufferList const*, AudioTimeStamp const&, AudioBufferList*)
                      5 AUGenericOutputEntry
                        3 AUGenericOutputEntry
                        1 0x70005bbb
                          1 0x70006d48
                            1 0x700085ff
                              1 0x70009021
                                1 0x19a150d7
                                  1 0x19a15090
                                    1 FMOD::GeometryI::getMemoryInfo(unsigned int, unsigned int, unsigned int*, FMOD_MEMORY_USAGE_DETAILS*)
                                      1 FMOD::DSPConnection::getInput(FMOD::DSP**)
                                        1 FMOD::DSPConnection::getInput(FMOD::DSP**)
                                          1 FMOD::DSPConnection::getInput(FMOD::DSP**)
                                            1 FMOD::DSPConnection::getInput(FMOD::DSP**)
                        1 AUMultibandCompressorEntry
                          1 pthread_self
    2343 Thread_220092
      2343 thread_start
        2343 _pthread_start
          2343 FMOD::SystemI::createSoundInternal(char const*, unsigned int, unsigned int, unsigned int, FMOD_CREATESOUNDEXINFO*, FMOD::File**, bool, FMOD::SoundI**)
            2324 0x19a144a6
              2324 usleep
                2324 nanosleep
                  2321 mach_wait_until
                  2 nanosleep
                  1 mach_absolute_time
            10 FMOD::SystemI::createSoundInternal(char const*, unsigned int, unsigned int, unsigned int, FMOD_CREATESOUNDEXINFO*, FMOD::File**, bool, FMOD::SoundI**)
            9 FMOD::SystemI::createChannelGroupInternal(char const*, FMOD::ChannelGroupI**, bool, bool)
              8 FMOD::SystemI::createSoundInternal(char const*, unsigned int, unsigned int, unsigned int, FMOD_CREATESOUNDEXINFO*, FMOD::File**, bool, FMOD::SoundI**)
                7 FMOD::SystemI::createSoundInternal(char const*, unsigned int, unsigned int, unsigned int, FMOD_CREATESOUNDEXINFO*, FMOD::File**, bool, FMOD::SoundI**)
                1 0x19a1452a
                  1 gettimeofday
              1 0x19a0cec9
                1 pthread_mutex_lock

Total number in stack (recursive counted multiple, when >=5):
        5       PyEval_EvalFrameEx

Sort by top of stack, same collapsed (when >= 5):
        semaphore_wait_signal_trap        4686
        kevent        2343
        mach_wait_until        2321
        semaphore_timedwait_signal_trap        2200
        iokit_user_client_trap        129
        FMOD::SystemI::createSoundInternal(char const*, unsigned int, unsigned int, unsigned int, FMOD_CREATESOUNDEXINFO*, FMOD::File**, bool, FMOD::SoundI**)        17
Sample analysis of process 1862 written to file /dev/stdout