I am running into a problem with my Panda build, where the game will crash after “running around” in the game for 30 seconds or so when running on my co-workers computers.
About my build:
Using source code from CVS, Sept. 4th.
I am using makepanda, with the following commandline:
The crash seems to be related to FMOD, if I set “audio-library-name null” it does not crash. I do have some changes to the FMOD code, but they worked in 1.7.0, and FMOD code only had one minor change since then that I saw.
The module is “libpanda.dll”.
After installing Visual C++ 2008 (express) on the machine it does not crash.
I did get the game to crash once on my machine while debugging in Visual C++, and it pointed to Thread::force_yield(), specifically the line “ThreadImpl::yield();”. I don’t use threading of any kind in Python, but maybe Panda uses some threads by default.
I was wondering if anyone has run into a similar problem or could help shed light on this. Have I built some kind of debug build by accident? Has the build system changed in some way recently?
Hmm, that’s pretty mysterious all right. How certain are you that the game does not crash in the named crashless situations? It seems strange that a crash in libpanda.dll could be related to FMod, or that installing Visual Studio has any effect on the game after it has started up. Nothing’s impossible, of course, but it is certainly strange, and if these are truly related, then this is quite baffling.
It is in Python.
I was getting the crashing myself, until I started debugging in Visual C++ at which point the problem mysteriously went away.
I tried it on my second machine and got the same crashing, until I installed Visual C++ on it. It would crash usually in no more than a minute, and with this I was able to run around for 15 minutes no problem. Same scenario with audio library set to null. After uninstalling Visual C++ and rebooting, the crashing returns.
Only thing I can think of is Visual C++ is installing some different DLL files or something like that.
Yeah I think I know what’s causing this, it’s makepanda’s fault and the runtime dlls it includes, or rather, the outdated thirdparty package, it’s currently a mess.
For example, the thirdparty package for windows currently includes version 9.00.21022.8 of msvcr90.dll. But my visual studio 2008 SP-1 comes with version 9.00.30729.1. It seems to me that even though the official distribution is built with Visual studio 2008 SP-1, the runtime dll’s hasn’t been updated and they are vanilla 2008. This would explain why you can get your app to run correctly when you install visual studio (because the right dlls are getting registered). Why they fail for you and not for everybody is a mystery, but I guess that the old DLL’s are just buggy or that Microsoft has introduced something through window update that breaks the non-SP1 dlls, or w/e. IMHO we need an overhaul of the build process, preferably remove ppremake and makepanda so that we can have an universal build system that won’t get outdated so easily.
Anyway, about your original issue, try replacing the runtime dlls found in “panda3d\thirdparty\win-libs-vc9\extras\bin” with those found in your visual studio dir at “C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\redist\x86”. Then let makepanda copy the dlls it fancies again and try.
Well it definitely seems like a DLL file version problem.
Looks like the DLL files in the Panda directory are not being used at all. I used the program “Who’s Locking” to find out which DLL file was being used, and on my computer it is loading
“C:\WINDOWS\WinSxS\x86_Microsoft.VC90.CRT_1fc8b3b9a1e18e3b_9.0.30729.4148_x-ww_d495ac4e\msvcr90.dll”,
whereas on my co-worker’s computer it is loading
“C:\Windows\winsxs\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.21022.8_none_bcb86ed6ac711f91\msvcr90.dll”.
Infact I can delete all of the msvc*.dll files and the result is the same.
I am using XP, and he is using Vista.
Right so notice that in your computer you are running the 2008-SP1 runtime dlls (see the 9.0.30729 in the string) whereas your co-worker’s computer only has the vanilla 2008 dlls (9.0.21022).
So I would venture that your co-worker computer is producing the crash, right? Which isn’t surprising because panda wasn’t built with that version of visual studio.
You have to redistribute the runtime with your app, copy the proper dlls along with the manifest files to your app dir. That or make your installer install the proper redistributable package.
Looks like the manifest files were actually pointing to the 9.0.21022.8 version. I guess they didn’t get overwritten when I rebuilt.
So it seems his computer was loading the DLLs as defined by the manifest file (rightly loading the wrong file), whereas mine was loading the newest version (wrongly loading the right file), which is probably the result of installing Visual C++.
I cleaned out the built directory and am rebuilding again, looks like the manifest files are correct so far. Will let you know how it turns out.
Well that seems to have done the trick!
I still find it strange that with Visual C++ it was loading a different version of the DLL, but I’m satisfied with this solution.
Thanks for pointing me in the right direction with the DLL versions, I owe ya one!
Well, perhaps I spoke too soon, the crashing came back like a… well like crashing.
The real problem does look like an FMOD/threads problem, and the Visual C++ “fix” seems to have been more related to rebooting after installing it.
Not all bad news, however! The cause seems to be this little bit of PRC data:
client-cpu-affinity-mask 1
The game runs in a client-server model (two separate processes, no threads) and communicates with datagrams even for the single player component, which is why the client was being forced onto the first core, and the server onto the second.
I think there is no need to force the client though since it will go on the first core by default.
So while I think I might consider this a bug, hopefully I have a real workaround for it now.
I even managed to snag a screenshot of this elusive beast.
If anything, I hope someone with the same problem will stumble upon this thread and save themselves a bit of headache.
Oh boy, another false solution. That seems to have just made it occur less often.
I find it very suspect that there is a line in the source commented out saying “we can’t yield here”, followed by Panda crashing on a function called “force_yield”.
What exactly is Panda trying to do, as seen in my screen capture, when it crashes?
OK I think I actually really got it to stop crashing now.
I compiled with threads disabled by setting “HAVE_THREADS”, “SIMPLE_THREADS”, and “OS_SIMPLE_THREADS” to ‘UNDEF’ in makepanda.py.
Also it is important to note that setting ‘support-threads 0’ with PRC did NOT stop the crashing.
So if in reality this has fixed it, looks like there is some instability using FMOD with threads enabled.
Ah, indeed. Now that I look at the code, it does appear that the FModAudioManager is making callbacks into Panda to read the audio data from vfs, which is going to cause problems if this callback is made in a sub-thread due to FMod’s own operations.
Completely disabling threads probably isn’t the best way to solve the problem, because now you might have a race condition within Panda if the audio thread happens to access the vfs at the same time as the Panda main thread. A better workaround should be to compile Panda with true threading support, which means HAVE_THREADS should be ‘1’ and SIMPLE_THREADS should be ‘UNDEF’.
An even better solution will be to ask FMod not to use threads, if possible. Failing that, we’ll need to add code to the FModAudioManager not to use the callback approach except in the case of true threads being enabled in Panda. This will, of course, prevent loading audio files from a mf file or p3d file, which is unfortunate.
