MSVC2010, successful compilation but corrupt heap

Mrakoplaz · May 17, 2012, 11:25am

Right, against all recommendation and common sense, I am attempting to run Panda3D 1.8.0 with Visual Studio 2010. I’ve had good times when compiling it in 2008, but that version does not support any 0x features, which our new codebase requires.
To avoid the thirdparty library issues (until I get a bare-bones version working), I’ve ran makepanda with only the absolute minimum settings (–nothing --use-gl --optimize 1).
Everything works well, nothing suspicious, but when I try a bare minimum program, it crashes on framework.open_window(), with the CRT debugger reporting a corrupt heap. Furthermore, the console is filled with messages like:

Non utf-8 byte in string: 0xff, string is '        ⁿ  ☻åwê↓)       )  !å'

Various permutations on this error occur with all the different --optimize values and debug/release configurations.

Here is the call stack:

 	msvcr90d.dll!5fea0d3b() 	
 	[Frames below may be incorrect and/or missing, no symbols loaded for msvcr90d.dll]	
 	msvcr90d.dll!5fea09e0() 	
 	msvcr90d.dll!5fea8990() 	
>	libp3dtool_d.dll!MemoryHook::heap_free_array(void * ptr)  Line 377 + 0xc bytes	C++
 	libpandaexpress_d.dll!MemoryUsage::heap_free_array(void * ptr)  Line 363 + 0xf bytes	C++
 	libpandagl_d.dll!GLGraphicsStateGuardian::reset()  Line 746 + 0x1d bytes	C++
 	libpandagl_d.dll!wglGraphicsStateGuardian::reset()  Line 458	C++
 	libpanda_d.dll!GraphicsStateGuardian::reset_if_new()  Line 845 + 0x12 bytes	C++
 	libpandagl_d.dll!wglGraphicsWindow::open_window()  Line 309 + 0xb bytes	C++
 	libpanda_d.dll!GraphicsWindow::set_properties_now(WindowProperties & properties)  Line 644 + 0x12 bytes	C++
 	libp3windisplay_d.dll!WinGraphicsWindow::set_properties_now(WindowProperties & properties)  Line 273 + 0xf bytes	C++
 	libpanda_d.dll!GraphicsWindow::process_events()  Line 604 + 0x19 bytes	C++
 	libp3windisplay_d.dll!WinGraphicsWindow::process_events()  Line 224 + 0xb bytes	C++
 	libpanda_d.dll!GraphicsEngine::process_events(const ov_set<PointerTo<GraphicsOutput>,IndirectLess<GraphicsOutput> > & wlist, Thread * current_thread)  Line 1616 + 0x25 bytes	C++
 	libpanda_d.dll!GraphicsEngine::WindowRenderer::do_windows(GraphicsEngine * engine, Thread * current_thread)  Line 2561	C++
 	libpanda_d.dll!GraphicsEngine::open_windows()  Line 955	C++
 	libpanda_d.dll!GraphicsEngine::make_output(GraphicsPipe * pipe, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & name, int sort, const FrameBufferProperties & fb_prop, const WindowProperties & win_prop, int flags, GraphicsStateGuardian * gsg, GraphicsOutput * host)  Line 425	C++
 	libp3framework_d.dll!WindowFramework::open_window(const WindowProperties & props, int flags, GraphicsEngine * engine, GraphicsPipe * pipe, GraphicsStateGuardian * gsg)  Line 161 + 0x36 bytes	C++
 	libp3framework_d.dll!PandaFramework::open_window(const WindowProperties & props, int flags, GraphicsPipe * pipe, GraphicsStateGuardian * gsg)  Line 453 + 0x28 bytes	C++
 	libp3framework_d.dll!PandaFramework::open_window(GraphicsPipe * pipe, GraphicsStateGuardian * gsg)  Line 419 + 0x1e bytes	C++
 	libp3framework_d.dll!PandaFramework::open_window()  Line 371 + 0xe bytes	C++
 	panda_test.exe!main(int argc, char * * argv)  Line 22	C++
 	panda_test.exe!__tmainCRTStartup()  Line 555 + 0x19 bytes	C
 	panda_test.exe!mainCRTStartup()  Line 371	C
 	kernel32.dll!74eb339a() 	
 	ntdll.dll!77799ef2() 	
 	ntdll.dll!77799ec5()

One last thing I noted, in the list of loaded modules, there is both the vc100 dlls (e.g. msvcr100d) AND the vc90 ones (e.g. msvcr90d). Is this a Very Bad Thing?

Thanks in advance for anything that could be even vaguely helpful

drwr · May 17, 2012, 4:45pm

First, make sure that you are linking and running with your dll’s that you compiled and not with the standard Panda dll’s (probably just uninstall any other versions of Panda that you have installed).

But the error message seems like a conflict between Unicode and ANSI builds. Make sure your project is set to ANSI; I think this is the way that makepanda compiles Panda.

David

Mrakoplaz · May 17, 2012, 5:10pm

ANSI is the “Multi byte character set”, right? I can confirm that it’s enabled (no unicode anywhere), and that I’ve deleted all other versions of Panda from the computer.

Having said all that, however, there’s now been an update to my situation:
When trying to uninstall/reinstall some of the vc redistributables + WinSDK, I’ve found out that I had a corrupt installation (which just wouldn’t uninstall from the Add/Remove programs). Simple enough, I get a program from Microsoft’s support pages to eliminate these stuck-uninstallers, and get rid of it.

But now, I am back to square one, as Panda no longer builds, even after re-installing WinSDK back. I get this nice error:

Cannot read built/bin/libp3dtool.dll.manifest

And indeed, if we check the bin directory, there’s the dll/pdb files, but no manifest.
After a bit of digging around in makepanda, I gathered that it seems to fail when the linker is called with the following settings (on the default optimize 3):

link /nologo /NOD:MFC90.LIB /NOD:MFC80.LIB /NOD:LIBCMT /NOD:LIBCI.LIB /DEBUG /nod:libc /nod:libcmtd /nod:atlthunk /nod:atls /DLL /MAP:NUL /NOD:MSVCRTD.LIB /NOD:MSVCPRTD.LIB /NOD:MSVCIRTD.LIB /FIXED:NO /OPT:REF /STACK:4194304 /INCREMENTAL:NO  /OUT:built/bin/libp3dtool.dll /IMPLIB:built/lib /libp3dtool.lib built/tmp/p3dtool_dtool.obj built/tmp/p3dtoolutil_composite1.obj built/tmp/p3dtoolutil_composite2.obj built/tmp/p3dtoolbase_composite1.obj built /tmp/p3dtoolbase_composite2.obj built/tmp/p3dtoolbase_indent.obj built/tmp/p3dtoolbase_lookup3.obj kernel32.lib shell32.lib advapi32.lib

This happens when I run it as Administrator, so I don’t think it’s a permissions issue.
Any idea where that one comes from? Other people seem to have reported it too, judging by the other MSVC2010 thread in the forum (had I known the libs were corrupt before, I’d have fixed that first, then posted there).

This is all on Win7-64, by the way.

EDIT 1: I added:

 cmd+= " /MANIFEST"

to makepanda’s CompileLink(), and it’s running now. I’ll edit again to inform of the results…

EDIT 2: It’s compiled now, but gives the exact same error (heap corruption). Even the exact same stack! However, there are no more “non utf-8 byte” messages in the console. Furthermore, I also notice that the new Panda libraries are now all linked against vc100 dlls (i.e. msvcr100d instead of msvcr90d)

Mrakoplaz · May 18, 2012, 6:56pm

Okay, I feel like an idiot now. I spent so much time trying to install the new Windows SDK (whose installer doesn’t work, and which wasn’t actually needed in the end, since MSVC2010 now includes all the necessary headers out of the box), that I missed the most obvious. Updated the graphics card drivers, and everything’s good now.
So, yeah. Thanks for help, drwr, despite me clearly not deserving it!

PS: I still think that the /MANIFEST command should be added to makepanda’s CompileLink() when MSVC2010 is detected (since it doesn’t link otherwise, returning with the “cannot find ***.dll.manifest” error).