Problems with MacOS Mojave

I just noticed that the latest valid build of Panda I made for mac os was with High Sierra. On Mojave, I had trouble compiling Panda and in the end, the sdk is unusable as it crashes when instancing CocoaGraphicsPipe()

I’m building using the following command, as I’m using Homebrew Python :

python3 makepanda/makepanda.py --everything --threads 4 --osxtarget=10.7 --python-incdir  /usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/include --python-libdir  /usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib --installer

First, building with bullet fails with the following errors :

[T3] Building C++ object built/tmp/p3bullet_composite.o
In file included from panda/src/bullet/p3bullet_composite.cxx:1:
In file included from panda/src/bullet/config_bullet.cxx:14:
In file included from panda/src/bullet/config_bullet.h:26:
In file included from panda/src/bullet/bulletWorld.h:19:
In file included from panda/src/bullet/bullet_includes.h:19:
In file included from thirdparty/darwin-libs-a/bullet/include/btBulletDynamicsCommon.h:20:
In file included from thirdparty/darwin-libs-a/bullet/include/btBulletCollisionCommon.h:22:
In file included from thirdparty/darwin-libs-a/bullet/include/BulletCollision/CollisionDispatch/btCollisionWorld.h:80:
thirdparty/darwin-libs-a/bullet/include/LinearMath/btVector3.h:330:7: error: argument value 10880 is outside the valid range [0, 255] [-Wargument-outside-range]
                y = bt_splat_ps(y, 0x80);
                    ^~~~~~~~~~~~~~~~~~~~
thirdparty/darwin-libs-a/bullet/include/LinearMath/btVector3.h:46:32: note: expanded from macro 'bt_splat_ps'
#define bt_splat_ps( _a, _i )  bt_pshufd_ps((_a), BT_SHUFFLE(_i,_i,_i,_i) )
                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
thirdparty/darwin-libs-a/bullet/include/LinearMath/btVector3.h:44:35: note: expanded from macro 'bt_pshufd_ps'
#define bt_pshufd_ps( _a, _mask ) _mm_shuffle_ps((_a), (_a), (_mask) )
                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.0/include/xmmintrin.h:2608:11: note: expanded from macro '_mm_shuffle_ps'
  (__m128)__builtin_ia32_shufps((__v4sf)(__m128)(a), (__v4sf)(__m128)(b), \
          ^
In file included from panda/src/bullet/p3bullet_composite.cxx:1:
In file included from panda/src/bullet/config_bullet.cxx:14:
In file included from panda/src/bullet/config_bullet.h:26:
In file included from panda/src/bullet/bulletWorld.h:19:
In file included from panda/src/bullet/bullet_includes.h:19:
In file included from thirdparty/darwin-libs-a/bullet/include/btBulletDynamicsCommon.h:20:
In file included from thirdparty/darwin-libs-a/bullet/include/btBulletCollisionCommon.h:22:
In file included from thirdparty/darwin-libs-a/bullet/include/BulletCollision/CollisionDispatch/btCollisionWorld.h:81:
In file included from thirdparty/darwin-libs-a/bullet/include/LinearMath/btTransform.h:21:
thirdparty/darwin-libs-a/bullet/include/LinearMath/btMatrix3x3.h:882:17: error: argument value 10880 is outside the valid range [0, 255] [-Wargument-outside-range]
    __m128 vk = bt_splat_ps(_mm_load_ss((float *)&k), 0x80);
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
thirdparty/darwin-libs-a/bullet/include/LinearMath/btVector3.h:46:32: note: expanded from macro 'bt_splat_ps'
#define bt_splat_ps( _a, _i )  bt_pshufd_ps((_a), BT_SHUFFLE(_i,_i,_i,_i) )
                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
thirdparty/darwin-libs-a/bullet/include/LinearMath/btVector3.h:44:35: note: expanded from macro 'bt_pshufd_ps'
#define bt_pshufd_ps( _a, _mask ) _mm_shuffle_ps((_a), (_a), (_mask) )
                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.0/include/xmmintrin.h:2608:11: note: expanded from macro '_mm_shuffle_ps'
  (__m128)__builtin_ia32_shufps((__v4sf)(__m128)(a), (__v4sf)(__m128)(b), \
          ^

Rebuilding using --no-bullet is successful, but with several warnings I hadn’t in the past related to the third party libraries (which are the latest though) :

[T1] Linking dynamic library built/lib/libpandaode.dylib

clang: **warning:** libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]

ld: warning: building for macOS, but linking in object file (thirdparty/darwin-libs-a/ode/lib/libode.a(ode.cpp.o)) built for

ld: warning: building for macOS, but linking in object file (thirdparty/darwin-libs-a/ode/lib/libode.a(collision_kernel.cpp.o)) built for

ld: warning: building for macOS, but linking in object file (thirdparty/darwin-libs-a/ode/lib/libode.a(box.cpp.o)) built for

...

ld: warning: building for macOS, but linking in object file (thirdparty/darwin-libs-a/ode/lib/libode.a(IceHPoint.cpp.o)) built for

With verbose, the correct SDK is selected though :

clang++ -undefined dynamic_lookup -dynamiclib -install_name @loader_path/../lib/libpandaexpress.1.10.dylib -compatibility_version 1.10 -current_version 1.10.5 -o built/lib/libpandaexpress.1.10.dylib -Lbuilt/lib -Lbuilt/tmp built/tmp/pandaexpress_pandaexpress.o built/tmp/p3downloader_composite1.o built/tmp/p3downloader_composite2.o built/tmp/p3express_composite1.o built/tmp/p3express_composite2.o built/tmp/p3pandabase_pandabase.o built/lib/libp3dtool.dylib built/lib/libp3dtoolconfig.dylib -headerpad_max_install_names -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk -Wl,-syslibroot,/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk -mmacosx-version-min=10.7 thirdparty/darwin-libs-a/openssl/lib/libssl.a thirdparty/darwin-libs-a/openssl/lib/libcrypto.a thirdparty/darwin-libs-a/zlib/lib/libz.a -framework AppKit -ldl

FWIW, I’m able to compile C++ modules for Panda on Mojave using the 10.7 SDK

I will investigate a bit, but any clue is welcome :slight_smile:

Curious, I’ve never seen this before. You’re building the 1.10 branch, right? Do you have a stack trace of the crash?

Here is the stack trace with debug information :

* thread #2, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xb8)
  * frame #0: 0x0000000103c318eb libpanda.1.10.dylib`GraphicsPipe::GraphicsPipe() [inlined] std::string::_Alloc_hider::_Alloc_hider(this=, __a=) at basic_string.h:262:17
    frame #1: 0x0000000103c318d6 libpanda.1.10.dylib`GraphicsPipe::GraphicsPipe() [inlined] std::string::_Alloc_hider::_Alloc_hider(this=, __a=) at basic_string.h:262
    frame #2: 0x0000000103c318d6 libpanda.1.10.dylib`GraphicsPipe::GraphicsPipe() [inlined] std::basic_string<char, std::char_traits, std::allocator >::basic_string(this=) at basic_string.h:2067
    frame #3: 0x0000000103c318d6 libpanda.1.10.dylib`GraphicsPipe::GraphicsPipe() [inlined] std::basic_string<char, std::char_traits, std::allocator >::basic_string(this=) at basic_string.h:2067
    frame #4: 0x0000000103c318d6 libpanda.1.10.dylib`GraphicsPipe::GraphicsPipe() [inlined] DisplayInformation::DisplayInformation(this=0x0000000000000000) at displayInformation.cxx:76
    frame #5: 0x0000000103c318d6 libpanda.1.10.dylib`GraphicsPipe::GraphicsPipe() [inlined] DisplayInformation::DisplayInformation(this=0x0000000000000000) at displayInformation.cxx:76
    frame #6: 0x0000000103c318d6 libpanda.1.10.dylib`GraphicsPipe::GraphicsPipe(this=0x0000000105e18810) at graphicsPipe.cxx:128
    frame #7: 0x0000000107c56415 libpandagl.dylib`CocoaGraphicsPipe::CocoaGraphicsPipe(this=0x0000000105e18810, display=69734406) at cocoaGraphicsPipe.mm:36:1
    frame #8: 0x0000000107c54dc8 libpandagl.dylib`CocoaGraphicsPipe::pipe_constructor() [inlined] CocoaGraphicsPipe::CocoaGraphicsPipe(this=, display=) at cocoaGraphicsPipe.mm:36:66
    frame #9: 0x0000000107c54dbe libpandagl.dylib`CocoaGraphicsPipe::pipe_constructor() at cocoaGraphicsPipe.mm:183
    frame #10: 0x0000000103c425e6 libpanda.1.10.dylib`GraphicsPipeSelection::make_default_pipe(this=0x0000000106f18820) at graphicsPipeSelection.cxx:286:29
    frame #11: 0x0000000102a76edb core.cpython-37m-darwin.so`Dtool_GraphicsPipeSelection_make_default_pipe_932(self=, (null)=) at libp3display_igate.cxx:30201:44
    frame #12: 0x00000001000e958e Python`_PyMethodDef_RawFastCallKeywords + 590
    frame #13: 0x00000001000eddd1 Python`_PyMethodDescr_FastCallKeywords + 81
    frame #14: 0x000000010017df27 Python`call_function + 801
    frame #15: 0x0000000100176c57 Python`_PyEval_EvalFrameDefault + 6396
    frame #16: 0x000000010017e7a6 Python`_PyEval_EvalCodeWithName + 1870
    frame #17: 0x00000001000e8a77 Python`_PyFunction_FastCallKeywords + 225
    frame #18: 0x000000010017def7 Python`call_function + 753
    frame #19: 0x0000000100176c57 Python`_PyEval_EvalFrameDefault + 6396
    frame #20: 0x000000010017e7a6 Python`_PyEval_EvalCodeWithName + 1870
    frame #21: 0x00000001000e8a77 Python`_PyFunction_FastCallKeywords + 225
    frame #22: 0x000000010017def7 Python`call_function + 753
    frame #23: 0x0000000100176db3 Python`_PyEval_EvalFrameDefault + 6744
    frame #24: 0x000000010017e7a6 Python`_PyEval_EvalCodeWithName + 1870
    frame #25: 0x00000001000e8a77 Python`_PyFunction_FastCallKeywords + 225
    frame #26: 0x000000010017def7 Python`call_function + 753
    frame #27: 0x0000000100176d0b Python`_PyEval_EvalFrameDefault + 6576
    frame #28: 0x000000010017e7a6 Python`_PyEval_EvalCodeWithName + 1870
    frame #29: 0x00000001000e86e5 Python`_PyFunction_FastCallDict + 441
    frame #30: 0x00000001000e9829 Python`_PyObject_Call_Prepend + 150
    frame #31: 0x00000001000e8bbf Python`PyObject_Call + 136
    frame #32: 0x0000000100176f43 Python`_PyEval_EvalFrameDefault + 7144
    frame #33: 0x000000010017e7a6 Python`_PyEval_EvalCodeWithName + 1870
    frame #34: 0x00000001000e86e5 Python`_PyFunction_FastCallDict + 441
    frame #35: 0x00000001000e9829 Python`_PyObject_Call_Prepend + 150
    frame #36: 0x00000001000e8bbf Python`PyObject_Call + 136
    frame #37: 0x0000000100176f43 Python`_PyEval_EvalFrameDefault + 7144
    frame #38: 0x000000010017e7a6 Python`_PyEval_EvalCodeWithName + 1870
    frame #39: 0x00000001000e8a77 Python`_PyFunction_FastCallKeywords + 225
    frame #40: 0x000000010017def7 Python`call_function + 753
    frame #41: 0x0000000100176db3 Python`_PyEval_EvalFrameDefault + 6744
    frame #42: 0x000000010017e7a6 Python`_PyEval_EvalCodeWithName + 1870
    frame #43: 0x00000001000e86e5 Python`_PyFunction_FastCallDict + 441
    frame #44: 0x00000001000e9829 Python`_PyObject_Call_Prepend + 150
    frame #45: 0x0000000100126f7d Python`slot_tp_init + 80
    frame #46: 0x0000000100123c20 Python`type_call + 172
    frame #47: 0x00000001000e8918 Python`_PyObject_FastCallKeywords + 358
    frame #48: 0x000000010017def0 Python`call_function + 746
    frame #49: 0x0000000100176d0b Python`_PyEval_EvalFrameDefault + 6576
    frame #50: 0x000000010017e7a6 Python`_PyEval_EvalCodeWithName + 1870
    frame #51: 0x00000001001752b8 Python`PyEval_EvalCode + 51
    frame #52: 0x00000001001a394b Python`run_mod + 54
    frame #53: 0x00000001001a2975 Python`PyRun_FileExFlags + 163
    frame #54: 0x00000001001a201b Python`PyRun_SimpleFileExFlags + 263
    frame #55: 0x00000001001ba89e Python`pymain_main + 5389
    frame #56: 0x00000001001baf80 Python`_Py_UnixMain + 56
    frame #57: 0x00007fff7c2723d5 libdyld.dylib`start + 1

The line causing the crash is :

* thread #2, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xb8)
    frame #0: 0x0000000103c318eb libpanda.1.10.dylib`GraphicsPipe::GraphicsPipe() [inlined] std::string::_Alloc_hider::_Alloc_hider(this=, __a=) at basic_string.h:262:17
   259 	      struct _Alloc_hider : _Alloc
   260 	      {
   261 		_Alloc_hider(_CharT* __dat, const _Alloc& __a)
-> 262 		: _Alloc(__a), _M_p(__dat) { }
   263 	
   264 		_CharT* _M_p; // The actual data.
   265 	      };

Though the root cause is probably at frame #6 where the constructor of DisplayInformation is called with a null pointer (or there is a stack corruption)

I checked the libs, they are correctly linked with the 10.7 SDK :

 $ otool -L /Developer/Panda3D/lib/libpanda.1.10.dylib 
/Developer/Panda3D/lib/libpanda.1.10.dylib:
	@loader_path/../lib/libpanda.1.10.dylib (compatibility version 1.10.0, current version 1.10.5)
	@loader_path/../lib/libpandaexpress.1.10.dylib (compatibility version 1.10.0, current version 1.10.5)
	@loader_path/../lib/libp3dtoolconfig.1.10.dylib (compatibility version 1.10.0, current version 1.10.5)
	@loader_path/../lib/libp3dtool.1.10.dylib (compatibility version 1.10.0, current version 1.10.5)
	@loader_path/../Frameworks/Cg.framework/Cg (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1138.47.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)
	/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 52.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 635.21.0)

I can confirm that I get the same problem on MacOS 10.14 and 10.15, though to be honest I’ve tried compiling bullet by itself on MacOS in the past and have never gotten it to work…

I had Panda (up to 1.10.3) compiling successfuly in the past, with SDK 10.7 and the official third-party dependencies, and I don’t remember having to disable bullet (I have the dog still available, I will check them to be sure)

I think some of the include files are not fetch from the 10.7 SDK but instead from the actual system, and we get either compilation errors or unusable build.

Does this crash occur with the official thirdparty packages or ones you built yourself?

I wonder if it might be an stdlib issue again, especially because it appears to be going wrong in std::string. Perhaps when compiling Panda it chose one version of the standard library, but it linked against another. It does seem that it was correctly linked against the GNU libc, but… You could try setting export CXXFLAGS=-stdlib=libc++ and export LDFLAGS=-stdlib=libc++ and recompiling Panda cleanly, and see if that helps.

The crash appears with the official thirdparty packages.

It’s indeed probably a problem with the c++ library, though I checked and the panda libs are correctly build against libstdc++ and there are no reference to libc++, so the problem might be an incompatibility between different version of the libstdc++ (or it might include the wrong headers at some point).

I will try and recompile the thirdparty packages with SDK 10.7 and see if I have the same problem

I took a different approach and instead tried first to downgrade Xcode to 3.2.6, and the compilation and execution are successful ! During the compilation I don’t have those strange warnings anymore :

ld: warning: building for macOS, but linking in object file (thirdparty/darwin-libs-a/ode/lib/libode.a(box.cpp.o)) built for

So clearly there was a compatibility problem between the thirdparty package and the toolchain used. Also, I’m able to build with bullet enabled (though I haven’t tested if it works or not and there are lots of warnings during linking)

So something in Mojave (or latest Xcode) has changed that makes Xcode 4.x unusable or at least incompatible with code built with Xcode 3.x

But, installing Xcode 3.2.6 is not trivial, you have to modify one of the .dist file in the installer to allow installation on a newer mac, disable installation of some of the tools, … (see http://catacombae.blogspot.com/2011/07/installing-xcode-326-in-mac-os-x-lion.html) Also, part of it is still 32bits so it won’t work anymore on Catalina.

Isn’t it time to switch the oldest MacOS version supported from 10.6 to 10.7 ? (or even 10.9 and switch to libc++ to avoid those C++ libraries problem ?)

Past time, which is why we have indeed done so at the beginning of this year on the master branch:

I missed that one :slight_smile:

The sooner we can get rid of 10.6 the better ! (Maybe one should add in the Readme or manual that in order to use the compiled thirdparty package, one has to use Xcode 2.3.6 or rebuild them).

And raising the bar to 10.9 would be better as we would have a more mature libc++.

I intend to post an updated set of thirdparty packages compiled against libc++ at some point, but haven’t gotten around to it yet.

I agree that 10.9 is a good minimum to set and that aiming for 10.7 might be a waste of time, especially given that the 10.7 libc++ seems to be somewhat unstable. It looks like Python 3.8 is only available for 10.9 and above anyway.