I seem to think that I’ve seen drwr or some other such dev comment to that effect.
And indeed, looking around, I found this post:
Note that drwr there indicates that visible geometry is less efficient for collision than even collision geometry of the exact same shape. And further that collision geometry is optimised for the purpose.
I’d guess that Panda has a different internal structure for collision geometry than for visible geometry, allowing faster traversals.
[edit]
Here’s an even clearer post:
In particular, note this line:
Although it is true that simpler geometry is faster still, collision geometry would be about 100 times faster even if you exactly duplicated your visible geometry 1:1 in the scene graph.