Collision geometry is far, far cheaper than visible geometry. Although it is true that simpler geometry is faster still, collision geometry would be about 100 times faster even if you exactly duplicated your visible geometry 1:1 in the scene graph.
This is because the code path that supports visible geometry has to do it by creating a CollisionPolygon on the fly–each frame!–for each visible triangle in your mesh.
This is necessary because the visible triangles don’t store all of the data that is needed for the collision system to work, like the plane equation of the polygon.
Visible geometry as collision data was always intended as a quick hack to get something to work when you just need something now. It should not be considered a viable long-term solution.
David