Bullet gets very slow in meters

I am running physics for meshes. When using millimeter as the metrics, my frame rate reaches 60fps. But when I change them to meters, the number lowers down to 1fps or less. I am wondering if I did a wrong option.

Figures below:
Meter, slow:

Millimeter, fast:

I wonder if it has something to do with this:

Thanks for the article. It very clearly explained the problem.
The prebuilt on Ubuntu had the collision error. Mac is not tested yet.
I will see if toggling off -ffast-math helps.