Welcome to the wonderful world of limited floating point precision 
This is sadly a common issue when you have objects far far away from the origin, you reach the limit of the single precision floating point variables : because you have huge values, the “lower” digits are simply truncated to fit into the mantissa (roughly speaking in single precision you can store about 6-7 digits in the mantissa). So, the actual position becomes jerky due to this truncation.
Two solutions:
Move the origin as close as possible to your object to make the position value more compatible limited precision. Usually this is done by putting the camera at the origin and move everything relative to it.
Rebuild Panda3D to use double precision by default, but you would still have precision issue when getting really far away, also this does not solve precision issue inside the GPU, mainly the depth buffer, where everything will still be 24/32 bits.
In my app Cosmonium I had to implement both solution (as well as some even more complicated) because of this.