Adding and Removing colliders

When using the built-in collision system, is it at all expensive to add colliders to or remove them from a collision traverser and a CollisionHandlerPusher?

I have in mind a bit of optimisation that involves objects only being considered for collision if they’re not “on the floor” (my floors being handled by separate code), simply removing them from the traverser and handler when they change to being “on the floor” and adding them back when they leave the floor.

Does this seem like a good idea?