Any (not so obvious) downsides to using a terrain mesh?

I was aware of the LOD bit.

However. Can’t this be slightly alleviated by reorganizing the mesh back into an octree, using EggOctree?

My main beef with simple, height map based terrains is that one can’t possibly hope to design all of the terrain in it. In our engine, I can first create a terrain from a height map (or any random noise) and then, if I find a spot I’d like to have a cave at, "punch in a few holes.