Could you construct your GUI as four frames, encircling your “cutout”? If the textures are tiled carefully you may be able to do so without seams.
Otherwise, you could perhaps model your frame–complete with central hole–in a 3D modelling program and then provide that to your GUI via the “geom” keyword-parameter.
And finally, I think that a ScissorEffect may allow you to mask out the relevant portion of your GUI–see this thread for more.