I came up with the idea of using an additional flag for mouse button handlers. You can do the same for the keyboard