Isn't that how CAD works already? you select something either by mouse clicke or window it will be highlighted hold shift to deselect it the same way.
I know if you have to many things selected they are no longer highlighted.
I think you would need to test for 'shift' being pressed and the test for a left mouse entity selection.
Do a 'princ' on your grread loop to display what you are doing, shift and select something - which should give you what you want to test for.