Since I work with NX (about 10 years) I have had this problem. Why does the selection tool not work like it should work?
When I want to select components which are inside the selection box it also selects components which are partly inside the box. And when I use crossing it does not select all the components that are crossing the box.
I also use the setting "no selection filter" and "entire assembly".
I find it always a lot of work to make good selections although it should be able to do it quickly if this function works properly. Really anoying.
Have you looked at the options in
Preferences -> Selection -> Selection rule...
Production: NX10.0.3.5 MP16/TC11.2
I'd rather be e-steamed than e-diseaseled
Are your assembly constraints displayed in the graphics window? I find that if a constraint falls within the rectangle selection, the component will be selected as well, however, if the constraint is hidden, then the same selection gesture will not select the unwanted component. See example below: