You can use the assembly load options (see the "reference sets" group) to specify which reference set to use for the components when an assembly is loaded.
"Is there a way to make it so that all children of a given component are at the same referene set as they if they are at the assembly or Part level."
I don't quite understand the question above. Reference sets are defined at the part level, but are only used when the part is added to an assembly as a component. In other words, there is no reference set in use at the part level. For the current display part, you are essentially dealing with the "entire part" reference set, with no option to change it. You cannot use reference sets at the part level to filter what is shown in the part.