You could use the same profile and use the combine tool instead. This would allow you to select the ace you want to keep. I wiil try to give you feedback!
It is NX 1847
So I describe my solution:
You don't need to make all of your tool with swept body. Make just half a turn in your helix to avoid self intersection. Then make a pattern feautre of swept with helix layout of the same pitch. 3 instances in 1 mm. That's the trick.
A solid body is not solid but a shell of surfaces.
To become a solid, all surfaces are trimmed together into a "watertight" shell. The faces of the solid shares the edges. If these surfaces are not completely closed, "watertight", it is a sheet body.
There is no material inside the volume, it is a shell where the faces have zero thickness.
All faces has a positive and a negative side, the face normal is ( shown) on the positive side.
The definition of "the volume" is that all faces of the volume have the positive side of the faces "out" from the volume. - the "negative side is the material side".
When You boolean one solid to/from another, the solid modeler will trim the faces of the two volumes together.
for this operation to be successful, the volumes must be clean enough or logic enough for the system to find the result that you , the user, expect.
You first example is not, for the solid modeler, logic enough. It cannot tell what volumes should be removed or left due to the helical sweeps self interference. You must create a more logical and "clean" sweep such that NX "understands" your intentions.