It seems that NX 8.5 doesn't use the Anti Aliasing functions that my Nvidia Quadro P4000 offers. See screenshots below. Even after going into the Nvidia control panel and doing an "override" on the 3D settings of NX (setting Antialiasing to max) it shows now improvement in line quality.
However, going into the NX settings (Preferences > Visualisation > Visual) and turning off "Line Antialiasing" as well as "Full Scene Antialiasing" results in an even uglier line quality.
How can I improve line quality further?
NX 18.104.22.168 MP11 Rev. A
GM TcE v22.214.171.124
GM GPDL v11-A.3.5.1
As @TimF mentioned, it's really hard to tell the issue from the small images, but it does seem that you have some "stair step" of the black edges in the last image?
For that, I would look at the Edge settings in the same Perference where you set Anti-Alias. Make sure it's set to "Smooth". On my laptop, there seems to be subtle change when I toggle this, but not a huge one (not like the "steps" seens when Anti-Alias is OFF in NX).
I have an NVIDIA Quadro M1000M, so different card, but in my settings, the Anti-Alias are mostly "Application-Controlled". Setting to "Override" seems like it would tell the card to do what it wants, instead of having NX tell the card what it needs?
The other thing that comes to mind (which may not matter) is that the P4000 seems to be pretty new while you're on an old NX version that does not even support Win 10, so maybe there's some conflicting new technology you are dealing with?