You can't change the display in the dialogs as far as I know.
The system will always consider double precision.
The post-processor will output to what ever precision you tell him to.
Production: NX10.0.3, VERICUT 8.0, FBM, MRL 3.1.4 | TcUA 10.1 MP7 Patch 0 (10.1.7.0) | TcVis 10.1
Development: VB.NET, Tcl/Tk Testing: NX11.0 EAP, NX12.0 EAP
How to Get the Most from Your Signature in the Community
The system stores double precision - you cannot change that. Like @Stefan_Pendl said, it is the job of the post to round off to the correct precision for the control.
The display in the dialog is an NX-wide setting:
Menu > Preferences > User Interface > Options > Dialog Boxes > Decimal Places.
Also found in Customer Defaults > Gateway > ...