did you set the UGII_JAVA_HOME environment variable?
From NX8.5 java path location isn't default set in NX.
try to set the UGII_JAVA_HOME in the system environment settings (point it to the location of the JAVA install)
after that try to start PTS again.
Also, make sure you're pointing UGII_JAVA_HOME to a 64-bit version of Java.
Another trick is to open the NX Command Prompt and enter "pts". You can manually define the path to UGII_JAVA_HOME and you'll get more system feedback in the command window than you would if you attempt to launch PTS from the Windows Start menu.
Even better - upgrade to NX 11 (or newer). The PTS Author application is embedded in NX now.