NX will not let you do that. The designer of NX consider dialog size is in their own ball park. Currently, NX enforces thin and long dialogs, with only items above having an impact on items below them. This is a design rule and explains why dialog size cannot be customized. This way, all NX dialogs have the same look and feel, no matter if they are "native" or customized.
It depends on which UI Styler you are asking about. In the older UI Styler, which is now considered a legcy product, but can still be used, you can control the width of the dialog with the UF_STYLER_DIALOG_WIDTH attribute. It can be used with UF_STYLER_set_value() to control the width of a UIStyler dialog.
The code might look something like this, in a UIStyler callback:
set_value.item_attr = UF_STYLER_DIALOG_WIDTH;
set_value.item_id = UF_STYLER_DIALOG_INDEX;
set_value.value.integer = 400; /** number of pixels for dialog width **/
However if you are using the newer product, (Block UI Styler), you might want to contact GTAC and ask them to add you to the Enhancement Request that is on file asking for a way to control the size of a Block Styler dialog from within a program.