Showing results for 
Search instead for 
Do you mean 
Reply
Solved! Go to solution

multithreaded Python extension modules

[ Edited ]

I am trying to import sklearn.cluster (scikit-learn) and scipy.spatial into the NX Open API and I was hoping the community could provide some thoughts. I have heard of Python's GIL problem, which said that there can only be one Python thread executing at one time in a multicore machine but it is my understanding that Python 3.2+ has a new GIL implementation. 

 http://jeffknupp.com/blog/2013/06/30/pythons-hardest-problem-revisited/

 

I successfully created a virtual environment with Anaconda for Python 3.3.2 (conda create -n py33) and I installed scikit-learn via conda. I am using Windows 7 on an Intel 64 bit machine.

 

For the most part, I have been able to use numpy methods and attributes successfully although some (numpy.array_equiv) will lock up NX.

 

When I run a python file with `import sklearn.cluster` or "from sklearn.cluster import Kmeans", it will crash NX. I have not used any sklearn classes or methods yet. The import line alone will crash NX. I also face a similar issue with "import scipy.spatial" and "from scipy.spatial import ConvexHull". I did not use any scipy.spatial methods or classes.

 

According to the documentation, putting a comment "#nx:threaded "at the very top of the python file should resolve the issue but unfortunately, it did not. My syslog showed that the sklearn and scipy libraries were never loaded.

 

https://docs.plm.automation.siemens.com/tdoc/nx/10.0.3/release_notes/#uid:xid385122

 

I also went into the ugii_env_ug.dat file and tried different values for UGII_SMP_ENABLE = 0/1

 

It is my understanding that  Python 3.4 supports an Intel library called MKL that might be able to resolve this but NX only supports Python 3.3

 

I also looked into Iron Python which has no Global Interpreter Lock but it only has a version for Python 2.7

 

Running threaded extension modules with Python The embedded Python interpreter in NX runs Python scripts using subinterpreter threads to isolate the execution environments of different scripts that are running simultaneously. For example, you can use the startup script, utilize user exits, and explicitly run a journal in a session. Running each of these scripts in a separate subinterpreter keeps each of these environments separate from each other to avoid possible illegal access and collisions. However, this approach has some drawbacks. There are a few third-party extension modules (non-NXOpen extension modules, such as matplotlib) that use C threads to perform operations. These extension modules could be imported safely, but when a function is called that starts a C thread, the subinterpreter hangs or crashes. These extensions run safely only in the main interpreter thread. Unfortunately, Python does not provide a safe way to run such extension modules in subinterpreter threads and does not provide any way to switch from a subinterpreter thread to the main interpreter thread when a script is already executing.

To support such threaded extension modules, NX must know if a Python script is using any of these modules before preparing its interpreter. So if the script is using these kinds of threaded extension modules or importing a module that is using threaded extension modules directly or indirectly, you should add a comment with the text nx:threaded anywhere in the first three lines. For example:

# nx:threaded

# some comments nx:threaded some comments

# some comments
# nx:threaded
# some comments
This instructs NX to prepare its embedded Python interpreter to run the script in the main thread instead of the subinterpreter thread to avoid a possible problem. Pure Python threads do not have those kinds of problems with subinterpreters and should be used without this extra comment. This comment could be added to any Python script whether it is a startup script, a user exit script, or a normal journal. Do not use this comment unnecessarily. It runs all the scripts in the main interpreter thread and may exhibit some unusual behavior, such as illegal data access and object deallocation. Use this comment only when threaded extension modules are imported and used.

 

2 REPLIES

Re: multithreaded Python extension modules

MKL library might be a solution but I cannot find one that supports Python 3.3.2

http://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy

Solution
Solution
Accepted by topic author AlexLee
‎05-26-2016 03:23 PM

Re: multithreaded Python extension modules

[ Edited ]

it's such a simple solution. There needs to be a space between "#nx: " and "threaded"

 

The online documentation does not mention this space

https://docs.plm.automation.siemens.com/tdoc/nx/10.0.3/release_notes/#uid:xid385122

 

 

#nx: threaded

#make sure there is a space between nx and threaded import NXOpen import NXOpen.BlockStyler import NXOpen.Features import NXOpen.UF import NXOpen.GeometricAnalysis import NXOpen.Facet import math from random import random global big_list big_list = [[random() for i in range(10)] for j in range(10000)] # compute eight representative points from sklearn.cluster import KMeans class generic_class: # Constructor for NX Styler class # ------------------------------------------------------------------------------ def __init__(self): # class members self.theSession = None self.theUI = None self.theLw= None self.theDialogName = "" self.theDialog = None #Get the UF session self.theUfSession = None try: self.theSession = NXOpen.Session.GetSession() self.theUI = NXOpen.UI.GetUI() self.theLw = self.theSession.ListingWindow #create print window self.theUfSession = NXOpen.UF.UFSession.GetUFSession() except Exception as ex: # ---- Enter your exception handling code here ----- raise ex def hello_world(self): self.theLw.Open() self.theLw.WriteLine("Hello NX World!") self.theLw.WriteLine(str(big_list)) model = KMeans(n_clusters=8) model.fit(big_list) centers = model.cluster_centers_ self.theLw.WriteLine(str(centers.shape)) # should give (8, 10), (number of clusters, length of list) def main(): try: #create instance of class myclass = generic_class() myclass.hello_world() except Exception as ex: # ---- Enter your exception handling code here ----- NXOpen.UI.GetUI().NXMessageBox.Show("Block Styler", NXOpen.NXMessageBox.DialogType.Error, str(ex)) if __name__ == "__main__": main()