[Paraview-developers] Scipy with ParaView is broken (and Py_NewInterpreter discussion)

[Pat Marion]

I am using numpy 1.6.1 and scipy 0.9.0 via system packages python-numpy and
python-scipy on Ubuntu 12.04.  In both the paraview python console and
programmable filter, importing certain scipy modules, for example,
scipy.stats, causes paraview to hang.  This issue is related to the python
global interpreter lock (GIL), the program hangs on PyGILState_Ensure().
Here is the backtrace:


#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
#1  0x00007ffff1e90c28 in PyThread_acquire_lock () from
/usr/lib/libpython2.7.so.1.0
#2  0x00007ffff1e91b34 in PyEval_RestoreThread () from
/usr/lib/libpython2.7.so.1.0
#3  0x00007ffff1e58d08 in PyGILState_Ensure () from
/usr/lib/libpython2.7.so.1.0
#4  0x00007fffca31ffc6 in ?? () from
/usr/lib/python2.7/dist-packages/numpy/core/umath.so
#5  0x00007fffca320357 in ?? () from
/usr/lib/python2.7/dist-packages/numpy/core/umath.so
#6  0x00007fffca337633 in ?? () from
/usr/lib/python2.7/dist-packages/numpy/core/umath.so
#7  0x00007fffca337ac6 in ?? () from
/usr/lib/python2.7/dist-packages/numpy/core/umath.so
#8  0x00007ffff1e47e83 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0
...


This could be a bug in scipy/numpy.  The issue does not appear for older
versions of numpy/scipy.  But, it is likely caused by the interaction with
ParaView's usage of Py_NewInterpreter.  The paraview class
vtkPVPythonInterpretor uses the Python C-API function Py_NewInterpreter in
order to create insulated interpreters.  The documentation for
Py_NewInterpreter warns users to be very careful mixing it with the GIL.
For example, it says:

"Note that the PyGILState_*() functions assume there is only one global
interpreter (created automatically by Py_Initialize()). Python supports the
creation of additional interpreters (using Py_NewInterpreter()), but mixing
multiple interpreters and the PyGILState_*() API is unsupported."

and

"Also note that combining this functionality with PyGILState_*() APIs is
delicate, because these APIs assume a bijection between Python thread
states and OS-level threads, an assumption broken by the presence of
sub-interpreters."

(  docs.python.org/c-api/init.html  )


Perhaps it is possible to work around this particular hangup by compiling
numpy/scipy with threading disabled or some other flag to disable their
usage of the GIL.  But, ParaView's usage of Py_NewInterpreter also makes it
difficult to use other libraries correctly, like PyQt4 or PySide for
example.  I that at some in the future we might want to consider removal of
Py_NewInterpreter from paraview.


Pat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/paraview-developers/attachments/20120803/2a92a45c/attachment.htm>