[Paraview-developers] Scipy with ParaView is broken (and Py_NewInterpreter discussion)

[Utkarsh Ayachit]

Pat,

I am not surprised. I remember running into this ages ago. If I
remember correctly, here are the two use-cases for Python in ParaView:

* Qt Python testing uses the python interpreter from a separate thread
* Python shell, Programmable Filter use Python interpreter from master
thread but they need separate scopes.

So long as these use-cases can be satisfied, it would make sense to
not rely on Py_NewInter... if possible.

Utkarsh

p.s. flying today, so we can talk more on Monday, if you'd like.

On Fri, Aug 3, 2012 at 2:16 PM, Pat Marion <pat.marion at kitware.com> wrote:
> I am using numpy 1.6.1 and scipy 0.9.0 via system packages python-numpy and
> python-scipy on Ubuntu 12.04.  In both the paraview python console and
> programmable filter, importing certain scipy modules, for example,
> scipy.stats, causes paraview to hang.  This issue is related to the python
> global interpreter lock (GIL), the program hangs on PyGILState_Ensure().
> Here is the backtrace:
>
>
> #0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
> #1  0x00007ffff1e90c28 in PyThread_acquire_lock () from
> /usr/lib/libpython2.7.so.1.0
> #2  0x00007ffff1e91b34 in PyEval_RestoreThread () from
> /usr/lib/libpython2.7.so.1.0
> #3  0x00007ffff1e58d08 in PyGILState_Ensure () from
> /usr/lib/libpython2.7.so.1.0
> #4  0x00007fffca31ffc6 in ?? () from
> /usr/lib/python2.7/dist-packages/numpy/core/umath.so
> #5  0x00007fffca320357 in ?? () from
> /usr/lib/python2.7/dist-packages/numpy/core/umath.so
> #6  0x00007fffca337633 in ?? () from
> /usr/lib/python2.7/dist-packages/numpy/core/umath.so
> #7  0x00007fffca337ac6 in ?? () from
> /usr/lib/python2.7/dist-packages/numpy/core/umath.so
> #8  0x00007ffff1e47e83 in PyObject_Call () from /usr/lib/libpython2.7.so.1.0
> ...
>
>
> This could be a bug in scipy/numpy.  The issue does not appear for older
> versions of numpy/scipy.  But, it is likely caused by the interaction with
> ParaView's usage of Py_NewInterpreter.  The paraview class
> vtkPVPythonInterpretor uses the Python C-API function Py_NewInterpreter in
> order to create insulated interpreters.  The documentation for
> Py_NewInterpreter warns users to be very careful mixing it with the GIL.
> For example, it says:
>
> "Note that the PyGILState_*() functions assume there is only one global
> interpreter (created automatically by Py_Initialize()). Python supports the
> creation of additional interpreters (using Py_NewInterpreter()), but mixing
> multiple interpreters and the PyGILState_*() API is unsupported."
>
> and
>
> "Also note that combining this functionality with PyGILState_*() APIs is
> delicate, because these APIs assume a bijection between Python thread states
> and OS-level threads, an assumption broken by the presence of
> sub-interpreters."
>
> (  docs.python.org/c-api/init.html  )
>
>
> Perhaps it is possible to work around this particular hangup by compiling
> numpy/scipy with threading disabled or some other flag to disable their
> usage of the GIL.  But, ParaView's usage of Py_NewInterpreter also makes it
> difficult to use other libraries correctly, like PyQt4 or PySide for
> example.  I that at some in the future we might want to consider removal of
> Py_NewInterpreter from paraview.
>
>
> Pat
>
> _______________________________________________
> Paraview-developers mailing list
> Paraview-developers at paraview.org
> http://public.kitware.com/mailman/listinfo/paraview-developers
>