[vtk-developers] Python 3.7 behavior change, GIL, and threading in C/C++ with Python

[Marcus D. Hanwell]

Hi,

I have seen several fixes for Python 3.7, but have been tracking down why
Tomviz deadlocks on when using vtkPythonScopeGilEnsurer, and it appears to
be related to the Python specific flags we are using and a behavior change
in Python 3.7. This may mainly be a question for David Gobbi and/or
Utkarsh. but I send it to the list so that others have a chance to see it,
and it gets indexed should others hit this issue.

We use VTK_PYTHON_FULL_THREADSAFE set to ON, and VTK_NO_PYTHON_THREADS set
to OFF. This means that when vtkPythonInterpreter::Initialize is called
that the following code will be called:

#ifdef VTK_PYTHON_FULL_THREADSAFE
    int threadInit = PyEval_ThreadsInitialized();
    PyEval_InitThreads(); // safe to call this multiple time
    if (!threadInit)
    {
      PyEval_SaveThread(); // release GIL
    }
#endif

This was effectively always calling PyEval_SaveThread() before Python 3.7.
but this is now never being called as starting with Python 3.7
Py_Initialize will initialize the GIL, see
https://docs.python.org/3/c-api/init.html#c.PyEval_InitThreads for details.

It looks like this check was done to ensure that PyEval_SaveThread would
only be called once, now that PyEval_InitThreads doesn't need to be called,
moving PyEval_SaveThread outside of the conditional gets rid of the
deadlock for me. It looks like something needs to be changed for 3.7+, but
I am not clear on how necessary this logic is.

This reminds me that VTK_PYTHON_FULL_THREADSAFE is used in VTK, but the
CMake option is in ParaView only as far as I can see. Should that be moved
down into VTK, it seems odd.

Thanks,

Marcus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://public.kitware.com/pipermail/vtk-developers/attachments/20181126/83bdc039/attachment-0001.html>