[vtk-developers] Python 3.7 behavior change, GIL, and threading in C/C++ with Python

[Marcus D. Hanwell]

On Mon, Nov 26, 2018 at 11:59 AM Marcus D. Hanwell <
marcus.hanwell at kitware.com> wrote:

> Hi,
>
> I have seen several fixes for Python 3.7, but have been tracking down why
> Tomviz deadlocks on when using vtkPythonScopeGilEnsurer, and it appears to
> be related to the Python specific flags we are using and a behavior change
> in Python 3.7. This may mainly be a question for David Gobbi and/or
> Utkarsh. but I send it to the list so that others have a chance to see it,
> and it gets indexed should others hit this issue.
>
> We use VTK_PYTHON_FULL_THREADSAFE set to ON, and VTK_NO_PYTHON_THREADS set
> to OFF. This means that when vtkPythonInterpreter::Initialize is called
> that the following code will be called:
>
> #ifdef VTK_PYTHON_FULL_THREADSAFE
>     int threadInit = PyEval_ThreadsInitialized();
>     PyEval_InitThreads(); // safe to call this multiple time
>     if (!threadInit)
>     {
>       PyEval_SaveThread(); // release GIL
>     }
> #endif
>
> This was effectively always calling PyEval_SaveThread() before Python 3.7.
> but this is now never being called as starting with Python 3.7
> Py_Initialize will initialize the GIL, see
> https://docs.python.org/3/c-api/init.html#c.PyEval_InitThreads for
> details.
>
> It looks like this check was done to ensure that PyEval_SaveThread would
> only be called once, now that PyEval_InitThreads doesn't need to be called,
> moving PyEval_SaveThread outside of the conditional gets rid of the
> deadlock for me. It looks like something needs to be changed for 3.7+, but
> I am not clear on how necessary this logic is.
>

I submitted https://gitlab.kitware.com/vtk/vtk/merge_requests/4921 for
review, it seems to check out in VTK, and a related ParaView branch moving
VTK forward.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://public.kitware.com/pipermail/vtk-developers/attachments/20181128/a2850ff3/attachment.html>