[vtk-developers] Patch to vtkWrapPython.c to make wrapped Python code GIL-friendly

[Prabhu Ramachandran]

>>>>> "Charl" == Charl P Botha <cpbotha at gmail.com> writes:

    >> I'm going to have to think about this more; it's almost
    >> beginning to look like there's no straight-forward and
    >> fail-safe way to do this, but that it all has to be terribly
    >> complex by definition. :) I'd appreciate any more insights that
    >> you might have.

    Charl> Wait, maybe you're not right on this point after all. :)
    Charl> Remember with my patch, ALL VTK calls release the GIL.  So,
    Charl> if observer 'fa' (which has the lock when it's executing)
    Charl> modifies any other object 'b', the lock is released for
    Charl> EVERY VTK method that's called.  My reasoning is based on
    Charl> the assumption that vtkPythonCommand::Execute() can ONLY be
    Charl> called from VTK C++ code, and that all VTK C++ code is
    Charl> running WITHOUT GIL due to my patch.  This means that
    Charl> ::Execute() HAS to acquire the lock.

    Charl> What do you think?

I see what you are saying but still am not sure it will work always.
Py_BEGIN_ALLOW_THREADS and its counterpart use PyEval_SaveThread and
PyEval_RestoreThread.  These are documented as follows:

PyThreadState* PyEval_SaveThread()

    Release the interpreter lock (if it has been created and thread
    support is enabled) and reset the thread state to NULL, returning
    the previous thread state (which is not NULL). If the lock has
    been created, the current thread must have acquired it. (This
    function is available even when thread support is disabled at
    compile time.)

void PyEval_RestoreThread(PyThreadState *tstate)

    Acquire the interpreter lock (if it has been created and thread
    support is enabled) and set the thread state to tstate, which must
    not be NULL. If the lock has been created, the current thread must
    not have acquired it, otherwise deadlock ensues. (This function is
    available even when thread support is disabled at compile time.)


The trouble being that you can only save the state if the current
thread is the one that acquired the lock.  This may or may not be
true.  So if all VTK objects are created in one thread, I think we are
OK, but if that is not the case, I don't think this will work.  So if
`b` lives on a different thread, I think this won't work.  I think a
simple test case could be engineered to check this out.  I could be
wrong though because I haven't thought through it enough.

I'm thinking aloud here, but would this work? Create a
vtkThreadManager that manages acquiring and releasing the GIL.  When a
VTK method is invoked and the GIL needs to be released, the thread
state is saved to a stack and the GIL released.  Now when the callback
is called, it will acquire the GIL and insert its thread state into
the stack.  This callback now modifies another VTK object `b` in
another thread.  When b's method is called it asks the manager to
release the GIL and then when its done, re-aquires it.  The manager
simply pops the stack and sets that thread state and acquires the GIL.
Maybe this is crazy.  What do you think?

cheers,
prabhu