[vtk-developers] Py_NewInterpreter and vtkPythonCallback issue

pat marion pat.marion at kitware.com
Thu Jun 18 10:55:36 EDT 2009


Hi vtk-devs,

Sorry for the long email...

I hit a bug in vtkPythonCallback when used in combination with
Py_NewInterpreter.  The Py_NewInterpreter api is used in paraview by
vtkPVPythonInterpretor, but the bug could appear in any project using
Py_NewInterpreter.  Here is some documentation on this method and
related data structures...

--------------------------
(documentation snippets from http://docs.python.org/c-api/init.html)

PyThreadState* Py_NewInterpreter():
Create a new sub-interpreter. This is an (almost) totally separate
environment for the execution of Python code. In particular, the new
interpreter has separate, independent versions of all imported
modules, including the fundamental modules __builtin__, __main__ and
sys. [...] The return value points to the first thread state created
in the new sub-interpreter. [...] Note that no actual thread is
created; see the discussion of thread states below.

PyThreadState:
    This data structure represents the state of a single thread. The
only public data member is PyInterpreterState *interp, which points to
this thread’s interpreter state.

PyInterpreterState:
    This data structure represents the state shared by a number of
cooperating threads. Threads belonging to the same interpreter share
their module administration and a few other internal items. There are
no public members in this structure.  Threads belonging to different
interpreters initially share nothing, except process state like
available memory, open file descriptors and such. The global
interpreter lock is also shared by all threads, regardless of to which
interpreter they belong.
--------------------------

In paraview the python shell, python programmable filter, and python
calculator each have a vtkPVPythonInterpretor instance.  When
vtkPVPythonInterpretor executes a string of python code it first makes
its interpreter the active interpreter by calling its member function
MakeCurrent(), and later calls ReleaseControl().  These methods call
PyThreadState_Swap(PyThreadState *tstate).  MakeCurrent swaps in the
PyThreadState representing its interpreter, and ReleaseControl swaps
back the previous PyThreadState.  In this context I believe the calls
have nothing to do with threads, they just ensure that the correct
PyInterpreterState is active.

Now say I give the interpreter this code:

    def myCallback(obj, eventId): print "callback"

    c = vtkCamera()
    c.AddObserver("ModifiedEvent", myCallback)
    c.Modified()

It's going to create a vtkPythonCommand and when c.Modified() executes
we will jump into vtkPythonCommand::Execute(), get the PyObject for
myCallback, and call it with PyEval_CallObject.


Now here is the bug:  let's say I don't evaluate c.Modified() through
the interpreter, instead Modified is called from the c++ world, maybe
because the camera was zoomed in a render window.  In this case, the
PyInterpreterState that was active when myCallback was added might not
be active when myCallback is called.  This is because
vtkPVPythonInterpretor called ReleaseControl() which swapped in a
different interpreter.  The result is that python executes the
callback in restricted mode.  This can lead to runtime errors such as
"RuntimeError: classes are read-only in restricted mode".  Python
falls into restricted mode because the __builtins__ module for the
current interpreter is not the same __builtins__ module used by
myCallback.  The check for restricted mode (in python 2.6.2) is found
in Include/frameobject.h:

    #define PyFrame_IsRestricted(f) \
      ((f)->f_builtins != (f)->f_tstate->interp->builtins)

Attached is a patch that can fix the problem by swapping threadstates
in vtkPythonCommand::Execute().  (tested on linux with python 2.6.2
and vtk cvs)

vtkPythonCommand::Execute also calls PyGILState_Ensure().  This is
usually blocked by ifdefs when vtk is compiled from paraview, and I'm
not sure how it might work in combination with PyThreadState_Swap, but
the python documentation suggests it is safe.

Pat
-------------- next part --------------
A non-text attachment was scrubbed...
Name: python-restricted-mode-fix.diff
Type: text/x-patch
Size: 3113 bytes
Desc: not available
URL: <http://public.kitware.com/pipermail/vtk-developers/attachments/20090618/0e0ebcb7/attachment-0001.bin>


More information about the vtk-developers mailing list