[vtkusers] bug in vtkPythonUnmangePointer

[David Gobbi]

Hi Scott,

This is a good place for reporting problems like this.  And if you
ever report a bug on the list and don't get a response, you can go one
step further by adding it to the bugtracker.

Your patch is straightforward and fixes something that's sure to
impact my code, too, so I'll get it pushed into VTK before 5.6 is
branched.  Just a note, though, it's good to follow
http://www.vtk.org/Wiki/VTK_Coding_Standards when modifying or writing
VTK classes.  That helps to accelerate the adoption of patches and new
classes into VTK.

   David


On Fri, Feb 5, 2010 at 8:55 AM, Scott Swarts <swarts at enthought.com> wrote:
>
> We've run into a crash in vtkPythonUnmanglePointer. From python, we are
> calling SetVoidArray on a vtk array in order to share memory with a numpy
> array. The problem seems to be that SetVoidArray calls
> vtkPythonUnmanglePointer in case it is a SWIG pointer being passed in.
> vtkPythonUnmanglePointer treats the memory as a string, but there is no
> terminating null. This leads to a buffer overflow and crash. I've attached a
> script that, at least on my windows machine, causes the crash. I've also
> attached a patch to VTK5.4.2 that I'm using to work around the problem.
> Can we get a fix put into the next release?  Is there someplace else I
> should be reporting this problem?
>
> Thanks,
>
> Scott
>
>
> import sys
> import numpy as np
> import vtk
>
> # Make a memory mapped array so we will segfault if we overflow it
> mapped_array = np.memmap('foo.np', np.int8, 'w+',  shape=(4096,))
> # Fill the array it has no '\0's
> mapped_array[:] = ord('0')
> if sys.platform != 'win32':
>    # Make it look like a swig pointer. NOTE: on Windows this is not
> necessary
>    # since sscanf(s, ...) turns into _snscanf(s, strlen(s), ...) and the
>    # strlen will overflow the buffer anyway.
>    mapped_array[0] = ord('_')
>    mapped_array[8] = ord('_')
>
> # Make a small array that shared memory with the mapped array
> small_array = mapped_array[:24]
>
> # Make a vtk array
> vtk_array = vtk.vtkCharArray()
> vtk_array.SetNumberOfComponents(1)
> vtk_array.SetNumberOfTuples(24)
>
> # Attempt to use the buffer protocol to share memory with the small array
> vtk_array.SetVoidArray(np.getbuffer(small_array), 24, 1)
>
> print "Done"
>
> diff -r cbc1b2d8ea86 Common/vtkPythonUtil.cxx
> --- a/Common/vtkPythonUtil.cxx  Sat Jan 16 13:22:39 2010 -0600
> +++ b/Common/vtkPythonUtil.cxx  Sat Jan 16 13:34:41 2010 -0600
> @@ -1648,18 +1648,27 @@
>   int i;
>   void *ptr;
>   char typeCheck[128];
> -  if (*len < 128)
> +  // Do some minimal checks that it might be a swig pointer.
> +  if (*len < 128 && *len > 4 && ptrText[0] == '_')
>     {
> -    i = sscanf(ptrText,"_%lx_%s",(long *)&ptr,typeCheck);
> -    if (strcmp(type,typeCheck) == 0)
> -      { // sucessfully unmangle
> -      *len = 0;
> -      return ptr;
> -      }
> -    else if (i == 2)
> -      { // mangled pointer of wrong type
> -      *len = -1;
> -      return NULL;
> +      // We need to avoid a buffer overflow, but snscanf is not always
> available
> +      // so we look for a terminating null starting at the end.
> +      for (i=*len-1; i>=0; --i)
> +        if (ptrText[i] == '\0')
> +          break;
> +      if (i >= 0) {
> +        // We have a terminating null, so the sscanf
> +        i = sscanf(ptrText,"_%lx_%s",(long *)&ptr,typeCheck);
> +        if (strcmp(type,typeCheck) == 0)
> +          { // sucessfully unmangle
> +            *len = 0;
> +            return ptr;
> +          }
> +        else if (i == 2)
> +          { // mangled pointer of wrong type
> +            *len = -1;
> +            return NULL;
> +          }
>       }
>     }
>   // couldn't unmangle: return string as void pointer if it didn't look
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the VTK FAQ at:
> http://www.vtk.org/Wiki/VTK_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.vtk.org/mailman/listinfo/vtkusers
>
>