[vtk-developers] zero-copy mixed language support in vtkDataArray

[Burlen Loring]

Hi Guys,

I've refactored the patch as David suggested adding 
vtkCommand::PointerFreeEvent which is fired when the pointer passed to 
SetArray is no longer needed and the delete method is 
VTK_DATA_ARRAY_CALLBACK. Would you mind taking another look at it?
http://review.source.kitware.com/#/c/14072/5

Burlen

On 01/19/2014 03:43 PM, David Gobbi wrote:
> I meant vtkCommand, not vtkCallback.  Invoke a FreeEvent and catch it
> on the other side with a vtkCommand.
>
> On Sun, Jan 19, 2014 at 4:41 PM, David Gobbi <david.gobbi at gmail.com> wrote:
>> Hi Burlen,
>>
>> If that trick won't work, then maybe the easiest thing is to make the
>> vtkDataArray generate a specific event (e.g. create a new event called
>> a FreeEvent) that is called when the memory is freed.  That way you
>> are taking advantage of the existing vtkCallback support, which is
>> already wrapped by all of the wrapper languages.
>>
>>    David
>>
>> On Sun, Jan 19, 2014 at 4:22 PM, Burlen Loring <burlen.loring at gmail.com> wrote:
>>> thanks, that's a neat trick!
>>>
>>> it certainly keeps one safe : the numpy object is around while the array is.
>>> I don't think that this will do the right thing if the vtk array is resized,
>>> or if SetVoidPointer is called again. In both cases, the vtk data array is
>>> still alive and well, but the numpy array's ref count should be decremented.
>>>
>>> a python only solution wont work for me here, because I'm working in the
>>> C/C++ "glue layer" sitting in between the python app(s) (which aren't mine)
>>> and a legacy VTK based app. VTK objects aren't used in the app's python API
>>> at all,  only lists, tuples, and I'm adding support for numpy arrays. Using
>>> VTK objects in the API is not an option.
>>>
>>>
>>> On 1/19/2014 8:43 AM, David Gobbi wrote:
>>>> Hi Burlen,
>>>>
>>>> There might be a simpler way to get VTK to free the numpy array,
>>>> that won't require any changes to the VTK code at all: just add
>>>> the numpy array to the VTK array as an attribute.  Here is some
>>>> example code:
>>>>
>>>> == BEGIN EXAMPLE ==
>>>>
>>>> # An example of using numpy arrays in VTK
>>>> import vtk
>>>> import numpy
>>>> import weakref
>>>> import gc
>>>>
>>>> # Use a numpy array as a VTK array
>>>> z = numpy.arange(0,10,1,float)
>>>> a = vtk.vtkDoubleArray()
>>>> a.SetVoidArray(z,10,1)
>>>>
>>>> # Make the VTK array track the numpy array
>>>> a.array = z
>>>>
>>>> # Create weakrefs to check for deletion
>>>> rz = weakref.ref(z)
>>>> ra = weakref.ref(a)
>>>>
>>>> # Decref the numpy array, it won't be deleted yet
>>>> del z
>>>> gc.collect()
>>>> if rz() == None:
>>>>       print "numpy array is deleted (#1)"
>>>>
>>>> # Decref the VTK array, numpy array will be decref'd
>>>> del a
>>>> gc.collect()
>>>> if rz() == None:
>>>>       print "numpy array is deleted (#2)"
>>>>
>>>> == END EXAMPLE ==
>>>>
>>>> This works because the attributes of a VTK-Python object are kept
>>>> alive for as long as the corresponding VTK-C++ object exists.
>>>>
>>>>     David
>>>>
>>>> On Sun, Jan 19, 2014 at 12:09 AM, Burlen Loring <burlen.loring at gmail.com>
>>>> wrote:
>>>>> To illustrate the callback in action I wrote an example where data is
>>>>> transferred to VTK from numpy ndarrays. I hope this will help. README
>>>>> file
>>>>> explains the example. https://github.com/burlen/TestZeroCopyPython
>>>>>
>>>>>
>>>>>>> OK. as fas as VTK is concerned callbackData is intended to be a key
>>>>>>> for use by the callback, and VTK does nothing with it beyond passing
>>>>>>> it to the callback. I'll add a note about this to clarify.
>>>>>> Thanks. The problem with callback context ownership is that if it's a
>>>>>> one-time, the callback "owns" the context, but if it's a notification
>>>>>> kind of thing, the *notifier* owns the context (since the callback may
>>>>>> never be called) and needs an additional function to free the context.
>>>>>>
>>>>> I'm not following your complaint here. We're concerned about a pointer to
>>>>> some array that must be kept alive while VTK uses it but must be released
>>>>> when VTK is done. In this situation VTK "no longer needs the data" only
>>>>> ever
>>>>> once. and just as VTK always will call free when the pointer is passed in
>>>>> through the existing SetArray w/ VTK_DATA_ARRAY_FREE flag. VTK will
>>>>> always
>>>>> call the callback, either when the vtkDataArray is resized or when it is
>>>>> Delete'd. The callback and callback data are pointers, as far as VTK is
>>>>> concerned they're just values, and VTK need not worry about free'ing
>>>>> them.
>>>>> Note that this doesn't prevent one from passing an instance of an
>>>>> arbitrary
>>>>> class in callbackData. In that case the instance should be deleted in the
>>>>> callback.
>>>>>
>>>>>
>>>>>>> Works for me, although I'm not sure NULL callbackData is ever
>>>>>>> valid/usefull.
>>>>>> As an example, if you have a LocalFree or whatever on Windows, you just
>>>>>> need the array parameter, so NULL callbackData is needed.
>>>>> NULL callbackData would generally not work out. the callback needs
>>>>> non-NULL
>>>>> callbackData passed to it in order to identify the memory to free. In
>>>>> your
>>>>> example callbackData is the array pointer itself, not NULL. This is not a
>>>>> problem.
>>>>>
>>>>>>     Where array
>>>>>> and callbackData might be needed is something like where you need the
>>>>>> array, but callbackData is the arena to free it from. Currently, you
>>>>>> only get one or the other.
>>>>> actually I think currently you could get both without issue. don't forget
>>>>> callbackData is very flexible, it can be (point to) anything. In the case
>>>>> you're describing: write a class containing the arena and the array. pass
>>>>> a
>>>>> heap allocated instance of your class as callbackData when you call
>>>>> SetArray. When VTK calls the callback, in addition to releasing the array
>>>>> data, delete the instance to your class.
>>>>>
>>>>> Would you be happier with a polymorphic reference counted functor, rather
>>>>> than a callback? Initially I thought this would be overkill, but I'd be
>>>>> willing to go that way if people are against the callback idea.
>>>>>
>>>>> Burlen
>>>>>
>>>>>
>>>>> On 01/17/2014 09:19 PM, Ben Boeckel wrote:
>>>>>> On Fri, Jan 17, 2014 at 15:21:42 -0800, Burlen Loring wrote:
>>>>>>> Yes, my intention is that for now they'd be used from glue code
>>>>>>> rather than exposed directly. In the future the API could be exposed
>>>>>>> in the wrapped languages if there were a compelling use case for it.
>>>>>>> Until then it's not worth the effort since I doubt VTK wrapping codes
>>>>>>> would handle it correctly.
>>>>>> That's valid; just making sure :) .
>>>>>>
>>>>>>> OK. as fas as VTK is concerned callbackData is intended to be a key
>>>>>>> for use by the callback, and VTK does nothing with it beyond passing
>>>>>>> it to the callback. I'll add a note about this to clarify.
>>>>>> Thanks. The problem with callback context ownership is that if it's a
>>>>>> one-time, the callback "owns" the context, but if it's a notification
>>>>>> kind of thing, the *notifier* owns the context (since the callback may
>>>>>> never be called) and needs an additional function to free the context.
>>>>>>
>>>>>>> Works for me, although I'm not sure NULL callbackData is ever
>>>>>>> valid/usefull.
>>>>>> As an example, if you have a LocalFree or whatever on Windows, you just
>>>>>> need the array parameter, so NULL callbackData is needed. Where array
>>>>>> and callbackData might be needed is something like where you need the
>>>>>> array, but callbackData is the arena to free it from. Currently, you
>>>>>> only get one or the other.
>>>>>>
>>>>>> --Ben