[vtk-developers] Memory-mapped files in VTK

[Burlen Loring]

Observing the stock delete event on a data array wouldn't prevent you 
from having the special memory management class. I'm sure you could 
accomplish your goals without modifying the vtkDataArray.

On 05/16/2014 11:08 AM, David Gobbi wrote:
> Hi Burlen,
>
> Thanks for the links.  I guess that I'm looking for deeper integration
> (though less general) than a callback.  The idea is to have something
> like a vtkMemoryMap class to take care of all the platform-specific
> details of creating/deleting memory maps, and to have vtkAbstractArray
> be aware of this class and use it to delete mmaps when appropriate.
>
>    David
>
> On Fri, May 16, 2014 at 11:56 AM, Burlen Loring <burlen.loring at gmail.com> wrote:
>> meant to say "no special memory management issues"
>> and (in case you actually would look at the code) mmap use is here:
>> https://github.com/burlen/SC11-100GB-Demo/blob/master/SC11DemoReader.h
>>
>> On 05/16/2014 10:50 AM, Burlen Loring wrote:
>>
>> Hi David,
>>
>> I've used mmap with VTK in C++ for the SC11 100GB network demo.
>>
>> http://dl.acm.org/citation.cfm?id=2287004
>> https://github.com/burlen/SC11-100GB-Demo
>>
>> This use was very ad-hoc and we wanted the data mapped for the life of the
>> app so there were special memory management issues.
>>
>> patch(http://review.source.kitware.com/#/c/14072/) would have handled
>> situations such as this through a callback invoked when the data array is
>> finished with the pointer. However, that patch was rejected for being
>> overkill. Instead Berk suggested that I simply use the delete event
>> observer, which will always result in data being released but perhaps not as
>> quickly as with the callback or the method that you proposed(eg nothing
>> happens if the array is resized). This approach has worked well for me. I
>> wonder if using the delete event observer would work well in your case?
>>
>> Burlen
>>
>>
>> On 05/16/2014 10:21 AM, David Gobbi wrote:
>>
>> Hi All,
>>
>> I've used memory mapped files with VTK in the past, using the python
>> "mmap" object to create the map and then passing the resulting pointer
>> to VTK via the SetVoidArray() method.  For large files (several GB) I've
>> found this to be many times faster than e.g. vtkImageReader2.
>>
>> My question: has anyone used mmap with VTK in C++?  Specifically,
>> what I'm interested in doing is adding a new "delete method" to the
>> SetVoidPointer() method in vtkDataArrayTemplate.h: it already has
>> "delete methods" for "malloc" and "new", so the idea is to add another
>> delete method for "mmap" that will call munmap on the pointer.
>>
>>     David