[vtk-developers] Memory-mapped files in VTK

David Gobbi david.gobbi at gmail.com
Fri May 16 14:28:00 EDT 2014


Hi Berk,

I was passing the data through a convolution filter immediately after
mapping it, and what I was timing was how long it took for that
operation to complete, for mmap vs. vtkImageReader2.

However, the biggest advantage with mmap is when you are using
the same file over and over again in different session (especially e.g.
when you are debugging).  If you have a machine with loads of RAM,
the whole file will remain in memory from one session to the next, so
with mmap loading the file in subsequent sessions comes at zero cost.
With a more typical reader, the file may still be cached in memory,
but the reader is forced to copy the data from disk cache to a new array.

  David


On Fri, May 16, 2014 at 12:16 PM, Berk Geveci <berk.geveci at kitware.com> wrote:
> I am not objecting to the concept but I have a question. Is it really
> faster? Don't you have to take compute time into account if you have a
> memory mapped file? That's when disk activity will actually occur. I am
> wondering if this approach doesn't simply postpone the inevitable till
> later. I understand this should be fast if processing only touches part of
> the data but then it shouldn't be loaded in the first place.
>
> -berk
>
>
>
> On Fri, May 16, 2014 at 2:08 PM, David Gobbi <david.gobbi at gmail.com> wrote:
>>
>> Hi Burlen,
>>
>> Thanks for the links.  I guess that I'm looking for deeper integration
>> (though less general) than a callback.  The idea is to have something
>> like a vtkMemoryMap class to take care of all the platform-specific
>> details of creating/deleting memory maps, and to have vtkAbstractArray
>> be aware of this class and use it to delete mmaps when appropriate.
>>
>>   David
>>
>> On Fri, May 16, 2014 at 11:56 AM, Burlen Loring <burlen.loring at gmail.com>
>> wrote:
>> > meant to say "no special memory management issues"
>> > and (in case you actually would look at the code) mmap use is here:
>> > https://github.com/burlen/SC11-100GB-Demo/blob/master/SC11DemoReader.h
>> >
>> > On 05/16/2014 10:50 AM, Burlen Loring wrote:
>> >
>> > Hi David,
>> >
>> > I've used mmap with VTK in C++ for the SC11 100GB network demo.
>> >
>> > http://dl.acm.org/citation.cfm?id=2287004
>> > https://github.com/burlen/SC11-100GB-Demo
>> >
>> > This use was very ad-hoc and we wanted the data mapped for the life of
>> > the
>> > app so there were special memory management issues.
>> >
>> > patch(http://review.source.kitware.com/#/c/14072/) would have handled
>> > situations such as this through a callback invoked when the data array
>> > is
>> > finished with the pointer. However, that patch was rejected for being
>> > overkill. Instead Berk suggested that I simply use the delete event
>> > observer, which will always result in data being released but perhaps
>> > not as
>> > quickly as with the callback or the method that you proposed(eg nothing
>> > happens if the array is resized). This approach has worked well for me.
>> > I
>> > wonder if using the delete event observer would work well in your case?
>> >
>> > Burlen
>> >
>> >
>> > On 05/16/2014 10:21 AM, David Gobbi wrote:
>> >
>> > Hi All,
>> >
>> > I've used memory mapped files with VTK in the past, using the python
>> > "mmap" object to create the map and then passing the resulting pointer
>> > to VTK via the SetVoidArray() method.  For large files (several GB) I've
>> > found this to be many times faster than e.g. vtkImageReader2.
>> >
>> > My question: has anyone used mmap with VTK in C++?  Specifically,
>> > what I'm interested in doing is adding a new "delete method" to the
>> > SetVoidPointer() method in vtkDataArrayTemplate.h: it already has
>> > "delete methods" for "malloc" and "new", so the idea is to add another
>> > delete method for "mmap" that will call munmap on the pointer.
>> >
>> >    David
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.vtk.org/mailman/listinfo/vtk-developers
>>
>



More information about the vtk-developers mailing list