[vtk-developers] Memory-mapped files in VTK

[Berk Geveci]

I am not objecting to the concept but I have a question. Is it really
faster? Don't you have to take compute time into account if you have a
memory mapped file? That's when disk activity will actually occur. I am
wondering if this approach doesn't simply postpone the inevitable till
later. I understand this should be fast if processing only touches part of
the data but then it shouldn't be loaded in the first place.

-berk



On Fri, May 16, 2014 at 2:08 PM, David Gobbi <david.gobbi at gmail.com> wrote:

> Hi Burlen,
>
> Thanks for the links.  I guess that I'm looking for deeper integration
> (though less general) than a callback.  The idea is to have something
> like a vtkMemoryMap class to take care of all the platform-specific
> details of creating/deleting memory maps, and to have vtkAbstractArray
> be aware of this class and use it to delete mmaps when appropriate.
>
>   David
>
> On Fri, May 16, 2014 at 11:56 AM, Burlen Loring <burlen.loring at gmail.com>
> wrote:
> > meant to say "no special memory management issues"
> > and (in case you actually would look at the code) mmap use is here:
> > https://github.com/burlen/SC11-100GB-Demo/blob/master/SC11DemoReader.h
> >
> > On 05/16/2014 10:50 AM, Burlen Loring wrote:
> >
> > Hi David,
> >
> > I've used mmap with VTK in C++ for the SC11 100GB network demo.
> >
> > http://dl.acm.org/citation.cfm?id=2287004
> > https://github.com/burlen/SC11-100GB-Demo
> >
> > This use was very ad-hoc and we wanted the data mapped for the life of
> the
> > app so there were special memory management issues.
> >
> > patch(http://review.source.kitware.com/#/c/14072/) would have handled
> > situations such as this through a callback invoked when the data array is
> > finished with the pointer. However, that patch was rejected for being
> > overkill. Instead Berk suggested that I simply use the delete event
> > observer, which will always result in data being released but perhaps
> not as
> > quickly as with the callback or the method that you proposed(eg nothing
> > happens if the array is resized). This approach has worked well for me. I
> > wonder if using the delete event observer would work well in your case?
> >
> > Burlen
> >
> >
> > On 05/16/2014 10:21 AM, David Gobbi wrote:
> >
> > Hi All,
> >
> > I've used memory mapped files with VTK in the past, using the python
> > "mmap" object to create the map and then passing the resulting pointer
> > to VTK via the SetVoidArray() method.  For large files (several GB) I've
> > found this to be many times faster than e.g. vtkImageReader2.
> >
> > My question: has anyone used mmap with VTK in C++?  Specifically,
> > what I'm interested in doing is adding a new "delete method" to the
> > SetVoidPointer() method in vtkDataArrayTemplate.h: it already has
> > "delete methods" for "malloc" and "new", so the idea is to add another
> > delete method for "mmap" that will call munmap on the pointer.
> >
> >    David
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://www.vtk.org/mailman/listinfo/vtk-developers
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/vtk-developers/attachments/20140516/4a0eac37/attachment-0002.html>