<div dir="ltr">Makes sense. Thanks for clarifying.</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, May 16, 2014 at 2:28 PM, David Gobbi <span dir="ltr"><<a href="mailto:david.gobbi@gmail.com" target="_blank">david.gobbi@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Berk,<br>
<br>
I was passing the data through a convolution filter immediately after<br>
mapping it, and what I was timing was how long it took for that<br>
operation to complete, for mmap vs. vtkImageReader2.<br>
<br>
However, the biggest advantage with mmap is when you are using<br>
the same file over and over again in different session (especially e.g.<br>
when you are debugging). If you have a machine with loads of RAM,<br>
the whole file will remain in memory from one session to the next, so<br>
with mmap loading the file in subsequent sessions comes at zero cost.<br>
With a more typical reader, the file may still be cached in memory,<br>
but the reader is forced to copy the data from disk cache to a new array.<br>
<span class="HOEnZb"><font color="#888888"><br>
David<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
On Fri, May 16, 2014 at 12:16 PM, Berk Geveci <<a href="mailto:berk.geveci@kitware.com">berk.geveci@kitware.com</a>> wrote:<br>
> I am not objecting to the concept but I have a question. Is it really<br>
> faster? Don't you have to take compute time into account if you have a<br>
> memory mapped file? That's when disk activity will actually occur. I am<br>
> wondering if this approach doesn't simply postpone the inevitable till<br>
> later. I understand this should be fast if processing only touches part of<br>
> the data but then it shouldn't be loaded in the first place.<br>
><br>
> -berk<br>
><br>
><br>
><br>
> On Fri, May 16, 2014 at 2:08 PM, David Gobbi <<a href="mailto:david.gobbi@gmail.com">david.gobbi@gmail.com</a>> wrote:<br>
>><br>
>> Hi Burlen,<br>
>><br>
>> Thanks for the links. I guess that I'm looking for deeper integration<br>
>> (though less general) than a callback. The idea is to have something<br>
>> like a vtkMemoryMap class to take care of all the platform-specific<br>
>> details of creating/deleting memory maps, and to have vtkAbstractArray<br>
>> be aware of this class and use it to delete mmaps when appropriate.<br>
>><br>
>> David<br>
>><br>
>> On Fri, May 16, 2014 at 11:56 AM, Burlen Loring <<a href="mailto:burlen.loring@gmail.com">burlen.loring@gmail.com</a>><br>
>> wrote:<br>
>> > meant to say "no special memory management issues"<br>
>> > and (in case you actually would look at the code) mmap use is here:<br>
>> > <a href="https://github.com/burlen/SC11-100GB-Demo/blob/master/SC11DemoReader.h" target="_blank">https://github.com/burlen/SC11-100GB-Demo/blob/master/SC11DemoReader.h</a><br>
>> ><br>
>> > On 05/16/2014 10:50 AM, Burlen Loring wrote:<br>
>> ><br>
>> > Hi David,<br>
>> ><br>
>> > I've used mmap with VTK in C++ for the SC11 100GB network demo.<br>
>> ><br>
>> > <a href="http://dl.acm.org/citation.cfm?id=2287004" target="_blank">http://dl.acm.org/citation.cfm?id=2287004</a><br>
>> > <a href="https://github.com/burlen/SC11-100GB-Demo" target="_blank">https://github.com/burlen/SC11-100GB-Demo</a><br>
>> ><br>
>> > This use was very ad-hoc and we wanted the data mapped for the life of<br>
>> > the<br>
>> > app so there were special memory management issues.<br>
>> ><br>
>> > patch(<a href="http://review.source.kitware.com/#/c/14072/" target="_blank">http://review.source.kitware.com/#/c/14072/</a>) would have handled<br>
>> > situations such as this through a callback invoked when the data array<br>
>> > is<br>
>> > finished with the pointer. However, that patch was rejected for being<br>
>> > overkill. Instead Berk suggested that I simply use the delete event<br>
>> > observer, which will always result in data being released but perhaps<br>
>> > not as<br>
>> > quickly as with the callback or the method that you proposed(eg nothing<br>
>> > happens if the array is resized). This approach has worked well for me.<br>
>> > I<br>
>> > wonder if using the delete event observer would work well in your case?<br>
>> ><br>
>> > Burlen<br>
>> ><br>
>> ><br>
>> > On 05/16/2014 10:21 AM, David Gobbi wrote:<br>
>> ><br>
>> > Hi All,<br>
>> ><br>
>> > I've used memory mapped files with VTK in the past, using the python<br>
>> > "mmap" object to create the map and then passing the resulting pointer<br>
>> > to VTK via the SetVoidArray() method. For large files (several GB) I've<br>
>> > found this to be many times faster than e.g. vtkImageReader2.<br>
>> ><br>
>> > My question: has anyone used mmap with VTK in C++? Specifically,<br>
>> > what I'm interested in doing is adding a new "delete method" to the<br>
>> > SetVoidPointer() method in vtkDataArrayTemplate.h: it already has<br>
>> > "delete methods" for "malloc" and "new", so the idea is to add another<br>
>> > delete method for "mmap" that will call munmap on the pointer.<br>
>> ><br>
>> > David<br>
>> _______________________________________________<br>
>> Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
>><br>
>> Visit other Kitware open-source projects at<br>
>> <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
>><br>
>> Follow this link to subscribe/unsubscribe:<br>
>> <a href="http://www.vtk.org/mailman/listinfo/vtk-developers" target="_blank">http://www.vtk.org/mailman/listinfo/vtk-developers</a><br>
>><br>
><br>
</div></div></blockquote></div><br></div>