<div dir="ltr">Hello Bradley,<br><br><div class="gmail_extra">Thank you for your suggestions! I'm glad somebody is working on tif streaming :) I answer below: <br><br><br></div><div class="gmail_extra"><div class="gmail_quote">2014-11-03 21:38 GMT+01:00 Bradley Lowekamp <span dir="ltr"><<a href="mailto:blowekamp@mail.nih.gov" target="_blank">blowekamp@mail.nih.gov</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello,<br>
<br>
You might find this article useful I wrote on streaming ImageIO [1].<br>
<span class=""><br></span></blockquote><div><br><div>I've read the article early this afternoon and I found it very good and encouraging. Specially <i>"The other difference is that the reader fully supports streaming and only reads the required region from the file."</i><br></div> <br></div><div>That is exactly what we need. I was just wondering wether it splitted the volume and took the chunks that intersected with the requested one or if it really just loaded the requested one. I replicated that code taking out the processing and just having a reader and a writer, but I didn't know how to test that indeed it was just loading that requested piece.<br><br></div><div>So after reading your comments here I think streaming is the way to go. Can you assert that only the requested region will be read? <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="">
<br>
> 1.- How to set the RequestedRegion on a writer.<br>
<br>
</span>I am not sure what you are wanting to do. Are you wanting to extract a region from the input file and write a partial sub image?<br>
<br>
There are two modes for the writer to drive this. 1) streaming - the sequential writing of sub-region of the whole image 2) pasting - updating a sub-region of an existing image file.<br>
<span class=""><br></span></blockquote><br></div><div class="gmail_quote">I didn't expect that some would support reading but not writing. Actually it's not mandatory for what we intend to do:<br><br></div><div class="gmail_quote">implifying, we have a huge image that we can't (and won't) load to disk. Fortunatelly we are only interested in some specific chunks of it. So we need to find a way to read those chunks, compute a simple operation and then we can discard them. I wanted to write to check that indeed we were retrieving the ROI and not the full image. We can't afford the reader to read the whole image. We don't want to process the full image.<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="">
<br>
> 2.- Will the pipeline only load the requested region, including the reader?<br>
<br>
</span>The pipeline has a series of steps which include propagating a requested region, and giving each filter the opportunity to enlarge this region. I have frequently assemble pipelines which perform out-of-core processing as you desire. However all filters in the pipeline must support streaming, and not enlarging the requested region to the largest possible region.<br>
<br></blockquote><div><br></div><div>Yes, I've seen that not many support streaming. But in our case we would just need a reader that can read a specific region. <br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Additionally the ImageIO for the reader and writer must also support streaming. There are several ImageIO's which support streaming reading (MRC,MetaIO, VTK, Nift, JPEG2000, etc), I believe that only MRC, VTK and MetaIO fully support it for writing. So TIFF is currently not there.<br></blockquote><div><br></div><div>Yeah, we were thinking on using HDF5, but maybe you have some idea on what would be best. That's another topic thought.<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Recently I have been working on improving the itkTIFFImageIO [2], and have performance gains of upto 3-5X from the prior version. However, I haven't gotten to adding streaming yet. It's is a back burner project for me to add support for stream reading to it, so that may be coming shortly ( this poking helps ). However, adding stream and paste writing is not planned.<br>
<br></blockquote><div><br></div><div>Good work! 3-5x! That might be of interest actually for something else we have. Can we pull your code?<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Let's look at your code.<br>
<div><div class="h5"><br>
> I've tried myself but I can't get to write just the requested region.<br>
><br>
> The code:<br>
><br>
> int main( int argc, char* argv[] )<br>
> {<br>
><br>
> typedef unsigned char PixelType;<br>
> typedef itk::Image< PixelType, 3 > ImageType;<br>
><br>
> typedef itk::ImageFileReader< ImageType > ReaderFilterType;<br>
> ReaderFilterType::Pointer reader = ReaderFilterType::New();<br>
> reader->SetFileName("../data/<br>
> Madrid_Train.tif");<br>
> ImageType::RegionType largest = reader->GetOutput()->GetLargestPossibleRegion();<br>
> ImageType::SizeType size = largest.GetSize();<br>
> ImageType::IndexType index = largest.GetIndex();<br>
><br>
> ImageType::RegionType half;<br>
> half.SetIndex(0,index[0] + 0.25*size[0]);<br>
> half.SetIndex(1,index[1] + 0.25*size[1]);<br>
> half.SetIndex(2,index[2] + 0.25*size[2]);<br>
><br>
> half.SetSize(0,0.5*size[0]);<br>
> half.SetSize(1,0.5*size[1]);<br>
> half.SetSize(2,0.5*size[2]);<br>
><br>
> typedef itk::ImageFileWriter< ImageType > WriterFilterType;<br>
> WriterFilterType::Pointer writer = WriterFilterType::New();<br>
> writer->SetFileName("image.tif");<br>
> reader->GetOutput()->SetRequestedRegion(half);<br>
> writer->SetInput(reader->GetOutput());<br>
><br>
> try<br>
> {<br>
> //streamingFilter->Update();<br>
> writer->Update();<br>
> }<br>
> catch( itk::ExceptionObject & error )<br>
> {<br>
> std::cerr << "Error: " << error << std::endl;<br>
> return EXIT_FAILURE;<br>
> }<br>
<br>
</div></div>Unfortunately this is not the way the pipeline works. The reader does not actually read and update the image until the pipeline is updated. I would recommend additional reading about how a pipeline architecture works, perhaps from the ITK software guide or an VTK book.<br>
<br>
For this case you should add in an ExtractImageFilter, or a CropImageFilter.<br>
<br></blockquote><div><br></div><div>But if I use Extract or Crop, the reader is still loading the full image to disk, isn't it?<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
Brad<br>
<br>
</div>Thank you for the article, I'll reproduce and let you know how it went.<br></blockquote><div><br></div><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
[1] <a href="http://www.kitware.com/media/html/IOStreamingInITK.html" target="_blank">http://www.kitware.com/media/html/IOStreamingInITK.html</a><br>
[2] <a href="https://github.com/InsightSoftwareConsortium/ITK/commits/master/Modules/IO/TIFF" target="_blank">https://github.com/InsightSoftwareConsortium/ITK/commits/master/Modules/IO/TIFF</a><br>
[3] <a href="http://www.itk.org/Doxygen/html/IO_2VisibleHumanStreamReadWrite_8cxx-example.html" target="_blank">http://www.itk.org/Doxygen/html/IO_2VisibleHumanStreamReadWrite_8cxx-example.html</a><br>
<span class=""><br>
<br>
On Nov 3, 2014, at 2:06 PM, Pol Monsó Purtí <<a href="mailto:lluna.nova@gmail.com">lluna.nova@gmail.com</a>> wrote:<br>
<br>
> Hello everyone.<br>
><br>
> Two simple questions:<br>
><br>
><br>
</span><span class="">> By reading the itksoftware manual, section 8.3 Streaming Large Data it looks like by default reader filters only load the requested region.<br>
><br>
> We want to extract a specific region of a large 3D tiff image that does not fit in memory. This has to be as fast as possible.<br>
><br>
> I've taken a look at streaming, althought it's not exactly what we want since we don't want to process the full image.<br>
><br>
</span>> return EXIT_SUCCESS;<br>
> }<br>
> _____________________________________<br>
> Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
><br>
> Visit other Kitware open-source projects at<br>
> <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
><br>
> Kitware offers ITK Training Courses, for more information visit:<br>
> <a href="http://www.kitware.com/products/protraining.php" target="_blank">http://www.kitware.com/products/protraining.php</a><br>
><br>
> Please keep messages on-topic and check the ITK FAQ at:<br>
> <a href="http://www.itk.org/Wiki/ITK_FAQ" target="_blank">http://www.itk.org/Wiki/ITK_FAQ</a><br>
><br>
> Follow this link to subscribe/unsubscribe:<br>
> <a href="http://public.kitware.com/mailman/listinfo/insight-users" target="_blank">http://public.kitware.com/mailman/listinfo/insight-users</a><br>
<br>
</blockquote></div><br></div></div>