<div dir="ltr"><div><div>Hello matt,<br><br>I wasn't aware of that possibility, thanks for the heads up!It doesn't apply to our case becaues I haven't created a filter but rather a class that takes a region and gives me the splitted regions that later on I feed to an extract filter (or the copy algorithm Bradley pointed out to repaste the results back). Creating a filter doesn't really fit well, the 'low-level' is based on machine learning so it trains and then predicts, and it requires several minutes depending on the size of the image etc. I didn't judge appropriate to create a Filter out of it. <br><br></div>Could you elaborate a little bit on what you were proposing? Let's say I have a set of preprocessing filters, my processing and a set of postprocessing filters. The output would be a set of connected components. The volume doesn't necessarily fit in memory (specially the processing) so we have to stream. Because the chunks need to overlap, are you saying that I could place, in the pipeline, a Filter that is derived from the StreamingImageFilter that re-implements GeneratInputRequestedRegion() to have an overlap of 30 voxels, and that would should do it? where should I place such a filter and how does interact with the pre/post-processing?<br><br></div><div>I will re-read the article on streaming, but maybe it would help me if you pinpoint the particularities of the approach you suggested.<br><br></div><div>Cheers,<br><br></div><div>Pol<br></div><div><br><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2015-03-17 20:22 GMT+01:00 Matt McCormick <span dir="ltr"><<a href="mailto:matt.mccormick@kitware.com" target="_blank">matt.mccormick@kitware.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Pol,<br>
<br>
You may already be aware of this, but the chunks will overlap if you<br>
implement a GenerateInputRequestedRegion() method on your filter that<br>
pads the output RequestedRegion.<br>
<br>
HTH,<br>
Matt<br>
<div class="HOEnZb"><div class="h5"><br>
On Tue, Mar 17, 2015 at 1:13 PM, Pol Monsó Purtí <<a href="mailto:lluna.nova@gmail.com">lluna.nova@gmail.com</a>> wrote:<br>
> Hello Matt,<br>
><br>
> As I've told Bradley, I can't use streaming as-is because I need overlap<br>
> between chunks and because the low-level library would need a proper itk<br>
> image with correct size, region, etc.<br>
> which I guess it wouldn't be provided by streaming.<br>
><br>
> But most definetely, right now I use 'streaming' by using the ExtractFilter<br>
> at the end. I believe that streams too. I was actually more interested in<br>
> having a filter/object that computes the splits for me. I've implemented it,<br>
> but it's far from perfect. For example, it broke when the input image had a<br>
> non-zero index. (It's fixed now, but still, I'm sure the itk's Splitter<br>
> filter does a better job, if only could I have overlap ;))<br>
><br>
> I guess the way to go would be to take the splitter (probably<br>
> ImageRegionSplitterMultidimensional now that I look at it) and extend it to<br>
> support overlap. If I do it, I'll let you know in case it is of interest to<br>
> anyone. For now I'll stick to my class.<br>
><br>
> Cheers,<br>
><br>
> Pol<br>
><br>
> 2015-03-16 16:05 GMT+01:00 Matt McCormick <<a href="mailto:matt.mccormick@kitware.com">matt.mccormick@kitware.com</a>>:<br>
>><br>
>> Hi Pol,<br>
>><br>
>> As a side note, limitation of memory usage is built into the streaming<br>
>> pipeline design of ITK. All computation of appropriate regions should<br>
>> occur if the filters are implemented correctly. To take advantage of<br>
>> streaming, place the StreamingImageFilter [1] at the end of your<br>
>> pipeline. To use less memory, set the ReleaseDataFlagOn() [2] on the<br>
>> component filters.<br>
>><br>
>> HTH,<br>
>> Matt<br>
>><br>
>> [1] <a href="http://www.itk.org/Doxygen/html/classitk_1_1StreamingImageFilter.html" target="_blank">http://www.itk.org/Doxygen/html/classitk_1_1StreamingImageFilter.html</a><br>
>><br>
>> [2]<br>
>> <a href="http://www.itk.org/Doxygen/html/classitk_1_1ProcessObject.html#ae732e9163879ae559ccd0b5957141f97" target="_blank">http://www.itk.org/Doxygen/html/classitk_1_1ProcessObject.html#ae732e9163879ae559ccd0b5957141f97</a><br>
>><br>
>> On Mon, Mar 16, 2015 at 10:20 AM, Bradley Lowekamp<br>
>> <<a href="mailto:blowekamp@mail.nih.gov">blowekamp@mail.nih.gov</a>> wrote:<br>
>> > You may find this method useful:<br>
>> ><br>
>> ><br>
>> > <a href="http://www.itk.org/Doxygen/html/structitk_1_1ImageAlgorithm.html#a5ec5bcac992cb3b1302840636530ba20" target="_blank">http://www.itk.org/Doxygen/html/structitk_1_1ImageAlgorithm.html#a5ec5bcac992cb3b1302840636530ba20</a><br>
>> ><br>
>> > Brad<br>
>> ><br>
>> > On Mar 16, 2015, at 10:16 AM, Pol Monsó Purtí <<a href="mailto:lluna.nova@gmail.com">lluna.nova@gmail.com</a>><br>
>> > wrote:<br>
>> ><br>
>> > Mmh, I believe I misread the RegionOfInterestImageFilter documentation,<br>
>> > confusing dimension with size. That should work.<br>
>> ><br>
>> > I still have the computation of the splitting regions problem to handle,<br>
>> > but<br>
>> > maybe with this filter is easier than with the ExtractImageFilter. Or<br>
>> > maybe<br>
>> > it just removes step 7?<br>
>> ><br>
>> > Is there a way to ease the repasting back to the original region with<br>
>> > it?<br>
>> > How should I use the physical space index?<br>
>> ><br>
>> > 2015-03-16 14:25 GMT+01:00 Pol Monsó Purtí <<a href="mailto:lluna.nova@gmail.com">lluna.nova@gmail.com</a>>:<br>
>> >><br>
>> >> Hello Bradley,<br>
>> >><br>
>> >> Thank you for your answer and suggestion.<br>
>> >><br>
>> >> Answering your question: It's not for multi-threading purposes, I have<br>
>> >> to<br>
>> >> split the image due to memory limitations.<br>
>> >><br>
>> >> I would use extractImage directly and do streaming, but the client and<br>
>> >> the<br>
>> >> low-level libraries might not work correctly if I do that. I have to<br>
>> >> provide<br>
>> >> itk images disconnected from the pipeline and with proper indexes and<br>
>> >> sizes.<br>
>> >><br>
>> >> The RegionOfInterestImageFilter would have the same memory footprint,<br>
>> >> and<br>
>> >> it would still be tricky to compute the regions of interest, due to the<br>
>> >> overlapping.<br>
>> >><br>
>> >><br>
>> >><br>
>> >> 2015-03-16 14:00 GMT+01:00 Bradley Lowekamp <<a href="mailto:blowekamp@mail.nih.gov">blowekamp@mail.nih.gov</a>>:<br>
>> >>><br>
>> >>> Hello,<br>
>> >>><br>
>> >>> The RegionOfInterestImageFilter[1] is very similar to the<br>
>> >>> ExtractImageFilter, but returns an image whose starting index is<br>
>> >>> always<br>
>> >>> zeros.<br>
>> >>><br>
>> >>> The process you described is remarkable close to what occur in each<br>
>> >>> filter for the multi-threading process. Also what occurs in the<br>
>> >>> StreamingImageFilter is very similar, but just copies the region.<br>
>> >>><br>
>> >>> As you didn't mention you motivation for chunking the data this way be<br>
>> >>> it<br>
>> >>> for threading or memory requirements, I can't fully advise.<br>
>> >>><br>
>> >>> Brad<br>
>> >>><br>
>> >>><br>
>> >>> [1<br>
>> >>><br>
>> >>> ]<a href="http://www.itk.org/Doxygen/html/classitk_1_1RegionOfInterestImageFilter.html" target="_blank">http://www.itk.org/Doxygen/html/classitk_1_1RegionOfInterestImageFilter.html</a><br>
>> >>> [2]<br>
>> >>><br>
>> >>> <a href="https://github.com/InsightSoftwareConsortium/ITK/blob/b184194869ac6407fcbb4e309710894cfc84466f/Modules/Core/Common/include/itkStreamingImageFilter.hxx#L194-L213" target="_blank">https://github.com/InsightSoftwareConsortium/ITK/blob/b184194869ac6407fcbb4e309710894cfc84466f/Modules/Core/Common/include/itkStreamingImageFilter.hxx#L194-L213</a><br>
>> >>><br>
>> >>><br>
>> >>> On Mar 16, 2015, at 7:37 AM, Pol Monsó Purtí <<a href="mailto:lluna.nova@gmail.com">lluna.nova@gmail.com</a>><br>
>> >>> wrote:<br>
>> >>><br>
>> >>> > Hello everyone,<br>
>> >>> ><br>
>> >>> > We need to split a volume into pieces, overlaped by a certain<br>
>> >>> > amount,<br>
>> >>> > do some computation and then repaste them.<br>
>> >>> ><br>
>> >>> > The computation requires the pieces' region indexes to be set to<br>
>> >>> > zero.<br>
>> >>> > The original volume might not have index at zero, but be a subregion<br>
>> >>> > itself.<br>
>> >>> ><br>
>> >>> > What is the best way to achieve this?<br>
>> >>> ><br>
>> >>> > I've thought of the following pipeline:<br>
>> >>> ><br>
>> >>> > 1. imageRegionSplitterSlowDimension<br>
>> >>> > 2. Store the index and size of each region<br>
>> >>> > 3. manually increase the size of each region (and modify index)<br>
>> >>> > 4. crop with the largestPossibleRegion of the original volume (to<br>
>> >>> > prevent requesting more than available)<br>
>> >>> > iterate over regions:<br>
>> >>> > 5. extractImageFilter<br>
>> >>> > 6. disconnect pipeline for each region<br>
>> >>> > 7. reset indexes<br>
>> >>> > 8. perform computation<br>
>> >>> > 9. crop with the regions in step 1 to remove the added overlap<br>
>> >>> > 10. restore indexes<br>
>> >>> > 11. pasteImageFilter<br>
>> >>> ><br>
>> >>> > I'll have to think of a smart way to deal with the index<br>
>> >>> > restore/cropping of steps 9 and 10.<br>
>> >>> ><br>
>> >>> > I somehow feel somebody will have already dealt with this... And<br>
>> >>> > there's certainly a better way to do it.<br>
>> >>> > _____________________________________<br>
>> >>> > Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
>> >>> ><br>
>> >>> > Visit other Kitware open-source projects at<br>
>> >>> > <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
>> >>> ><br>
>> >>> > Kitware offers ITK Training Courses, for more information visit:<br>
>> >>> > <a href="http://www.kitware.com/products/protraining.php" target="_blank">http://www.kitware.com/products/protraining.php</a><br>
>> >>> ><br>
>> >>> > Please keep messages on-topic and check the ITK FAQ at:<br>
>> >>> > <a href="http://www.itk.org/Wiki/ITK_FAQ" target="_blank">http://www.itk.org/Wiki/ITK_FAQ</a><br>
>> >>> ><br>
>> >>> > Follow this link to subscribe/unsubscribe:<br>
>> >>> > <a href="http://public.kitware.com/mailman/listinfo/insight-users" target="_blank">http://public.kitware.com/mailman/listinfo/insight-users</a><br>
>> >>><br>
>> >><br>
>> >><br>
>> >> 2015-03-16 14:00 GMT+01:00 Bradley Lowekamp <<a href="mailto:blowekamp@mail.nih.gov">blowekamp@mail.nih.gov</a>>:<br>
>> >>><br>
>> >>> Hello,<br>
>> >>><br>
>> >>> The RegionOfInterestImageFilter[1] is very similar to the<br>
>> >>> ExtractImageFilter, but returns an image whose starting index is<br>
>> >>> always<br>
>> >>> zeros.<br>
>> >>><br>
>> >>> The process you described is remarkable close to what occur in each<br>
>> >>> filter for the multi-threading process. Also what occurs in the<br>
>> >>> StreamingImageFilter is very similar, but just copies the region.<br>
>> >>><br>
>> >>> As you didn't mention you motivation for chunking the data this way be<br>
>> >>> it<br>
>> >>> for threading or memory requirements, I can't fully advise.<br>
>> >>><br>
>> >>> Brad<br>
>> >>><br>
>> >>><br>
>> >>> [1<br>
>> >>><br>
>> >>> ]<a href="http://www.itk.org/Doxygen/html/classitk_1_1RegionOfInterestImageFilter.html" target="_blank">http://www.itk.org/Doxygen/html/classitk_1_1RegionOfInterestImageFilter.html</a><br>
>> >>> [2]<br>
>> >>><br>
>> >>> <a href="https://github.com/InsightSoftwareConsortium/ITK/blob/b184194869ac6407fcbb4e309710894cfc84466f/Modules/Core/Common/include/itkStreamingImageFilter.hxx#L194-L213" target="_blank">https://github.com/InsightSoftwareConsortium/ITK/blob/b184194869ac6407fcbb4e309710894cfc84466f/Modules/Core/Common/include/itkStreamingImageFilter.hxx#L194-L213</a><br>
>> >>><br>
>> >>><br>
>> >>> On Mar 16, 2015, at 7:37 AM, Pol Monsó Purtí <<a href="mailto:lluna.nova@gmail.com">lluna.nova@gmail.com</a>><br>
>> >>> wrote:<br>
>> >>><br>
>> >>> > Hello everyone,<br>
>> >>> ><br>
>> >>> > We need to split a volume into pieces, overlaped by a certain<br>
>> >>> > amount,<br>
>> >>> > do some computation and then repaste them.<br>
>> >>> ><br>
>> >>> > The computation requires the pieces' region indexes to be set to<br>
>> >>> > zero.<br>
>> >>> > The original volume might not have index at zero, but be a subregion<br>
>> >>> > itself.<br>
>> >>> ><br>
>> >>> > What is the best way to achieve this?<br>
>> >>> ><br>
>> >>> > I've thought of the following pipeline:<br>
>> >>> ><br>
>> >>> > 1. imageRegionSplitterSlowDimension<br>
>> >>> > 2. Store the index and size of each region<br>
>> >>> > 3. manually increase the size of each region (and modify index)<br>
>> >>> > 4. crop with the largestPossibleRegion of the original volume (to<br>
>> >>> > prevent requesting more than available)<br>
>> >>> > iterate over regions:<br>
>> >>> > 5. extractImageFilter<br>
>> >>> > 6. disconnect pipeline for each region<br>
>> >>> > 7. reset indexes<br>
>> >>> > 8. perform computation<br>
>> >>> > 9. crop with the regions in step 1 to remove the added overlap<br>
>> >>> > 10. restore indexes<br>
>> >>> > 11. pasteImageFilter<br>
>> >>> ><br>
>> >>> > I'll have to think of a smart way to deal with the index<br>
>> >>> > restore/cropping of steps 9 and 10.<br>
>> >>> ><br>
>> >>> > I somehow feel somebody will have already dealt with this... And<br>
>> >>> > there's certainly a better way to do it.<br>
>> >>> > _____________________________________<br>
>> >>> > Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
>> >>> ><br>
>> >>> > Visit other Kitware open-source projects at<br>
>> >>> > <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
>> >>> ><br>
>> >>> > Kitware offers ITK Training Courses, for more information visit:<br>
>> >>> > <a href="http://www.kitware.com/products/protraining.php" target="_blank">http://www.kitware.com/products/protraining.php</a><br>
>> >>> ><br>
>> >>> > Please keep messages on-topic and check the ITK FAQ at:<br>
>> >>> > <a href="http://www.itk.org/Wiki/ITK_FAQ" target="_blank">http://www.itk.org/Wiki/ITK_FAQ</a><br>
>> >>> ><br>
>> >>> > Follow this link to subscribe/unsubscribe:<br>
>> >>> > <a href="http://public.kitware.com/mailman/listinfo/insight-users" target="_blank">http://public.kitware.com/mailman/listinfo/insight-users</a><br>
>> >>><br>
>> >><br>
>> ><br>
>> ><br>
>> ><br>
>> > _____________________________________<br>
>> > Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
>> ><br>
>> > Visit other Kitware open-source projects at<br>
>> > <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
>> ><br>
>> > Kitware offers ITK Training Courses, for more information visit:<br>
>> > <a href="http://www.kitware.com/products/protraining.php" target="_blank">http://www.kitware.com/products/protraining.php</a><br>
>> ><br>
>> > Please keep messages on-topic and check the ITK FAQ at:<br>
>> > <a href="http://www.itk.org/Wiki/ITK_FAQ" target="_blank">http://www.itk.org/Wiki/ITK_FAQ</a><br>
>> ><br>
>> > Follow this link to subscribe/unsubscribe:<br>
>> > <a href="http://public.kitware.com/mailman/listinfo/insight-users" target="_blank">http://public.kitware.com/mailman/listinfo/insight-users</a><br>
>> ><br>
><br>
><br>
</div></div></blockquote></div><br></div>