[Insight-users] Lack of memory for segmentation; SetNumberOfThreads

[Atwood, Robert C]

Thanks,Jim, for the clarification. I was seeing that 'threading' was
happening when I expected 'streaming', this gives me the correct names
to use!  Also I did not realize that it requied a separate filter to
enable streaming. In the original suggestion I thought that you meant
any of the filters that support streaming needed to be downstream. 


However, the StreamingPage.html seems to not have any text on it? I
tried Firefox and also Microsoft Internet Explorer browsers in case it
was a browser problem, but all I see is the image illustrating the
concept but no text explaining how to use it. 

I have tried it now in what appears to be the correct way , also I
changed to GradientAnisotropicDiffusion which seems to be streamable and
gives similar results for my purposes. 

Unfortunately I still cannot process an image of greater than 1 gb on a
2gb memory node it cannot allocate the memory for the output image, so
it seems that the biggest image is half the memory, to enable the
original and the final whole images to be in memory despite having some
swap space set up? Does this make sense?



Thanks
Robert

 

 

-----Original Message-----
From: Miller, James V (GE, Research) [mailto:millerjv at crd.ge.com] 
Sent: 24 January 2006 21:56
To: Atwood, Robert C; insight-users at itk.org
Subject: RE: [Insight-users] Lack of memory for segmentation;
SetNumberOfThreads

Robert,

Streaming and threading are separate concepts that leverage the same
underlying architecture.

In threading, a filter will divide an image into num_processors
sections, 
and process all the section is parallel.  The MedianImageFilter and the 
CurvatureFlowImageFilter support threading.

In streaming, the problem is divided in N sections, and each filter in 
the pipeline is called N times, once for each section.  Within each 
filter execution, the section may be further divided for threading.
Not all filters support streaming.  For instance, any filter that
requires
global information can effectively stream.  The MedianImageFilter
supports
streaming but the CurvatureFlowImageFilter does not.

These concepts are explained further in the doxygen documentation.

http://www.itk.org/Doxygen16/html/ThreadingPage.html

http://www.itk.org/Doxygen16/html/StreamingPage.html


To use streaming, you need to place a StreamingImageFilter in the
pipeline.
If some of the filters you are using do not support streaming, you want
to 
place the StreamingImageFilter downstream from the filters that do
support
streaming and upstream from the filters that do not support streaming.

Jim


-----Original Message-----
From: Atwood, Robert C [mailto:r.atwood at imperial.ac.uk]
Sent: Tuesday, January 24, 2006 11:26 AM
To: Miller, James V (GE, Research); insight-users at itk.org
Subject: Re: [Insight-users] Lack of memory for segmentation;
SetNumberOfThreads


Hi,
I have finally got round to trying this. However, the implication I get
from the suggestion is that the entire pipleline will be run N times if
one of the filters is set to stream in N chunks.
I have a pipline like (minimized code snippet, Volmedian is
itkMedianImageFilter  and Curvflowdiff is itkCurvatureFlowImageFilter)

itk::SimpleFilterWatcher volmwatcher(Volmedian,"VolMedian");
itk::SimpleFilterWatcher pfwatcher(Curvflow,"PostFilter");

Volmedian->SetNumberOfThreads(nchunks);

Volmedian->SetInput(ExistingImage);
Curvflow->SetInput(Volmedian->GetOutput());
Writer->SetInput(Curvflow->GetOutput());

Writer->Update(); /* in a try/catch block */ 


The output of the SimpleFilterWatcher indicates that the number of
threads for the volume median is nchunks (8) as requested, but the
number of threads for the post-filter (Curvflow) is 2 (it seems to use
the number of processors, it goes to 4 on a similar machine with
'hyperthreading' enabled)


Is this what should happen? Did I misunderstand? From the watcher
otuput, it seems that the entire volume median filter completes
executing before the post-filter starts, or is this a misunderstanding
of the output again? 

Thanks in advance for clarifying these matters.

Robert

(sections of the output .. There is a lot of output ... )

-------- End MedianImageFilter "VolMedian" MedianImageFilter (0x8452538)

  .....

  Thread Count: 8
  Global Maximum Number Of Threads: 0
  Radius: [1, 1, 1]

-------- Start CurvatureFlowImageFilter "PostFilter"
CurvatureFlowImageFilter (0x8459780)
 
.....

-------- End CurvatureFlowImageFilter "PostFilter"
CurvatureFlowImageFilter (0x8459780)

....
   Debug: Off
    Observers: 
      none
    Thread Count: 2
    Global Maximum Number Of Threads: 0
.....



_________________Original message quote regarding running out of memory
___________

Miller, James V (Research) 
Fri Nov 18 09:16:50 EST 2005
 
There are a number of strategies for managing memory.
....
2) StreamingImageFilter.  You can place a StreamingImageFilter at the
end of
your pipeline (or anywhere within the pipeline) and tell the
StreamingImageFilter
to divide the problem into N sections.  The pipeline will then execute N
times, 
....