[Insight-developers] Image memory model

[Dan Mueller]

Hi Karthik,

Thanks for exploring this issue, but I am very confused by some of
your suggestions.

> GetBufferPointer() as such is never (I think) used within ITK
> directly. However it provides a convinient way of exporting data to
> other libraries.

Image::GetBufferPointer() is used extensively within the toolkit. Take
a look at the constructors for ImageConstIterator,
ConstNeighborhoodIterator, ImageReverseConstIterator,
ImageConstIteratorWithIndex, etc. I have added a list to
    http://www.itk.org/Wiki/Proposals:Slice_contiguous_images
however I realise now that some of these are
ImportImageContainer::GetBufferPointer() (I'll fix this up when time
permits).

Unfortunately, while your suggestion of using the PixelAccessor and
PixelAccessorFunctor would have been easy, I think it is too good to
be true. The assumptions regarding contiguous memory allocation are
deeply entrenched.

Take for example the very first step you discussed, the act of passing
the input pixel value from the iterator.Get() function to the
PixelAccessorFunctor:
>   PixelType iterator.Get()   reads
>    { return pixelAccessorFunctor.Get(*(m_Buffer+m_Offset)); }

This deference *(m_Buffer+m_Offset) will access potentially invalid
memory in the slice-contiguous case (m_Buffer is the result of
m_Image->GetBufferPointer(), the start of the contiguous array, or the
start of the first slice for my case).

There are other places this assumption is also made: for example
ImageConstIteratorWithIndex::SetIndex() uses the buffer pointer to
compute the index.

Have I misinterpreted your meaning? I think either m_Buffer and/or
m_Offset would need to altered from their original meaning to work for
slice-contiguous memory...

Regards, Dan

2008/5/25 Karthik Krishnan <karthik.krishnan at kitware.com>:
> Volken, Dan:
>
> Thanks for the detailed explanation. Comments are inlined.
>
> On Sat, May 24, 2008 at 1:28 AM, Dan Mueller <dan.muel at gmail.com> wrote:
>> Thanks for engaging in this discussion with me.
>>
>> To cut to the chase, my actual problem is politics -- the "not
>> invented here syndrome". I am trying to introduce ITK into my project,
>> but have encountered resistance from other developers. The memory
>> model is their biggest concern; they seem blinded to the plethora of
>> advantages ITK offers. As such I am investigating the possibility of
>> removing this stumbling block. I guess suggesting a breaking change to
>> ITK due to the internal politics at my work is probably not a very
>> good idea :P Let me describe the problem further to see if there is
>> another way round.
>>
>> Some background. We have an extensive image processing application
>> which uses a slice-contiguous model (based on DICOM). Images are read
>> from disk, loaded into memory, image processing operations performed,
>> and results rendered to screen or written back to disk. There may be 5
>> or more 1GB images open at a time (many situations require less
>> memory, some require more). To integrate ITK I am currently 1.
>> importing each slice using the Import filter, copying the slices into
>> contiguous memory using the Tile filter, applying the desired ITK
>> algorithm(s), and converting the contiguous memory back to slices (if
>> required). The conversions consume unnecessary time and (more
>> importantly) memory, which is not acceptable in various circumstances
>> (having to consume an extra 5GB of memory to duplicate image slices is
>> not very nice). Some of the ITK operations I want to use in this
>> setting include (but are not limited to): registration, level set
>> segmentation, intensity-based segmentation, distance transforms, and
>> mesh operations (image to mesh, mesh to image).
>>
>> So, how exactly would the proposed itk::SliceContiguousImage subclass
>> work? I guess I would need to override the PixelContainer type (by
>> adding a new template argument to itk:Image, which defaults to
>> ImportImageContainer) and override the Set/GetPixel(..) methods. Then
>> I guess I would need to create a new ImportSlicedImagePixelContainer
>> to replace ImportImageContainer. I'm not sure I understand how the new
>> image type would interact with linear iterators -- these iterators use
>> m_Buffer+m_Offset throughout. Could I somehow override
>> ImageBase::ComputeOffsetTable(..) to return the correct offset based
>> on the slice memory model?
>
>> -OR-
>> 2. attempt to get linear *and* neighbor iterators to work with a slice
>> contiguous memory model.
>>
>
> Very much possible with the iterators.
>
> We already do this with the itk::VectorImage. More on it here
>  http://www.itk.org/Wiki/Proposals:VectorImage
>
> Any access to any pixel via the iterators goes through another class
> called the "PixelAccessor" and the "PixelAccessorFunctor".
>
> Each image may define its own PixelAccessor. ie its a trait of the Image.
>
> For instance the itk::VectorImage, which also internally has a
> different memory layout from a standard itk::Image, this is done by
> defining our own PixelAccessor. Please see
> itk::DefaultVectorPixelAccessor
>
> ------
>
> If you follow the code of Iterator's Set method:
>
> 1. Iterator delegates the actual job of access to the pixel to a
> "PixelAccessorFunctor".
>
>   PixelType iterator.Get()   reads
>    { return pixelAccessorFunctor.Get(*(m_Buffer+m_Offset)); }
>
>
> 2. PixelAccessorFunctor delegates it to the appropriate
> "PixelAccessor" for the image. For instance for the
> DefaultVectorPixelAccessorFunctor, the one used for the VectorImage,
> this reads:
>
>  inline ExternalPixelType Get( const InternalPixelType &input ) const
>    {
>    return m_PixelAccessor.Get( input, &input  - m_Begin );
>    }
>
> Note that we pass the offset in:  (&input  - m_Begin)
>
>
> 3. The PixelAccessor for a vector image uses the offset to compute the
> correct index into its memory layout and return the right pixel on the
> fly.
>
>  ExternalType Get( const InternalType & input, const unsigned long
> offset ) const
>    {
>    ExternalType output( (&input)+(offset*m_OffsetMultiplier) ,
> m_VectorLength );
>    return output;
>    }
>
> --------
>
> The same holds true for the iterator.Set(..) methods.
>
> You can see that PixelAccessors allow you to:
>
> (a) Plug in your own way of accessing the image, by computing the
> actual index from the iterator's offset (m_Offset).
>
> (b) You don't need this, but FYI. You can construct a different return
> type/ input type as well, so as to implement a facade. For the
> VectorImage, we not only have a different memory layout, but we wish
> to exhibit a facade to the actual pixel type (much like
> ImageAdaptors). Pixels retreived/set via Get/Set are VectorPixels,
> however the memory organization is that of multi-component Image<
> pixels >. (more on the wiki).
>
> Note that all the methods above are inlined for rapid access despite
> the delegation.
>
> --------
>
> So the gist is:
>
>  (a) You would write your own Image class: itk::SliceImage.
>
>  (b) You would write your own SliceImagePixelAccessor and
> SliceImagePixelAccessorFunctor
>
>  (c) These classes would be traits of your image.
>
> That's it. I also take back statements the statement we might have
> made that you would have to write your own iterator. I don't think
> this will be needed. All the current iterators in ITK will work fine
> on your image as they do for VectorImage.
>
> ------
>
> GetBufferPointer() as such is never (I think) used within ITK
> directly. However it provides a convinient way of exporting data to
> other libraries.
>
> Hope this helps.
>
> --
> Karthik Krishnan
> R&D Engineer,
> Kitware Inc.
>