[Insight-developers] HDF5 writer streaming

[David Froger]

Hi all,

I would like to add writer streaming support to the HDF5 module. Writer
streaming would includes:

1) Write a whole array in a new HDF5 file by iteration on subregion writting.
  For example: write NZ times an array of dimension (NY*NX) in a new HDF5 file
  that will finally store an array of dimensions (NZ*NY*NX).

2) Paste (override) a subregion in an existing HDF5 file. (This require to
open the file in a different mode).

I know how to do it in HDF5, I've looked at the source code of the HDF5 ITK
module, and have some questions on how to patch the HDF5 module.

To support reading and writing subregion, HDF5 manipulates simple arrays
defining the start and the size of the region to read/write. These arrays are
'HDFSize' and 'offset' in the method HDF5ImageIO::SetupStreaming of the file
itkHDF5ImageIO.cxx, and their values are set according to the size and index
of the region 'this->GetIORegion()':

    void
    HDF5ImageIO
    ::SetupStreaming(H5::DataSpace *imageSpace, H5::DataSpace *slabSpace)
    {
      ImageIORegion            regionToRead = this->GetIORegion();
      ImageIORegion::SizeType  size = regionToRead.GetSize();
      ImageIORegion::IndexType start = regionToRead.GetIndex();
      //
      int numComponents = this->GetNumberOfComponents();

      const int HDFDim(this->GetNumberOfDimensions() +
                       (numComponents > 1 ? 1 : 0));
      hsize_t *offset = new hsize_t[HDFDim];
      hsize_t *HDFSize = new hsize_t[HDFDim];
      int limit;
      //
      // fastest moving dimension is intra-voxel
      // index
      if(numComponents > 1)
        {
        limit = HDFDim - 1;
        offset[limit] = 0;
        HDFSize[limit] = numComponents;
        }
      else
        {
        limit = HDFDim;
        }
      for(int i=0; i < limit; i++)
        {
        offset[limit - i - 1] = start[i];
        HDFSize[limit - i - 1] = size[i];
        }
      slabSpace->setExtentSimple(HDFDim,HDFSize);
      imageSpace->selectHyperslab(H5S_SELECT_SET,HDFSize,offset);
      delete[] HDFSize;
      delete[] offset;
    }

When reading a file, 'this->GetIORegion()' can be modified by calling
    reader->GetOutput()->SetRequestedRegion(requestedRegion);
which enables input streaming.

But how to modifiy 'this->GetIORegion()' when writing in a file, for case 1)
and 2)?

Which example / source code can I look to get started with writer streaming?

Thanks for your help!

David