[Insight-developers] read stdin, write stdout
Bill Lorensen
bill.lorensen at gmail.com
Tue Oct 1 07:44:41 EDT 2013
We did a memory read/write imageio for Slicer4. The code is here:
http://viewvc.slicer.org/viewvc.cgi/Slicer4/trunk/Libs/MRML/IDImageIO/
A similar approach might be used to read/write pipes.
Bill
On Tue, Oct 1, 2013 at 1:43 AM, David Froger <david.froger at inria.fr> wrote:
> Dear all,
>
> We are looking for a proper way to integrate in ITK reading/writing
> in standard input/output HDF5 images.
>
> We write a "proove concept" that works, but involves a lot of code
> duplication that should be avoided. The rest of this long mail is divided
> into these sections:
> - Motivations
> - HDF5 and unix pipe
> - Building ITK with recent HDF5 version
> - itkHDF5unixpipeImageIO module
> - ImageUnixPipeReader module
> - Using the new modules
> - Questions
> - References
>
> We can create topic branch as described in the Git workflow [2] if need.
>
> Thanks for reading!
>
> Best regards,
> David Froger
>
>
> Motivations
> -----------
>
> We are rewriting command line tools based on the INRimage [1] image
> format/library so that they are now based ITK.
>
> A particularity of these command line tools is that they can be piped with unix
> pipe. For example, the 'so' command subtracts the pixel values of two images,
> while the 'ical' command computes min/max/average pixel values of a image. To
> test if two images are equals, we pipe the commands as follow:
> so image1.inr image2.inr | ical
>
> While we are aware of the ITK pipeline and that we can use it to pipe ITK
> sources/mappers/filters into one processus (as suggested in [3] for example),
> we want to keep the possibility of using unix pipe in order to keep a backward
> compatibilty because we use numerous bash and perl scripts.
>
> HDF5 and unix pipe
> ------------------
>
> The new file image operations library [4], introduced in HDF5 at the 1.8.9 version
> on May 2012 makes easy to read/write HDF5 images on standard input/output.
>
> An example of writting an HDF5 image to standard output is:
>
> #include <stdio.h>
> #include "hdf5.h"
> #include "H5Cpp.h"
> #include "H5LTpublic.h"
>
> // C array -> HDF5 image file -> buffer -> stdout
>
> #define NDATA 5
>
> int main()
> {
> // Create C array.
> int data[NDATA] = {2, 4, 8, 16, 32};
>
> // Open HDF5 image file.
> hbool_t backing_store = 0;
> H5::FileAccPropList fapl;
> fapl.setCore((size_t)1,backing_store);
> H5::H5File file("nosuch.h5",H5F_ACC_TRUNC,H5::FileCreatPropList::DEFAULT,fapl);
>
> // Write C array in HDF5 image file.
> hsize_t dims[1]= {NDATA};
> H5::DataSpace space(1,dims);
> H5::DataSet dset = file.createDataSet("pow2",H5::PredType::NATIVE_INT,space);
> dset.write(data, H5::PredType::NATIVE_INT);
>
> // Get HDF5 image file buffer.
> file.flush(H5F_SCOPE_GLOBAL);
> hid_t fileid = file.getId();
> size_t size = H5Fget_file_image(fileid, NULL, 0);
> char* buffer = new char[size];
> H5Fget_file_image(fileid, buffer, size);
>
> // Write buffer to standard output.
> fwrite(buffer,1,size,stdout);
> }
>
> Compile and run:
>
> g++ -o write_h5_stdout write_h5_stdout.cxx -lhdf5_hl -lhdf5 -lhdf5_cpp
> ./write_h5_stdout > f.hdf5
>
> A example of reading this f.hdf5 file from the standard input is:
>
> #include <stdio.h>
> #include <stdlib.h>
> #include "hdf5.h"
> #include "H5Cpp.h"
> #include "H5LTpublic.h"
>
> #define NDATA 5
> #define READ_SIZE 512
> #define CHUNCK_SIZE 8192
>
> // stdin -> buffer -> HDF5 image file -> C array
>
> main()
> {
> // size of buffer is buffer_size=CHUNCK_SIZE*nchunck, size of h5data is
> // h5data_size <= buffer_size
> if (CHUNCK_SIZE < READ_SIZE) {
> fprintf(stderr, "ERROR: CHUNCK_SIZE < READ_SIZE\n");
> exit(EXIT_FAILURE);
> }
>
> // Read stdin into buffer.
> size_t nchunck = 1, n;
> size_t buffer_size = CHUNCK_SIZE*nchunck;
> char* buffer = (char*) malloc(buffer_size);
> size_t h5data_size = 0;
> while (1) {
> // Make sure buffer is big enough to receive READ_SIZE new bytes.
> if (h5data_size+READ_SIZE > buffer_size) {
> nchunck += 1;
> buffer_size = nchunck*CHUNCK_SIZE;
> buffer = (char*) realloc(buffer,buffer_size);
> }
> // copy from stdin to buffer.
> n = fread(buffer+h5data_size,1,READ_SIZE,stdin);
> h5data_size += n;
>
> if (n <= 0) break;
> }
>
> // Create HDF5 image file from buffer.
> unsigned flags = H5LT_FILE_IMAGE_DONT_COPY | H5LT_FILE_IMAGE_DONT_RELEASE;
> hid_t fileid = H5LTopen_file_image(buffer, h5data_size, flags);
> H5::H5File *h5file;
> h5file = new H5::H5File;
> h5file->setId(fileid);
>
> // Read C array in HDF5 file image.
> int data[NDATA];
> H5::DataSet dset = h5file->openDataSet("pow2");
> dset.read(data,dset.getDataType());
>
> for (int i=0 ; i<NDATA ; i++) {
> printf("%d\n",data[i]);
> }
>
> free(buffer);
>
> return 0;
> }
>
> Compile and run:
>
> g++ -o read_h5_stdin read_h5_stdin.cxx -lhdf5_hl -lhdf5 -lhdf5_cpp
> cat f.hdf5 | ./read_h5_stdin
> ./write_h5_stdout | ./read_h5_stdin
>
> Building ITK with recent HDF5 version
> -------------------------------------
>
> ITK must be compiled and and linked with a HDF5 > 1.8.9 (the current ITK 4.4.0
> embeds HDF5==1.8.7).
>
> itkHDF5unixpipeImageIO module
> -----------------------------
>
> In order to read/write HDF5 in standard input/output, we have created a
> itkHDF5unixpipeImageIO module. To this purpose, we have copied the directory
> Modules/IO/HDF5 into Modules/External/HDF5unixpipe (a bad practice that should
> be fixed as it duplicates a lot of code).
>
> src/itkHDF5unixpipeImageIO.cxx is the same as src/itkHDF5ImageIO.cxx, except
> that:
>
> - CanWriteFile and CanReadFile test on file extensions .h5stdout and .h5stdin,
> CanReadFile no more test on file existance.
>
> - An helper function ReadStandardInput is added:
>
> void
> HDF5unixpipeImageIO
> ::ReadStandardInput()
> {
> // Do not call the function twice.
> if (this->m_StandardInputReaden) return;
>
> /* Size of m_H5FileImageBuffer is H5FileImageBufferSize = chunckSize*nChunck.
> * Size of actual data is m_H5FileImageDataSize <= H5FileImageBufferSize.
> */
>
> size_t readSize = 512;
> size_t chunckSize = 8192;
> size_t nChunck = 1, nBytesReaden;
> size_t H5FileImageBufferSize = chunckSize*nChunck;
>
> this->m_H5FileImageBuffer = (char*) malloc(H5FileImageBufferSize);
> this->m_H5FileImageDataSize = 0.;
>
> while (1) {
> /* make sure m_H5FileImageBuffer is big enough to receive readSize new bytes */
> if (this->m_H5FileImageDataSize+readSize > H5FileImageBufferSize) {
> nChunck += 1;
> H5FileImageBufferSize = nChunck*chunckSize;
> this->m_H5FileImageBuffer = (char*) realloc(this->m_H5FileImageBuffer,H5FileImageBufferSize);
> }
>
> /* copy from stdin to m_H5FileImageBuffer */
> nBytesReaden = fread(this->m_H5FileImageBuffer+this->m_H5FileImageDataSize,1,readSize,stdin);
> this->m_H5FileImageDataSize += nBytesReaden;
>
> if (nBytesReaden <= 0) break;
> }
>
> this->m_StandardInputReaden = true;
> }
>
> In ReadImageInformation():
>
> this->m_H5File = new H5::H5File(this->GetFileName(),
> H5F_ACC_RDONLY);
>
> is replaced with:
>
> // Read stdin into m_H5FileImageBuffer.
> this->ReadStandardInput();
>
> // Create HDF5 image file from buffer.
> unsigned flags = H5LT_FILE_IMAGE_DONT_COPY | H5LT_FILE_IMAGE_DONT_RELEASE;
> hid_t fileid = H5LTopen_file_image(this->m_H5FileImageBuffer, this->m_H5FileImageDataSize, flags);
> this->m_H5File = new H5::H5File;
> this->m_H5File->setId(fileid);
>
> in WriteImageInformation(void):
>
> this->m_H5File = new H5::H5File(this->GetFileName(),
> H5F_ACC_TRUNC);
>
> is replaced with:
>
> hbool_t backing_store = 0;
> H5::FileAccPropList fapl;
> fapl.setCore((size_t)1,backing_store);
> this->m_H5File = new H5::H5File("nosuch.h5",H5F_ACC_TRUNC,
> H5::FileCreatPropList::DEFAULT,fapl);
>
> In Write(const void *buffer), at this end of the try block, we add:
>
> // Get HDF5 image file buffer.
> this->m_H5File->flush(H5F_SCOPE_GLOBAL);
> hid_t fileid = this->m_H5File->getId();
> size_t imageFileBufferSize = H5Fget_file_image(fileid, NULL, 0);
> char* imageFileBuffer = (char*) malloc(imageFileBufferSize);
> H5Fget_file_image(fileid, imageFileBuffer, imageFileBufferSize);
>
> // Write imageFileBuffer to standard output.
> fwrite(imageFileBuffer,1,imageFileBufferSize,stdout);
>
>
> ImageUnixPipeReader module
> --------------------------
>
> In the directory ./IO/ImageBase/include, we copied itkImageFileReader.h
> in itkImageUnixPipeReader.h and itkImageFileReader.hxx in
> itkImageUnixPipeReader.hxx (again, a bad practice that should
> be fixed as it duplicates a lot of code).
>
> The only change is to remove the use of the TestFileExistanceAndReadability
> function.
>
> Note that itkHDF5unixpipe can be used directly with ImageFileWritter.
>
> Using the new modules
> ---------------------
>
> The new modules to read/write in standard input/output can now be used as the
> ones that read in files, but by specifying "virtual" file names with extension
> ".h5stdin" and ".h5stdout", so that ImageUnixPipeReader and ImageFileWritter
> know they can make use of itkHDF5unixpipe, if we want to add support of other
> image formats.
>
> Missing features
> ----------------
>
> A module ImageUnixPipeWriter should be added. ImageUnixPipeWriter and
> ImageUnixPipeReader, should not test on file extension (there is no file name),
> but require an ImageIO to be explicity passed.
>
> Questions
> ---------
>
> 1. Is there any chance that this functionnalty could be integrated in ITK?
> (if not, I suppose we should put everything in Modules/external).
>
> 2. In either case, how to avoid code duplication? Can we use class inheritance?
>
> References
> ----------
>
> [1] (in french) http://inrimage.gforge.inria.fr/WWW/index.html
> [2] http://www.itk.org/Wiki/ITK/Git/Develop
> [3] http://www.itk.org/pipermail/insight-users/2004-September/010274.html
> [4] http://www.hdfgroup.org/HDF5/doc/Advanced/FileImageOperations/HDF5FileImageOperations.pdf
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Kitware offers ITK Training Courses, for more information visit:
> http://kitware.com/products/protraining.php
>
> Please keep messages on-topic and check the ITK FAQ at:
> http://www.itk.org/Wiki/ITK_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.itk.org/mailman/listinfo/insight-developers
--
Unpaid intern in BillsBasement at noware dot com
More information about the Insight-developers
mailing list