[Insight-developers] read stdin, write stdout
David Froger
david.froger at inria.fr
Tue Oct 1 08:33:51 EDT 2013
Thanks Bill! I'll use it as a model.
----- Mail original -----
> De: "Bill Lorensen" <bill.lorensen at gmail.com>
> À: "David Froger" <david.froger at inria.fr>
> Cc: "Insight Developers" <insight-developers at itk.org>
> Envoyé: Mardi 1 Octobre 2013 13:44:41
> Objet: Re: [Insight-developers] read stdin, write stdout
>
> We did a memory read/write imageio for Slicer4. The code is here:
> http://viewvc.slicer.org/viewvc.cgi/Slicer4/trunk/Libs/MRML/IDImageIO/
> A similar approach might be used to read/write pipes.
>
> Bill
>
>
> On Tue, Oct 1, 2013 at 1:43 AM, David Froger <david.froger at inria.fr>
> wrote:
> > Dear all,
> >
> > We are looking for a proper way to integrate in ITK reading/writing
> > in standard input/output HDF5 images.
> >
> > We write a "proove concept" that works, but involves a lot of code
> > duplication that should be avoided. The rest of this long mail is
> > divided
> > into these sections:
> > - Motivations
> > - HDF5 and unix pipe
> > - Building ITK with recent HDF5 version
> > - itkHDF5unixpipeImageIO module
> > - ImageUnixPipeReader module
> > - Using the new modules
> > - Questions
> > - References
> >
> > We can create topic branch as described in the Git workflow [2] if
> > need.
> >
> > Thanks for reading!
> >
> > Best regards,
> > David Froger
> >
> >
> > Motivations
> > -----------
> >
> > We are rewriting command line tools based on the INRimage [1] image
> > format/library so that they are now based ITK.
> >
> > A particularity of these command line tools is that they can be
> > piped with unix
> > pipe. For example, the 'so' command subtracts the pixel values of
> > two images,
> > while the 'ical' command computes min/max/average pixel values of a
> > image. To
> > test if two images are equals, we pipe the commands as follow:
> > so image1.inr image2.inr | ical
> >
> > While we are aware of the ITK pipeline and that we can use it to
> > pipe ITK
> > sources/mappers/filters into one processus (as suggested in [3] for
> > example),
> > we want to keep the possibility of using unix pipe in order to keep
> > a backward
> > compatibilty because we use numerous bash and perl scripts.
> >
> > HDF5 and unix pipe
> > ------------------
> >
> > The new file image operations library [4], introduced in HDF5 at
> > the 1.8.9 version
> > on May 2012 makes easy to read/write HDF5 images on standard
> > input/output.
> >
> > An example of writting an HDF5 image to standard output is:
> >
> > #include <stdio.h>
> > #include "hdf5.h"
> > #include "H5Cpp.h"
> > #include "H5LTpublic.h"
> >
> > // C array -> HDF5 image file -> buffer -> stdout
> >
> > #define NDATA 5
> >
> > int main()
> > {
> > // Create C array.
> > int data[NDATA] = {2, 4, 8, 16, 32};
> >
> > // Open HDF5 image file.
> > hbool_t backing_store = 0;
> > H5::FileAccPropList fapl;
> > fapl.setCore((size_t)1,backing_store);
> > H5::H5File
> > file("nosuch.h5",H5F_ACC_TRUNC,H5::FileCreatPropList::DEFAULT,fapl);
> >
> > // Write C array in HDF5 image file.
> > hsize_t dims[1]= {NDATA};
> > H5::DataSpace space(1,dims);
> > H5::DataSet dset =
> > file.createDataSet("pow2",H5::PredType::NATIVE_INT,space);
> > dset.write(data, H5::PredType::NATIVE_INT);
> >
> > // Get HDF5 image file buffer.
> > file.flush(H5F_SCOPE_GLOBAL);
> > hid_t fileid = file.getId();
> > size_t size = H5Fget_file_image(fileid, NULL, 0);
> > char* buffer = new char[size];
> > H5Fget_file_image(fileid, buffer, size);
> >
> > // Write buffer to standard output.
> > fwrite(buffer,1,size,stdout);
> > }
> >
> > Compile and run:
> >
> > g++ -o write_h5_stdout write_h5_stdout.cxx -lhdf5_hl -lhdf5
> > -lhdf5_cpp
> > ./write_h5_stdout > f.hdf5
> >
> > A example of reading this f.hdf5 file from the standard input is:
> >
> > #include <stdio.h>
> > #include <stdlib.h>
> > #include "hdf5.h"
> > #include "H5Cpp.h"
> > #include "H5LTpublic.h"
> >
> > #define NDATA 5
> > #define READ_SIZE 512
> > #define CHUNCK_SIZE 8192
> >
> > // stdin -> buffer -> HDF5 image file -> C array
> >
> > main()
> > {
> > // size of buffer is buffer_size=CHUNCK_SIZE*nchunck, size
> > of h5data is
> > // h5data_size <= buffer_size
> > if (CHUNCK_SIZE < READ_SIZE) {
> > fprintf(stderr, "ERROR: CHUNCK_SIZE < READ_SIZE\n");
> > exit(EXIT_FAILURE);
> > }
> >
> > // Read stdin into buffer.
> > size_t nchunck = 1, n;
> > size_t buffer_size = CHUNCK_SIZE*nchunck;
> > char* buffer = (char*) malloc(buffer_size);
> > size_t h5data_size = 0;
> > while (1) {
> > // Make sure buffer is big enough to receive READ_SIZE
> > new bytes.
> > if (h5data_size+READ_SIZE > buffer_size) {
> > nchunck += 1;
> > buffer_size = nchunck*CHUNCK_SIZE;
> > buffer = (char*) realloc(buffer,buffer_size);
> > }
> > // copy from stdin to buffer.
> > n = fread(buffer+h5data_size,1,READ_SIZE,stdin);
> > h5data_size += n;
> >
> > if (n <= 0) break;
> > }
> >
> > // Create HDF5 image file from buffer.
> > unsigned flags = H5LT_FILE_IMAGE_DONT_COPY |
> > H5LT_FILE_IMAGE_DONT_RELEASE;
> > hid_t fileid = H5LTopen_file_image(buffer, h5data_size,
> > flags);
> > H5::H5File *h5file;
> > h5file = new H5::H5File;
> > h5file->setId(fileid);
> >
> > // Read C array in HDF5 file image.
> > int data[NDATA];
> > H5::DataSet dset = h5file->openDataSet("pow2");
> > dset.read(data,dset.getDataType());
> >
> > for (int i=0 ; i<NDATA ; i++) {
> > printf("%d\n",data[i]);
> > }
> >
> > free(buffer);
> >
> > return 0;
> > }
> >
> > Compile and run:
> >
> > g++ -o read_h5_stdin read_h5_stdin.cxx -lhdf5_hl -lhdf5
> > -lhdf5_cpp
> > cat f.hdf5 | ./read_h5_stdin
> > ./write_h5_stdout | ./read_h5_stdin
> >
> > Building ITK with recent HDF5 version
> > -------------------------------------
> >
> > ITK must be compiled and and linked with a HDF5 > 1.8.9 (the
> > current ITK 4.4.0
> > embeds HDF5==1.8.7).
> >
> > itkHDF5unixpipeImageIO module
> > -----------------------------
> >
> > In order to read/write HDF5 in standard input/output, we have
> > created a
> > itkHDF5unixpipeImageIO module. To this purpose, we have copied the
> > directory
> > Modules/IO/HDF5 into Modules/External/HDF5unixpipe (a bad practice
> > that should
> > be fixed as it duplicates a lot of code).
> >
> > src/itkHDF5unixpipeImageIO.cxx is the same as
> > src/itkHDF5ImageIO.cxx, except
> > that:
> >
> > - CanWriteFile and CanReadFile test on file extensions .h5stdout
> > and .h5stdin,
> > CanReadFile no more test on file existance.
> >
> > - An helper function ReadStandardInput is added:
> >
> > void
> > HDF5unixpipeImageIO
> > ::ReadStandardInput()
> > {
> > // Do not call the function twice.
> > if (this->m_StandardInputReaden) return;
> >
> > /* Size of m_H5FileImageBuffer is H5FileImageBufferSize =
> > chunckSize*nChunck.
> > * Size of actual data is m_H5FileImageDataSize <=
> > H5FileImageBufferSize.
> > */
> >
> > size_t readSize = 512;
> > size_t chunckSize = 8192;
> > size_t nChunck = 1, nBytesReaden;
> > size_t H5FileImageBufferSize = chunckSize*nChunck;
> >
> > this->m_H5FileImageBuffer = (char*)
> > malloc(H5FileImageBufferSize);
> > this->m_H5FileImageDataSize = 0.;
> >
> > while (1) {
> > /* make sure m_H5FileImageBuffer is big enough to
> > receive readSize new bytes */
> > if (this->m_H5FileImageDataSize+readSize >
> > H5FileImageBufferSize) {
> > nChunck += 1;
> > H5FileImageBufferSize = nChunck*chunckSize;
> > this->m_H5FileImageBuffer = (char*)
> > realloc(this->m_H5FileImageBuffer,H5FileImageBufferSize);
> > }
> >
> > /* copy from stdin to m_H5FileImageBuffer */
> > nBytesReaden =
> > fread(this->m_H5FileImageBuffer+this->m_H5FileImageDataSize,1,readSize,stdin);
> > this->m_H5FileImageDataSize += nBytesReaden;
> >
> > if (nBytesReaden <= 0) break;
> > }
> >
> > this->m_StandardInputReaden = true;
> > }
> >
> > In ReadImageInformation():
> >
> > this->m_H5File = new H5::H5File(this->GetFileName(),
> > H5F_ACC_RDONLY);
> >
> > is replaced with:
> >
> > // Read stdin into m_H5FileImageBuffer.
> > this->ReadStandardInput();
> >
> > // Create HDF5 image file from buffer.
> > unsigned flags = H5LT_FILE_IMAGE_DONT_COPY |
> > H5LT_FILE_IMAGE_DONT_RELEASE;
> > hid_t fileid = H5LTopen_file_image(this->m_H5FileImageBuffer,
> > this->m_H5FileImageDataSize, flags);
> > this->m_H5File = new H5::H5File;
> > this->m_H5File->setId(fileid);
> >
> > in WriteImageInformation(void):
> >
> > this->m_H5File = new H5::H5File(this->GetFileName(),
> > H5F_ACC_TRUNC);
> >
> > is replaced with:
> >
> > hbool_t backing_store = 0;
> > H5::FileAccPropList fapl;
> > fapl.setCore((size_t)1,backing_store);
> > this->m_H5File = new H5::H5File("nosuch.h5",H5F_ACC_TRUNC,
> > H5::FileCreatPropList::DEFAULT,fapl);
> >
> > In Write(const void *buffer), at this end of the try block, we add:
> >
> > // Get HDF5 image file buffer.
> > this->m_H5File->flush(H5F_SCOPE_GLOBAL);
> > hid_t fileid = this->m_H5File->getId();
> > size_t imageFileBufferSize = H5Fget_file_image(fileid, NULL,
> > 0);
> > char* imageFileBuffer = (char*) malloc(imageFileBufferSize);
> > H5Fget_file_image(fileid, imageFileBuffer,
> > imageFileBufferSize);
> >
> > // Write imageFileBuffer to standard output.
> > fwrite(imageFileBuffer,1,imageFileBufferSize,stdout);
> >
> >
> > ImageUnixPipeReader module
> > --------------------------
> >
> > In the directory ./IO/ImageBase/include, we copied
> > itkImageFileReader.h
> > in itkImageUnixPipeReader.h and itkImageFileReader.hxx in
> > itkImageUnixPipeReader.hxx (again, a bad practice that should
> > be fixed as it duplicates a lot of code).
> >
> > The only change is to remove the use of the
> > TestFileExistanceAndReadability
> > function.
> >
> > Note that itkHDF5unixpipe can be used directly with
> > ImageFileWritter.
> >
> > Using the new modules
> > ---------------------
> >
> > The new modules to read/write in standard input/output can now be
> > used as the
> > ones that read in files, but by specifying "virtual" file names
> > with extension
> > ".h5stdin" and ".h5stdout", so that ImageUnixPipeReader and
> > ImageFileWritter
> > know they can make use of itkHDF5unixpipe, if we want to add
> > support of other
> > image formats.
> >
> > Missing features
> > ----------------
> >
> > A module ImageUnixPipeWriter should be added. ImageUnixPipeWriter
> > and
> > ImageUnixPipeReader, should not test on file extension (there is no
> > file name),
> > but require an ImageIO to be explicity passed.
> >
> > Questions
> > ---------
> >
> > 1. Is there any chance that this functionnalty could be integrated
> > in ITK?
> > (if not, I suppose we should put everything in Modules/external).
> >
> > 2. In either case, how to avoid code duplication? Can we use class
> > inheritance?
> >
> > References
> > ----------
> >
> > [1] (in french) http://inrimage.gforge.inria.fr/WWW/index.html
> > [2] http://www.itk.org/Wiki/ITK/Git/Develop
> > [3]
> > http://www.itk.org/pipermail/insight-users/2004-September/010274.html
> > [4]
> > http://www.hdfgroup.org/HDF5/doc/Advanced/FileImageOperations/HDF5FileImageOperations.pdf
> > _______________________________________________
> > Powered by www.kitware.com
> >
> > Visit other Kitware open-source projects at
> > http://www.kitware.com/opensource/opensource.html
> >
> > Kitware offers ITK Training Courses, for more information visit:
> > http://kitware.com/products/protraining.php
> >
> > Please keep messages on-topic and check the ITK FAQ at:
> > http://www.itk.org/Wiki/ITK_FAQ
> >
> > Follow this link to subscribe/unsubscribe:
> > http://www.itk.org/mailman/listinfo/insight-developers
>
>
>
> --
> Unpaid intern in BillsBasement at noware dot com
>
More information about the Insight-developers
mailing list