[Insight-developers] read stdin, write stdout
Williams, Norman K
norman-k-williams at uiowa.edu
Tue Oct 1 10:18:17 EDT 2013
This is pretty interesting. It raises some questions:
1. How big is the advantage relative to simply reading std::cin into a
temporary file and the using conventional ImageIO? That doubles the IO
and requires temp file storage, but it requires less code to implement.
2. If you're going to do this, is there a way to leverage HDF5 streaming?
Holding an arbitrarily large HDF5 image in memory for writing means you
have 2 copies of a large data blob allocated.
I haven't looked at Slicer's memory I/O mechanism.
--
Kent Williams norman-k-williams at uiowa.edu
On 10/1/13 7:33 AM, "David Froger" <david.froger at inria.fr> wrote:
>Thanks Bill! I'll use it as a model.
>
>----- Mail original -----
>> De: "Bill Lorensen" <bill.lorensen at gmail.com>
>> À: "David Froger" <david.froger at inria.fr>
>> Cc: "Insight Developers" <insight-developers at itk.org>
>> Envoyé: Mardi 1 Octobre 2013 13:44:41
>> Objet: Re: [Insight-developers] read stdin, write stdout
>>
>> We did a memory read/write imageio for Slicer4. The code is here:
>> http://viewvc.slicer.org/viewvc.cgi/Slicer4/trunk/Libs/MRML/IDImageIO/
>> A similar approach might be used to read/write pipes.
>>
>> Bill
>>
>>
>> On Tue, Oct 1, 2013 at 1:43 AM, David Froger <david.froger at inria.fr>
>> wrote:
>> > Dear all,
>> >
>> > We are looking for a proper way to integrate in ITK reading/writing
>> > in standard input/output HDF5 images.
>> >
>> > We write a "proove concept" that works, but involves a lot of code
>> > duplication that should be avoided. The rest of this long mail is
>> > divided
>> > into these sections:
>> > - Motivations
>> > - HDF5 and unix pipe
>> > - Building ITK with recent HDF5 version
>> > - itkHDF5unixpipeImageIO module
>> > - ImageUnixPipeReader module
>> > - Using the new modules
>> > - Questions
>> > - References
>> >
>> > We can create topic branch as described in the Git workflow [2] if
>> > need.
>> >
>> > Thanks for reading!
>> >
>> > Best regards,
>> > David Froger
>> >
>> >
>> > Motivations
>> > -----------
>> >
>> > We are rewriting command line tools based on the INRimage [1] image
>> > format/library so that they are now based ITK.
>> >
>> > A particularity of these command line tools is that they can be
>> > piped with unix
>> > pipe. For example, the 'so' command subtracts the pixel values of
>> > two images,
>> > while the 'ical' command computes min/max/average pixel values of a
>> > image. To
>> > test if two images are equals, we pipe the commands as follow:
>> > so image1.inr image2.inr | ical
>> >
>> > While we are aware of the ITK pipeline and that we can use it to
>> > pipe ITK
>> > sources/mappers/filters into one processus (as suggested in [3] for
>> > example),
>> > we want to keep the possibility of using unix pipe in order to keep
>> > a backward
>> > compatibilty because we use numerous bash and perl scripts.
>> >
>> > HDF5 and unix pipe
>> > ------------------
>> >
>> > The new file image operations library [4], introduced in HDF5 at
>> > the 1.8.9 version
>> > on May 2012 makes easy to read/write HDF5 images on standard
>> > input/output.
>> >
>> > An example of writting an HDF5 image to standard output is:
>> >
>> > #include <stdio.h>
>> > #include "hdf5.h"
>> > #include "H5Cpp.h"
>> > #include "H5LTpublic.h"
>> >
>> > // C array -> HDF5 image file -> buffer -> stdout
>> >
>> > #define NDATA 5
>> >
>> > int main()
>> > {
>> > // Create C array.
>> > int data[NDATA] = {2, 4, 8, 16, 32};
>> >
>> > // Open HDF5 image file.
>> > hbool_t backing_store = 0;
>> > H5::FileAccPropList fapl;
>> > fapl.setCore((size_t)1,backing_store);
>> > H5::H5File
>> >
>>file("nosuch.h5",H5F_ACC_TRUNC,H5::FileCreatPropList::DEFAULT,fapl);
>> >
>> > // Write C array in HDF5 image file.
>> > hsize_t dims[1]= {NDATA};
>> > H5::DataSpace space(1,dims);
>> > H5::DataSet dset =
>> > file.createDataSet("pow2",H5::PredType::NATIVE_INT,space);
>> > dset.write(data, H5::PredType::NATIVE_INT);
>> >
>> > // Get HDF5 image file buffer.
>> > file.flush(H5F_SCOPE_GLOBAL);
>> > hid_t fileid = file.getId();
>> > size_t size = H5Fget_file_image(fileid, NULL, 0);
>> > char* buffer = new char[size];
>> > H5Fget_file_image(fileid, buffer, size);
>> >
>> > // Write buffer to standard output.
>> > fwrite(buffer,1,size,stdout);
>> > }
>> >
>> > Compile and run:
>> >
>> > g++ -o write_h5_stdout write_h5_stdout.cxx -lhdf5_hl -lhdf5
>> > -lhdf5_cpp
>> > ./write_h5_stdout > f.hdf5
>> >
>> > A example of reading this f.hdf5 file from the standard input is:
>> >
>> > #include <stdio.h>
>> > #include <stdlib.h>
>> > #include "hdf5.h"
>> > #include "H5Cpp.h"
>> > #include "H5LTpublic.h"
>> >
>> > #define NDATA 5
>> > #define READ_SIZE 512
>> > #define CHUNCK_SIZE 8192
>> >
>> > // stdin -> buffer -> HDF5 image file -> C array
>> >
>> > main()
>> > {
>> > // size of buffer is buffer_size=CHUNCK_SIZE*nchunck, size
>> > of h5data is
>> > // h5data_size <= buffer_size
>> > if (CHUNCK_SIZE < READ_SIZE) {
>> > fprintf(stderr, "ERROR: CHUNCK_SIZE < READ_SIZE\n");
>> > exit(EXIT_FAILURE);
>> > }
>> >
>> > // Read stdin into buffer.
>> > size_t nchunck = 1, n;
>> > size_t buffer_size = CHUNCK_SIZE*nchunck;
>> > char* buffer = (char*) malloc(buffer_size);
>> > size_t h5data_size = 0;
>> > while (1) {
>> > // Make sure buffer is big enough to receive READ_SIZE
>> > new bytes.
>> > if (h5data_size+READ_SIZE > buffer_size) {
>> > nchunck += 1;
>> > buffer_size = nchunck*CHUNCK_SIZE;
>> > buffer = (char*) realloc(buffer,buffer_size);
>> > }
>> > // copy from stdin to buffer.
>> > n = fread(buffer+h5data_size,1,READ_SIZE,stdin);
>> > h5data_size += n;
>> >
>> > if (n <= 0) break;
>> > }
>> >
>> > // Create HDF5 image file from buffer.
>> > unsigned flags = H5LT_FILE_IMAGE_DONT_COPY |
>> > H5LT_FILE_IMAGE_DONT_RELEASE;
>> > hid_t fileid = H5LTopen_file_image(buffer, h5data_size,
>> > flags);
>> > H5::H5File *h5file;
>> > h5file = new H5::H5File;
>> > h5file->setId(fileid);
>> >
>> > // Read C array in HDF5 file image.
>> > int data[NDATA];
>> > H5::DataSet dset = h5file->openDataSet("pow2");
>> > dset.read(data,dset.getDataType());
>> >
>> > for (int i=0 ; i<NDATA ; i++) {
>> > printf("%d\n",data[i]);
>> > }
>> >
>> > free(buffer);
>> >
>> > return 0;
>> > }
>> >
>> > Compile and run:
>> >
>> > g++ -o read_h5_stdin read_h5_stdin.cxx -lhdf5_hl -lhdf5
>> > -lhdf5_cpp
>> > cat f.hdf5 | ./read_h5_stdin
>> > ./write_h5_stdout | ./read_h5_stdin
>> >
>> > Building ITK with recent HDF5 version
>> > -------------------------------------
>> >
>> > ITK must be compiled and and linked with a HDF5 > 1.8.9 (the
>> > current ITK 4.4.0
>> > embeds HDF5==1.8.7).
>> >
>> > itkHDF5unixpipeImageIO module
>> > -----------------------------
>> >
>> > In order to read/write HDF5 in standard input/output, we have
>> > created a
>> > itkHDF5unixpipeImageIO module. To this purpose, we have copied the
>> > directory
>> > Modules/IO/HDF5 into Modules/External/HDF5unixpipe (a bad practice
>> > that should
>> > be fixed as it duplicates a lot of code).
>> >
>> > src/itkHDF5unixpipeImageIO.cxx is the same as
>> > src/itkHDF5ImageIO.cxx, except
>> > that:
>> >
>> > - CanWriteFile and CanReadFile test on file extensions .h5stdout
>> > and .h5stdin,
>> > CanReadFile no more test on file existance.
>> >
>> > - An helper function ReadStandardInput is added:
>> >
>> > void
>> > HDF5unixpipeImageIO
>> > ::ReadStandardInput()
>> > {
>> > // Do not call the function twice.
>> > if (this->m_StandardInputReaden) return;
>> >
>> > /* Size of m_H5FileImageBuffer is H5FileImageBufferSize =
>> > chunckSize*nChunck.
>> > * Size of actual data is m_H5FileImageDataSize <=
>> > H5FileImageBufferSize.
>> > */
>> >
>> > size_t readSize = 512;
>> > size_t chunckSize = 8192;
>> > size_t nChunck = 1, nBytesReaden;
>> > size_t H5FileImageBufferSize = chunckSize*nChunck;
>> >
>> > this->m_H5FileImageBuffer = (char*)
>> > malloc(H5FileImageBufferSize);
>> > this->m_H5FileImageDataSize = 0.;
>> >
>> > while (1) {
>> > /* make sure m_H5FileImageBuffer is big enough to
>> > receive readSize new bytes */
>> > if (this->m_H5FileImageDataSize+readSize >
>> > H5FileImageBufferSize) {
>> > nChunck += 1;
>> > H5FileImageBufferSize = nChunck*chunckSize;
>> > this->m_H5FileImageBuffer = (char*)
>> >
>>realloc(this->m_H5FileImageBuffer,H5FileImageBufferSize);
>> > }
>> >
>> > /* copy from stdin to m_H5FileImageBuffer */
>> > nBytesReaden =
>> >
>>fread(this->m_H5FileImageBuffer+this->m_H5FileImageDataSize,1,readSize,st
>>din);
>> > this->m_H5FileImageDataSize += nBytesReaden;
>> >
>> > if (nBytesReaden <= 0) break;
>> > }
>> >
>> > this->m_StandardInputReaden = true;
>> > }
>> >
>> > In ReadImageInformation():
>> >
>> > this->m_H5File = new H5::H5File(this->GetFileName(),
>> > H5F_ACC_RDONLY);
>> >
>> > is replaced with:
>> >
>> > // Read stdin into m_H5FileImageBuffer.
>> > this->ReadStandardInput();
>> >
>> > // Create HDF5 image file from buffer.
>> > unsigned flags = H5LT_FILE_IMAGE_DONT_COPY |
>> > H5LT_FILE_IMAGE_DONT_RELEASE;
>> > hid_t fileid = H5LTopen_file_image(this->m_H5FileImageBuffer,
>> > this->m_H5FileImageDataSize, flags);
>> > this->m_H5File = new H5::H5File;
>> > this->m_H5File->setId(fileid);
>> >
>> > in WriteImageInformation(void):
>> >
>> > this->m_H5File = new H5::H5File(this->GetFileName(),
>> > H5F_ACC_TRUNC);
>> >
>> > is replaced with:
>> >
>> > hbool_t backing_store = 0;
>> > H5::FileAccPropList fapl;
>> > fapl.setCore((size_t)1,backing_store);
>> > this->m_H5File = new H5::H5File("nosuch.h5",H5F_ACC_TRUNC,
>> > H5::FileCreatPropList::DEFAULT,fapl);
>> >
>> > In Write(const void *buffer), at this end of the try block, we add:
>> >
>> > // Get HDF5 image file buffer.
>> > this->m_H5File->flush(H5F_SCOPE_GLOBAL);
>> > hid_t fileid = this->m_H5File->getId();
>> > size_t imageFileBufferSize = H5Fget_file_image(fileid, NULL,
>> > 0);
>> > char* imageFileBuffer = (char*) malloc(imageFileBufferSize);
>> > H5Fget_file_image(fileid, imageFileBuffer,
>> > imageFileBufferSize);
>> >
>> > // Write imageFileBuffer to standard output.
>> > fwrite(imageFileBuffer,1,imageFileBufferSize,stdout);
>> >
>> >
>> > ImageUnixPipeReader module
>> > --------------------------
>> >
>> > In the directory ./IO/ImageBase/include, we copied
>> > itkImageFileReader.h
>> > in itkImageUnixPipeReader.h and itkImageFileReader.hxx in
>> > itkImageUnixPipeReader.hxx (again, a bad practice that should
>> > be fixed as it duplicates a lot of code).
>> >
>> > The only change is to remove the use of the
>> > TestFileExistanceAndReadability
>> > function.
>> >
>> > Note that itkHDF5unixpipe can be used directly with
>> > ImageFileWritter.
>> >
>> > Using the new modules
>> > ---------------------
>> >
>> > The new modules to read/write in standard input/output can now be
>> > used as the
>> > ones that read in files, but by specifying "virtual" file names
>> > with extension
>> > ".h5stdin" and ".h5stdout", so that ImageUnixPipeReader and
>> > ImageFileWritter
>> > know they can make use of itkHDF5unixpipe, if we want to add
>> > support of other
>> > image formats.
>> >
>> > Missing features
>> > ----------------
>> >
>> > A module ImageUnixPipeWriter should be added. ImageUnixPipeWriter
>> > and
>> > ImageUnixPipeReader, should not test on file extension (there is no
>> > file name),
>> > but require an ImageIO to be explicity passed.
>> >
>> > Questions
>> > ---------
>> >
>> > 1. Is there any chance that this functionnalty could be integrated
>> > in ITK?
>> > (if not, I suppose we should put everything in Modules/external).
>> >
>> > 2. In either case, how to avoid code duplication? Can we use class
>> > inheritance?
>> >
>> > References
>> > ----------
>> >
>> > [1] (in french) http://inrimage.gforge.inria.fr/WWW/index.html
>> > [2] http://www.itk.org/Wiki/ITK/Git/Develop
>> > [3]
>> > http://www.itk.org/pipermail/insight-users/2004-September/010274.html
>> > [4]
>> >
>>http://www.hdfgroup.org/HDF5/doc/Advanced/FileImageOperations/HDF5FileIma
>>geOperations.pdf
>> > _______________________________________________
>> > Powered by www.kitware.com
>> >
>> > Visit other Kitware open-source projects at
>> > http://www.kitware.com/opensource/opensource.html
>> >
>> > Kitware offers ITK Training Courses, for more information visit:
>> > http://kitware.com/products/protraining.php
>> >
>> > Please keep messages on-topic and check the ITK FAQ at:
>> > http://www.itk.org/Wiki/ITK_FAQ
>> >
>> > Follow this link to subscribe/unsubscribe:
>> > http://www.itk.org/mailman/listinfo/insight-developers
>>
>>
>>
>> --
>> Unpaid intern in BillsBasement at noware dot com
>>
>_______________________________________________
>Powered by www.kitware.com
>
>Visit other Kitware open-source projects at
>http://www.kitware.com/opensource/opensource.html
>
>Kitware offers ITK Training Courses, for more information visit:
>http://kitware.com/products/protraining.php
>
>Please keep messages on-topic and check the ITK FAQ at:
>http://www.itk.org/Wiki/ITK_FAQ
>
>Follow this link to subscribe/unsubscribe:
>http://www.itk.org/mailman/listinfo/insight-developers
________________________________
Notice: This UI Health Care e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential and may be legally privileged. If you are not the intended recipient, you are hereby notified that any retention, dissemination, distribution, or copying of this communication is strictly prohibited. Please reply to the sender that you have received the message in error, then delete it. Thank you.
________________________________
More information about the Insight-developers
mailing list