[Insight-developers] read stdin, write stdout
David Froger
david.froger at inria.fr
Tue Oct 1 01:43:39 EDT 2013
Dear all,
We are looking for a proper way to integrate in ITK reading/writing
in standard input/output HDF5 images.
We write a "proove concept" that works, but involves a lot of code
duplication that should be avoided. The rest of this long mail is divided
into these sections:
- Motivations
- HDF5 and unix pipe
- Building ITK with recent HDF5 version
- itkHDF5unixpipeImageIO module
- ImageUnixPipeReader module
- Using the new modules
- Questions
- References
We can create topic branch as described in the Git workflow [2] if need.
Thanks for reading!
Best regards,
David Froger
Motivations
-----------
We are rewriting command line tools based on the INRimage [1] image
format/library so that they are now based ITK.
A particularity of these command line tools is that they can be piped with unix
pipe. For example, the 'so' command subtracts the pixel values of two images,
while the 'ical' command computes min/max/average pixel values of a image. To
test if two images are equals, we pipe the commands as follow:
so image1.inr image2.inr | ical
While we are aware of the ITK pipeline and that we can use it to pipe ITK
sources/mappers/filters into one processus (as suggested in [3] for example),
we want to keep the possibility of using unix pipe in order to keep a backward
compatibilty because we use numerous bash and perl scripts.
HDF5 and unix pipe
------------------
The new file image operations library [4], introduced in HDF5 at the 1.8.9 version
on May 2012 makes easy to read/write HDF5 images on standard input/output.
An example of writting an HDF5 image to standard output is:
#include <stdio.h>
#include "hdf5.h"
#include "H5Cpp.h"
#include "H5LTpublic.h"
// C array -> HDF5 image file -> buffer -> stdout
#define NDATA 5
int main()
{
// Create C array.
int data[NDATA] = {2, 4, 8, 16, 32};
// Open HDF5 image file.
hbool_t backing_store = 0;
H5::FileAccPropList fapl;
fapl.setCore((size_t)1,backing_store);
H5::H5File file("nosuch.h5",H5F_ACC_TRUNC,H5::FileCreatPropList::DEFAULT,fapl);
// Write C array in HDF5 image file.
hsize_t dims[1]= {NDATA};
H5::DataSpace space(1,dims);
H5::DataSet dset = file.createDataSet("pow2",H5::PredType::NATIVE_INT,space);
dset.write(data, H5::PredType::NATIVE_INT);
// Get HDF5 image file buffer.
file.flush(H5F_SCOPE_GLOBAL);
hid_t fileid = file.getId();
size_t size = H5Fget_file_image(fileid, NULL, 0);
char* buffer = new char[size];
H5Fget_file_image(fileid, buffer, size);
// Write buffer to standard output.
fwrite(buffer,1,size,stdout);
}
Compile and run:
g++ -o write_h5_stdout write_h5_stdout.cxx -lhdf5_hl -lhdf5 -lhdf5_cpp
./write_h5_stdout > f.hdf5
A example of reading this f.hdf5 file from the standard input is:
#include <stdio.h>
#include <stdlib.h>
#include "hdf5.h"
#include "H5Cpp.h"
#include "H5LTpublic.h"
#define NDATA 5
#define READ_SIZE 512
#define CHUNCK_SIZE 8192
// stdin -> buffer -> HDF5 image file -> C array
main()
{
// size of buffer is buffer_size=CHUNCK_SIZE*nchunck, size of h5data is
// h5data_size <= buffer_size
if (CHUNCK_SIZE < READ_SIZE) {
fprintf(stderr, "ERROR: CHUNCK_SIZE < READ_SIZE\n");
exit(EXIT_FAILURE);
}
// Read stdin into buffer.
size_t nchunck = 1, n;
size_t buffer_size = CHUNCK_SIZE*nchunck;
char* buffer = (char*) malloc(buffer_size);
size_t h5data_size = 0;
while (1) {
// Make sure buffer is big enough to receive READ_SIZE new bytes.
if (h5data_size+READ_SIZE > buffer_size) {
nchunck += 1;
buffer_size = nchunck*CHUNCK_SIZE;
buffer = (char*) realloc(buffer,buffer_size);
}
// copy from stdin to buffer.
n = fread(buffer+h5data_size,1,READ_SIZE,stdin);
h5data_size += n;
if (n <= 0) break;
}
// Create HDF5 image file from buffer.
unsigned flags = H5LT_FILE_IMAGE_DONT_COPY | H5LT_FILE_IMAGE_DONT_RELEASE;
hid_t fileid = H5LTopen_file_image(buffer, h5data_size, flags);
H5::H5File *h5file;
h5file = new H5::H5File;
h5file->setId(fileid);
// Read C array in HDF5 file image.
int data[NDATA];
H5::DataSet dset = h5file->openDataSet("pow2");
dset.read(data,dset.getDataType());
for (int i=0 ; i<NDATA ; i++) {
printf("%d\n",data[i]);
}
free(buffer);
return 0;
}
Compile and run:
g++ -o read_h5_stdin read_h5_stdin.cxx -lhdf5_hl -lhdf5 -lhdf5_cpp
cat f.hdf5 | ./read_h5_stdin
./write_h5_stdout | ./read_h5_stdin
Building ITK with recent HDF5 version
-------------------------------------
ITK must be compiled and and linked with a HDF5 > 1.8.9 (the current ITK 4.4.0
embeds HDF5==1.8.7).
itkHDF5unixpipeImageIO module
-----------------------------
In order to read/write HDF5 in standard input/output, we have created a
itkHDF5unixpipeImageIO module. To this purpose, we have copied the directory
Modules/IO/HDF5 into Modules/External/HDF5unixpipe (a bad practice that should
be fixed as it duplicates a lot of code).
src/itkHDF5unixpipeImageIO.cxx is the same as src/itkHDF5ImageIO.cxx, except
that:
- CanWriteFile and CanReadFile test on file extensions .h5stdout and .h5stdin,
CanReadFile no more test on file existance.
- An helper function ReadStandardInput is added:
void
HDF5unixpipeImageIO
::ReadStandardInput()
{
// Do not call the function twice.
if (this->m_StandardInputReaden) return;
/* Size of m_H5FileImageBuffer is H5FileImageBufferSize = chunckSize*nChunck.
* Size of actual data is m_H5FileImageDataSize <= H5FileImageBufferSize.
*/
size_t readSize = 512;
size_t chunckSize = 8192;
size_t nChunck = 1, nBytesReaden;
size_t H5FileImageBufferSize = chunckSize*nChunck;
this->m_H5FileImageBuffer = (char*) malloc(H5FileImageBufferSize);
this->m_H5FileImageDataSize = 0.;
while (1) {
/* make sure m_H5FileImageBuffer is big enough to receive readSize new bytes */
if (this->m_H5FileImageDataSize+readSize > H5FileImageBufferSize) {
nChunck += 1;
H5FileImageBufferSize = nChunck*chunckSize;
this->m_H5FileImageBuffer = (char*) realloc(this->m_H5FileImageBuffer,H5FileImageBufferSize);
}
/* copy from stdin to m_H5FileImageBuffer */
nBytesReaden = fread(this->m_H5FileImageBuffer+this->m_H5FileImageDataSize,1,readSize,stdin);
this->m_H5FileImageDataSize += nBytesReaden;
if (nBytesReaden <= 0) break;
}
this->m_StandardInputReaden = true;
}
In ReadImageInformation():
this->m_H5File = new H5::H5File(this->GetFileName(),
H5F_ACC_RDONLY);
is replaced with:
// Read stdin into m_H5FileImageBuffer.
this->ReadStandardInput();
// Create HDF5 image file from buffer.
unsigned flags = H5LT_FILE_IMAGE_DONT_COPY | H5LT_FILE_IMAGE_DONT_RELEASE;
hid_t fileid = H5LTopen_file_image(this->m_H5FileImageBuffer, this->m_H5FileImageDataSize, flags);
this->m_H5File = new H5::H5File;
this->m_H5File->setId(fileid);
in WriteImageInformation(void):
this->m_H5File = new H5::H5File(this->GetFileName(),
H5F_ACC_TRUNC);
is replaced with:
hbool_t backing_store = 0;
H5::FileAccPropList fapl;
fapl.setCore((size_t)1,backing_store);
this->m_H5File = new H5::H5File("nosuch.h5",H5F_ACC_TRUNC,
H5::FileCreatPropList::DEFAULT,fapl);
In Write(const void *buffer), at this end of the try block, we add:
// Get HDF5 image file buffer.
this->m_H5File->flush(H5F_SCOPE_GLOBAL);
hid_t fileid = this->m_H5File->getId();
size_t imageFileBufferSize = H5Fget_file_image(fileid, NULL, 0);
char* imageFileBuffer = (char*) malloc(imageFileBufferSize);
H5Fget_file_image(fileid, imageFileBuffer, imageFileBufferSize);
// Write imageFileBuffer to standard output.
fwrite(imageFileBuffer,1,imageFileBufferSize,stdout);
ImageUnixPipeReader module
--------------------------
In the directory ./IO/ImageBase/include, we copied itkImageFileReader.h
in itkImageUnixPipeReader.h and itkImageFileReader.hxx in
itkImageUnixPipeReader.hxx (again, a bad practice that should
be fixed as it duplicates a lot of code).
The only change is to remove the use of the TestFileExistanceAndReadability
function.
Note that itkHDF5unixpipe can be used directly with ImageFileWritter.
Using the new modules
---------------------
The new modules to read/write in standard input/output can now be used as the
ones that read in files, but by specifying "virtual" file names with extension
".h5stdin" and ".h5stdout", so that ImageUnixPipeReader and ImageFileWritter
know they can make use of itkHDF5unixpipe, if we want to add support of other
image formats.
Missing features
----------------
A module ImageUnixPipeWriter should be added. ImageUnixPipeWriter and
ImageUnixPipeReader, should not test on file extension (there is no file name),
but require an ImageIO to be explicity passed.
Questions
---------
1. Is there any chance that this functionnalty could be integrated in ITK?
(if not, I suppose we should put everything in Modules/external).
2. In either case, how to avoid code duplication? Can we use class inheritance?
References
----------
[1] (in french) http://inrimage.gforge.inria.fr/WWW/index.html
[2] http://www.itk.org/Wiki/ITK/Git/Develop
[3] http://www.itk.org/pipermail/insight-users/2004-September/010274.html
[4] http://www.hdfgroup.org/HDF5/doc/Advanced/FileImageOperations/HDF5FileImageOperations.pdf
More information about the Insight-developers
mailing list