MantisBT - ITK
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0009320ITKpublic2009-07-27 09:432010-10-21 12:31
ReporterBradley Lowekamp 
Assigned ToBradley Lowekamp 
PrioritynormalSeverityminorReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version 
Resolution Date
Sprint
Sprint Status
Summary0009320: MetaImageIO itkLargeImageWriterReadTests fails on apple, due to 2GB IO limitation
DescriptionOn osx with even 64-bits builds, there is a limit to ~2GB for file reads and writes at once. Reads and writes in MetaIO should use a method which reads in smaller chunks. In the additional information section I have included code which could be used.

    The itkLargeImageWriteReadTests were added to test to see if itk::Image can deal with image greater then 2GB or 4GB. However, they fail on Mac OSX despite it being 64-bits with a 8-byte long. The apple OS is limited to only being able to read or write a maximum of 2GB at a time (regardless of 32 or 64-bit build). I don't know if this is the only OS with this limitation. The MetaIO library writes the files in one large block and therefore runs into this limitation. However the NRRD file format appears to read and write in smaller blocks, so by modifying the test to use NRRD it passes on my system. An alternative is to use streamed reading and writing (actually I am not sure if that would work to get around this for writing, as I think the ImageFileWriter writes the whole image if it is available even if it was requested to stream).
Steps To Reproduce
Additional Information
bool ReadBufferAsBinary( std::istream& is, void *buffer, StreamingImageIOBase::SizeType num )
{
  
  // some systems have a limit of 2GB to be read at once
  const SizeType maxChunk = 1024*1024*1024;
  
  std::streamsize bufferOffset = 0;
  std::streamsize bytesRemaining = static_cast<std::streamsize>( num );
  
  while (bytesRemaining)
    {
    
    SizeType bytesToRead = bytesRemaining > maxChunk ? maxChunk : bytesRemaining;
    
    itkDebugMacro(<< "Reading " << bytesToRead << " of " << bytesRemaining << " bytes for " << m_FileName);
    
    is.read( static_cast<char *>( buffer ) + bufferOffset, bytesToRead );
    
    if ( (is.gcount() != bytesToRead) || is.fail() )
      {
      return false;
      }
    bufferOffset += bytesToRead;
    bytesRemaining -= bytesToRead;
    }
  
  return true;
}


bool WriteBufferAsBinary( std::ostream& os, const void *buffer, StreamingImageIOBase::SizeType num )
{
  // some systems have a limit of 2GB to be read at once
  const SizeType maxChunk = 1024*1024*1024;

  std::streamsize bufferOffset = 0;
  std::streamsize bytesRemaining = num;
  while (bytesRemaining) {
    
    SizeType bytesToWrite = bytesRemaining > maxChunk ? maxChunk : bytesRemaining;

    itkDebugMacro(<< "Writing " << bytesToWrite << " of " << bytesRemaining << " bytes for " << m_FileName);
    
    os.write(static_cast<const char*>(buffer)+bufferOffset , bytesToWrite);
    if ( os.fail() )
      {
      return false;
      }

    bufferOffset += bytesToWrite;
    bytesRemaining -= bytesToWrite;
  }
 
  return true;
}
TagsNo tags attached.
Relationships
Attached Filespatch bug9320.patch (10,534) 2009-11-12 11:37
https://public.kitware.com/Bug/file/2653/bug9320.patch
patch bug9320-2.patch (690) 2009-11-16 11:24
https://public.kitware.com/Bug/file/2658/bug9320-2.patch
Issue History
Date ModifiedUsernameFieldChange
2009-07-27 09:43Bradley LowekampNew Issue
2009-09-08 12:01Bradley LowekampNote Added: 0017330
2009-09-08 12:10Bradley LowekampNote Added: 0017332
2009-09-08 12:50Sean McBrideNote Added: 0017333
2009-09-08 14:00Bradley LowekampNote Added: 0017336
2009-09-08 14:34Bradley LowekampNote Added: 0017340
2009-09-11 18:28Sean McBrideNote Added: 0017396
2009-11-02 16:18Bradley LowekampStatusnew => assigned
2009-11-02 16:18Bradley LowekampAssigned To => Bradley Lowekamp
2009-11-12 10:57Bradley LowekampNote Added: 0018401
2009-11-12 11:37Bradley LowekampFile Added: bug9320.patch
2009-11-12 11:54Julien JomierNote Added: 0018402
2009-11-16 11:24Bradley LowekampFile Added: bug9320-2.patch
2009-11-16 11:26Bradley LowekampNote Added: 0018443
2009-11-16 11:29Julien JomierNote Added: 0018444
2009-12-07 13:55Bradley LowekampNote Added: 0018713
2009-12-07 13:55Bradley LowekampStatusassigned => resolved
2009-12-07 13:55Bradley LowekampResolutionopen => fixed
2010-10-21 12:31Gabe HartStatusresolved => closed

Notes
(0017330)
Bradley Lowekamp   
2009-09-08 12:01   
I have used the following test program for determining if the OS is limited:

#include <fstream>
#include <iostream>


int main(void) {
 std::streamsize large = std::streamsize(4)*1024*1024*1024+1;
 char *foo = new char[large];

 std::cout << "ptr size: " << sizeof(void*) << " streamsize:" << sizeof(std::streamsize) << " " << large << std::endl;

 std::ofstream file("foo.dat", std::ofstream::binary);

 file.write((char *)foo, large);

 std::cout << "io good: " << file.good() << std::endl;

 file.close();

 return 0;
};
(0017332)
Bradley Lowekamp   
2009-09-08 12:10   
On Apple OSX, the 2GB IO limitation is limited by 32-bit kernels.With Snow Leopard, a 64-bit kernel may be available for some systems.

 I am unsure if this need to be resolved in the MetaIO library, since it's a system limitation...
(0017333)
Sean McBride   
2009-09-08 12:50   
I'm pretty sure that the limitation you speak of is at the STL level. On 10.5, the STL is not fully 64 bit clean. No doubt on 10.6 improvements have been made in that regard. I doubt changing between the 32 and 64 kernel would affect this.
(0017336)
Bradley Lowekamp   
2009-09-08 14:00   
I have run code above in the same executable with run with a 32-bit kernel:

ptr size: 8 streamsize:8 4294967297
io good: 0

-rw-r--r-- 1 blowek1 wheel 0 Sep 8 12:00 foo.dat

and a 64-bit:

ptr size: 8 streamsize:8 4294967297
io good: 1

-rw-r--r-- 1 blowek1 wheel 4294967297 Sep 8 11:58 foo.dat

The executable was build with Xcode 3.2 on OS 10.6 with the following command:

g++ test4gbio2.C -arch x86_64 -o test4gbio
(0017340)
Bradley Lowekamp   
2009-09-08 14:34   
Extending the test to read, caused the input file to not be "good":


int main(void)
{
  const char* filename = "foo.dat";

 std::streamsize large = std::streamsize(4)*1024*1024*1024+1;
 char *foo = new char[large];

 std::cout << "ptr size: " << sizeof(void*) << " streamsize:" << sizeof(std::streamsize) << " " << large << std::endl;

 std::ofstream file(filename, std::ofstream::binary);
 
 std::cout << "writing file" << std::endl;
 file.write((char *)foo, large);
 std::cout << "io good: " << file.good() << std::endl;

 delete [] foo;

 file.close();

 foo = new char[large];
 

 std::ifstream ifile(filename, std::ofstream::binary);

 std::cout << "reading file" << std::endl;
 ifile.read((char *)foo, large);

 std::cout << "io good: " << ifile.good() << std::endl;
 std::cout << "io gcount: " << ifile.gcount() << std::endl;

 ifile.close();

 return 0;
};
(0017396)
Sean McBride   
2009-09-11 18:28   
It seems to me this should work both with the 32 and 64 bit kernels. You should file a bug with Apple at

https://bugreport.apple.com [^]
(0018401)
Bradley Lowekamp   
2009-11-12 10:57   
Attached patch for MetaIO to use loops a 1GB read.

Submitted experimental build:
http://www.cdash.org/CDash/buildSummary.php?buildid=470359 [^]
(0018402)
Julien Jomier   
2009-11-12 11:54   
Patch submitted.
(0018443)
Bradley Lowekamp   
2009-11-16 11:26   
The original patch contained a bug which caused the following error to be incorrectly and excessively reported:
MetaImage: M_ReadElementsData: data not read completely

I have attached at patch for the patch to correct this issue.
(0018444)
Julien Jomier   
2009-11-16 11:29   
patch has been applied. Thanks!
(0018713)
Bradley Lowekamp   
2009-12-07 13:55   
That patch appears to have fixed the problem.