View Issue Details [ Jump to Notes ] [ Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0009320ITKpublic2009-07-27 09:432010-10-21 12:31
ReporterBradley Lowekamp 
Assigned ToBradley Lowekamp 
PrioritynormalSeverityminorReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version 
Summary0009320: MetaImageIO itkLargeImageWriterReadTests fails on apple, due to 2GB IO limitation
DescriptionOn osx with even 64-bits builds, there is a limit to ~2GB for file reads and writes at once. Reads and writes in MetaIO should use a method which reads in smaller chunks. In the additional information section I have included code which could be used.

    The itkLargeImageWriteReadTests were added to test to see if itk::Image can deal with image greater then 2GB or 4GB. However, they fail on Mac OSX despite it being 64-bits with a 8-byte long. The apple OS is limited to only being able to read or write a maximum of 2GB at a time (regardless of 32 or 64-bit build). I don't know if this is the only OS with this limitation. The MetaIO library writes the files in one large block and therefore runs into this limitation. However the NRRD file format appears to read and write in smaller blocks, so by modifying the test to use NRRD it passes on my system. An alternative is to use streamed reading and writing (actually I am not sure if that would work to get around this for writing, as I think the ImageFileWriter writes the whole image if it is available even if it was requested to stream).
Additional Information
bool ReadBufferAsBinary( std::istream& is, void *buffer, StreamingImageIOBase::SizeType num )
{
  
  // some systems have a limit of 2GB to be read at once
  const SizeType maxChunk = 1024*1024*1024;
  
  std::streamsize bufferOffset = 0;
  std::streamsize bytesRemaining = static_cast<std::streamsize>( num );
  
  while (bytesRemaining)
    {
    
    SizeType bytesToRead = bytesRemaining > maxChunk ? maxChunk : bytesRemaining;
    
    itkDebugMacro(<< "Reading " << bytesToRead << " of " << bytesRemaining << " bytes for " << m_FileName);
    
    is.read( static_cast<char *>( buffer ) + bufferOffset, bytesToRead );
    
    if ( (is.gcount() != bytesToRead) || is.fail() )
      {
      return false;
      }
    bufferOffset += bytesToRead;
    bytesRemaining -= bytesToRead;
    }
  
  return true;
}


bool WriteBufferAsBinary( std::ostream& os, const void *buffer, StreamingImageIOBase::SizeType num )
{
  // some systems have a limit of 2GB to be read at once
  const SizeType maxChunk = 1024*1024*1024;

  std::streamsize bufferOffset = 0;
  std::streamsize bytesRemaining = num;
  while (bytesRemaining) {
    
    SizeType bytesToWrite = bytesRemaining > maxChunk ? maxChunk : bytesRemaining;

    itkDebugMacro(<< "Writing " << bytesToWrite << " of " << bytesRemaining << " bytes for " << m_FileName);
    
    os.write(static_cast<const char*>(buffer)+bufferOffset , bytesToWrite);
    if ( os.fail() )
      {
      return false;
      }

    bufferOffset += bytesToWrite;
    bytesRemaining -= bytesToWrite;
  }
 
  return true;
}
TagsNo tags attached.
Resolution Date
Sprint
Sprint Status
Attached Filespatch file icon bug9320.patch [^] (10,534 bytes) 2009-11-12 11:37 [Show Content]
patch file icon bug9320-2.patch [^] (690 bytes) 2009-11-16 11:24 [Show Content]

 Relationships

  Notes
(0017330)
Bradley Lowekamp (developer)
2009-09-08 12:01

 I have used the following test program for determining if the OS is limited:

#include <fstream>
#include <iostream>


int main(void) {
 std::streamsize large = std::streamsize(4)*1024*1024*1024+1;
 char *foo = new char[large];

 std::cout << "ptr size: " << sizeof(void*) << " streamsize:" << sizeof(std::streamsize) << " " << large << std::endl;

 std::ofstream file("foo.dat", std::ofstream::binary);

 file.write((char *)foo, large);

 std::cout << "io good: " << file.good() << std::endl;

 file.close();

 return 0;
};
(0017332)
Bradley Lowekamp (developer)
2009-09-08 12:10

 On Apple OSX, the 2GB IO limitation is limited by 32-bit kernels.With Snow Leopard, a 64-bit kernel may be available for some systems.

 I am unsure if this need to be resolved in the MetaIO library, since it's a system limitation...
(0017333)
Sean McBride (developer)
2009-09-08 12:50

 I'm pretty sure that the limitation you speak of is at the STL level. On 10.5, the STL is not fully 64 bit clean. No doubt on 10.6 improvements have been made in that regard. I doubt changing between the 32 and 64 kernel would affect this.
(0017336)
Bradley Lowekamp (developer)
2009-09-08 14:00

 I have run code above in the same executable with run with a 32-bit kernel:

ptr size: 8 streamsize:8 4294967297
io good: 0

-rw-r--r-- 1 blowek1 wheel 0 Sep 8 12:00 foo.dat

and a 64-bit:

ptr size: 8 streamsize:8 4294967297
io good: 1

-rw-r--r-- 1 blowek1 wheel 4294967297 Sep 8 11:58 foo.dat

The executable was build with Xcode 3.2 on OS 10.6 with the following command:

g++ test4gbio2.C -arch x86_64 -o test4gbio
(0017340)
Bradley Lowekamp (developer)
2009-09-08 14:34

 Extending the test to read, caused the input file to not be "good":


int main(void)
{
  const char* filename = "foo.dat";

 std::streamsize large = std::streamsize(4)*1024*1024*1024+1;
 char *foo = new char[large];

 std::cout << "ptr size: " << sizeof(void*) << " streamsize:" << sizeof(std::streamsize) << " " << large << std::endl;

 std::ofstream file(filename, std::ofstream::binary);
 
 std::cout << "writing file" << std::endl;
 file.write((char *)foo, large);
 std::cout << "io good: " << file.good() << std::endl;

 delete [] foo;

 file.close();

 foo = new char[large];
 

 std::ifstream ifile(filename, std::ofstream::binary);

 std::cout << "reading file" << std::endl;
 ifile.read((char *)foo, large);

 std::cout << "io good: " << ifile.good() << std::endl;
 std::cout << "io gcount: " << ifile.gcount() << std::endl;

 ifile.close();

 return 0;
};
(0017396)
Sean McBride (developer)
2009-09-11 18:28

 It seems to me this should work both with the 32 and 64 bit kernels. You should file a bug with Apple at

https://bugreport.apple.com [^]
(0018401)
Bradley Lowekamp (developer)
2009-11-12 10:57

 Attached patch for MetaIO to use loops a 1GB read.

Submitted experimental build:
http://www.cdash.org/CDash/buildSummary.php?buildid=470359 [^]
(0018402)
Julien Jomier (developer)
2009-11-12 11:54

 Patch submitted.
(0018443)
Bradley Lowekamp (developer)
2009-11-16 11:26

 The original patch contained a bug which caused the following error to be incorrectly and excessively reported:
MetaImage: M_ReadElementsData: data not read completely

I have attached at patch for the patch to correct this issue.
(0018444)
Julien Jomier (developer)
2009-11-16 11:29

 patch has been applied. Thanks!
(0018713)
Bradley Lowekamp (developer)
2009-12-07 13:55

 That patch appears to have fixed the problem.

 Issue History
Date Modified Username Field Change
2009-07-27 09:43 Bradley Lowekamp New Issue
2009-09-08 12:01 Bradley Lowekamp Note Added: 0017330
2009-09-08 12:10 Bradley Lowekamp Note Added: 0017332
2009-09-08 12:50 Sean McBride Note Added: 0017333
2009-09-08 14:00 Bradley Lowekamp Note Added: 0017336
2009-09-08 14:34 Bradley Lowekamp Note Added: 0017340
2009-09-11 18:28 Sean McBride Note Added: 0017396
2009-11-02 16:18 Bradley Lowekamp Status new => assigned
2009-11-02 16:18 Bradley Lowekamp Assigned To => Bradley Lowekamp
2009-11-12 10:57 Bradley Lowekamp Note Added: 0018401
2009-11-12 11:37 Bradley Lowekamp File Added: bug9320.patch
2009-11-12 11:54 Julien Jomier Note Added: 0018402
2009-11-16 11:24 Bradley Lowekamp File Added: bug9320-2.patch
2009-11-16 11:26 Bradley Lowekamp Note Added: 0018443
2009-11-16 11:29 Julien Jomier Note Added: 0018444
2009-12-07 13:55 Bradley Lowekamp Note Added: 0018713
2009-12-07 13:55 Bradley Lowekamp Status assigned => resolved
2009-12-07 13:55 Bradley Lowekamp Resolution open => fixed
2010-10-21 12:31 Gabe Hart Status resolved => closed

Copyright © 2000 - 2018 MantisBT Team