[Insight-users] 2GB VTK IO limit

[Bradley Lowekamp]

Hello,
	
	Specifically the io bug I have found can be illustrated in the  
following with an apple x86_64:

int main(void) {
   std::streamsize large = std::streamsize(2)*1024*1024*1024;
   char *foo = new char[large];
   std::cout << "ptr size: " << sizeof(void*) << " streamsize:" <<  
sizeof(std::streamsize) << " " << large << std::endl;
   std::ofstream file("foo.dat", std::ofstream::binary);
   file.write((char *)foo, large);
   std::cout << "io good: " << file.good() << std::endl;
   file.close();
   return 0;
};


with the following output:

ptr size: 8 streamsize:8 2147483648
Filesize limit exceeded

An exception is thrown and size for foo.dat is 0.

I am not too worried about reading a 2GB file when I only have 4 GB  
memory, since you really can't do much with it after it's loaded.
I do have to wonder why "unsigned long" was used below. I guess there  
were not enough SizeType defined in ITK. In this particular case we  
are not dealing with a file offset or anything so a memory size type  
is appropriate.

On Oct 9, 2008, at 6:44 PM, Sean McBride wrote:

> On 10/10/08 12:31 AM, Mathieu Malaterre said:
>
>> Looking at the code I would think this is coming from piece such as:
>>
>> itkVTKImageIO.cxx:
>> ...
>> ByteSwapper<short>::SwapRangeFromSystemToBigEndian((short *)buffer,
>> static_cast<unsigned long>(this->GetImageSizeInComponents()) );
>> ...
>>
>> What is the size of an 'unsigned long' on macosx +gcc ? Anyway if VTK
>> support larger file, this piece of code (and other cast) should be
>> fixed.
>
> The Mac OS ABI specifies that 'unsigned long' is 32bit when building  
> as
> 32bit, and is 64bit when building as 64bit.  It is one of the (few)
> basic types the changes size between ABIs.
>
> In any case, 'unsigned long' is indeed the wrong type to use for  
> things
> like file length.  POSIX file APIs tend to use types like off_t.  On  
> OS
> X, off_t is 64bit on both the 32bit and 64bit ABIs.
>
> Writing portable 32/64 bit-safe code is surprisingly non-trivial.  ITK
> and VTK are riddled with such errors, as a cursory inspection reveals.
>
> --
> ____________________________________________________________
> Sean McBride, B. Eng                 sean at rogue-research.com
> Rogue Research                        www.rogue-research.com
> Mac Software Developer              Montréal, Québec, Canada
>
>