[vtkusers] Problem in loading big images using vtkimagereader
Michael Jackson
mike.jackson at bluequartz.net
Wed Aug 11 10:52:35 EDT 2010
Simply compiling with VTK_USE_64BIT_IDS set to "ON" will not solve the
problem which takes out 1. There shouldn't have to be a "special"
definition for Windows which takes out 3.
The issue is code like the following which is found in the VTK/IO/
vtkImageReader2.cxx at line 557.
void vtkImageReader2::ComputeDataIncrements()
{
int idx;
unsigned long fileDataLength;
.....
In that function alone if your file is larger than 4GB you are going
to have problems on windows.
and again at line 671:
void vtkImageReader2::SeekFile(int i, int j, int k)
{
unsigned long streamStart;
The longest stream you are going to be able to read is 4GB. The actual
code needs to be written in such a way as to have "64 bit" file access
even when compiling in 32 bit mode. This would cause all sorts of API
changes I am sure and even more "defensive" programming techniques
such that you really can not load up a 6 GB file on a 32 bit system
since the largest array one could have is the largest value of size_t
on that platform.
Like I said, the most draconian fix is to just outlaw the use of
"long". Use the standard vtktypes as they are defined in Common/
vtkType.h. vtkTypeUInt64, vtkTypeInt64 and those types.
___________________________________________________________
Mike Jackson www.bluequartz.net
On Aug 11, 2010, at 10:15 AM, Arunachalam Kana wrote:
> Hi VtkUsers,
>
> First i was thinking this was a problem faced by me and Mike in the
> past. But now i see more response for this problem. If there is more
> response for this problem, which means a group exist which has 64
> bit m/c and windows xp and want to work with large images. i think
> something has to be done, so that future programmers does not face
> the same problem and waste few days in finding the bug.
>
> I understand that it takes some effort to make this change and to
> change the respective files. But as we already have some solution,
> we can think of more possible solutions.
>
> Possible solutions:
> 1. John: Building vtk with 64bit.
> 2. Mike: changing from "long" to vtkIdType.
> 3. Kana: vtkIdtype defined speciall for windows system and can
> switch between 32 bit and 64 bit. Which can be selected during cmake.
>
> If there is any other vtk user interested to participate, please
> post your opinions.
>
> Kana
>
>
> -----Original Message-----
> From: vtkusers-bounces at vtk.org [mailto:vtkusers-bounces at vtk.org] On
> Behalf Of Michael Jackson
> Sent: 11 August 2010 15:31
> To: VTK Users
> Subject: Re: [vtkusers] Problem in loading big images using
> vtkimagereader
>
> http://public.kitware.com/pipermail/paraview/2008-March/007422.html
>
> Is when I came across the issue. I did not keep track of what the
> eventual "fixes" were but somewhere (probably a private email) I think
> it was said that a fix was difficult because the API for a lot of
> filters would need to be changed.
> In the most draconian fix the use of "long" should just be banned,
> but this would cause all sorts of backwards compatibility issues.
> ___________________________________________________________
> Mike Jackson www.bluequartz.net
>
>
> On Aug 11, 2010, at 8:57 AM, Jim Peterson wrote:
>
>> All,
>> Would this be a situation where the enthusiastic risk taking
>> programmer might copy some subset of vtk modules to the vtklocal
>> directory to make local modification for 64 bit windows builds? in
>> this case converting the 32bit integer file size fields to 64bit for
>> the affected file access modules in the windows builds.
>>
>> Does anyone have a starting list of affected modules to begin such a
>> task with?
>>
>> Thanks in advance.
>> Jim
>>
>> Michael Jackson wrote:
>>> Because vtkimagereader2 uses the "long" type for calculations in the
>>> data extents and other places. On windows "long" evaluates to a 32
>>> bit
>>> integer no matter how you compile it. On Linux and OS X which use
>>> the
>>> LP64 model "long" will evaluate to a 64 bit integer. The added range
>>> of the 64 bit integer allows the larger images to be loaded.
>>> For now you will have to find another way to load your images into
>>> vtk until a fix is committed to vtk.
>>> These types of problems are not new to vtk as I have reported a few
>>> in the past.
>>>
>>> -----
>>> Mike Jackson www.bluequartz.net
>>> Principal Software Engineer mike.jackson at bluequartz.net
>>> BlueQuartz Software Dayton, Ohio
>>>
>>> On Aug 11, 2010, at 5:34, "Arunachalam Kana"
>>> <Kana.Arunachalam at fh-wels.at> wrote:
>>>
>>>
>>>> Hi Mike,
>>>>
>>>> I have tested the same code in linux system and it works. I am
>>>> able to load dataset >3Gb.
>>>> I think your theory is right. Can you please explain why this
>>>> happens and how can i solve this problem to load large image data
>>>> set in windows system.
>>>>
>>>> Kana
>>>>
>>>> -----Original Message-----
>>>> From: vtkusers-bounces at vtk.org [mailto:vtkusers-bounces at vtk.org]
>>>> On Behalf Of Arunachalam Kana
>>>> Sent: 11 August 2010 10:00
>>>> To: vtkusers at vtk.org
>>>> Subject: Re: [vtkusers] Problem in loading big images using
>>>> vtkimagereader
>>>>
>>>> Hi Mike,
>>>>
>>>> I have not tried the code on linux OS, but going to try it now. I
>>>> would like to know your theory, which makes you think that
>>>> vtkimagereader2.cxx will work on linux 64 bit machine and not in
>>>> windows xp 64 bit machine. Which part of the code in
>>>> vtkimagereader2.cxx makes you to think this ?
>>>>
>>>> I would like to point out that, in our algorithms I was using
>>>> vtkmetaimagereader.cxx to read files. And I had no problem in
>>>> loading image >4Gb size in a windows xp 64 bit m/c. Is there a
>>>> difference between the vtkmetaimagereader.cxx and
>>>> vtkimagereader2.cxx ?
>>>>
>>>> Kana
>>>>
>>>> -----Original Message-----
>>>> From: vtkusers-bounces at vtk.org [mailto:vtkusers-bounces at vtk.org]
>>>> On Behalf Of Michael Jackson
>>>> Sent: 10 August 2010 19:42
>>>> To: VTK Users
>>>> Subject: Re: [vtkusers] Problem in loading big images using
>>>> vtkimagereader
>>>>
>>>> Are you able to run your code/data on a 64 bit Linux/OSX box to
>>>> see if
>>>> it works there? I have a theory but it may be a bad theory based
>>>> off a
>>>> very casual look at vtkImageReader2.cxx file
>>>>
>>>> Mike Jackson
>>>>
>>>> On Aug 10, 2010, at 1:27 PM, Arunachalam Kana wrote:
>>>>
>>>>
>>>>> Hi Michael,
>>>>>
>>>>> My system is a 64bit with windows xp. I used cmake to generate
>>>>> visual studio project for
>>>>> Vtk. I generated the projects using option "visual studio 9 2008
>>>>> Win64". So i think the vtk libraries are also built for 64 bit
>>>>> machine.
>>>>>
>>>>> Kana.
>>>>>
>>>>> -----Original Message-----
>>>>> From: vtkusers-bounces at vtk.org [mailto:vtkusers-bounces at vtk.org]
>>>>> On
>>>>> Behalf Of Michael Jackson
>>>>> Sent: 10 August 2010 18:42
>>>>> To: VTK Users
>>>>> Subject: Re: [vtkusers] Problem in loading big images using
>>>>> vtkimagereader
>>>>>
>>>>> Are you compiling VTK as a 64 bit set of libraries? Possibly
>>>>> sounds
>>>>> like there is a 32/64 bit threshold being passed.
>>>>>
>>>>> ___________________________________________________________
>>>>> Mike Jackson www.bluequartz.net
>>>>> Principal Software Engineer mike.jackson at bluequartz.net
>>>>> BlueQuartz Software Dayton, Ohio
>>>>>
>>>>> On Aug 10, 2010, at 11:51 AM, Arunachalam Kana wrote:
>>>>>
>>>>>
>>>>>> Hi VtkUsers,
>>>>>>
>>>>>> I have problem in the vtkimagereader2. Following below is the
>>>>>> code
>>>>>> used to read image.
>>>>>>
>>>>>> vtkImageReader2 * reader = vtkImageReader2::New();
>>>>>> reader->SetFileName( file.toAscii().data() );
>>>>>> reader->SetDataScalarType( scalarType );
>>>>>> reader->SetHeaderSize( headersize );
>>>>>> reader->SetDataExtent( extent );
>>>>>> reader->SetDataSpacing( spacing );
>>>>>> reader->SetDataOrigin( origin );
>>>>>> reader->SetFileDimensionality( dim );
>>>>>> reader->UpdateWholeExtent();
>>>>>> reader->Update();
>>>>>>
>>>>>> The code works well when I read an image <= 3Gb(approx). But it
>>>>>> does
>>>>>> not read the full
>>>>>> Image if the image is >= 3Gb. The image data set i handle is >
>>>>>> 3Gb.
>>>>>>
>>>>>> I would like to have some suggestion to use vtkimagereader and
>>>>>> solve this problem.
>>>>>>
>>>>>> Thank you,
>>>>>> Regards,
>>>>>> Kana Arunachalam Kannappan
>>>>>> Research Associate
>>>>>> FH OÖ Forschungs & Entwicklungs GmbH
>>>>>> Stelzhamer Strasse 23,
>>>>>> 4600 Wels,
>>>>>> Austria.
>>>>>> Phone: +43 (0)7242 72811 -4420
>>>>>> kana.arunachalam at fh-wels.at
>>>>>> www.fh-ooe.at; www.3dct.at
>>>>>>
>>>>>> _______________________________________________
>>>>>> Powered by www.kitware.com
>>>>>>
>>>>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>>>>>
>>>>>> Please keep messages on-topic and check the VTK FAQ at: http://www.vtk.org/Wiki/VTK_FAQ
>>>>>>
>>>>>> Follow this link to subscribe/unsubscribe:
>>>>>> http://www.vtk.org/mailman/listinfo/vtkusers
>>>>>>
>>>>> _______________________________________________
>>>>> Powered by www.kitware.com
>>>>>
>>>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>>>>
>>>>> Please keep messages on-topic and check the VTK FAQ at: http://www.vtk.org/Wiki/VTK_FAQ
>>>>>
>>>>> Follow this link to subscribe/unsubscribe:
>>>>> http://www.vtk.org/mailman/listinfo/vtkusers
>>>>>
>>>> _______________________________________________
>>>> Powered by www.kitware.com
>>>>
>>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>>>
>>>> Please keep messages on-topic and check the VTK FAQ at: http://www.vtk.org/Wiki/VTK_FAQ
>>>>
>>>> Follow this link to subscribe/unsubscribe:
>>>> http://www.vtk.org/mailman/listinfo/vtkusers
>>>> _______________________________________________
>>>> Powered by www.kitware.com
>>>>
>>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>>>
>>>> Please keep messages on-topic and check the VTK FAQ at: http://www.vtk.org/Wiki/VTK_FAQ
>>>>
>>>> Follow this link to subscribe/unsubscribe:
>>>> http://www.vtk.org/mailman/listinfo/vtkusers
>>>>
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>>
>>> Please keep messages on-topic and check the VTK FAQ at: http://www.vtk.org/Wiki/VTK_FAQ
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://www.vtk.org/mailman/listinfo/vtkusers
>>>
>>>
>>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the VTK FAQ at: http://www.vtk.org/Wiki/VTK_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.vtk.org/mailman/listinfo/vtkusers
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the VTK FAQ at: http://www.vtk.org/Wiki/VTK_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.vtk.org/mailman/listinfo/vtkusers
More information about the vtkusers
mailing list