[Insight-developers] [Insight-users] Image offset is giving bad pointer for large datasets (7Gb)

Luis Ibanez luis.ibanez at kitware.com
Fri Jul 10 12:30:12 EDT 2009


Hi Kana,

Thanks a lot for looking into this and sharing your findings.

I just confirmed that in Windows 64bits, the "long" type is
only 4 bytes.

Here is the program I used:

#include <iostream>
#include "itkOffset.h"
#include "itkNumericTraits.h"

int main()
{
  unsigned long tt;
  std::cout << "size = " << sizeof(tt) << std::endl;
  tt  =  -1;
  std::cout << "tt = " << tt << std::endl;

  typedef itk::Offset<3>   OffsetType;
  typedef OffsetType::OffsetValueType   OffsetValueType;

  OffsetValueType  offsetValue;

  std::cout << "sizeof(offsetValue) = " << sizeof( offsetValue ) <<
std::endl;

  offsetValue = itk::NumericTraits< OffsetValueType >::max();

  std::cout << "OffsetValueType max() = " << offsetValue << std::endl;

  return EXIT_SUCCESS;
}


with this CMakeLists.txt file


CMAKE_MINIMUM_REQUIRED(VERSION 2.4)
IF(COMMAND CMAKE_POLICY)
  CMAKE_POLICY(SET CMP0003 NEW)
ENDIF(COMMAND CMAKE_POLICY)


PROJECT(64bitsTest)

FIND_PACKAGE(ITK REQUIRED)
INCLUDE(${ITK_USE_FILE})

ADD_EXECUTABLE(typesTest typesTest.cxx )

TARGET_LINK_LIBRARIES(typesTest ITKCommon)



---------


So, it seems that we have to introduce a new ITK type,
that will be the longest int available in the platform.


We currently have the following declarations in the file:

           Insight/Code/Common/itkIntTypes.h


  /** Convenient and more descriptive integer types. */
  typedef char      ITK_INT8;
  typedef int       ITK_INT32;

#ifndef _WIN32
  typedef long long   ITK_INT64;
#endif

#ifdef _WIN32
  typedef long      ITK_INT64;
#endif

  typedef unsigned char   ITK_UINT8;
  typedef unsigned short  ITK_UINT16;
  typedef unsigned        ITK_UINT32;

#ifndef _WIN32
  typedef unsigned long long  ITK_UINT64;
#endif

#ifdef _WIN32
  typedef unsigned long ITK_UINT64;
#endif

  typedef int       ITK_INTPTR;
  typedef unsigned  ITK_UINTPTR;

#ifdef __cplusplus
}
#endif



One option that comes to mind is that we
should simply use


                        size_t
 or                    std::size_t

Which is the type expected by the "mem"
methods, {memcpy,memdup...}, and the
allocation "new" method.

I have verified that "size_t" will have 8 bits
on Windows 64.

 std::cout << "sizeof( size_t ) = " << sizeof( size_t ) << std::endl;



Unless there are any objections, I'll suggest
that we replace the type of all integer variables
related to image offsets and image size, with
the type "size_t".


BTW: note that there is also "size_type"

It seems that "size_type" will be the type used by STL
containers.


   Any comments ?


       Thanks


             Luis


------------------------------------------------------------------------------------------------------------
On Thu, Jul 9, 2009 at 2:53 AM, Arunachalam Kana <
Kana.Arunachalam at fh-wels.at> wrote:

> Hi Luis,
>
> Thank you for your response. I ran the test program and the given the
> results are given below along with detailed
> System information and what option i used to compile itk.
>
> System Information
> ------------------
> Time of this report: 7/8/2009, 17:16:29
>       Machine name: CT-DELL
>   Operating System: Windows XP Professional x64 Edition (5.2, Build
> 3790) Service Pack 2 (3790.srv03_sp2_gdr.090319-1204)
>           Language: English (Regional Setting: German)
> System Manufacturer: Dell Inc.
>       System Model: Precision WorkStation T7400
>               BIOS: Default System BIOS
>          Processor: Intel(R) Pentium(R) III Xeon-Prozessor (8 CPUs),
> ~3.2GHz
>             Memory: 65534MB RAM
>          Page File: 717MB used, 65267MB available
>        Windows Dir: C:\WINDOWS
>    DirectX Version: DirectX 9.0c (4.09.0000.0904)
> DX Setup Parameters: Not found
>     DxDiag Version: 5.03.3790.3959 32bit Unicode
>
>
> Itk compilation
> ---------------------
> Itk compiled using Microsoft visual studio 2009 x64 (option).
>
> Microsoft visual studio 2008
> ---------------------
> Configuration manager details:
> Active Solution Configuration: Debug
> Active Solution Platform: x64
>
> Test run details:
> 1. Program:
>     unsigned long tt;
>    std::cout << "size = " << sizeof(tt) << std::endl;
>    tt  =  -1;
>    std::cout << "tt = " << tt << std::endl;
>     output: size = 4; tt = 4294967295
>
> 2. Program:
>    unsigned long long tt;
>     std::cout << "size = " << sizeof(tt) << std::endl;
>    tt  =  -1;
>    std::cout << "tt = " << tt << std::endl;
>     output: size = 8; tt = 18446744073709551615
>
> I changed the configuration details:
> --------------------------------------
> Active Solution Configuration: Debug
> Active Solution Platform: Win32
>
> Test run details:
> 1. Program:
>     unsigned long tt;
>    std::cout << "size = " << sizeof(tt) << std::endl;
>    tt  =  -1;
>    std::cout << "tt = " << tt << std::endl;
>     output: size = 4; tt = 4294967295
>
> 2. Program:
>    unsigned long long tt;
>     std::cout << "size = " << sizeof(tt) << std::endl;
>    tt  =  -1;
>    std::cout << "tt = " << tt << std::endl;
>     output: size = 8; tt = 18446744073709551615
>
> For both Win32 and x64 i get the same result.
> Unsigned long is 4 byte and unsigned long long is 8 byte.
>
> I was a little confused after the result, so i searched for data type
> ranges in msdn. Link below:
>
> http://msdn.microsoft.com/en-us/library/s3f49ktz.aspx
>
> From the information given in above link, i understand that for 32 bit
> and 64 bit compiler unsigned long is 4 byte and unsigned long long is 8
> byte.
>
> I checked in the wiki for 64 bit machines (link below)
>
> http://en.wikipedia.org/wiki/64-bit
>
> In the link above, under topic "scientific data model": It is given that
> Microsoft win64 uses the LLP data model. From the above links, I
> understand the following:
>
> 1.      A 64bit machine with Microsoft Windows x64 Operating system has
> 4 byte for long and 8 byte for long long.
> 2.      Microsoft visual studio 2009 has long as 4 byte and long long as
> 8 bytes for both win32 and x64 compiler option.
>
> What's next?
>
> Thank you,
> Regards,
> Kana
>
> -----Original Message-----
> From: Luis Ibanez [mailto:luis.ibanez at kitware.com]
> Sent: 08 July 2009 16:10
> To: Arunachalam Kana
> Cc: insight-users at itk.org
> Subject: Re: [Insight-users] Image offset is giving bad pointer for
> large datasets (7Gb)
>
>
>
> Hi Arunachalam,
>
>
>          Thanks for your detailed post.
>
>
>          Let's discuss first ITK by itself,
>
>
> (before we go into the integration with VTK and VTKEdge)
>
>
>
> As you pointed out, the type
>
>               OffsetValueType
>
> is currently defined as "long" in ITK.
>
>
>
> However, your assumptiong that "long" types in 64 bits
> platforms, are storing values up to 2^32 is incorrect.
>
>
> An unsigned long type in a 64bits platforms will have
> 8 bytes, and therefore can store values up to 2^64.
> which is:
>
>             18446744073709551615 L
>
>
> You can verify this with the following code:
>
>
> #include <iostream>
> int main()
> {
>    unsigned long tt;
>    std::cout << "size = " << sizeof(tt) << std::endl;
>    tt  =  -1;
>    std::cout << "tt = " << tt << std::endl;
> }
>
>
> The output of this program, in a 64bits machine will be:
>
>
>          size = 8
>          tt = 18446744073709551615
>
>
> The same code, when compiled in a 32-bits machine
> will produce as output:
>
>
>          size = 4
>          tt = 4294967295
>
>
> Therefore, if your OffsetValueType is getting saturated,
> it is likely that you are *not* compiling for 64 bits.
>
>
> Are you sure that when you configured ITK with CMake,
> you selected the "64 bits" version of your Visual Studio
> compiler.
>
>
> A simple way to double-check is to compile the code
> above and run it.
>
>
>
>    Please do so, and let us konw what you find.
>
>
>       Thanks
>
>
>
>          Luis
>
>
>
> ----------------------------------
> Arunachalam Kana wrote:
> > Hi,
> >
> >
> >
> > Machine details: Windows XP, visual studio 2008,  64 bit platform,
> 64Gb Ram
> >
> > Image size: 1442 x 1566 x 1657 ( approx 7Gb )
> >
> > Other libraries: Along with ITK, VTK and VTK Edge is also used. VTK
> Edge
> > is used for changing vtk image to itk image and vice versa.
> >
> >
> >
> > Problem:
> >
> > In ITK the OffsetValueType is long. Range of long datatype:
> > -2,147,483,648 to 2,147,483,647
> >
> > But the image m_OffsetTable values of type OffsetValueType are : 1;
> > 1442; 2258172;  3,741,791,004
> >
> > In the above, it is clearly seen that the image offset value exceeds
> the
> > OffsetValueType range.
> >
> >
> >
> > Correction attempts:
> >
> > 1.       I changed the OffsetValueType to unsigned long to increase
> the
> > range. Unsigned long range : 0 to 4,294,967,295. But
> >
> > this gives problems on conversion back to vtk. The vtkdataarray
> created
> > has a variable maxID which is of VtkIDType.
> >
> > VtkIDType is of long long. The maxID contains the value =
> -553,176,292.
> >
> >
> >
> > Correlation between  3,741,791,004 and -553,176,292.
> >
> > Decimal                                               Hexa
> >
> > 3,741,791,003                                    DF07331B
> >
> > -553,176,292                  FFFFFFFF DF07331B
> >
> >
> >
> > So, the program crashes due to bad pointer when it used vtkdata array
> > details to draw histogram.
> >
> > I think the probelm here is assigning of unsigned long to long long. I
>
> > don't want to change the
> >
> > Vtk data type to unsigned long, so in next attempt i change the itk
> > OffsetValueType to long long.
> >
> >
> >
> > 2.       I changed the itk OffsetValueType to long long, but this
> leads
> > to change of lot of several other
> >
> > variable type in different file. Specially some of the basic image
> > container variables like TElementIdentifier, TElement have to be
> changed.
> >
> > For now I only can give these 2 variable, but i am afraid more
> variable
> > data types have to be changed in the future.
> >
> >
> >
> > I would be glad to have some help to solve this problem.
> >
> >
> >
> > Regards,
> >
> > Kana
> >
> >
> >
> ------------------------------------------------------------------------
> >
> > _____________________________________
> > Powered by www.kitware.com
> >
> > Visit other Kitware open-source projects at
> > http://www.kitware.com/opensource/opensource.html
> >
> > Please keep messages on-topic and check the ITK FAQ at:
> http://www.itk.org/Wiki/ITK_FAQ
> >
> > Follow this link to subscribe/unsubscribe:
> > http://www.itk.org/mailman/listinfo/insight-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.itk.org/mailman/private/insight-developers/attachments/20090710/fc89e45a/attachment.htm>


More information about the Insight-developers mailing list