[vtkusers] timings for vtkImageReader2

Mathieu Malaterre mathieu.malaterre at kitware.com
Wed Oct 13 13:25:31 EDT 2004


Kate,

	Are you sure your customer has as little memory as you do ? If not the 
problem desepear.

	Since AMIRA is a closed source software it is difficult to know what 
they are doing. Are you sure they are loading the *whole* data, they 
could do some striding to remove part which won't need to be displayed.

	Another thing you could do, is compress the raw data(*). then in your 
program read it, uncompress it and use. Since the IO is a slow part in 
the process you should be able to save some time depending on how good 
the compression was.

HTH,
Mathieu

#include "vtkImageNoiseSource.h"
#include "vtkImageData.h"
#include "vtkImageCast.h"
#include "vtkXMLImageDataWriter.h"
#include "vtkXMLImageDataReader.h"

void gen()
{
   vtkImageNoiseSource *noise = vtkImageNoiseSource::New();
   noise->SetWholeExtent(0,511,0,511,0,511);
   noise->SetMinimum(0.0);
   noise->SetMaximum(1.0);
   noise->ReleaseDataFlagOff();
   noise->Update();

//  noise->GetOutput()->Print( cout );

   vtkImageCast *cast = vtkImageCast::New();
   cast->SetInput( noise->GetOutput() );
   cast->SetOutputScalarTypeToUnsignedShort ();

   vtkXMLImageDataWriter *writer = vtkXMLImageDataWriter::New();
   writer->SetInput( cast->GetOutput() );
   writer->SetFileName( "bench.vti" );
   writer->Write();

   noise->Delete();
   cast->Delete();
   writer->Delete();

}

int main()
{
   //gen();

   vtkXMLImageDataReader *reader = vtkXMLImageDataReader::New();
   reader->SetFileName( "bench.vti" );
   reader->Update();

//  reader->GetOutput()->Print ( cout );

   reader->Delete();

   return 0;
}

Kate Kerekes wrote:
> Thank you to all for the comments.  
> 
> 1.  I am developing this app for windows because the final user will be
> using windows--although I much prefer linux :)
> 
> 2.  I understand that I don't have a lot of memory to run such a large
> dataset through, but as I said in an earlier message I have read the same
> dataset with AMIRA, and it is significantly faster (sorry cannot remember
> exact time, but if you look back through the list you can find it)  So why
> is AMIRA faster?  I suspect that AMIRA takes advantage of memory mapping in
> windows--although it is hard to tell because it is a commercial app and does
> not provide source code.  Is there any way to do this with vtk or a vtk
> class that already does this?
> 
> kate
> 
> -----Original Message-----
> From: Sean McInerney [mailto:seanm at nmr.mgh.harvard.edu] 
> Sent: Friday, October 08, 2004 1:15 PM
> To: Mathieu Malaterre
> Cc: Budd Hirons; vtk-users; kerekes at fastmail.fm
> Subject: Re: [vtkusers] timings for vtkImageReader2
> 
> Hi Kate,
> 
>    Again I'll second Mathieu. You have far too little memory to be 
> handling that much data. In memory your data alone occupies 268Mb before 
> you even do anything to it. The memory occupied by your application and 
> the VTK libraries that it links in are also significant ... so your 
> application and data may be occupying around 350Mb in memory! ... and 
> that is a conservative estimate.
> 
>    Since Windows XP is not known for being particularly lightweight, you 
> are certainly causing a whole lot of its processes to be written out to 
> swap. This massive slowdown is even compounded if the swap space is on 
> the same disk from which you are reading.
> 
>    Using Linux will certainly improve your quality of life, but I think 
> investing in, at least, another 512Mb of RAM is compulsory. I used to 
> use 1Gb of RAM as standard for any machine expected to do any serious 
> imaging work. With the advent of higher field scanners with increased 
> resolutions and considering the relatively low price of RAM, I would 
> suggest using 2Gb if possible. That will give you the room you need to 
> support pipeline operations on your data once it is read in.
> 
> -Sean
> 
> Mathieu Malaterre wrote:
> 
>>Budd,
>>
>>    512x512x512x2 = 268,435,456 so I don't see any problem with a 3s 
>>reading time. I admit a have 1gig of memory so there is very few chance 
>>I start swapping. Also to be honest the first time I run the c++ code it 
>>take 8s, the second time it takes 3s
>>
>>$ time ./bench                                                 ./bench  
>>1.05s user 1.97s system 99% cpu 3.021 total
>>
>>
>>    My guess is:
>>- you are swapping a lot (if you didn't ran defrag in a while swapping 
>>would be very slow).
>>- Close any other apps you are using to avoid having to swap.
>>- Use linux :P
>>
>>Mathieu
>>Ps: sys info:
>>
>>$ cat /proc/cpuinfo
>>processor       : 0
>>vendor_id       : GenuineIntel
>>cpu family      : 15
>>model           : 2
>>model name      : Intel(R) Pentium(R) 4 CPU 2.80GHz
>>stepping        : 9
>>cpu MHz         : 2793.067
>>cache size      : 512 KB
>>physical id     : 0
>>siblings        : 2
>>fdiv_bug        : no
>>hlt_bug         : no
>>f00f_bug        : no
>>coma_bug        : no
>>fpu             : yes
>>fpu_exception   : yes
>>cpuid level     : 2
>>wp              : yes
>>flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
>>mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid
>>runqueue        : 0
>>
>>bogomips        : 5570.56
>>
>>processor       : 1
>>vendor_id       : GenuineIntel
>>cpu family      : 15
>>model           : 2
>>model name      : Intel(R) Pentium(R) 4 CPU 2.80GHz
>>stepping        : 9
>>cpu MHz         : 2793.067
>>cache size      : 512 KB
>>physical id     : 0
>>siblings        : 2
>>fdiv_bug        : no
>>hlt_bug         : no
>>f00f_bug        : no
>>coma_bug        : no
>>fpu             : yes
>>fpu_exception   : yes
>>cpuid level     : 2
>>wp              : yes
>>flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
>>mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid
>>runqueue        : 0
>>
>>bogomips        : 5583.66
>>
>>
>>/sbin/hdparm -t /dev/sda2
>>
>>/dev/sda2:
>> Timing buffered disk reads:  160 MB in  3.00 seconds =  53.33 MB/sec
>>
>>
>>
>>Budd Hirons wrote:
>>
>>
>>>3 seconds! you have to let me in on your secret!
>>>
>>>UltraATA maximum transfer in the DMA mode 5 is only 100MB/s, so I 
>>>guess it is possible, but unlikely, to read this in 3 seconds.  I ran 
>>>this exercise on two machines, compiled from C++.
>>>
>>>machine 1 (p4,ultraATA drive on board,DMA,DDR ram) - 18s read
>>>machine 2 (dual Xeon,SCSI on PCI adapter,no DMA,pc133 ram) - 66s read
>>>
>>>We regularly deal with volumes that exceed this one in size, and range 
>>>to 800 megs on disk.  Load times to get very large chunks of data into 
>>>can ram vary wildly, but it should not be expected to happen 
>>>instantaneously.  45 seconds for larger datasets would not be off the 
>>>mark in release on random equipment and with other processes likely 
>>>running.
>>>
>>>Cheers,
>>>Budd.
>>>
>>>
>>>
>>>Kate Kerekes wrote:
>>>
>>>
>>>>Hello again,
>>>>
>>>>I apologize for the length of this message.  I am still having 
>>>>problems with
>>>>the time it takes vtkImageReader2 to read 268 MB of data (a series of 
>>>>512
>>>>files with 512x512 shorts each.)  I ran the following test program (from
>>>>Mathieu Malaterre  translated to tcl by me) to see how long a read 
>>>>takes:
>>>>
>>>>#!/bin/sh
>>>>load vtkCommonTCL.dll
>>>>load vtkFilteringTCL.dll
>>>>load vtkGraphicsTCL.dll
>>>>load vtkIOTCL.dll
>>>>load vtkImagingTCL.dll
>>>>load vtkRenderingTCL.dll
>>>>load vtkHybridTCL.dll
>>>>source Interactor.tcl
>>>>source bindings-rw.tcl
>>>>source bindings-iw.tcl
>>>>source bindings.tcl
>>>>source setget.tcl
>>>> 
>>>>console show
>>>>
>>>>set createImage yes
>>>>
>>>>if {$createImage=="yes"} {
>>>>vtkImageNoiseSource noise
>>>>   noise SetWholeExtent 0 511 0 511 0 511
>>>>   noise SetMinimum 0.0
>>>>   noise SetMaximum 1.0
>>>>   [noise GetOutput] ReleaseDataFlagOn
>>>>   puts "about to update vtkImageNoiseSource"
>>>>   noise Update
>>>>   puts "finished updating vtkImageNoiseSource"
>>>> 
>>>>   vtkImageCast cast
>>>>   cast SetInput [noise GetOutput]
>>>>   cast SetOutputScalarTypeToUnsignedShort     vtkImageWriter writer
>>>>   writer SetInput [cast GetOutput]
>>>>   writer SetFileDimensionality 3
>>>>   writer SetFileName "bench.img"
>>>>   puts "about to update writer"
>>>>   writer Write
>>>>   puts "finished updating writer"
>>>>}
>>>> 
>>>>   vtkImageReader2 reader
>>>>   [reader GetOutput] ReleaseDataFlagOn
>>>>   reader SetDataExtent 0 511 0 511 0 511
>>>>   reader SetDataScalarTypeToUnsignedShort
>>>>   reader SetDataByteOrderToLittleEndian
>>>>   reader SetFileName "bench.img"
>>>>
>>>>   puts "about to update reader"
>>>>   set time1 [clock clicks -milliseconds]
>>>>   reader Update
>>>>   set time2 [clock clicks -milliseconds]
>>>>   puts "finished updating reader"
>>>>   puts "reader update took [expr $time2-$time1] milliseconds"
>>>>
>>>>The read takes 15 seconds on my machine (Windows XP with service pack 
>>>>2, AMD
>>>>athlon 64 chipset 3000+, with 512 megs of memory).  I am reading data 
>>>>from
>>>>the IDE harddrive.  Mathieu's timing was 3 seconds.  This of course 
>>>>is not
>>>>the newest machine out there, but it is not that old either.
>>>>
>>>>Then when I try to read my series of slices, my machine takes 45 
>>>>seconds.
>>>>The only difference in reading the slice files and the bench.img 
>>>>created by
>>>>ImageNoiseSource is that the bench.img is all three dimensions in one 
>>>>file
>>>>(rather than a series of slices), and the bench.img file uses unsigned
>>>>shorts rather than shorts (but both are 2 bytes, right?).
>>>>
>>>>So here are my thoughts as to why the problem could be occurring--any
>>>>feedback would be appreciated:
>>>>1.  my computer is too slow--I don't think this is it
>>>>2.  the bench.img file only reads faster because it has just been 
>>>>written to
>>>>memory and is probably still in the cache and therefore takes less 
>>>>time to
>>>>access
>>>>3.  the fact that my slices are in series and 512 different files 
>>>>have to be
>>>>read is the slow down
>>>>4.  the code running in tcl is slower than code running in c (I can 
>>>>fix this
>>>>by calling a c function from tcl--but only if I think this is really the
>>>>problem)
>>>>5.  there is some other problem that I am unaware of--this is my 
>>>>worst fear
>>>>:(
>>>>
>>>>Please help!  All of the other feedback I have gotten before says 
>>>>that the
>>>>read should take no more than a few seconds.  45 seconds is way too 
>>>>long for
>>>>our customer and this is not even the largest filesize that will be 
>>>>used.
>>>>
>>>>Thanks,
>>>>Kate Kerekes
>>>>
>>>>
>>>>
>>>>_______________________________________________
>>>>This is the private VTK discussion list. Please keep messages 
>>>>on-topic. Check the FAQ at: <http://public.kitware.com/cgi-bin/vtkfaq>
>>>>Follow this link to subscribe/unsubscribe:
>>>>http://www.vtk.org/mailman/listinfo/vtkusers
>>>>
>>>
>>>_______________________________________________
>>>This is the private VTK discussion list. Please keep messages 
>>>on-topic. Check the FAQ at: <http://public.kitware.com/cgi-bin/vtkfaq>
>>>Follow this link to subscribe/unsubscribe:
>>>http://www.vtk.org/mailman/listinfo/vtkusers
>>>
>>
>>
>>
>>_______________________________________________
>>This is the private VTK discussion list. Please keep messages on-topic. 
>>Check the FAQ at: <http://public.kitware.com/cgi-bin/vtkfaq>
>>Follow this link to subscribe/unsubscribe:
>>http://www.vtk.org/mailman/listinfo/vtkusers
>>
> 
> 
> _______________________________________________
> This is the private VTK discussion list. 
> Please keep messages on-topic. Check the FAQ at: <http://public.kitware.com/cgi-bin/vtkfaq>
> Follow this link to subscribe/unsubscribe:
> http://www.vtk.org/mailman/listinfo/vtkusers
> 






More information about the vtkusers mailing list