[Insight-developers] itk performance numbers
Rupert Brooks
rupert.brooks at gmail.com
Wed Jul 25 12:14:14 EDT 2012
Interesting re Otsu. Theres a bug in the stdev calc - update will arrive
soon. Thanks for pointing out the results didnt make sense.
For my specific needs, its the slowdown in the metrics that trouble me the
most. However, there may be improvements in result quality that would
justify a slowdown. In general, i dont think the results are numerically
identical (have not tested this specific case).
Rupert
--------------------------------------------------------------
Rupert Brooks
rupert.brooks at gmail.com
On Wed, Jul 25, 2012 at 12:02 PM, Bradley Lowekamp
<blowekamp at mail.nih.gov>wrote:
> Rupert,
>
>
> I took out the OtsuThreshold filter and the resulting timings for ITKv3
> and ITKv4 were comparable for your Filter Chain. More than half the time
> was spend in this one filter.
>
> Also the computed stddev results do not appear to be correct. However
> looking at the code it looks right.
>
> Brad
>
> On Jul 25, 2012, at 11:53 AM, Rupert Brooks wrote:
>
> Brad,
>
> Agreed, VS2005 is not my favorite either, which is why im doing tests also
> with a recent gcc. If the performance degradation was limited to VS2005,
> that would not be a big deal in my view. I'm stuck with it at the moment,
> but I can hardly expect you folks to do your best work for it.
>
> If and when i give other compilers a shot, i will keep you posted.
>
> Rupert
> --------------------------------------------------------------
> Rupert Brooks
> rupert.brooks at gmail.com
>
>
>
> On Wed, Jul 25, 2012 at 10:52 AM, Bradley Lowekamp <blowekamp at mail.nih.gov
> > wrote:
>
>> Hello Rupert,
>>
>> I see you are still using Visual Studio 8 2005, I am curious how VS10
>> would compare. I know that there are a couple special case optimization
>> added to ITKv4 which VS8 is not able to take advantage off.
>>
>> As others have reported we are begining to address some of these issues.
>>
>> Thanks for sharing!
>> Brad
>>
>> On Jul 25, 2012, at 10:12 AM, Rupert Brooks wrote:
>>
>> Hi,
>>
>> I was commenting the other day that ITK4 seems to have lower performance
>> than ITK3. I put together a little benchmark to test this objectively.
>> Results for a couple of systems are below. The benchmark is neither
>> particularly sophisticated nor exact, but i have tried to make it typical
>> of what i use ITK for. I have not tried it with Hans' patch as that got
>> abandoned. Will follow up separately on that.
>>
>> In the numbers below, you will see that performance seems to have
>> degraded across the board. One exception is the 2D image test. In this
>> test i was lazy, and i left the image axis aligned with unit spacing. I am
>> suspecting a special case optimization in ITK4. At the moment, this blocks
>> my ability to upgrade my projects to ITK4, as they are performance
>> critical. Any hints and tips to get those milliseconds back will be much
>> appreciated.
>>
>> The code is at https://github.com/rupertbrooks/itkbench, i'd be curious
>> if others get the same results, and of course, criticism and improvements
>> are welcome.
>>
>> Note that the output about what processor / cores is on the system is
>> bogus. The windows system is a core i7 with 4 cores, the linux one is a
>> Core2 Quad. I suspect a bug in itksys::SystemInformation, but i will
>> follow up on that separately also.
>>
>> In all itk builds, ITK_USE_REVIEW is on. In the itk3 builds,
>> USE_OPTIMZIZED_REGISTRATION, and the new statistics framework are turned on.
>>
>> Cheers,
>> Rupert
>>
>> Data follows.....
>>
>> itk3 Windows XP 32bit Visual Studio Pro 2005 Build RelWithDebInfo
>> System: CAMD5C5PMN1
>> Processor: Pentium III (0.18 micron) With 1 Or 2 MB On-Die L2 Cache
>> Serial #:
>> Cache: -1
>> Clock: 2800
>> Cores: 4 cpus x 1 Cores = 4
>> OSName: Windows
>> Release: XP Professional
>> Version: Service Pack 3 (Build 2600)
>> Platform: x86
>> Operating System is 32 bit
>> ITK Version: 3.20.0
>> Virtual Memory: Total: 2047 Available: 2021
>> Physical Memory: Total:3581 Available: 1457
>> Probe Name: Count Min Mean
>> Stdev Max Total
>> FilterChain_1_threads 10 0.0345688 0.0361805
>> 0.0381523 0.0375099 0.361805
>> FilterChain_2_threads 10 0.0257301 0.0265144
>> 0.0279574 0.0282478 0.265144
>> FilterChain_3_threads 10 0.022007 0.0232452
>> 0.0245125 0.0240822 0.232452
>> FilterChain_4_threads 10 0.0199661 0.0221756
>> 0.0235205 0.0288773 0.221756
>> Image2D 10 0.144909 0.14665
>> 0.154586 0.148136 1.4665
>> Image3D 10 0.19125 0.193378
>> 0.203846 0.196495 1.93378
>> MeanSquares_1_threads 10 0.612728 0.616159
>> 0.649503 0.627789 6.16159
>> MeanSquares_2_threads 10 0.359703 0.368051
>> 0.388106 0.392696 3.68051
>> MeanSquares_3_threads 10 0.268024 0.280672
>> 0.296142 0.312557 2.80672
>> MeanSquares_4_threads 10 0.199249 0.213209
>> 0.224918 0.2243 2.13209
>>
>>
>> itk4 Windows XP 32bit Visual Studio Pro 2005 Build RelWithDebInfo
>>
>> System: CAMD5C5PMN1
>> Processor: Pentium III (0.18 micron) With 1 Or 2 MB On-Die L2 Cache
>> Serial #:
>> Cache: -1
>> Clock: 2800
>> Cores: 4 cpus x 1 Cores = 4
>> OSName: Windows
>> Release: XP Professional
>> Version: Service Pack 3 (Build 2600)
>> Platform: x86
>> Operating System is 32 bit
>> ITK Version: 4.3.0
>> Virtual Memory: Total: 2047 Available: 2008
>> Physical Memory: Total:3581 Available: 1574
>> Probe Name: Count Min Mean
>> Stdev Max Total
>> FilterChain_1_threads 10 0.0797005 0.0808155
>> 0.0851898 0.0820313 0.808155
>> FilterChain_2_threads 10 0.0670509 0.0921925
>> 0.0991914 0.109177 0.921925
>> FilterChain_3_threads 10 0.0648499 0.113566
>> 0.120975 0.125893 1.13566
>> FilterChain_4_threads 10 0.0536919 0.104662
>> 0.114558 0.132114 1.04662
>> Image2D 10 0.0953979 0.0965282
>> 0.101753 0.0978088 0.965282
>> Image3D 10 0.196468 0.198288
>> 0.209017 0.200562 1.98288
>> MeanSquares_1_threads 10 0.982071 0.986089
>> 1.03944 0.996445 9.86089
>> MeanSquares_2_threads 10 0.686028 0.728621
>> 0.768821 0.799675 7.28621
>> MeanSquares_3_threads 10 0.534885 0.554387
>> 0.5846 0.578133 5.54387
>> MeanSquares_4_threads 10 0.441551 0.463463
>> 0.488903 0.502502 4.63463
>>
>>
>> itk3 Linux 64 bit gcc 4.6.1 Build Release
>> System: morrigan
>> Processor: Intel(R) Core(TM)2 Duo CPU T9500 @ 2.60GHz
>> Serial #:
>> Cache: 2048
>> Clock: 2003
>> Cores: 4 cpus x 4 Cores = 16
>> OSName: Linux
>> Release: 3.0.0-21-generic
>> Version: #35-Ubuntu SMP Fri May 25 17:57:41 UTC 2012
>> Platform: x86_64
>> Operating System is 64 bit
>> ITK Version: 3.20.1
>> Virtual Memory: Total: 20795 Available: 20795
>> Physical Memory: Total:8001 Available: 3416
>> Probe Name: Count Min Mean
>> Stdev
>> Max Total
>> FilterChain_1_threads 10 0.0309269 0.031097
>> 0.0327796 0.031528 0.31097
>> FilterChain_2_threads 10 0.021672 0.0248219
>> 0.0261976 0.026264 0.248219
>> FilterChain_3_threads 10 0.0213931 0.0217796
>> 0.0229634 0.0228851 0.217796
>> FilterChain_4_threads 10 0.0198419 0.0207421
>> 0.0218725 0.0216701 0.207421
>> Image2D 10 0.0263629 0.0264159
>> 0.0278451 0.0267591 0.264159
>> Image3D 10 0.0263779 0.0263906
>> 0.0278181 0.026463 0.263906
>> MeanSquares_1_threads 10 0.463909 0.465901
>> 0.491113 0.473644 4.65901
>> MeanSquares_2_threads 10 0.349138 0.40529
>> 0.429438 0.487785 4.0529
>> MeanSquares_3_threads 10 0.331908 0.357685
>> 0.377834 0.400941 3.57685
>> MeanSquares_4_threads 10 0.299428 0.336914
>> 0.356878 0.395144 3.36914
>>
>> itk4 Linux 64-bit gcc 4.6.1 Build Release
>> System: morrigan
>> Processor: Intel(R) Core(TM)2 Duo CPU T9500 @ 2.60GHz
>> Serial #:
>> Cache: 2048
>> Clock: 2003
>> Cores: 4 cpus x 4 Cores = 16
>> OSName: Linux
>> Release: 3.0.0-21-generic
>> Version: #35-Ubuntu SMP Fri May 25 17:57:41 UTC 2012
>> Platform: x86_64
>> Operating System is 64 bit
>> ITK Version: 4.3.0
>> Virtual Memory: Total: 20795 Available: 20795
>> Physical Memory: Total:8001 Available: 3400
>> Probe Name: Count Min Mean
>> Stdev
>> Max Total
>> FilterChain_1_threads 10 0.0663331 0.0668247
>> 0.0704416 0.0681288 0.668247
>> FilterChain_2_threads 10 0.0353 0.0393441
>> 0.042048 0.0534279 0.393441
>> FilterChain_3_threads 10 0.025275 0.0265869
>> 0.0282408 0.0364931 0.265869
>> FilterChain_4_threads 10 0.0208249 0.0222288
>> 0.0235333 0.028336 0.222288
>> Image2D 10 0.026361 0.026421
>> 0.0278503 0.0266371 0.26421
>> Image3D 10 0.0263991 0.0266509
>> 0.0280954 0.0274949 0.266509
>> MeanSquares_1_threads 10 0.69005 0.69459
>> 0.732169 0.701676 6.9459
>> MeanSquares_2_threads 10 0.590396 0.699826
>> 0.739994 0.766702 6.99826
>> MeanSquares_3_threads 10 0.527146 0.538844
>> 0.568047 0.548791 5.38844
>> MeanSquares_4_threads 10 0.417008 0.454278
>> 0.480139 0.533593 4.54278
>> --------------------------------------------------------------
>> Rupert Brooks
>> rupert.brooks at gmail.com
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Kitware offers ITK Training Courses, for more information visit:
>> http://kitware.com/products/protraining.php
>>
>> Please keep messages on-topic and check the ITK FAQ at:
>> http://www.itk.org/Wiki/ITK_FAQ
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.itk.org/mailman/listinfo/insight-developers
>>
>>
>> ========================================================
>> Bradley Lowekamp
>> Medical Science and Computing for
>> Office of High Performance Computing and Communications
>> National Library of Medicine
>> blowekamp at mail.nih.gov
>>
>>
>>
>>
>
> ========================================================
>
> Bradley Lowekamp
>
> Medical Science and Computing for
>
> Office of High Performance Computing and Communications
>
> National Library of Medicine
>
> blowekamp at mail.nih.gov
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.itk.org/pipermail/insight-developers/attachments/20120725/f52a856a/attachment.htm>
More information about the Insight-developers
mailing list