[Insight-developers] itk performance numbers

Bradley Lowekamp blowekamp at mail.nih.gov
Wed Jul 25 12:02:35 EDT 2012


Rupert,


I took out the OtsuThreshold filter and the resulting timings for ITKv3 and ITKv4 were comparable for your Filter Chain. More than half the time was spend in this one filter.

Also the computed stddev results do not appear to be correct. However looking at the code it looks right.

Brad

On Jul 25, 2012, at 11:53 AM, Rupert Brooks wrote:

> Brad,
> 
> Agreed, VS2005 is not my favorite either, which is why im doing tests also with a recent gcc.  If the performance degradation was limited to VS2005, that would not be a big deal in my view.   I'm stuck with it at the moment, but I can hardly expect you folks to do your best work for it.
> 
> If and when i give other compilers a shot, i will keep you posted.
> 
> Rupert
> --------------------------------------------------------------
> Rupert Brooks
> rupert.brooks at gmail.com
> 
> 
> 
> On Wed, Jul 25, 2012 at 10:52 AM, Bradley Lowekamp <blowekamp at mail.nih.gov> wrote:
> Hello Rupert,
> 
> I see you are still using Visual Studio 8 2005, I am curious how VS10 would compare. I know that there are a couple special case optimization added to ITKv4 which VS8 is not able to take advantage off. 
> 
> As others have reported we are begining to address some of these issues.
> 
> Thanks for sharing!
> Brad
> 
> On Jul 25, 2012, at 10:12 AM, Rupert Brooks wrote:
> 
>> Hi,
>> 
>> I was commenting the other day that ITK4 seems to have lower performance than ITK3.  I put together a little benchmark to test this objectively.  Results for a couple of systems are below.  The benchmark is neither particularly sophisticated nor exact, but i have tried to make it typical of what i use ITK for.  I have not tried it with Hans' patch as that got abandoned.  Will follow up separately on that.
>> 
>> In the numbers below, you will see that performance seems to have degraded across the board.  One exception is the 2D image test.  In this test i was lazy, and i left the image axis aligned with unit spacing.  I am suspecting a special case optimization in ITK4.  At the moment, this blocks my ability to upgrade my projects to ITK4, as they are performance critical.  Any hints and tips to get those milliseconds back will be much appreciated.
>> 
>> The code is at https://github.com/rupertbrooks/itkbench, i'd be curious if others get the same results, and of course, criticism and improvements are welcome. 
>> 
>> Note that the output about what processor / cores is on the system is bogus. The windows system is a core i7 with 4 cores, the linux one is a Core2 Quad.  I suspect a bug in itksys::SystemInformation, but i will follow up on that separately also.  
>> 
>> In all itk builds, ITK_USE_REVIEW is on.  In the itk3 builds, USE_OPTIMZIZED_REGISTRATION, and the new statistics framework are turned on.
>> 
>> Cheers,
>> Rupert
>> 
>> Data follows.....
>> 
>> itk3 Windows XP 32bit Visual Studio Pro 2005  Build RelWithDebInfo 
>> System: CAMD5C5PMN1
>> Processor: Pentium III (0.18 micron) With 1 Or 2 MB On-Die L2 Cache 
>>  Serial #: 
>>     Cache: -1
>>     Clock: 2800
>>     Cores: 4 cpus x 1 Cores = 4
>> OSName:     Windows
>>   Release:  XP Professional
>>   Version:  Service Pack 3 (Build 2600)
>>   Platform: x86
>>   Operating System is 32 bit
>> ITK Version: 3.20.0
>> Virtual Memory: Total: 2047 Available: 2021
>> Physical Memory: Total:3581 Available: 1457
>>            Probe Name:        Count          Min           Mean         Stdev            Max        Total 
>>  FilterChain_1_threads            10     0.0345688     0.0361805     0.0381523     0.0375099      0.361805
>>  FilterChain_2_threads            10     0.0257301     0.0265144     0.0279574     0.0282478      0.265144
>>  FilterChain_3_threads            10      0.022007     0.0232452     0.0245125     0.0240822      0.232452
>>  FilterChain_4_threads            10     0.0199661     0.0221756     0.0235205     0.0288773      0.221756
>>                Image2D            10      0.144909       0.14665      0.154586      0.148136        1.4665
>>                Image3D            10       0.19125      0.193378      0.203846      0.196495       1.93378
>>  MeanSquares_1_threads            10      0.612728      0.616159      0.649503      0.627789       6.16159
>>  MeanSquares_2_threads            10      0.359703      0.368051      0.388106      0.392696       3.68051
>>  MeanSquares_3_threads            10      0.268024      0.280672      0.296142      0.312557       2.80672
>>  MeanSquares_4_threads            10      0.199249      0.213209      0.224918        0.2243       2.13209
>> 
>> 
>> itk4 Windows XP 32bit Visual Studio Pro 2005  Build RelWithDebInfo
>> 
>> System: CAMD5C5PMN1
>> Processor: Pentium III (0.18 micron) With 1 Or 2 MB On-Die L2 Cache 
>>  Serial #: 
>>     Cache: -1
>>     Clock: 2800
>>     Cores: 4 cpus x 1 Cores = 4
>> OSName:     Windows
>>   Release:  XP Professional
>>   Version:  Service Pack 3 (Build 2600)
>>   Platform: x86
>>   Operating System is 32 bit
>> ITK Version: 4.3.0
>> Virtual Memory: Total: 2047 Available: 2008
>> Physical Memory: Total:3581 Available: 1574
>>            Probe Name:        Count          Min           Mean         Stdev            Max        Total 
>>  FilterChain_1_threads            10     0.0797005     0.0808155     0.0851898     0.0820313      0.808155
>>  FilterChain_2_threads            10     0.0670509     0.0921925     0.0991914      0.109177      0.921925
>>  FilterChain_3_threads            10     0.0648499      0.113566      0.120975      0.125893       1.13566
>>  FilterChain_4_threads            10     0.0536919      0.104662      0.114558      0.132114       1.04662
>>                Image2D            10     0.0953979     0.0965282      0.101753     0.0978088      0.965282
>>                Image3D            10      0.196468      0.198288      0.209017      0.200562       1.98288
>>  MeanSquares_1_threads            10      0.982071      0.986089       1.03944      0.996445       9.86089
>>  MeanSquares_2_threads            10      0.686028      0.728621      0.768821      0.799675       7.28621
>>  MeanSquares_3_threads            10      0.534885      0.554387        0.5846      0.578133       5.54387
>>  MeanSquares_4_threads            10      0.441551      0.463463      0.488903      0.502502       4.63463
>> 
>> 
>> itk3 Linux 64 bit gcc 4.6.1 Build Release
>> System: morrigan
>> Processor: Intel(R) Core(TM)2 Duo CPU     T9500  @ 2.60GHz
>>  Serial #:
>>     Cache: 2048
>>     Clock: 2003
>>     Cores: 4 cpus x 4 Cores = 16
>> OSName:     Linux
>>   Release:  3.0.0-21-generic
>>   Version:  #35-Ubuntu SMP Fri May 25 17:57:41 UTC 2012
>>   Platform: x86_64
>>   Operating System is 64 bit
>> ITK Version: 3.20.1
>> Virtual Memory: Total: 20795 Available: 20795
>> Physical Memory: Total:8001 Available: 3416
>>            Probe Name:        Count          Min           Mean         Stdev
>>          Max        Total
>>  FilterChain_1_threads            10     0.0309269      0.031097     0.0327796    0.031528       0.31097
>>  FilterChain_2_threads            10      0.021672     0.0248219     0.0261976    0.026264      0.248219
>>  FilterChain_3_threads            10     0.0213931     0.0217796     0.0229634   0.0228851      0.217796
>>  FilterChain_4_threads            10     0.0198419     0.0207421     0.0218725   0.0216701      0.207421
>>                Image2D            10     0.0263629     0.0264159     0.0278451   0.0267591      0.264159
>>                Image3D            10     0.0263779     0.0263906     0.0278181    0.026463      0.263906
>>  MeanSquares_1_threads            10      0.463909      0.465901      0.491113    0.473644       4.65901
>>  MeanSquares_2_threads            10      0.349138       0.40529      0.429438    0.487785        4.0529
>>  MeanSquares_3_threads            10      0.331908      0.357685      0.377834    0.400941       3.57685
>>  MeanSquares_4_threads            10      0.299428      0.336914      0.356878    0.395144       3.36914
>> 
>> itk4 Linux 64-bit gcc 4.6.1 Build Release
>> System: morrigan
>> Processor: Intel(R) Core(TM)2 Duo CPU     T9500  @ 2.60GHz
>>  Serial #:
>>     Cache: 2048
>>     Clock: 2003
>>     Cores: 4 cpus x 4 Cores = 16
>> OSName:     Linux
>>   Release:  3.0.0-21-generic
>>   Version:  #35-Ubuntu SMP Fri May 25 17:57:41 UTC 2012
>>   Platform: x86_64
>>   Operating System is 64 bit
>> ITK Version: 4.3.0
>> Virtual Memory: Total: 20795 Available: 20795
>> Physical Memory: Total:8001 Available: 3400
>>            Probe Name:        Count          Min           Mean         Stdev
>>          Max        Total
>>  FilterChain_1_threads            10     0.0663331     0.0668247     0.0704416   0.0681288      0.668247
>>  FilterChain_2_threads            10        0.0353     0.0393441      0.042048   0.0534279      0.393441
>>  FilterChain_3_threads            10      0.025275     0.0265869     0.0282408   0.0364931      0.265869
>>  FilterChain_4_threads            10     0.0208249     0.0222288     0.0235333    0.028336      0.222288
>>                Image2D            10      0.026361      0.026421     0.0278503   0.0266371       0.26421
>>                Image3D            10     0.0263991     0.0266509     0.0280954   0.0274949      0.266509
>>  MeanSquares_1_threads            10       0.69005       0.69459      0.732169    0.701676        6.9459
>>  MeanSquares_2_threads            10      0.590396      0.699826      0.739994    0.766702       6.99826
>>  MeanSquares_3_threads            10      0.527146      0.538844      0.568047    0.548791       5.38844
>>  MeanSquares_4_threads            10      0.417008      0.454278      0.480139    0.533593       4.54278
>> --------------------------------------------------------------
>> Rupert Brooks
>> rupert.brooks at gmail.com
>> 
>> _______________________________________________
>> Powered by www.kitware.com
>> 
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>> 
>> Kitware offers ITK Training Courses, for more information visit:
>> http://kitware.com/products/protraining.php
>> 
>> Please keep messages on-topic and check the ITK FAQ at:
>> http://www.itk.org/Wiki/ITK_FAQ
>> 
>> Follow this link to subscribe/unsubscribe:
>> http://www.itk.org/mailman/listinfo/insight-developers
> 
> ========================================================
> Bradley Lowekamp  
> Medical Science and Computing for
> Office of High Performance Computing and Communications
> National Library of Medicine 
> blowekamp at mail.nih.gov
> 
> 
> 
> 

========================================================
Bradley Lowekamp  
Medical Science and Computing for
Office of High Performance Computing and Communications
National Library of Medicine 
blowekamp at mail.nih.gov



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.itk.org/pipermail/insight-developers/attachments/20120725/8dc126e5/attachment.htm>


More information about the Insight-developers mailing list