[Insight-developers] itk performance numbers
Matt McCormick
matt.mccormick at kitware.com
Wed Jul 25 13:26:53 EDT 2012
Hi Rupert,
Thanks for the feedback.
Matt
On Wed, Jul 25, 2012 at 3:48 PM, Rupert Brooks <rupert.brooks at gmail.com> wrote:
> Matt,
>
> I just upgraded master to current (which i think includes these changes),
> rebuilt and reran on both systems. I cannot see a statisically significant
> difference. To truly check we would have to run multiple times and
> compare, but its safe to say if theres a difference, its subtle. However, i
> did not particularly choose my filter chain to have the GetInput/Output
> problem. I threw together something more or less similar to something i
> already did, so this cant be considered conclusive.
>
> Rupert
>
> New results:
> itk4, Visual 2005
> System: CAMD5C5PMN1
> Processor: Pentium III (0.18 micron) With 1 Or 2 MB On-Die L2 Cache
> Serial #:
> Cache: -1
> Clock: 2800
> Cores: 4 cpus x 1 Cores = 4
> OSName: Windows
> Release: XP Professional
> Version: Service Pack 3 (Build 2600)
> Platform: x86
> Operating System is 32 bit
> ITK Version: 4.3.0
> Virtual Memory: Total: 2047 Available: 2008
> Physical Memory: Total:3581 Available: 1370
> Probe Name: Count Min Mean
> Stdev Max Total
> FilterChain_1_threads 10 0.0799255 0.0812134
> 0.0856113 0.0824928 0.812134
> FilterChain_2_threads 10 0.0585442 0.101192
> 0.107718 0.1078 1.01192
> FilterChain_3_threads 10 0.0670929 0.107661
> 0.115547 0.124683 1.07661
> FilterChain_4_threads 10 0.0634041 0.109252
> 0.117602 0.125126 1.09252
> Image2D 10 0.0952263 0.0958263
> 0.101011 0.0965271 0.958263
> Image3D 10 0.195629 0.197874
> 0.208581 0.200047 1.97874
> MeanSquares_1_threads 10 0.979141 0.98356
> 1.03678 0.997391 9.8356
> MeanSquares_2_threads 10 0.685131 0.712635
> 0.751585 0.749096 7.12635
> MeanSquares_3_threads 10 0.525139 0.549089
> 0.579045 0.57336 5.49089
> MeanSquares_4_threads 10 0.439487 0.452317
> 0.476833 0.458878 4.52317
>
> Linux, gcc 4.6.1
> ./itkbench
> System: morrigan
> Processor: Intel(R) Core(TM)2 Duo CPU T9500 @ 2.60GHz
> Serial #:
> Cache: 2048
> Clock: 2003
> Cores: 4 cpus x 4 Cores = 16
> OSName: Linux
> Release: 3.0.0-21-generic
> Version: #35-Ubuntu SMP Fri May 25 17:57:41 UTC 2012
> Platform: x86_64
> Operating System is 64 bit
> ITK Version: 4.3.0
> Virtual Memory: Total: 20795 Available: 20695
> Physical Memory: Total:8001 Available: 3538
> Probe Name: Count Min Mean
> Stdev Max Total
> FilterChain_1_threads 10 0.0684249 0.0686138
> 0.0723256 0.0689371 0.686138
> FilterChain_2_threads 10 0.0367901 0.048942
> 0.0519886 0.0535438 0.48942
> FilterChain_3_threads 10 0.0256481 0.0262128
> 0.0276426 0.0280211 0.262128
> FilterChain_4_threads 10 0.02005 0.0246643
> 0.0269426 0.0390608 0.246643
> Image2D 10 0.026371 0.0265399
> 0.0279799 0.0279481 0.265399
> Image3D 10 0.0263801 0.0264313
> 0.0278611 0.0266159 0.264313
> MeanSquares_1_threads 10 0.683572 0.686879
> 0.724048 0.696691 6.86879
> MeanSquares_2_threads 10 0.613772 0.6994
> 0.739107 0.766093 6.994
> MeanSquares_3_threads 10 0.53469 0.565863
> 0.596594 0.579526 5.65863
> MeanSquares_4_threads 10 0.429782 0.463741
> 0.489788 0.50595 4.63741
> --------------------------------------------------------------
> Rupert Brooks
> rupert.brooks at gmail.com
>
>
>
>
> On Wed, Jul 25, 2012 at 10:47 AM, Matt McCormick
> <matt.mccormick at kitware.com> wrote:
>>
>> Hi Rupert,
>>
>> Very interesting. I am curious if the recent GetInput/GetOutput patch
>> helps with the quantities.
>>
>> Thanks,
>> Matt
>>
>> On Wed, Jul 25, 2012 at 2:12 PM, Rupert Brooks <rupert.brooks at gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I was commenting the other day that ITK4 seems to have lower performance
>> > than ITK3. I put together a little benchmark to test this objectively.
>> > Results for a couple of systems are below. The benchmark is neither
>> > particularly sophisticated nor exact, but i have tried to make it
>> > typical of
>> > what i use ITK for. I have not tried it with Hans' patch as that got
>> > abandoned. Will follow up separately on that.
>> >
>> > In the numbers below, you will see that performance seems to have
>> > degraded
>> > across the board. One exception is the 2D image test. In this test i
>> > was
>> > lazy, and i left the image axis aligned with unit spacing. I am
>> > suspecting
>> > a special case optimization in ITK4. At the moment, this blocks my
>> > ability
>> > to upgrade my projects to ITK4, as they are performance critical. Any
>> > hints
>> > and tips to get those milliseconds back will be much appreciated.
>> >
>> > The code is at https://github.com/rupertbrooks/itkbench, i'd be curious
>> > if
>> > others get the same results, and of course, criticism and improvements
>> > are
>> > welcome.
>> >
>> > Note that the output about what processor / cores is on the system is
>> > bogus.
>> > The windows system is a core i7 with 4 cores, the linux one is a Core2
>> > Quad.
>> > I suspect a bug in itksys::SystemInformation, but i will follow up on
>> > that
>> > separately also.
>> >
>> > In all itk builds, ITK_USE_REVIEW is on. In the itk3 builds,
>> > USE_OPTIMZIZED_REGISTRATION, and the new statistics framework are turned
>> > on.
>> >
>> > Cheers,
>> > Rupert
>> >
>> > Data follows.....
>> >
>> > itk3 Windows XP 32bit Visual Studio Pro 2005 Build RelWithDebInfo
>> > System: CAMD5C5PMN1
>> > Processor: Pentium III (0.18 micron) With 1 Or 2 MB On-Die L2 Cache
>> > Serial #:
>> > Cache: -1
>> > Clock: 2800
>> > Cores: 4 cpus x 1 Cores = 4
>> > OSName: Windows
>> > Release: XP Professional
>> > Version: Service Pack 3 (Build 2600)
>> > Platform: x86
>> > Operating System is 32 bit
>> > ITK Version: 3.20.0
>> > Virtual Memory: Total: 2047 Available: 2021
>> > Physical Memory: Total:3581 Available: 1457
>> > Probe Name: Count Min Mean
>> > Stdev Max Total
>> > FilterChain_1_threads 10 0.0345688 0.0361805
>> > 0.0381523 0.0375099 0.361805
>> > FilterChain_2_threads 10 0.0257301 0.0265144
>> > 0.0279574 0.0282478 0.265144
>> > FilterChain_3_threads 10 0.022007 0.0232452
>> > 0.0245125 0.0240822 0.232452
>> > FilterChain_4_threads 10 0.0199661 0.0221756
>> > 0.0235205 0.0288773 0.221756
>> > Image2D 10 0.144909 0.14665
>> > 0.154586 0.148136 1.4665
>> > Image3D 10 0.19125 0.193378
>> > 0.203846 0.196495 1.93378
>> > MeanSquares_1_threads 10 0.612728 0.616159
>> > 0.649503 0.627789 6.16159
>> > MeanSquares_2_threads 10 0.359703 0.368051
>> > 0.388106 0.392696 3.68051
>> > MeanSquares_3_threads 10 0.268024 0.280672
>> > 0.296142 0.312557 2.80672
>> > MeanSquares_4_threads 10 0.199249 0.213209
>> > 0.224918 0.2243 2.13209
>> >
>> >
>> > itk4 Windows XP 32bit Visual Studio Pro 2005 Build RelWithDebInfo
>> >
>> > System: CAMD5C5PMN1
>> > Processor: Pentium III (0.18 micron) With 1 Or 2 MB On-Die L2 Cache
>> > Serial #:
>> > Cache: -1
>> > Clock: 2800
>> > Cores: 4 cpus x 1 Cores = 4
>> > OSName: Windows
>> > Release: XP Professional
>> > Version: Service Pack 3 (Build 2600)
>> > Platform: x86
>> > Operating System is 32 bit
>> > ITK Version: 4.3.0
>> > Virtual Memory: Total: 2047 Available: 2008
>> > Physical Memory: Total:3581 Available: 1574
>> > Probe Name: Count Min Mean
>> > Stdev Max Total
>> > FilterChain_1_threads 10 0.0797005 0.0808155
>> > 0.0851898 0.0820313 0.808155
>> > FilterChain_2_threads 10 0.0670509 0.0921925
>> > 0.0991914 0.109177 0.921925
>> > FilterChain_3_threads 10 0.0648499 0.113566
>> > 0.120975 0.125893 1.13566
>> > FilterChain_4_threads 10 0.0536919 0.104662
>> > 0.114558 0.132114 1.04662
>> > Image2D 10 0.0953979 0.0965282
>> > 0.101753 0.0978088 0.965282
>> > Image3D 10 0.196468 0.198288
>> > 0.209017 0.200562 1.98288
>> > MeanSquares_1_threads 10 0.982071 0.986089
>> > 1.03944 0.996445 9.86089
>> > MeanSquares_2_threads 10 0.686028 0.728621
>> > 0.768821 0.799675 7.28621
>> > MeanSquares_3_threads 10 0.534885 0.554387
>> > 0.5846 0.578133 5.54387
>> > MeanSquares_4_threads 10 0.441551 0.463463
>> > 0.488903 0.502502 4.63463
>> >
>> >
>> > itk3 Linux 64 bit gcc 4.6.1 Build Release
>> > System: morrigan
>> > Processor: Intel(R) Core(TM)2 Duo CPU T9500 @ 2.60GHz
>> > Serial #:
>> > Cache: 2048
>> > Clock: 2003
>> > Cores: 4 cpus x 4 Cores = 16
>> > OSName: Linux
>> > Release: 3.0.0-21-generic
>> > Version: #35-Ubuntu SMP Fri May 25 17:57:41 UTC 2012
>> > Platform: x86_64
>> > Operating System is 64 bit
>> > ITK Version: 3.20.1
>> > Virtual Memory: Total: 20795 Available: 20795
>> > Physical Memory: Total:8001 Available: 3416
>> > Probe Name: Count Min Mean
>> > Stdev
>> > Max Total
>> > FilterChain_1_threads 10 0.0309269 0.031097
>> > 0.0327796 0.031528 0.31097
>> > FilterChain_2_threads 10 0.021672 0.0248219
>> > 0.0261976 0.026264 0.248219
>> > FilterChain_3_threads 10 0.0213931 0.0217796
>> > 0.0229634 0.0228851 0.217796
>> > FilterChain_4_threads 10 0.0198419 0.0207421
>> > 0.0218725 0.0216701 0.207421
>> > Image2D 10 0.0263629 0.0264159
>> > 0.0278451 0.0267591 0.264159
>> > Image3D 10 0.0263779 0.0263906
>> > 0.0278181 0.026463 0.263906
>> > MeanSquares_1_threads 10 0.463909 0.465901
>> > 0.491113 0.473644 4.65901
>> > MeanSquares_2_threads 10 0.349138 0.40529
>> > 0.429438 0.487785 4.0529
>> > MeanSquares_3_threads 10 0.331908 0.357685
>> > 0.377834 0.400941 3.57685
>> > MeanSquares_4_threads 10 0.299428 0.336914
>> > 0.356878 0.395144 3.36914
>> >
>> > itk4 Linux 64-bit gcc 4.6.1 Build Release
>> > System: morrigan
>> > Processor: Intel(R) Core(TM)2 Duo CPU T9500 @ 2.60GHz
>> > Serial #:
>> > Cache: 2048
>> > Clock: 2003
>> > Cores: 4 cpus x 4 Cores = 16
>> > OSName: Linux
>> > Release: 3.0.0-21-generic
>> > Version: #35-Ubuntu SMP Fri May 25 17:57:41 UTC 2012
>> > Platform: x86_64
>> > Operating System is 64 bit
>> > ITK Version: 4.3.0
>> > Virtual Memory: Total: 20795 Available: 20795
>> > Physical Memory: Total:8001 Available: 3400
>> > Probe Name: Count Min Mean
>> > Stdev
>> > Max Total
>> > FilterChain_1_threads 10 0.0663331 0.0668247
>> > 0.0704416 0.0681288 0.668247
>> > FilterChain_2_threads 10 0.0353 0.0393441
>> > 0.042048 0.0534279 0.393441
>> > FilterChain_3_threads 10 0.025275 0.0265869
>> > 0.0282408 0.0364931 0.265869
>> > FilterChain_4_threads 10 0.0208249 0.0222288
>> > 0.0235333 0.028336 0.222288
>> > Image2D 10 0.026361 0.026421
>> > 0.0278503 0.0266371 0.26421
>> > Image3D 10 0.0263991 0.0266509
>> > 0.0280954 0.0274949 0.266509
>> > MeanSquares_1_threads 10 0.69005 0.69459
>> > 0.732169 0.701676 6.9459
>> > MeanSquares_2_threads 10 0.590396 0.699826
>> > 0.739994 0.766702 6.99826
>> > MeanSquares_3_threads 10 0.527146 0.538844
>> > 0.568047 0.548791 5.38844
>> > MeanSquares_4_threads 10 0.417008 0.454278
>> > 0.480139 0.533593 4.54278
>> > --------------------------------------------------------------
>> > Rupert Brooks
>> > rupert.brooks at gmail.com
>> >
>> >
>> > _______________________________________________
>> > Powered by www.kitware.com
>> >
>> > Visit other Kitware open-source projects at
>> > http://www.kitware.com/opensource/opensource.html
>> >
>> > Kitware offers ITK Training Courses, for more information visit:
>> > http://kitware.com/products/protraining.php
>> >
>> > Please keep messages on-topic and check the ITK FAQ at:
>> > http://www.itk.org/Wiki/ITK_FAQ
>> >
>> > Follow this link to subscribe/unsubscribe:
>> > http://www.itk.org/mailman/listinfo/insight-developers
>> >
>
>
More information about the Insight-developers
mailing list