[Insight-developers] itk performance numbers
Rupert Brooks
rupert.brooks at gmail.com
Wed Jul 25 11:48:05 EDT 2012
Matt,
I just upgraded master to current (which i think includes these changes),
rebuilt and reran on both systems. I cannot see a statisically significant
difference. To truly check we would have to run multiple times and
compare, but its safe to say if theres a difference, its subtle. However, i
did not particularly choose my filter chain to have the GetInput/Output
problem. I threw together something more or less similar to something i
already did, so this cant be considered conclusive.
Rupert
New results:
itk4, Visual 2005
System: CAMD5C5PMN1
Processor: Pentium III (0.18 micron) With 1 Or 2 MB On-Die L2 Cache
Serial #:
Cache: -1
Clock: 2800
Cores: 4 cpus x 1 Cores = 4
OSName: Windows
Release: XP Professional
Version: Service Pack 3 (Build 2600)
Platform: x86
Operating System is 32 bit
ITK Version: 4.3.0
Virtual Memory: Total: 2047 Available: 2008
Physical Memory: Total:3581 Available: 1370
Probe Name: Count Min Mean
Stdev Max Total
FilterChain_1_threads 10 0.0799255 0.0812134
0.0856113 0.0824928 0.812134
FilterChain_2_threads 10 0.0585442 0.101192
0.107718 0.1078 1.01192
FilterChain_3_threads 10 0.0670929 0.107661
0.115547 0.124683 1.07661
FilterChain_4_threads 10 0.0634041 0.109252
0.117602 0.125126 1.09252
Image2D 10 0.0952263 0.0958263
0.101011 0.0965271 0.958263
Image3D 10 0.195629 0.197874
0.208581 0.200047 1.97874
MeanSquares_1_threads 10 0.979141 0.98356
1.03678 0.997391 9.8356
MeanSquares_2_threads 10 0.685131 0.712635
0.751585 0.749096 7.12635
MeanSquares_3_threads 10 0.525139 0.549089
0.579045 0.57336 5.49089
MeanSquares_4_threads 10 0.439487 0.452317
0.476833 0.458878 4.52317
Linux, gcc 4.6.1
./itkbench
System: morrigan
Processor: Intel(R) Core(TM)2 Duo CPU T9500 @ 2.60GHz
Serial #:
Cache: 2048
Clock: 2003
Cores: 4 cpus x 4 Cores = 16
OSName: Linux
Release: 3.0.0-21-generic
Version: #35-Ubuntu SMP Fri May 25 17:57:41 UTC 2012
Platform: x86_64
Operating System is 64 bit
ITK Version: 4.3.0
Virtual Memory: Total: 20795 Available: 20695
Physical Memory: Total:8001 Available: 3538
Probe Name: Count Min Mean
Stdev Max Total
FilterChain_1_threads 10 0.0684249 0.0686138
0.0723256 0.0689371 0.686138
FilterChain_2_threads 10 0.0367901 0.048942
0.0519886 0.0535438 0.48942
FilterChain_3_threads 10 0.0256481 0.0262128
0.0276426 0.0280211 0.262128
FilterChain_4_threads 10 0.02005 0.0246643
0.0269426 0.0390608 0.246643
Image2D 10 0.026371 0.0265399
0.0279799 0.0279481 0.265399
Image3D 10 0.0263801 0.0264313
0.0278611 0.0266159 0.264313
MeanSquares_1_threads 10 0.683572 0.686879
0.724048 0.696691 6.86879
MeanSquares_2_threads 10 0.613772 0.6994
0.739107 0.766093 6.994
MeanSquares_3_threads 10 0.53469 0.565863
0.596594 0.579526 5.65863
MeanSquares_4_threads 10 0.429782 0.463741
0.489788 0.50595 4.63741
--------------------------------------------------------------
Rupert Brooks
rupert.brooks at gmail.com
On Wed, Jul 25, 2012 at 10:47 AM, Matt McCormick <matt.mccormick at kitware.com
> wrote:
> Hi Rupert,
>
> Very interesting. I am curious if the recent GetInput/GetOutput patch
> helps with the quantities.
>
> Thanks,
> Matt
>
> On Wed, Jul 25, 2012 at 2:12 PM, Rupert Brooks <rupert.brooks at gmail.com>
> wrote:
> > Hi,
> >
> > I was commenting the other day that ITK4 seems to have lower performance
> > than ITK3. I put together a little benchmark to test this objectively.
> > Results for a couple of systems are below. The benchmark is neither
> > particularly sophisticated nor exact, but i have tried to make it
> typical of
> > what i use ITK for. I have not tried it with Hans' patch as that got
> > abandoned. Will follow up separately on that.
> >
> > In the numbers below, you will see that performance seems to have
> degraded
> > across the board. One exception is the 2D image test. In this test i
> was
> > lazy, and i left the image axis aligned with unit spacing. I am
> suspecting
> > a special case optimization in ITK4. At the moment, this blocks my
> ability
> > to upgrade my projects to ITK4, as they are performance critical. Any
> hints
> > and tips to get those milliseconds back will be much appreciated.
> >
> > The code is at https://github.com/rupertbrooks/itkbench, i'd be curious
> if
> > others get the same results, and of course, criticism and improvements
> are
> > welcome.
> >
> > Note that the output about what processor / cores is on the system is
> bogus.
> > The windows system is a core i7 with 4 cores, the linux one is a Core2
> Quad.
> > I suspect a bug in itksys::SystemInformation, but i will follow up on
> that
> > separately also.
> >
> > In all itk builds, ITK_USE_REVIEW is on. In the itk3 builds,
> > USE_OPTIMZIZED_REGISTRATION, and the new statistics framework are turned
> on.
> >
> > Cheers,
> > Rupert
> >
> > Data follows.....
> >
> > itk3 Windows XP 32bit Visual Studio Pro 2005 Build RelWithDebInfo
> > System: CAMD5C5PMN1
> > Processor: Pentium III (0.18 micron) With 1 Or 2 MB On-Die L2 Cache
> > Serial #:
> > Cache: -1
> > Clock: 2800
> > Cores: 4 cpus x 1 Cores = 4
> > OSName: Windows
> > Release: XP Professional
> > Version: Service Pack 3 (Build 2600)
> > Platform: x86
> > Operating System is 32 bit
> > ITK Version: 3.20.0
> > Virtual Memory: Total: 2047 Available: 2021
> > Physical Memory: Total:3581 Available: 1457
> > Probe Name: Count Min Mean
> > Stdev Max Total
> > FilterChain_1_threads 10 0.0345688 0.0361805
> > 0.0381523 0.0375099 0.361805
> > FilterChain_2_threads 10 0.0257301 0.0265144
> > 0.0279574 0.0282478 0.265144
> > FilterChain_3_threads 10 0.022007 0.0232452
> > 0.0245125 0.0240822 0.232452
> > FilterChain_4_threads 10 0.0199661 0.0221756
> > 0.0235205 0.0288773 0.221756
> > Image2D 10 0.144909 0.14665
> > 0.154586 0.148136 1.4665
> > Image3D 10 0.19125 0.193378
> > 0.203846 0.196495 1.93378
> > MeanSquares_1_threads 10 0.612728 0.616159
> > 0.649503 0.627789 6.16159
> > MeanSquares_2_threads 10 0.359703 0.368051
> > 0.388106 0.392696 3.68051
> > MeanSquares_3_threads 10 0.268024 0.280672
> > 0.296142 0.312557 2.80672
> > MeanSquares_4_threads 10 0.199249 0.213209
> > 0.224918 0.2243 2.13209
> >
> >
> > itk4 Windows XP 32bit Visual Studio Pro 2005 Build RelWithDebInfo
> >
> > System: CAMD5C5PMN1
> > Processor: Pentium III (0.18 micron) With 1 Or 2 MB On-Die L2 Cache
> > Serial #:
> > Cache: -1
> > Clock: 2800
> > Cores: 4 cpus x 1 Cores = 4
> > OSName: Windows
> > Release: XP Professional
> > Version: Service Pack 3 (Build 2600)
> > Platform: x86
> > Operating System is 32 bit
> > ITK Version: 4.3.0
> > Virtual Memory: Total: 2047 Available: 2008
> > Physical Memory: Total:3581 Available: 1574
> > Probe Name: Count Min Mean
> > Stdev Max Total
> > FilterChain_1_threads 10 0.0797005 0.0808155
> > 0.0851898 0.0820313 0.808155
> > FilterChain_2_threads 10 0.0670509 0.0921925
> > 0.0991914 0.109177 0.921925
> > FilterChain_3_threads 10 0.0648499 0.113566
> > 0.120975 0.125893 1.13566
> > FilterChain_4_threads 10 0.0536919 0.104662
> > 0.114558 0.132114 1.04662
> > Image2D 10 0.0953979 0.0965282
> > 0.101753 0.0978088 0.965282
> > Image3D 10 0.196468 0.198288
> > 0.209017 0.200562 1.98288
> > MeanSquares_1_threads 10 0.982071 0.986089
> > 1.03944 0.996445 9.86089
> > MeanSquares_2_threads 10 0.686028 0.728621
> > 0.768821 0.799675 7.28621
> > MeanSquares_3_threads 10 0.534885 0.554387
> > 0.5846 0.578133 5.54387
> > MeanSquares_4_threads 10 0.441551 0.463463
> > 0.488903 0.502502 4.63463
> >
> >
> > itk3 Linux 64 bit gcc 4.6.1 Build Release
> > System: morrigan
> > Processor: Intel(R) Core(TM)2 Duo CPU T9500 @ 2.60GHz
> > Serial #:
> > Cache: 2048
> > Clock: 2003
> > Cores: 4 cpus x 4 Cores = 16
> > OSName: Linux
> > Release: 3.0.0-21-generic
> > Version: #35-Ubuntu SMP Fri May 25 17:57:41 UTC 2012
> > Platform: x86_64
> > Operating System is 64 bit
> > ITK Version: 3.20.1
> > Virtual Memory: Total: 20795 Available: 20795
> > Physical Memory: Total:8001 Available: 3416
> > Probe Name: Count Min Mean
> > Stdev
> > Max Total
> > FilterChain_1_threads 10 0.0309269 0.031097
> > 0.0327796 0.031528 0.31097
> > FilterChain_2_threads 10 0.021672 0.0248219
> > 0.0261976 0.026264 0.248219
> > FilterChain_3_threads 10 0.0213931 0.0217796
> > 0.0229634 0.0228851 0.217796
> > FilterChain_4_threads 10 0.0198419 0.0207421
> > 0.0218725 0.0216701 0.207421
> > Image2D 10 0.0263629 0.0264159
> > 0.0278451 0.0267591 0.264159
> > Image3D 10 0.0263779 0.0263906
> > 0.0278181 0.026463 0.263906
> > MeanSquares_1_threads 10 0.463909 0.465901
> > 0.491113 0.473644 4.65901
> > MeanSquares_2_threads 10 0.349138 0.40529
> > 0.429438 0.487785 4.0529
> > MeanSquares_3_threads 10 0.331908 0.357685
> > 0.377834 0.400941 3.57685
> > MeanSquares_4_threads 10 0.299428 0.336914
> > 0.356878 0.395144 3.36914
> >
> > itk4 Linux 64-bit gcc 4.6.1 Build Release
> > System: morrigan
> > Processor: Intel(R) Core(TM)2 Duo CPU T9500 @ 2.60GHz
> > Serial #:
> > Cache: 2048
> > Clock: 2003
> > Cores: 4 cpus x 4 Cores = 16
> > OSName: Linux
> > Release: 3.0.0-21-generic
> > Version: #35-Ubuntu SMP Fri May 25 17:57:41 UTC 2012
> > Platform: x86_64
> > Operating System is 64 bit
> > ITK Version: 4.3.0
> > Virtual Memory: Total: 20795 Available: 20795
> > Physical Memory: Total:8001 Available: 3400
> > Probe Name: Count Min Mean
> > Stdev
> > Max Total
> > FilterChain_1_threads 10 0.0663331 0.0668247
> > 0.0704416 0.0681288 0.668247
> > FilterChain_2_threads 10 0.0353 0.0393441
> > 0.042048 0.0534279 0.393441
> > FilterChain_3_threads 10 0.025275 0.0265869
> > 0.0282408 0.0364931 0.265869
> > FilterChain_4_threads 10 0.0208249 0.0222288
> > 0.0235333 0.028336 0.222288
> > Image2D 10 0.026361 0.026421
> > 0.0278503 0.0266371 0.26421
> > Image3D 10 0.0263991 0.0266509
> > 0.0280954 0.0274949 0.266509
> > MeanSquares_1_threads 10 0.69005 0.69459
> > 0.732169 0.701676 6.9459
> > MeanSquares_2_threads 10 0.590396 0.699826
> > 0.739994 0.766702 6.99826
> > MeanSquares_3_threads 10 0.527146 0.538844
> > 0.568047 0.548791 5.38844
> > MeanSquares_4_threads 10 0.417008 0.454278
> > 0.480139 0.533593 4.54278
> > --------------------------------------------------------------
> > Rupert Brooks
> > rupert.brooks at gmail.com
> >
> >
> > _______________________________________________
> > Powered by www.kitware.com
> >
> > Visit other Kitware open-source projects at
> > http://www.kitware.com/opensource/opensource.html
> >
> > Kitware offers ITK Training Courses, for more information visit:
> > http://kitware.com/products/protraining.php
> >
> > Please keep messages on-topic and check the ITK FAQ at:
> > http://www.itk.org/Wiki/ITK_FAQ
> >
> > Follow this link to subscribe/unsubscribe:
> > http://www.itk.org/mailman/listinfo/insight-developers
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.itk.org/pipermail/insight-developers/attachments/20120725/fe69e56e/attachment-0001.htm>
More information about the Insight-developers
mailing list