[Insight-developers] Kitware Robots still appear to be dead

Sat Oct 1 11:55:41 EDT 2011

Is there a way to profile CTest? Are we certain that the delays we see
are due to test output length? Do we even know if the test output
length makes any difference at all?

On Sat, Oct 1, 2011 at 11:54 AM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
> Funny, two of the tests that I sometimes lookat are culprits:
> itkSystemInformationTest - this dumps the CMakeCache.txt file and others
> itkNumericLimitsTest
>
> On Sat, Oct 1, 2011 at 11:35 AM, David Cole <david.cole at kitware.com> wrote:
>> These 11 tests have more than 99,999 bytes of output when passing:
>> (the "len=" part of the following strings...)
>>
>> $ grep -E "len=[0-9][0-9][0-9][0-9][0-9][0-9]"
>> /c/xampp/htdocs/CDash-trunk/backup/cdash.log
>> [2011-10-01T11:24:47][INFO][pid=2680](end_timer):
>> Test->Insert(name=vnl_test_alignment,len=405582), 0.014 seconds
>> [2011-10-01T11:24:48][INFO][pid=2680](end_timer):
>> Test->Insert(name=itkImageRegionExclusionIteratorWithIndexTest,len=356615),
>> 0.013 seconds
>> [2011-10-01T11:24:48][INFO][pid=2680](end_timer):
>> Test->Insert(name=itkNumericTraitsTest,len=399485), 0.016 seconds
>> [2011-10-01T11:24:48][INFO][pid=2680](end_timer):
>> Test->Insert(name=itkSliceIteratorTest,len=142697), 0.006 seconds
>> [2011-10-01T11:24:48][INFO][pid=2680](end_timer):
>> Test->Insert(name=itkSystemInformationTest,len=957625), 0.035 seconds
>> [2011-10-01T11:24:51][INFO][pid=2680](end_timer):
>> Test->Insert(name=itkCheckerBoardImageFilterTest,len=128040), 0.006
>> seconds
>> [2011-10-01T11:24:52][INFO][pid=2680](end_timer):
>> Test->Insert(name=itkRegionGrow2DTest,len=106984), 0.006 seconds
>> [2011-10-01T11:24:53][INFO][pid=2680](end_timer):
>> Test->Insert(name=itkTriangleMeshToBinaryImageFilterTest2,len=110042),
>>  0.007 seconds
>> [2011-10-01T11:24:54][INFO][pid=2680](end_timer):
>> Test->Insert(name=itkImageRegistrationMethodTest_13,len=144907), 0.016
>>  seconds
>> [2011-10-01T11:24:55][INFO][pid=2680](end_timer):
>> Test->Insert(name=itkSampleToHistogramFilterTest5,len=252555), 0.019 s
>> econds
>> [2011-10-01T11:24:55][INFO][pid=2680](end_timer):
>> Test->Insert(name=itkSampleToHistogramFilterTest4,len=1000093), 0.032
>> seconds
>>
>>
>> There are ~150 additional tests that have more than 9,999 bytes of output...
>> And there are ~850 more that have more than 999...
>>
>> So.... Literally HALF of the ITK tests are sending more output than
>> the other projects hosted on the CDash server.
>>
>> Combine that with the fact that ITK has far more tests than any other
>> project, and it is not really surprising that ITK is the first project
>> to have processing problems when many, many builds are being thrown at
>> the CDash server.
>>
>>
>>
>> On Sat, Oct 1, 2011 at 10:46 AM, Bradley Lowekamp
>> <blowekamp at mail.nih.gov> wrote:
>>>
>>> As this delay appears to have gotten worse this week. Is there an easy way to check which tests are outputting excessive amounts of information when they are not failing?
>>>
>>> On Oct 1, 2011, at 9:55 AM, David Cole wrote:
>>>
>>>> On Fri, Sep 30, 2011 at 1:24 PM, Bill Hoffman <bill.hoffman at kitware.com> wrote:
>>>>> On 9/29/2011 10:19 PM, Johnson, Hans J wrote:
>>>>>>
>>>>>> Is there an ETA on when the Gerrit Kitware Robots are going to start
>>>>>> processing patches again?
>>>>>>
>>>>>
>>>>> OK, the robots are not really dead!  They are actually alive and well (long
>>>>> live the robots).  However, CDash is falling behind again in ITK.  Right now
>>>>> it seems to be about 5 hours behind in processing data.
>>>>>
>>>>> Couple of things:
>>>>>
>>>>> http://review.source.kitware.com/#change,2610
>>>>>
>>>>> There is a bug in CDash that makes these links not work, if CDash is behind:
>>>>>
>>>>> Linux build: http://www.cdash.org/CDash/scheduleSummary.php?scheduleid=9760
>>>>>
>>>>> Mac build: http://www.cdash.org/CDash/scheduleSummary.php?scheduleid=9761
>>>>>
>>>>> Windows build:
>>>>> http://www.cdash.org/CDash/scheduleSummary.php?scheduleid=9762
>>>>>
>>>>>
>>>>> However, when the build is done, it will show up here:
>>>>>
>>>>>
>>>>> CDash filtered page:
>>>>> http://www.cdash.org/CDash/index.php?project=Insight&filtercount=2&field1=buildname/string&compare1=63&value1=confocal_seg-4&field2=buildstarttime/date&compare2=83&value2=2011-9-30
>>>>>
>>>>>
>>>>> One thing that might be causing this problem with ITK, is this:
>>>>>
>>>>> SET(CTEST_CUSTOM_MAXIMUM_PASSED_TEST_OUTPUT_SIZE 1000000)
>>>>>
>>>>> The default for CTest is 1K of output per passed test.  So, ITK is an
>>>>> outlier.
>>>>>
>>>>> -Bill
>>>>> _______________________________________________
>>>>> Powered by www.kitware.com
>>>>>
>>>>> Visit other Kitware open-source projects at
>>>>> http://www.kitware.com/opensource/opensource.html
>>>>>
>>>>> Kitware offers ITK Training Courses, for more information visit:
>>>>> http://kitware.com/products/protraining.html
>>>>>
>>>>> Please keep messages on-topic and check the ITK FAQ at:
>>>>> http://www.itk.org/Wiki/ITK_FAQ
>>>>>
>>>>> Follow this link to subscribe/unsubscribe:
>>>>> http://www.itk.org/mailman/listinfo/insight-developers
>>>>>
>>>>
>>>> I have eliminated this line:
>>>>
>>>>  SET(CTEST_CUSTOM_MAXIMUM_PASSED_TEST_OUTPUT_SIZE 1000000)
>>>>
>>>> in the latest patch set of this gerrit change:
>>>>
>>>>  http://review.source.kitware.com/#change,3015
>>>>
>>>> Please review and approve, and I'll get it merged in.
>>>>
>>>>
>>>> Also: the ITK CDash dashboard is all caught up on its processing as of
>>>> 1:00 am this morning. Things are current as of right now.
>>>>
>>>>
>>>> Thanks,
>>>> David
>>>> _______________________________________________
>>>> Powered by www.kitware.com
>>>>
>>>> Visit other Kitware open-source projects at
>>>> http://www.kitware.com/opensource/opensource.html
>>>>
>>>> Kitware offers ITK Training Courses, for more information visit:
>>>> http://kitware.com/products/protraining.html
>>>>
>>>> Please keep messages on-topic and check the ITK FAQ at:
>>>> http://www.itk.org/Wiki/ITK_FAQ
>>>>
>>>> Follow this link to subscribe/unsubscribe:
>>>> http://www.itk.org/mailman/listinfo/insight-developers
>>>
>>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Kitware offers ITK Training Courses, for more information visit:
>> http://kitware.com/products/protraining.html
>>
>> Please keep messages on-topic and check the ITK FAQ at:
>> http://www.itk.org/Wiki/ITK_FAQ
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.itk.org/mailman/listinfo/insight-developers
>>
>