[Cdash] CDash submission problem

Wed Nov 26 14:51:26 UTC 2008

Wow! this is just great.

I have actually started to work on the SAX parser to backup/restore 
CDash submissions, but didn't have a chance to modify the main parsing. 
I'll review your modifications (hopefully next week) and see how it 
goes. Regarding the submission time, it might actually take less time to 
submit since the submission starts right when the streaming begins 
(without having to wait for the complete XML to be uploaded).

On a related note. We are modifying CDash to use a MVC 
(model/view/controller) approach, which should make things easier for 
developers in the long run.

Thanks again for providing the patch,
Julien

Matthias Brantner wrote:
> Hello Julien, hello Emmanuel,
> 
> some time has passed but we finally managed to come up with a patch that 
> changes the CDash submission mechanism. William Candillon (in CC) is a 
> colleague of mine at 28msec, Inc. He made the following two changes to 
> CDash:
> 
> (1) remove the current test submission parse mechanism that stores all 
> tests in an array
> (2) instead there is one sax handler for each test submission (e.g. 
> Build, Testing, Coverage). As a result, the incoming data is fully 
> streamed and there is no need to load the complete input.
> 
> We have carefully tested the changes in our environment (approx. 16000 
> tests per submission) and with the test scenarios that you provide (see 
> http://public.kitware.com/Wiki/CDash:XML). Everything seems to work fine 
> and we were able to make a submission with ca. 16000 tests that required 
> less then 8MB of memory. However, we had to pay the price that a 
> submission might take slightly longer (but we think it's worth it 
> because we are now able to reliably process our huge submissions).
> 
> We created a diff against the current svn head (Rev. 1267). You can find 
> the diff at http://www.28msec.com/CDashSAX_rev1267.diff.
> 
> Could you please review this patch (if you find the time) and let us 
> know what you think. As we have seen in your FAQ, there seem to be a lot 
> of people that have the same problem as we had. Hence, it would be great 
> if you could consider to integrate our changes in a next CDash release.
> 
> Best regards,
> 
> Matthias
> 
> 
> On 07.10.2008, at 17:39, Julien Jomier wrote:
> 
>> Matthias,
>>
>> We have seen this behavior for very large submission and since tests 
>> are not split by CTest the submission can be very big.
>>
>> I think we can improve this by customizing the XML parser in PHP and 
>> instead of storing in an array, do the processing at parsing time (as 
>> you are suggesting).
>>
>> If you have any experience with this, feel free to share :)
>>
>> I'll look at this when I get a chance. Thanks for the input,
>> Julien
>>
>> Matthias Brantner wrote:
>>> Julien,
>>> unfortunately, I also encountered this problem for our project.
>>> We have several nightly submissions each having approx. 15000 tests.
>>> I already changed the PHP variables to
>>> max_execution_time = 1500;
>>> max_input_time = 1500;
>>> memory_limit = 1000M;
>>> With these settings the submission is sometimes parsed successfully. 
>>> However,
>>> it does not always work.
>>> I looked at the code and I've seen that the full submission is stored 
>>> in an array
>>> which is afterwards written into the database. Would it be an option 
>>> to use a SAX parser and
>>> immediately write each submission into the database, i.e. without 
>>> creating the big array but do
>>> it in a streaming fashion? I think this could dramatically reduce the 
>>> submission time and the
>>> main memory usage.
>>> Maybe our test submissions are a to big for the scenarios CDash was 
>>> designed for.
>>> What do you think?
>>> Thanks in advance.
>>> Matthias
>>> On 07.10.2008, at 13:46, Julien Jomier wrote:
>>>> Emmanuel,
>>>>
>>>> You should increase the memory requirement for PHP otherwise it runs 
>>>> out of memory when trying to parse large submission:
>>>>
>>>> Try changing these variables in your php.ini file (and restart the 
>>>> web server)
>>>>
>>>> max_execution_time = 30;
>>>> max_input_time = 60;
>>>> memory_limit = 200M;
>>>>
>>>> http://public.kitware.com/Wiki/CDash:FAQ#PHP_cannot_parse_my_submission
>>>>
>>>> Let us know if that was the problem,
>>>> Julien
>>>>
>>>> Emmanuel Christophe wrote:
>>>>> Hi,
>>>>> I'm trying to set up a cdash dashboard. It seems to be working pretty
>>>>> well for small submissions (*-Experimental_Test.xml about 1.5 Mo of
>>>>> size) as I can see
>>>>> http://www.orfeo-toolbox.org/Dashboard/index.php?project=OTB&date=20081006 
>>>>>
>>>>> for the test done at 14:12:57 CEST. This test is done without the
>>>>> DATA_ROOT set for the test data, resulting in many fails but shorter
>>>>> xml file.
>>>>> However, when I set properly the DATA_ROOT, the xml file is about 6 Mo
>>>>> and something is wrong during the submission: as you can see on the
>>>>> 14:33:20 CEST test, no test result appear.
>>>>> I add a similar problem at the beginning, with some error in apache
>>>>> log showing a lack of memory for php to parse the xml file and the
>>>>> *-Experimental_Test.xml not appearing in the backup directory. After
>>>>> increasing the memory allowed to php, the *-Experimental_Test.xml file
>>>>> now does appear in the backup directory, there is no more error in the
>>>>> apache log, but the test result do not display (even after rerunning
>>>>> the tests).
>>>>> My CTestConfig.cmake is :
>>>>> SET(CTEST_PROJECT_NAME "OTB")
>>>>> SET(CTEST_NIGHTLY_START_TIME "00:00:00 EST")
>>>>> IF(NOT DEFINED CTEST_DROP_METHOD)
>>>>> SET(CTEST_DROP_METHOD "http")
>>>>> ENDIF(NOT DEFINED CTEST_DROP_METHOD)
>>>>> IF(CTEST_DROP_METHOD STREQUAL "http")
>>>>> SET(CTEST_DROP_SITE "www.orfeo-toolbox.org")
>>>>> SET(CTEST_DROP_LOCATION "/Dashboard/submit.php?project=OTB")
>>>>> SET(CTEST_TRIGGER_SITE "")
>>>>> ENDIF(CTEST_DROP_METHOD STREQUAL "http")
>>>>> I'm using CDash-1.0.2.zip, php5, apache2.
>>>>> Any suggestions? Where should I look to find out the source of the 
>>>>> problem?
>>>>> Thanks,
>>>>> Emmanuel
>>>>> _______________________________________________
>>>>> Cdash mailing list
>>>>> Cdash at public.kitware.com
>>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>>>> _______________________________________________
>>>> Cdash mailing list
>>>> Cdash at public.kitware.com
>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
> 
> 