[Cdash] CDash submission problem

[Matthias Brantner]

Hello Julien, hello Emmanuel,

some time has passed but we finally managed to come up with a patch  
that changes the CDash submission mechanism. William Candillon (in CC)  
is a colleague of mine at 28msec, Inc. He made the following two  
changes to CDash:

(1) remove the current test submission parse mechanism that stores all  
tests in an array
(2) instead there is one sax handler for each test submission (e.g.  
Build, Testing, Coverage). As a result, the incoming data is fully  
streamed and there is no need to load the complete input.

We have carefully tested the changes in our environment (approx. 16000  
tests per submission) and with the test scenarios that you provide  
(see http://public.kitware.com/Wiki/CDash:XML). Everything seems to  
work fine and we were able to make a submission with ca. 16000 tests  
that required less then 8MB of memory. However, we had to pay the  
price that a submission might take slightly longer (but we think it's  
worth it because we are now able to reliably process our huge  
submissions).

We created a diff against the current svn head (Rev. 1267). You can  
find the diff at http://www.28msec.com/CDashSAX_rev1267.diff.

Could you please review this patch (if you find the time) and let us  
know what you think. As we have seen in your FAQ, there seem to be a  
lot of people that have the same problem as we had. Hence, it would be  
great if you could consider to integrate our changes in a next CDash  
release.

Best regards,

Matthias


On 07.10.2008, at 17:39, Julien Jomier wrote:

> Matthias,
>
> We have seen this behavior for very large submission and since tests  
> are not split by CTest the submission can be very big.
>
> I think we can improve this by customizing the XML parser in PHP and  
> instead of storing in an array, do the processing at parsing time  
> (as you are suggesting).
>
> If you have any experience with this, feel free to share :)
>
> I'll look at this when I get a chance. Thanks for the input,
> Julien
>
> Matthias Brantner wrote:
>> Julien,
>> unfortunately, I also encountered this problem for our project.
>> We have several nightly submissions each having approx. 15000 tests.
>> I already changed the PHP variables to
>> max_execution_time = 1500;
>> max_input_time = 1500;
>> memory_limit = 1000M;
>> With these settings the submission is sometimes parsed  
>> successfully. However,
>> it does not always work.
>> I looked at the code and I've seen that the full submission is  
>> stored in an array
>> which is afterwards written into the database. Would it be an  
>> option to use a SAX parser and
>> immediately write each submission into the database, i.e. without  
>> creating the big array but do
>> it in a streaming fashion? I think this could dramatically reduce  
>> the submission time and the
>> main memory usage.
>> Maybe our test submissions are a to big for the scenarios CDash was  
>> designed for.
>> What do you think?
>> Thanks in advance.
>> Matthias
>> On 07.10.2008, at 13:46, Julien Jomier wrote:
>>> Emmanuel,
>>>
>>> You should increase the memory requirement for PHP otherwise it  
>>> runs out of memory when trying to parse large submission:
>>>
>>> Try changing these variables in your php.ini file (and restart the  
>>> web server)
>>>
>>> max_execution_time = 30;
>>> max_input_time = 60;
>>> memory_limit = 200M;
>>>
>>> http://public.kitware.com/Wiki/CDash:FAQ#PHP_cannot_parse_my_submission
>>>
>>> Let us know if that was the problem,
>>> Julien
>>>
>>> Emmanuel Christophe wrote:
>>>> Hi,
>>>> I'm trying to set up a cdash dashboard. It seems to be working  
>>>> pretty
>>>> well for small submissions (*-Experimental_Test.xml about 1.5 Mo of
>>>> size) as I can see
>>>> http://www.orfeo-toolbox.org/Dashboard/index.php?project=OTB&date=20081006
>>>> for the test done at 14:12:57 CEST. This test is done without the
>>>> DATA_ROOT set for the test data, resulting in many fails but  
>>>> shorter
>>>> xml file.
>>>> However, when I set properly the DATA_ROOT, the xml file is about  
>>>> 6 Mo
>>>> and something is wrong during the submission: as you can see on the
>>>> 14:33:20 CEST test, no test result appear.
>>>> I add a similar problem at the beginning, with some error in apache
>>>> log showing a lack of memory for php to parse the xml file and the
>>>> *-Experimental_Test.xml not appearing in the backup directory.  
>>>> After
>>>> increasing the memory allowed to php, the *-Experimental_Test.xml  
>>>> file
>>>> now does appear in the backup directory, there is no more error  
>>>> in the
>>>> apache log, but the test result do not display (even after  
>>>> rerunning
>>>> the tests).
>>>> My CTestConfig.cmake is :
>>>> SET(CTEST_PROJECT_NAME "OTB")
>>>> SET(CTEST_NIGHTLY_START_TIME "00:00:00 EST")
>>>> IF(NOT DEFINED CTEST_DROP_METHOD)
>>>> SET(CTEST_DROP_METHOD "http")
>>>> ENDIF(NOT DEFINED CTEST_DROP_METHOD)
>>>> IF(CTEST_DROP_METHOD STREQUAL "http")
>>>> SET(CTEST_DROP_SITE "www.orfeo-toolbox.org")
>>>> SET(CTEST_DROP_LOCATION "/Dashboard/submit.php?project=OTB")
>>>> SET(CTEST_TRIGGER_SITE "")
>>>> ENDIF(CTEST_DROP_METHOD STREQUAL "http")
>>>> I'm using CDash-1.0.2.zip, php5, apache2.
>>>> Any suggestions? Where should I look to find out the source of  
>>>> the problem?
>>>> Thanks,
>>>> Emmanuel
>>>> _______________________________________________
>>>> Cdash mailing list
>>>> Cdash at public.kitware.com
>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>>> _______________________________________________
>>> Cdash mailing list
>>> Cdash at public.kitware.com
>>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash