[Cdash] CDash submission problem

[Julien Jomier]

Mathias,

I have reviewed your modifications over the weekend. Very nice work.
I had to change a couple of things to make it work with our current 
client (CTest), but these were minor changes. I have also added the 
support for backup of file when they arrived to the system.

The plan is to switch to using the new model classes. I hope to have 
time to work on this, this week.

Thanks again,
Julien

Matthias Brantner wrote:
> Hello Julien, hello Emmanuel,
> 
> some time has passed but we finally managed to come up with a patch that 
> changes the CDash submission mechanism. William Candillon (in CC) is a 
> colleague of mine at 28msec, Inc. He made the following two changes to 
> CDash:
> 
> (1) remove the current test submission parse mechanism that stores all 
> tests in an array
> (2) instead there is one sax handler for each test submission (e.g. 
> Build, Testing, Coverage). As a result, the incoming data is fully 
> streamed and there is no need to load the complete input.
> 
> We have carefully tested the changes in our environment (approx. 16000 
> tests per submission) and with the test scenarios that you provide (see 
> http://public.kitware.com/Wiki/CDash:XML). Everything seems to work fine 
> and we were able to make a submission with ca. 16000 tests that required 
> less then 8MB of memory. However, we had to pay the price that a 
> submission might take slightly longer (but we think it's worth it 
> because we are now able to reliably process our huge submissions).
> 
> We created a diff against the current svn head (Rev. 1267). You can find 
> the diff at http://www.28msec.com/CDashSAX_rev1267.diff.
> 
> Could you please review this patch (if you find the time) and let us 
> know what you think. As we have seen in your FAQ, there seem to be a lot 
> of people that have the same problem as we had. Hence, it would be great 
> if you could consider to integrate our changes in a next CDash release.
> 
> Best regards,
> 
> Matthias
> 
> 
> On 07.10.2008, at 17:39, Julien Jomier wrote:
> 
>> Matthias,
>>
>> We have seen this behavior for very large submission and since tests 
>> are not split by CTest the submission can be very big.
>>
>> I think we can improve this by customizing the XML parser in PHP and 
>> instead of storing in an array, do the processing at parsing time (as 
>> you are suggesting).
>>
>> If you have any experience with this, feel free to share :)
>>
>> I'll look at this when I get a chance. Thanks for the input,
>> Julien
>>
>> Matthias Brantner wrote:
>>> Julien,
>>> unfortunately, I also encountered this problem for our project.
>>> We have several nightly submissions each having approx. 15000 tests.
>>> I already changed the PHP variables to
>>> max_execution_time = 1500;
>>> max_input_time = 1500;
>>> memory_limit = 1000M;
>>> With these settings the submission is sometimes parsed successfully. 
>>> However,
>>> it does not always work.
>>> I looked at the code and I've seen that the full submission is stored 
>>> in an array
>>> which is afterwards written into the database. Would it be an option 
>>> to use a SAX parser and
>>> immediately write each submission into the database, i.e. without 
>>> creating the big array but do
>>> it in a streaming fashion? I think this could dramatically reduce the 
>>> submission time and the
>>> main memory usage.
>>> Maybe our test submissions are a to big for the scenarios CDash was 
>>> designed for.
>>> What do you think?
>>> Thanks in advance.
>>> Matthias
>>> On 07.10.2008, at 13:46, Julien Jomier wrote:
>>>> Emmanuel,
>>>>
>>>> You should increase the memory requirement for PHP otherwise it runs 
>>>> out of memory when trying to parse large submission:
>>>>
>>>> Try changing these variables in your php.ini file (and restart the 
>>>> web server)
>>>>
>>>> max_execution_time = 30;
>>>> max_input_time = 60;
>>>> memory_limit = 200M;
>>>>
>>>> http://public.kitware.com/Wiki/CDash:FAQ#PHP_cannot_parse_my_submission
>>>>
>>>> Let us know if that was the problem,
>>>> Julien
>>>>
>>>> Emmanuel Christophe wrote:
>>>>> Hi,
>>>>> I'm trying to set up a cdash dashboard. It seems to be working pretty
>>>>> well for small submissions (*-Experimental_Test.xml about 1.5 Mo of
>>>>> size) as I can see
>>>>> http://www.orfeo-toolbox.org/Dashboard/index.php?project=OTB&date=20081006 
>>>>>
>>>>> for the test done at 14:12:57 CEST. This test is done without the
>>>>> DATA_ROOT set for the test data, resulting in many fails but shorter
>>>>> xml file.
>>>>> However, when I set properly the DATA_ROOT, the xml file is about 6 Mo
>>>>> and something is wrong during the submission: as you can see on the
>>>>> 14:33:20 CEST test, no test result appear.
>>>>> I add a similar problem at the beginning, with some error in apache
>>>>> log showing a lack of memory for php to parse the xml file and the
>>>>> *-Experimental_Test.xml not appearing in the backup directory. After
>>>>> increasing the memory allowed to php, the *-Experimental_Test.xml file
>>>>> now does appear in the backup directory, there is no more error in the
>>>>> apache log, but the test result do not display (even after rerunning
>>>>> the tests).
>>>>> My CTestConfig.cmake is :
>>>>> SET(CTEST_PROJECT_NAME "OTB")
>>>>> SET(CTEST_NIGHTLY_START_TIME "00:00:00 EST")
>>>>> IF(NOT DEFINED CTEST_DROP_METHOD)
>>>>> SET(CTEST_DROP_METHOD "http")
>>>>> ENDIF(NOT DEFINED CTEST_DROP_METHOD)
>>>>> IF(CTEST_DROP_METHOD STREQUAL "http")
>>>>> SET(CTEST_DROP_SITE "www.orfeo-toolbox.org")
>>>>> SET(CTEST_DROP_LOCATION "/Dashboard/submit.php?project=OTB")
>>>>> SET(CTEST_TRIGGER_SITE "")
>>>>> ENDIF(CTEST_DROP_METHOD STREQUAL "http")
>>>>> I'm using CDash-1.0.2.zip, php5, apache2.
>>>>> Any suggestions? Where should I look to find out the source of the 
>>>>> problem?
>>>>> Thanks,
>>>>> Emmanuel
>>>>> _______________________________________________
>>>>> Cdash mailing list
>>>>> Cdash at public.kitware.com
>>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>>>> _______________________________________________
>>>> Cdash mailing list
>>>> Cdash at public.kitware.com
>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
> 
>