[Cdash] CDash submission problem

[Julien Jomier]

Matthias,

We have seen this behavior for very large submission and since tests are 
not split by CTest the submission can be very big.

I think we can improve this by customizing the XML parser in PHP and 
instead of storing in an array, do the processing at parsing time (as 
you are suggesting).

If you have any experience with this, feel free to share :)

I'll look at this when I get a chance. Thanks for the input,
Julien

Matthias Brantner wrote:
> Julien,
> 
> unfortunately, I also encountered this problem for our project.
> We have several nightly submissions each having approx. 15000 tests.
> I already changed the PHP variables to
> 
> max_execution_time = 1500;
> max_input_time = 1500;
> memory_limit = 1000M;
> 
> With these settings the submission is sometimes parsed successfully. 
> However,
> it does not always work.
> 
> I looked at the code and I've seen that the full submission is stored in 
> an array
> which is afterwards written into the database. Would it be an option to 
> use a SAX parser and
> immediately write each submission into the database, i.e. without 
> creating the big array but do
> it in a streaming fashion? I think this could dramatically reduce the 
> submission time and the
> main memory usage.
> 
> Maybe our test submissions are a to big for the scenarios CDash was 
> designed for.
> What do you think?
> 
> Thanks in advance.
> 
> Matthias
> 
> 
> On 07.10.2008, at 13:46, Julien Jomier wrote:
> 
>> Emmanuel,
>>
>> You should increase the memory requirement for PHP otherwise it runs 
>> out of memory when trying to parse large submission:
>>
>> Try changing these variables in your php.ini file (and restart the web 
>> server)
>>
>> max_execution_time = 30;
>> max_input_time = 60;
>> memory_limit = 200M;
>>
>> http://public.kitware.com/Wiki/CDash:FAQ#PHP_cannot_parse_my_submission
>>
>> Let us know if that was the problem,
>> Julien
>>
>> Emmanuel Christophe wrote:
>>> Hi,
>>> I'm trying to set up a cdash dashboard. It seems to be working pretty
>>> well for small submissions (*-Experimental_Test.xml about 1.5 Mo of
>>> size) as I can see
>>> http://www.orfeo-toolbox.org/Dashboard/index.php?project=OTB&date=20081006 
>>>
>>> for the test done at 14:12:57 CEST. This test is done without the
>>> DATA_ROOT set for the test data, resulting in many fails but shorter
>>> xml file.
>>> However, when I set properly the DATA_ROOT, the xml file is about 6 Mo
>>> and something is wrong during the submission: as you can see on the
>>> 14:33:20 CEST test, no test result appear.
>>> I add a similar problem at the beginning, with some error in apache
>>> log showing a lack of memory for php to parse the xml file and the
>>> *-Experimental_Test.xml not appearing in the backup directory. After
>>> increasing the memory allowed to php, the *-Experimental_Test.xml file
>>> now does appear in the backup directory, there is no more error in the
>>> apache log, but the test result do not display (even after rerunning
>>> the tests).
>>> My CTestConfig.cmake is :
>>> SET(CTEST_PROJECT_NAME "OTB")
>>> SET(CTEST_NIGHTLY_START_TIME "00:00:00 EST")
>>> IF(NOT DEFINED CTEST_DROP_METHOD)
>>>  SET(CTEST_DROP_METHOD "http")
>>> ENDIF(NOT DEFINED CTEST_DROP_METHOD)
>>> IF(CTEST_DROP_METHOD STREQUAL "http")
>>>  SET(CTEST_DROP_SITE "www.orfeo-toolbox.org")
>>>  SET(CTEST_DROP_LOCATION "/Dashboard/submit.php?project=OTB")
>>>  SET(CTEST_TRIGGER_SITE "")
>>> ENDIF(CTEST_DROP_METHOD STREQUAL "http")
>>> I'm using CDash-1.0.2.zip, php5, apache2.
>>> Any suggestions? Where should I look to find out the source of the 
>>> problem?
>>> Thanks,
>>> Emmanuel
>>> _______________________________________________
>>> Cdash mailing list
>>> Cdash at public.kitware.com
>>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>> _______________________________________________
>> Cdash mailing list
>> Cdash at public.kitware.com
>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
> 
>