[Cdash] CDash submission problem

[Emmanuel Christophe]

Hi Julien,

xml files are accessible at
http://www.orfeo-toolbox.org/Dashboard/backup/ as well as the
cdash.log
Files from the 08-Oct-2008 23:39 are the one posing problem (as shown
herehttp://www.orfeo-toolbox.org/Dashboard/index.php?project=OTB&date=20081008
).

I realize the size went up to 12 MB this time (after pushing the limit
from 128M to 200M ?). I'll try to increase the limit again but it
might be a much better solution to change the parsing strategy.

I haven't noticed it before but the backup folder is cleaned regurlarly?

Let me know if you find something wrong in the xml files and/or if you
have any suggestion to improve the test process.

Thanks,
Emmanuel

2008/10/10 Julien Jomier <julien.jomier at kitware.com>:
> Emmanuel,
>
> Do you think you can put the XML files somewhere so I can test them?. One
> other option is too look at the backup/cdash.log to see if there is any
> suspicious entries.
>
> Thanks again for the report,
> Julien
>
> Emmanuel Christophe wrote:
>>
>> Hi,
>>
>> Took me a while to rerun the test after pushing the limit to 200M but
>> this didn't change anything. I had already increased the limit to 128M
>> few days ago.
>>
>> I'm not convince that the problem is coming from here: when the limit
>> was 16M originally the following error message appeared in the apache
>> log:
>>
>> Fatal error: Allowed memory size of 16777216 bytes exhausted (tried to
>> allocate 64 bytes) in /var/www/Dashboard/common.php on line 235
>>
>> Which is pretty explicit. With the limit set up to 128M or 200M, there
>> is no more such message.
>>
>> What is also significant is that after I raised the limit to 128M, the
>> *-Test.xml file did appear in the backup directory. Given the
>> instructions on the submit.php file:
>>
>> [...]
>> // Parse the XML
>> $xml_array = parse_XML($contents);
>> // Backup the XML file
>> backup_xml_file($xml_array,$contents,$projectid);
>> unset($contents);
>> [...]
>>
>> the xml parsing seems to be OK and reach the backup_xml_file stage at
>> least.
>>
>> Files submitted for the VTK project are slightly over my files (around
>> 6-7 MB) so it should be working.
>>
>> Could the problem be coming from the insertion in the mysql database?
>> Where could I look for more information?
>>
>> Thanks in advance,
>> Emmanuel
>>
>>
>>
>> 2008/10/7 Julien Jomier <julien.jomier at kitware.com>:
>>>
>>> Matthias,
>>>
>>> We have seen this behavior for very large submission and since tests are
>>> not
>>> split by CTest the submission can be very big.
>>>
>>> I think we can improve this by customizing the XML parser in PHP and
>>> instead
>>> of storing in an array, do the processing at parsing time (as you are
>>> suggesting).
>>>
>>> If you have any experience with this, feel free to share :)
>>>
>>> I'll look at this when I get a chance. Thanks for the input,
>>> Julien
>>>
>>> Matthias Brantner wrote:
>>>>
>>>> Julien,
>>>>
>>>> unfortunately, I also encountered this problem for our project.
>>>> We have several nightly submissions each having approx. 15000 tests.
>>>> I already changed the PHP variables to
>>>>
>>>> max_execution_time = 1500;
>>>> max_input_time = 1500;
>>>> memory_limit = 1000M;
>>>>
>>>> With these settings the submission is sometimes parsed successfully.
>>>> However,
>>>> it does not always work.
>>>>
>>>> I looked at the code and I've seen that the full submission is stored in
>>>> an array
>>>> which is afterwards written into the database. Would it be an option to
>>>> use a SAX parser and
>>>> immediately write each submission into the database, i.e. without
>>>> creating
>>>> the big array but do
>>>> it in a streaming fashion? I think this could dramatically reduce the
>>>> submission time and the
>>>> main memory usage.
>>>>
>>>> Maybe our test submissions are a to big for the scenarios CDash was
>>>> designed for.
>>>> What do you think?
>>>>
>>>> Thanks in advance.
>>>>
>>>> Matthias
>>>>
>>>>
>>>> On 07.10.2008, at 13:46, Julien Jomier wrote:
>>>>
>>>>> Emmanuel,
>>>>>
>>>>> You should increase the memory requirement for PHP otherwise it runs
>>>>> out
>>>>> of memory when trying to parse large submission:
>>>>>
>>>>> Try changing these variables in your php.ini file (and restart the web
>>>>> server)
>>>>>
>>>>> max_execution_time = 30;
>>>>> max_input_time = 60;
>>>>> memory_limit = 200M;
>>>>>
>>>>> http://public.kitware.com/Wiki/CDash:FAQ#PHP_cannot_parse_my_submission
>>>>>
>>>>> Let us know if that was the problem,
>>>>> Julien
>>>>>
>>>>> Emmanuel Christophe wrote:
>>>>>>
>>>>>> Hi,
>>>>>> I'm trying to set up a cdash dashboard. It seems to be working pretty
>>>>>> well for small submissions (*-Experimental_Test.xml about 1.5 Mo of
>>>>>> size) as I can see
>>>>>>
>>>>>>
>>>>>> http://www.orfeo-toolbox.org/Dashboard/index.php?project=OTB&date=20081006
>>>>>> for the test done at 14:12:57 CEST. This test is done without the
>>>>>> DATA_ROOT set for the test data, resulting in many fails but shorter
>>>>>> xml file.
>>>>>> However, when I set properly the DATA_ROOT, the xml file is about 6 Mo
>>>>>> and something is wrong during the submission: as you can see on the
>>>>>> 14:33:20 CEST test, no test result appear.
>>>>>> I add a similar problem at the beginning, with some error in apache
>>>>>> log showing a lack of memory for php to parse the xml file and the
>>>>>> *-Experimental_Test.xml not appearing in the backup directory. After
>>>>>> increasing the memory allowed to php, the *-Experimental_Test.xml file
>>>>>> now does appear in the backup directory, there is no more error in the
>>>>>> apache log, but the test result do not display (even after rerunning
>>>>>> the tests).
>>>>>> My CTestConfig.cmake is :
>>>>>> SET(CTEST_PROJECT_NAME "OTB")
>>>>>> SET(CTEST_NIGHTLY_START_TIME "00:00:00 EST")
>>>>>> IF(NOT DEFINED CTEST_DROP_METHOD)
>>>>>>  SET(CTEST_DROP_METHOD "http")
>>>>>> ENDIF(NOT DEFINED CTEST_DROP_METHOD)
>>>>>> IF(CTEST_DROP_METHOD STREQUAL "http")
>>>>>>  SET(CTEST_DROP_SITE "www.orfeo-toolbox.org")
>>>>>>  SET(CTEST_DROP_LOCATION "/Dashboard/submit.php?project=OTB")
>>>>>>  SET(CTEST_TRIGGER_SITE "")
>>>>>> ENDIF(CTEST_DROP_METHOD STREQUAL "http")
>>>>>> I'm using CDash-1.0.2.zip, php5, apache2.
>>>>>> Any suggestions? Where should I look to find out the source of the
>>>>>> problem?
>>>>>> Thanks,
>>>>>> Emmanuel
>>>>>> _______________________________________________
>>>>>> Cdash mailing list
>>>>>> Cdash at public.kitware.com
>>>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>>>>>
>>>>> _______________________________________________
>>>>> Cdash mailing list
>>>>> Cdash at public.kitware.com
>>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>>>>
>>
>