[Cdash] Upload to CDash fails for huge tests amount

Fri May 23 12:31:36 UTC 2014

Hi Miroslav,

Good catch. I have pushed your fix for monitor.php.

Julien

On 20/05/2014 10:22, Matějů Miroslav, Ing. wrote:
> Hi Julien,
>
> I’ve looked into CDash code and discovered following:
>
> * This variable CDASH_BASE_URL is not used at all in the function do_submit_asynchronous()
>    because of the hardcoded parameter true in the call of get_server_URI()
>    (cdash/do_submit.php, 161).
>
> * The variable CDASH_CURL_LOCALHOST_PREFIX seems not to be needed
>    as it is deduced from $_SERVER['REQUEST_URI'] which works well.
>
> * Server name can be guessed too, I only need to set CDASH_CURL_REQUEST_LOCALHOST to false.
>
> Using this I got the asynchronous submissions to work.
> My final config file, after removing unnecessary definitions, contains this:
>
> // Using HTTPS protocol to access CDash
> $CDASH_USE_HTTPS ='0';
> // Name of the server running CDash.
> // Leave empty to use current name and default port.
> $CDASH_SERVER_NAME = '';
> $CDASH_SERVER_PORT = '';
> // If the remote request should use localhost or the full name
> // This variable should be set to 1 in most of the server configurations
> $CDASH_CURL_REQUEST_LOCALHOST='0';
> $CDASH_CURL_LOCALHOST_PREFIX='';
> $CDASH_BASE_URL='';
>
> By the way, as I’ve used monitor.php to check the asynchronous submissions:
> The time_local shown in monitor.php looks a bit meaningless for someone
> not located in America. I found it’s hardcoded to UTC-4.
> I changed it to SYSTEM timezone and it looks much butter.
>
> --- monitor.php	Thu Mar 01 09:06:54 2012
> +++ monitor.php	Tue May 20 09:26:00 2014
> @@ -105,7 +105,7 @@ function echo_pending_submissions()
>   function echo_average_wait_time($projectid)
>   {
>     $sql_query = "SELECT TIMESTAMPDIFF(HOUR, created, UTC_TIMESTAMP) as hours_ago, ".
> -    "TIME_FORMAT(CONVERT_TZ(created, '+00:00', '-04:00'), '%l:00 %p') AS time_local, ".
> +    "TIME_FORMAT(CONVERT_TZ(created, '+00:00', 'SYSTEM'), '%l:00 %p') AS time_local, ".
>       "COUNT(created) AS num_files, ".
>       "ROUND(AVG(TIMESTAMPDIFF(SECOND, created, started))/3600, 1) AS avg_hours_delay, ".
>       "AVG(TIMESTAMPDIFF(SECOND, started, finished)) AS mean, ".
>
> This feature seems to be present in all versions of MySQL according to its manual,
> so its usage should be safe.
>
> Miroslav
>
> -----Original Message-----
> From: Julien Jomier [mailto:julien.jomier at kitware.com]
> Sent: Monday, May 19, 2014 4:03 PM
>
> Hi Miroslav,
>
> I agree with you, we need to upgrade the documentation.
>
> The best is to use the CDASH_BASE_URL variable. This should point to your CDash URL (where the index.php is called), so something like:
>
> $CDASH_BASE_URL='192.168.2.105/CDash';
>
> Julien
>
> On 19/05/2014 15:48, Matějů Miroslav, Ing. wrote:
>> Hi Julien,
>>
>> Thanks for your advice. I have set CDASH_ASYNCHRONOUS_SUBMISSION to true and my submissions are processed correctly now – but only after running cdash/processsubmissions.php manually (following http://public.kitware.com/pipermail/cdash/2014-January/001405.html). However, the processing does not start automatically. I tried to find some documentation about CDash Asynchronous submissions with no success. The only useful page is http://www.kitware.com/source/home/post/59 but it does not mention what (apart from the config file) should I do to process the asynchronous submissions. I’m supposing it should work automatically (without setting cron, for example) after reading the mail archive mentioned above.
>>
>> The server configuration does not allow to access localhost (or 127.0.0.1) using HTTP, I have to use the local IP address to access webpages. So I’ve set the following values in cdash/config.local.php:
>>
>> $CDASH_USE_HTTPS ='0';
>> // Name of the server running CDash.
>> // Leave empty to use current name and default port.
>> $CDASH_SERVER_NAME = '192.168.2.105';
>> $CDASH_SERVER_PORT = '';
>> // If the remote request should use localhost or the full name // This
>> variable should be set to 1 in most of the server configurations
>> $CDASH_CURL_REQUEST_LOCALHOST='0';
>> $CDASH_CURL_LOCALHOST_PREFIX='cdash';
>> $CDASH_BASE_URL='';
>> // Define the location of the local directory
>> $CDASH_USE_LOCAL_DIRECTORY = '0';
>>
>> Is that correct? I miss any documentation of these variables except the comments (which do not help when I’m not the “most”). I am not sure with CDASH_CURL_LOCALHOST_PREFIX, the full path to process submissions manually is http://192.168.2.105/cdash/cdash/processsubmissions.php.
>>
>> Best regards,
>>
>> Miroslav Matějů
>>
>> -----Original Message-----
>> From: Julien Jomier [mailto:julien.jomier at kitware.com]
>> Sent: Thursday, May 15, 2014 9:29 AM
>>
>> Hi Miroslav,
>>
>> Are you running CDash with Asynchronous submissions?
>> ($CDASH_ASYNCHRONOUS_SUBMISSION=true in the config.local.php)
>>
>> If not that might be the issue,
>> Julien
>>
>> On 15/05/2014 09:16, Matějů Miroslav, Ing. wrote:
>>> Hello there,
>>>
>>> I’ve got a problem with the upload from CTest to CDash. I have many
>>> test suites (considered as named builds for CDash), each of them with
>>> several tests. The update info is not transferred to CDash for some
>>> test suites with a huge amount of tests. To give you an idea what “huge amount”
>>> means: There is currently being a transfer to new hardware for both
>>> the CTest and CDash machines. For the older HW, it means more than
>>> 1000, and about 10000 for the newer one.
>>>
>>> Some useful info can be found in LastSubmit_*.log on the CTest
>>> machine (I shortened it where I put “< >” marks):
>>>
>>> Upload files:
>>>
>>> 0       Build.xml
>>>
>>> 1       Configure.xml
>>>
>>> 2       Coverage.xml
>>>
>>> 3       CoverageLog-0.xml
>>>
>>> 4       DynamicAnalysis.xml
>>>
>>> 5       Notes.xml
>>>
>>> 6       Test.xml
>>>
>>> 7       Update.xml
>>>
>>> Using drop method: http
>>>
>>>            Upload file: <0 to 5>
>>>
>>>            Upload file: /<mypath>/Testing/20140514-0000/Test.xml to
>>> <machine>___<build>___20140514-0000-Nightly___XML___Test.xml
>>>
>>>       Error when uploading file:
>>> /<mypath>/Testing/20140514-0000/Test.xml
>>>
>>>       Error message was: Operation too slow. Less than 1 bytes/sec
>>> transferred the last 120 seconds
>>>
>>>       Problems when submitting via HTTP
>>>
>>> <END OF FILE>
>>>
>>> Although CTest complains about sending Test.xml, it seems to be
>>> transferred completely. The tests (including the last ones) present
>>> in this file are shown correctly in CDash as well as configure,
>>> build, coverage etc. However, it does not seem to even try to upload
>>> Update.xml and there is no update info shown in CDash accordingly.
>>>
>>> I tried to enable repeated sending to CDash using
>>>
>>> CTEST_SUBMIT(RETRY_COUNT 10)
>>>
>>> in the CTest script. The only result was an overloaded CDash server
>>> and even more upload errors.
>>>
>>> Could you give me any hints what can I do to correct this issue? Or
>>> is it a bug which has to be corrected in the CDash itself? I’m
>>> running CTest 2.8.7 and sending results to CDash 2.0.2.
>>>
>>> Thanks in advance!
>>>
>>> Miroslav Matějů
>>>
>>>
>>>
>>> _______________________________________________
>>> Cdash mailing list
>>> Cdash at public.kitware.com
>>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>>>
>> _______________________________________________
>> Cdash mailing list
>> Cdash at public.kitware.com
>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>>
> _______________________________________________
> Cdash mailing list
> Cdash at public.kitware.com
> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>