[Cdash] Upload to CDash fails for huge tests amount
julien.jomier at kitware.com
Fri May 23 12:31:36 UTC 2014
Good catch. I have pushed your fix for monitor.php.
On 20/05/2014 10:22, Matějů Miroslav, Ing. wrote:
> Hi Julien,
> I’ve looked into CDash code and discovered following:
> * This variable CDASH_BASE_URL is not used at all in the function do_submit_asynchronous()
> because of the hardcoded parameter true in the call of get_server_URI()
> (cdash/do_submit.php, 161).
> * The variable CDASH_CURL_LOCALHOST_PREFIX seems not to be needed
> as it is deduced from $_SERVER['REQUEST_URI'] which works well.
> * Server name can be guessed too, I only need to set CDASH_CURL_REQUEST_LOCALHOST to false.
> Using this I got the asynchronous submissions to work.
> My final config file, after removing unnecessary definitions, contains this:
> // Using HTTPS protocol to access CDash
> $CDASH_USE_HTTPS ='0';
> // Name of the server running CDash.
> // Leave empty to use current name and default port.
> $CDASH_SERVER_NAME = '';
> $CDASH_SERVER_PORT = '';
> // If the remote request should use localhost or the full name
> // This variable should be set to 1 in most of the server configurations
> By the way, as I’ve used monitor.php to check the asynchronous submissions:
> The time_local shown in monitor.php looks a bit meaningless for someone
> not located in America. I found it’s hardcoded to UTC-4.
> I changed it to SYSTEM timezone and it looks much butter.
> --- monitor.php Thu Mar 01 09:06:54 2012
> +++ monitor.php Tue May 20 09:26:00 2014
> @@ -105,7 +105,7 @@ function echo_pending_submissions()
> function echo_average_wait_time($projectid)
> $sql_query = "SELECT TIMESTAMPDIFF(HOUR, created, UTC_TIMESTAMP) as hours_ago, ".
> - "TIME_FORMAT(CONVERT_TZ(created, '+00:00', '-04:00'), '%l:00 %p') AS time_local, ".
> + "TIME_FORMAT(CONVERT_TZ(created, '+00:00', 'SYSTEM'), '%l:00 %p') AS time_local, ".
> "COUNT(created) AS num_files, ".
> "ROUND(AVG(TIMESTAMPDIFF(SECOND, created, started))/3600, 1) AS avg_hours_delay, ".
> "AVG(TIMESTAMPDIFF(SECOND, started, finished)) AS mean, ".
> This feature seems to be present in all versions of MySQL according to its manual,
> so its usage should be safe.
> -----Original Message-----
> From: Julien Jomier [mailto:julien.jomier at kitware.com]
> Sent: Monday, May 19, 2014 4:03 PM
> Hi Miroslav,
> I agree with you, we need to upgrade the documentation.
> The best is to use the CDASH_BASE_URL variable. This should point to your CDash URL (where the index.php is called), so something like:
> On 19/05/2014 15:48, Matějů Miroslav, Ing. wrote:
>> Hi Julien,
>> Thanks for your advice. I have set CDASH_ASYNCHRONOUS_SUBMISSION to true and my submissions are processed correctly now – but only after running cdash/processsubmissions.php manually (following http://public.kitware.com/pipermail/cdash/2014-January/001405.html). However, the processing does not start automatically. I tried to find some documentation about CDash Asynchronous submissions with no success. The only useful page is http://www.kitware.com/source/home/post/59 but it does not mention what (apart from the config file) should I do to process the asynchronous submissions. I’m supposing it should work automatically (without setting cron, for example) after reading the mail archive mentioned above.
>> The server configuration does not allow to access localhost (or 127.0.0.1) using HTTP, I have to use the local IP address to access webpages. So I’ve set the following values in cdash/config.local.php:
>> $CDASH_USE_HTTPS ='0';
>> // Name of the server running CDash.
>> // Leave empty to use current name and default port.
>> $CDASH_SERVER_NAME = '192.168.2.105';
>> $CDASH_SERVER_PORT = '';
>> // If the remote request should use localhost or the full name // This
>> variable should be set to 1 in most of the server configurations
>> // Define the location of the local directory
>> $CDASH_USE_LOCAL_DIRECTORY = '0';
>> Is that correct? I miss any documentation of these variables except the comments (which do not help when I’m not the “most”). I am not sure with CDASH_CURL_LOCALHOST_PREFIX, the full path to process submissions manually is http://192.168.2.105/cdash/cdash/processsubmissions.php.
>> Best regards,
>> Miroslav Matějů
>> -----Original Message-----
>> From: Julien Jomier [mailto:julien.jomier at kitware.com]
>> Sent: Thursday, May 15, 2014 9:29 AM
>> Hi Miroslav,
>> Are you running CDash with Asynchronous submissions?
>> ($CDASH_ASYNCHRONOUS_SUBMISSION=true in the config.local.php)
>> If not that might be the issue,
>> On 15/05/2014 09:16, Matějů Miroslav, Ing. wrote:
>>> Hello there,
>>> I’ve got a problem with the upload from CTest to CDash. I have many
>>> test suites (considered as named builds for CDash), each of them with
>>> several tests. The update info is not transferred to CDash for some
>>> test suites with a huge amount of tests. To give you an idea what “huge amount”
>>> means: There is currently being a transfer to new hardware for both
>>> the CTest and CDash machines. For the older HW, it means more than
>>> 1000, and about 10000 for the newer one.
>>> Some useful info can be found in LastSubmit_*.log on the CTest
>>> machine (I shortened it where I put “< >” marks):
>>> Upload files:
>>> 0 Build.xml
>>> 1 Configure.xml
>>> 2 Coverage.xml
>>> 3 CoverageLog-0.xml
>>> 4 DynamicAnalysis.xml
>>> 5 Notes.xml
>>> 6 Test.xml
>>> 7 Update.xml
>>> Using drop method: http
>>> Upload file: <0 to 5>
>>> Upload file: /<mypath>/Testing/20140514-0000/Test.xml to
>>> Error when uploading file:
>>> Error message was: Operation too slow. Less than 1 bytes/sec
>>> transferred the last 120 seconds
>>> Problems when submitting via HTTP
>>> <END OF FILE>
>>> Although CTest complains about sending Test.xml, it seems to be
>>> transferred completely. The tests (including the last ones) present
>>> in this file are shown correctly in CDash as well as configure,
>>> build, coverage etc. However, it does not seem to even try to upload
>>> Update.xml and there is no update info shown in CDash accordingly.
>>> I tried to enable repeated sending to CDash using
>>> CTEST_SUBMIT(RETRY_COUNT 10)
>>> in the CTest script. The only result was an overloaded CDash server
>>> and even more upload errors.
>>> Could you give me any hints what can I do to correct this issue? Or
>>> is it a bug which has to be corrected in the CDash itself? I’m
>>> running CTest 2.8.7 and sending results to CDash 2.0.2.
>>> Thanks in advance!
>>> Miroslav Matějů
>>> Cdash mailing list
>>> Cdash at public.kitware.com
>> Cdash mailing list
>> Cdash at public.kitware.com
> Cdash mailing list
> Cdash at public.kitware.com
More information about the CDash