[Cdash] Upload to CDash fails for huge tests amount
Matějů Miroslav, Ing.
Mateju.Miroslav at azd.cz
Tue May 20 08:22:07 UTC 2014
Hi Julien,
I’ve looked into CDash code and discovered following:
* This variable CDASH_BASE_URL is not used at all in the function do_submit_asynchronous()
because of the hardcoded parameter true in the call of get_server_URI()
(cdash/do_submit.php, 161).
* The variable CDASH_CURL_LOCALHOST_PREFIX seems not to be needed
as it is deduced from $_SERVER['REQUEST_URI'] which works well.
* Server name can be guessed too, I only need to set CDASH_CURL_REQUEST_LOCALHOST to false.
Using this I got the asynchronous submissions to work.
My final config file, after removing unnecessary definitions, contains this:
// Using HTTPS protocol to access CDash
$CDASH_USE_HTTPS ='0';
// Name of the server running CDash.
// Leave empty to use current name and default port.
$CDASH_SERVER_NAME = '';
$CDASH_SERVER_PORT = '';
// If the remote request should use localhost or the full name
// This variable should be set to 1 in most of the server configurations
$CDASH_CURL_REQUEST_LOCALHOST='0';
$CDASH_CURL_LOCALHOST_PREFIX='';
$CDASH_BASE_URL='';
By the way, as I’ve used monitor.php to check the asynchronous submissions:
The time_local shown in monitor.php looks a bit meaningless for someone
not located in America. I found it’s hardcoded to UTC-4.
I changed it to SYSTEM timezone and it looks much butter.
--- monitor.php Thu Mar 01 09:06:54 2012
+++ monitor.php Tue May 20 09:26:00 2014
@@ -105,7 +105,7 @@ function echo_pending_submissions()
function echo_average_wait_time($projectid)
{
$sql_query = "SELECT TIMESTAMPDIFF(HOUR, created, UTC_TIMESTAMP) as hours_ago, ".
- "TIME_FORMAT(CONVERT_TZ(created, '+00:00', '-04:00'), '%l:00 %p') AS time_local, ".
+ "TIME_FORMAT(CONVERT_TZ(created, '+00:00', 'SYSTEM'), '%l:00 %p') AS time_local, ".
"COUNT(created) AS num_files, ".
"ROUND(AVG(TIMESTAMPDIFF(SECOND, created, started))/3600, 1) AS avg_hours_delay, ".
"AVG(TIMESTAMPDIFF(SECOND, started, finished)) AS mean, ".
This feature seems to be present in all versions of MySQL according to its manual,
so its usage should be safe.
Miroslav
-----Original Message-----
From: Julien Jomier [mailto:julien.jomier at kitware.com]
Sent: Monday, May 19, 2014 4:03 PM
Hi Miroslav,
I agree with you, we need to upgrade the documentation.
The best is to use the CDASH_BASE_URL variable. This should point to your CDash URL (where the index.php is called), so something like:
$CDASH_BASE_URL='192.168.2.105/CDash';
Julien
On 19/05/2014 15:48, Matějů Miroslav, Ing. wrote:
> Hi Julien,
>
> Thanks for your advice. I have set CDASH_ASYNCHRONOUS_SUBMISSION to true and my submissions are processed correctly now – but only after running cdash/processsubmissions.php manually (following http://public.kitware.com/pipermail/cdash/2014-January/001405.html). However, the processing does not start automatically. I tried to find some documentation about CDash Asynchronous submissions with no success. The only useful page is http://www.kitware.com/source/home/post/59 but it does not mention what (apart from the config file) should I do to process the asynchronous submissions. I’m supposing it should work automatically (without setting cron, for example) after reading the mail archive mentioned above.
>
> The server configuration does not allow to access localhost (or 127.0.0.1) using HTTP, I have to use the local IP address to access webpages. So I’ve set the following values in cdash/config.local.php:
>
> $CDASH_USE_HTTPS ='0';
> // Name of the server running CDash.
> // Leave empty to use current name and default port.
> $CDASH_SERVER_NAME = '192.168.2.105';
> $CDASH_SERVER_PORT = '';
> // If the remote request should use localhost or the full name // This
> variable should be set to 1 in most of the server configurations
> $CDASH_CURL_REQUEST_LOCALHOST='0';
> $CDASH_CURL_LOCALHOST_PREFIX='cdash';
> $CDASH_BASE_URL='';
> // Define the location of the local directory
> $CDASH_USE_LOCAL_DIRECTORY = '0';
>
> Is that correct? I miss any documentation of these variables except the comments (which do not help when I’m not the “most”). I am not sure with CDASH_CURL_LOCALHOST_PREFIX, the full path to process submissions manually is http://192.168.2.105/cdash/cdash/processsubmissions.php.
>
> Best regards,
>
> Miroslav Matějů
>
> -----Original Message-----
> From: Julien Jomier [mailto:julien.jomier at kitware.com]
> Sent: Thursday, May 15, 2014 9:29 AM
>
> Hi Miroslav,
>
> Are you running CDash with Asynchronous submissions?
> ($CDASH_ASYNCHRONOUS_SUBMISSION=true in the config.local.php)
>
> If not that might be the issue,
> Julien
>
> On 15/05/2014 09:16, Matějů Miroslav, Ing. wrote:
>> Hello there,
>>
>> I’ve got a problem with the upload from CTest to CDash. I have many
>> test suites (considered as named builds for CDash), each of them with
>> several tests. The update info is not transferred to CDash for some
>> test suites with a huge amount of tests. To give you an idea what “huge amount”
>> means: There is currently being a transfer to new hardware for both
>> the CTest and CDash machines. For the older HW, it means more than
>> 1000, and about 10000 for the newer one.
>>
>> Some useful info can be found in LastSubmit_*.log on the CTest
>> machine (I shortened it where I put “< >” marks):
>>
>> Upload files:
>>
>> 0 Build.xml
>>
>> 1 Configure.xml
>>
>> 2 Coverage.xml
>>
>> 3 CoverageLog-0.xml
>>
>> 4 DynamicAnalysis.xml
>>
>> 5 Notes.xml
>>
>> 6 Test.xml
>>
>> 7 Update.xml
>>
>> Using drop method: http
>>
>> Upload file: <0 to 5>
>>
>> Upload file: /<mypath>/Testing/20140514-0000/Test.xml to
>> <machine>___<build>___20140514-0000-Nightly___XML___Test.xml
>>
>> Error when uploading file:
>> /<mypath>/Testing/20140514-0000/Test.xml
>>
>> Error message was: Operation too slow. Less than 1 bytes/sec
>> transferred the last 120 seconds
>>
>> Problems when submitting via HTTP
>>
>> <END OF FILE>
>>
>> Although CTest complains about sending Test.xml, it seems to be
>> transferred completely. The tests (including the last ones) present
>> in this file are shown correctly in CDash as well as configure,
>> build, coverage etc. However, it does not seem to even try to upload
>> Update.xml and there is no update info shown in CDash accordingly.
>>
>> I tried to enable repeated sending to CDash using
>>
>> CTEST_SUBMIT(RETRY_COUNT 10)
>>
>> in the CTest script. The only result was an overloaded CDash server
>> and even more upload errors.
>>
>> Could you give me any hints what can I do to correct this issue? Or
>> is it a bug which has to be corrected in the CDash itself? I’m
>> running CTest 2.8.7 and sending results to CDash 2.0.2.
>>
>> Thanks in advance!
>>
>> Miroslav Matějů
>>
>>
>>
>> _______________________________________________
>> Cdash mailing list
>> Cdash at public.kitware.com
>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>>
> _______________________________________________
> Cdash mailing list
> Cdash at public.kitware.com
> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>
More information about the CDash
mailing list