[Cdash] Asynchronous submissions not processed anymore
Martin Apel
martin.apel at simpack.de
Tue Jan 7 12:43:51 UTC 2014
Hi Julien,
I tried this and got 'http://localhost/CDash' as output.
After adding a trailing slash, retrieving this URL via curl works
correctly. but it took rather
long (about 15 s), so I suspected that I was running into a timeout. The
do_submit script sets
the timeout to 1.
This long duration led me to the cause of the problem, which has three
aspects:
1. nsswitch.conf was misconfigured to ask DNS before consulting
/etc/hosts. This
caused a DNS request for localhost.
2. CDash uses a short timeout of 1 s and does not check the error code
of the curl operation
3. The configuration of our DNS servers has changed. This seems to have
triggered the problem
in the first place, though the misconfiguration was the real cause.
I am still waiting for the next submission to CDash to see, if the
change in nsswitch.conf
fixed the problem, but I am confident that this should work. At least
the long duration in the
interactive curl call has disappeared after the change.
I think CDash should check the error code of curl_exec in this case,
this would have made
finding the cause easier.
I will send another mail, when the next submission has occurred to
report, if this really did the
trick.
Martin
On 07/01/14 12:37, Julien Jomier wrote:
> Hi Martin,
>
> Could you try to hack one of the main CDash php pages and add the
> following line:
>
> echo get_server_URI(true);
>
> to see which URL you are getting.
>
> Then on the server try to do a "curl <the URL you got>" and see if that
> works.
>
> Julien
>
> On 07/01/2014 12:31, Martin Apel wrote:
>> Hi Julien,
>>
>> we have modified config.php instead of using config.local.php, which is
>> probably still a relict
>> from old CDash times.
>> The variable values are as follows:
>>
>> $CDASH_SERVER_NAME = 'simdev-server';
>> $CDASH_SERVER_PORT = '';
>> $CDASH_CURL_REQUEST_LOCALHOST='1';
>> $CDASH_CURL_LOCALHOST_PREFIX='';
>> $CDASH_BASE_URL='';
>>
>> Martin
>>
>> On 07/01/14 12:27, Julien Jomier wrote:
>>> Hi Martin,
>>>
>>> I have seen this issue in the past. It's cleary the asynchronous call
>>> using CURL which is the issue. Are you using https?
>>>
>>> Also can you send me the values of the following variables in your
>>> config.local.php:
>>>
>>> $CDASH_SERVER_NAME
>>> $CDASH_SERVER_PORT
>>> $CDASH_CURL_REQUEST_LOCALHOST
>>> $CDASH_CURL_LOCALHOST_PREFIX
>>> $CDASH_BASE_URL
>>>
>>> Julien
>>>
>>> On 07/01/2014 12:18, Martin Apel wrote:
>>>> Hi Julien,
>>>>
>>>> that's what we did the last days (well, the days before Christmas and
>>>> today, the rest was vacation ;-)
>>>> and this works. But you probably did not read my last update, when you
>>>> wrote your mail,
>>>> the processsubmissions is not called automatically.
>>>>
>>>> Martin
>>>>
>>>> On 07/01/14 12:15, Julien Jomier wrote:
>>>>> Hi Martin,
>>>>>
>>>>> Can you confirm that when you manually go to
>>>>>
>>>>> .../cdash/processsubmissions.php?projectid=1
>>>>>
>>>>> then the processing is done correctly?
>>>>>
>>>>> Julien
>>>>>
>>>>> On 07/01/2014 12:05, Martin Apel wrote:
>>>>>> Hi Julien,
>>>>>>
>>>>>> thanks for the quick reply.
>>>>>> Configuration changes were my initial idea as well. The latest
>>>>>> installation changes were in October and Apache was
>>>>>> restarted several times between October and December, so this rules
>>>>>> out
>>>>>> installation changes.
>>>>>> The disk is definitely not full, but I cannot rule out, that it
>>>>>> once was
>>>>>> full sometime in December.
>>>>>> It might be the case, that CDash could not correctly write at that
>>>>>> time,
>>>>>> and that we still see the symptom of this.
>>>>>> I have checked the access.log file of Apache and the accesses to the
>>>>>> processsubmissions.php script are there,
>>>>>> so it is called, but somehow does not process anything. Can I check
>>>>>> the
>>>>>> database somehow?
>>>>>> Any other ideas?
>>>>>>
>>>>>> Martin
>>>>>>
>>>>>> On 07/01/14 11:51, Julien Jomier wrote:
>>>>>>> Hi Martin,
>>>>>>>
>>>>>>> Did something change in the current configuration of the server (an
>>>>>>> Apache update?). Otherwise can you check that the disk is not full?
>>>>>>>
>>>>>>> Just ideas,
>>>>>>> Julien
>>>>>>>
>>>>>>> On 07/01/2014 11:26, Martin Apel wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> we have a CDash server 2.0.2 running for years now, but shortly
>>>>>>>> before
>>>>>>>> Christmas it stopped processing
>>>>>>>> its asynchronous submissions. The processing can be triggered by
>>>>>>>> hand by
>>>>>>>> visiting the URL .../cdash/processsubmissions.php?projectid=1,
>>>>>>>> but this used to happen automatically. I have no idea, why this
>>>>>>>> doesn't
>>>>>>>> happen automatically anymore, the log file does not reveal
>>>>>>>> anything useful.
>>>>>>>>
>>>>>>>> The monitoring page outputs the relevant statistics:
>>>>>>>>
>>>>>>>>
>>>>>>>> Currently Processing Submissions as of 2014-01-07
>>>>>>>> 10:18:13 UTC
>>>>>>>>
>>>>>>>> Nothing is currently processing...
>>>>>>>>
>>>>>>>>
>>>>>>>> Pending Submissions
>>>>>>>>
>>>>>>>> project.name, project.id, count of pending queued submissions
>>>>>>>>
>>>>>>>> SIMPACK NA, 1, 4
>>>>>>>>
>>>>>>>>
>>>>>>>> Average Wait Times
>>>>>>>>
>>>>>>>> projectid 1 wait time
>>>>>>>> hours_ago, time_local, num_files, avg_hours_delay, mean, shortest,
>>>>>>>> longest
>>>>>>>> 0, 5:00 AM, 11, 0.2, 1.5455, 1, 3
>>>>>>>> 1, 5:00 AM, 8, 0.1, 1.5000, 1, 3
>>>>>>>> 2, 4:00 AM, 6, 0.8, 2.0000, 1, 4
>>>>>>>> 3, 2:00 AM, 3, 2.1, 15.3333, 1, 44
>>>>>>>> 4, 1:00 AM, 3, 3.6, 1.0000, 1, 1
>>>>>>>> 13, 4:00 PM, 3, 12.5, 1.3333, 1, 2
>>>>>>>> 14, 4:00 PM, 3, 12.6, 15.0000, 1, 43
>>>>>>>> 27, 2:00 AM, 5, 26.3, 8.6000, 1, 38
>>>>>>>> 30, 11:00 PM, 3, 29.5, 13.0000, 1, 37
>>>>>>>> 38, 4:00 PM, 3, 36.6, 14.6667, 1, 41
>>>>>>>> 46, 7:00 AM, 3, 45.2, 13.3333, 1, 38
>>>>>>>> 50, 4:00 AM, 3, 48.8, 2.0000, 1, 4
>>>>>>>> 51, 3:00 AM, 3, 49.7, 13.3333, 1, 38
>>>>>>>> 53, 12:00 AM, 3, 52.4, 18.3333, 1, 53
>>>>>>>> 74, 3:00 AM, 3, 73.4, 1.6667, 1, 3
>>>>>>>> 75, 2:00 AM, 3, 74.0, 13.3333, 1, 38
>>>>>>>> 78, 11:00 PM, 3, 77.1, 13.0000, 1, 37
>>>>>>>> 86, 4:00 PM, 3, 84.5, 15.0000, 1, 43
>>>>>>>> 98, 3:00 AM, 3, 97.4, 2.0000, 1, 3
>>>>>>>> 99, 2:00 AM, 3, 97.9, 13.3333, 1, 38
>>>>>>>> 102, 11:00 PM, 3, 101.2, 13.0000, 1, 37
>>>>>>>> 110, 4:00 PM, 3, 108.5, 15.6667, 1, 45
>>>>>>>> 122, 3:00 AM, 3, 121.4, 1.6667, 1, 3
>>>>>>>> 123, 2:00 AM, 3, 122.0, 13.3333, 1, 37
>>>>>>>> 126, 11:00 PM, 3, 125.3, 12.3333, 1, 35
>>>>>>>> 134, 4:00 PM, 3, 132.4, 15.6667, 1, 45
>>>>>>>> 146, 3:00 AM, 3, 145.3, 1.6667, 1, 3
>>>>>>>> 147, 2:00 AM, 3, 145.9, 13.0000, 1, 37
>>>>>>>> 150, 11:00 PM, 3, 149.2, 13.0000, 1, 37
>>>>>>>> 158, 4:00 PM, 3, 156.4, 15.3333, 1, 44
>>>>>>>> 170, 3:00 AM, 3, 169.3, 1.6667, 1, 3
>>>>>>>> 171, 2:00 AM, 3, 169.9, 13.6667, 1, 38
>>>>>>>> 174, 11:00 PM, 3, 173.2, 13.3333, 1, 38
>>>>>>>> 182, 4:00 PM, 3, 180.4, 16.3333, 1, 47
>>>>>>>> 194, 3:00 AM, 3, 193.2, 2.0000, 1, 4
>>>>>>>> 195, 2:00 AM, 3, 193.9, 13.0000, 1, 37
>>>>>>>> 198, 11:00 PM, 3, 197.0, 12.6667, 1, 36
>>>>>>>> 206, 4:00 PM, 3, 204.3, 15.0000, 1, 42
>>>>>>>> 218, 4:00 AM, 3, 216.3, 1.6667, 1, 3
>>>>>>>> 219, 3:00 AM, 3, 217.3, 13.6667, 1, 39
>>>>>>>> 221, 12:00 AM, 3, 219.7, 18.6667, 1, 54
>>>>>>>> 242, 3:00 AM, 3, 241.2, 2.0000, 1, 4
>>>>>>>> 243, 2:00 AM, 3, 241.8, 13.0000, 1, 37
>>>>>>>> 246, 11:00 PM, 3, 245.0, 12.6667, 1, 36
>>>>>>>> 254, 4:00 PM, 3, 252.3, 15.0000, 1, 43
>>>>>>>> 266, 3:00 AM, 3, 265.1, 1.6667, 1, 3
>>>>>>>> 267, 2:00 AM, 3, 265.7, 14.0000, 1, 40
>>>>>>>> 270, 11:00 PM, 3, 269.1, 13.6667, 1, 39
>>>>>>>>
>>>>>>>>
>>>>>>>> Table `submissionprocessor` (one row per project)
>>>>>>>>
>>>>>>>> project.name, projectid, pid, lastupdated, locked
>>>>>>>>
>>>>>>>> SIMPACK NA, 1, 0, 2014-01-07 10:09:04, 1980-01-01 00:00:00
>>>>>>>>
>>>>>>>>
>>>>>>>> Table `submission` (most recently queued 25)
>>>>>>>>
>>>>>>>> id, filename, projectid, status, attempts, filesize, filemd5sum,
>>>>>>>> lastupdated, created, started, finished
>>>>>>>>
>>>>>>>> 226639, /var/www/CDash-2-0-2/backup/502859723.xml, 1, 0, 0, 3528,
>>>>>>>> 54e89efcd9908c4db29d64a912a2c2ff, 1980-01-01 00:00:00, 2014-01-07
>>>>>>>> 10:14:27, 1980-01-01 00:00:00, 1980-01-01 00:00:00
>>>>>>>> 226638, /var/www/CDash-2-0-2/backup/1420765816.xml, 1, 0, 0,
>>>>>>>> 5192294,
>>>>>>>> 4a6794bd2953a2cc255ddb1fa1dcdf40, 1980-01-01 00:00:00, 2014-01-07
>>>>>>>> 10:14:26, 1980-01-01 00:00:00, 1980-01-01 00:00:00
>>>>>>>> 226637, /var/www/CDash-2-0-2/backup/555091309.xml, 1, 0, 0, 1699,
>>>>>>>> 7abedb2437a6c6b963ece3f977304cd9, 1980-01-01 00:00:00, 2014-01-07
>>>>>>>> 10:14:25, 1980-01-01 00:00:00, 1980-01-01 00:00:00
>>>>>>>> 226636, /var/www/CDash-2-0-2/backup/1075704639.xml, 1, 0, 0, 1063,
>>>>>>>> 1d45170765e7380653bcc3f8f9a38846, 1980-01-01 00:00:00, 2014-01-07
>>>>>>>> 10:14:24, 1980-01-01 00:00:00, 1980-01-01 00:00:00
>>>>>>>> 226635, /var/www/CDash-2-0-2/backup/1722771811.xml, 1, 2, 1, 1584,
>>>>>>>> 3f1768c96932eb14c69404d69f0d55d0, 2014-01-07 10:09:04, 2014-01-07
>>>>>>>> 09:54:51, 2014-01-07 10:09:03, 2014-01-07 10:09:04
>>>>>>>> 226634, /var/www/CDash-2-0-2/backup/758486451.xml, 1, 2, 1, 5162754,
>>>>>>>> feb09272338e599884934f0937921084, 2014-01-07 10:09:03, 2014-01-07
>>>>>>>> 09:54:50, 2014-01-07 10:09:00, 2014-01-07 10:09:03
>>>>>>>> 226633, /var/www/CDash-2-0-2/backup/242524917.xml, 1, 2, 1, 1643,
>>>>>>>> df12695150ba0a9b6326069eca3617a9, 2014-01-07 10:09:00, 2014-01-07
>>>>>>>> 09:54:49, 2014-01-07 10:08:59, 2014-01-07 10:09:00
>>>>>>>> 226632, /var/www/CDash-2-0-2/backup/328287145.xml, 1, 2, 1, 1000,
>>>>>>>> 9e46d671ef9b1c17658f1235e14d9203, 2014-01-07 10:08:59, 2014-01-07
>>>>>>>> 09:54:48, 2014-01-07 10:08:58, 2014-01-07 10:08:59
>>>>>>>> 226631, /var/www/CDash-2-0-2/backup/1413398585.xml, 1, 2, 1, 1584,
>>>>>>>> 0a2471cf0471b5a9f0c9f9fafe4c5e6a, 2014-01-07 10:08:58, 2014-01-07
>>>>>>>> 09:53:53, 2014-01-07 10:08:57, 2014-01-07 10:08:58
>>>>>>>> 226630, /var/www/CDash-2-0-2/backup/1643613376.xml, 1, 2, 1,
>>>>>>>> 5165054,
>>>>>>>> 701d41e1b8a2e0756b1a30138c86f88c, 2014-01-07 10:08:57, 2014-01-07
>>>>>>>> 09:53:52, 2014-01-07 10:08:54, 2014-01-07 10:08:57
>>>>>>>> 226629, /var/www/CDash-2-0-2/backup/1580744380.xml, 1, 2, 1, 1643,
>>>>>>>> afd6ecbf0bca78d9700f7052f514bb6e, 2014-01-07 10:08:54, 2014-01-07
>>>>>>>> 09:53:51, 2014-01-07 10:08:53, 2014-01-07 10:08:54
>>>>>>>> 226628, /var/www/CDash-2-0-2/backup/516354354.xml, 1, 2, 1, 1000,
>>>>>>>> f2f3ce4da201305b25304acb45281c73, 2014-01-07 10:08:53, 2014-01-07
>>>>>>>> 09:53:50, 2014-01-07 10:08:52, 2014-01-07 10:08:53
>>>>>>>> 226627, /var/www/CDash-2-0-2/backup/347351203.xml, 1, 2, 1, 5190173,
>>>>>>>> b96b79efcb356f7cb699ff382542aaa8, 2014-01-07 09:52:50, 2014-01-07
>>>>>>>> 09:48:11, 2014-01-07 09:52:47, 2014-01-07 09:52:50
>>>>>>>> 226626, /var/www/CDash-2-0-2/backup/196135980.xml, 1, 2, 1, 7980,
>>>>>>>> b9d8fb37323efdd946042f7b9ea5d645, 2014-01-07 09:52:47, 2014-01-07
>>>>>>>> 09:48:10, 2014-01-07 09:52:46, 2014-01-07 09:52:47
>>>>>>>> 226625, /var/www/CDash-2-0-2/backup/540120385.xml, 1, 2, 1, 1064,
>>>>>>>> 856656e9f9f09a1ea80ad2265b692823, 2014-01-07 09:52:46, 2014-01-07
>>>>>>>> 09:48:09, 2014-01-07 09:52:45, 2014-01-07 09:52:46
>>>>>>>> 226624, /var/www/CDash-2-0-2/backup/822820666.xml, 1, 2, 1, 2495,
>>>>>>>> 19107c45a119eb4b32cad6f3f8819908, 2014-01-07 09:14:30, 2014-01-07
>>>>>>>> 09:10:30, 2014-01-07 09:14:29, 2014-01-07 09:14:30
>>>>>>>> 226623, /var/www/CDash-2-0-2/backup/1436949172.xml, 1, 2, 1,
>>>>>>>> 5161118,
>>>>>>>> d809e82fb32f1f6e1b88b19821b5e5f0, 2014-01-07 09:14:29, 2014-01-07
>>>>>>>> 09:10:29, 2014-01-07 09:14:26, 2014-01-07 09:14:29
>>>>>>>> 226622, /var/www/CDash-2-0-2/backup/868711020.xml, 1, 2, 1, 1643,
>>>>>>>> 85d1592d4082faf2963727e6b4ed2bc2, 2014-01-07 09:14:26, 2014-01-07
>>>>>>>> 09:10:28, 2014-01-07 09:14:25, 2014-01-07 09:14:26
>>>>>>>> 226621, /var/www/CDash-2-0-2/backup/1531771759.xml, 1, 2, 1, 1000,
>>>>>>>> 096731f87d4d4f826a199410c0f280f8, 2014-01-07 09:14:25, 2014-01-07
>>>>>>>> 09:10:27, 2014-01-07 09:14:24, 2014-01-07 09:14:25
>>>>>>>> 226620, /var/www/CDash-2-0-2/backup/1002205360.xml, 1, 2, 1, 2495,
>>>>>>>> 3d9698df2114a13199e2947cf2b91366, 2014-01-07 09:14:24, 2014-01-07
>>>>>>>> 09:07:10, 2014-01-07 09:14:23, 2014-01-07 09:14:24
>>>>>>>> 226619, /var/www/CDash-2-0-2/backup/1082455045.xml, 1, 2, 1,
>>>>>>>> 5164845,
>>>>>>>> c96cfcc43fb580ca7c099117c370d57f, 2014-01-07 09:14:23, 2014-01-07
>>>>>>>> 09:07:09, 2014-01-07 09:14:20, 2014-01-07 09:14:23
>>>>>>>> 226618, /var/www/CDash-2-0-2/backup/1147021750.xml, 1, 2, 1, 1643,
>>>>>>>> 371946b5a5e948744d64966d8da5ef69, 2014-01-07 09:14:20, 2014-01-07
>>>>>>>> 09:07:08, 2014-01-07 09:14:19, 2014-01-07 09:14:20
>>>>>>>> 226617, /var/www/CDash-2-0-2/backup/1238403471.xml, 1, 2, 1, 1000,
>>>>>>>> 522474f1cbfc4259b29d090f96bc84e1, 2014-01-07 09:14:19, 2014-01-07
>>>>>>>> 09:07:07, 2014-01-07 09:14:18, 2014-01-07 09:14:19
>>>>>>>> 226616, /var/www/CDash-2-0-2/backup/620749966.xml, 1, 2, 1, 5163405,
>>>>>>>> 1dae92592f807e250f0dc9fa5a7ec793, 2014-01-07 08:54:50, 2014-01-07
>>>>>>>> 08:08:30, 2014-01-07 08:54:46, 2014-01-07 08:54:50
>>>>>>>> 226615, /var/www/CDash-2-0-2/backup/849392602.xml, 1, 2, 1, 8002,
>>>>>>>> 960e2e3f8fd7a79144c316c8b7e2d897, 2014-01-07 08:54:46, 2014-01-07
>>>>>>>> 08:08:29, 2014-01-07 08:54:45, 2014-01-07 08:54:46
>>>>>>>>
>>>>>>>> Any ideas what goes wrong here would be very helpful.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Martin
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Cdash mailing list
>>>>>>>> Cdash at public.kitware.com
>>>>>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>>>>>>>>
>>
--
Martin Apel
Software Architect
Phone: + 49 8105 77266-53
E-Mail: martin.apel at simpack.de
SIMPACK AG
Friedrichshafener Strasse 1, 82205 Gilching, Germany
info at simpack.de, www.simpack.com
Phone: + 49 8105 77266-0
Fax: + 49 8105 77266-11
Executive Board: Dr. Alexander Eichberger, Dr. Lutz Mauer
Chair of Supervisory Board: Silvia Förster (CPA)
Commercial Register München HRB 181 229
More information about the CDash
mailing list