[Cdash] Asynchronous submissions not processed anymore
Martin Apel
martin.apel at simpack.de
Tue Jan 7 08:45:09 EST 2014
Hi Julien,
now it really seems to work, the next submission now triggered the
processing of the
queued submissions.
Thanks a lot for your help.
Martin
On 07/01/14 13:55, Julien Jomier wrote:
> Hi Martin,
>
> Thanks for investigating. I'm going to add a check for the curl timeout.
>
> Let us know if that did the trick,
> Julien
>
> On 07/01/2014 13:43, Martin Apel wrote:
>> Hi Julien,
>>
>> I tried this and got 'http://localhost/CDash' as output.
>> After adding a trailing slash, retrieving this URL via curl works
>> correctly. but it took rather
>> long (about 15 s), so I suspected that I was running into a timeout. The
>> do_submit script sets
>> the timeout to 1.
>>
>> This long duration led me to the cause of the problem, which has three
>> aspects:
>> 1. nsswitch.conf was misconfigured to ask DNS before consulting
>> /etc/hosts. This
>> caused a DNS request for localhost.
>> 2. CDash uses a short timeout of 1 s and does not check the error code
>> of the curl operation
>> 3. The configuration of our DNS servers has changed. This seems to have
>> triggered the problem
>> in the first place, though the misconfiguration was the real cause.
>>
>> I am still waiting for the next submission to CDash to see, if the
>> change in nsswitch.conf
>> fixed the problem, but I am confident that this should work. At least
>> the long duration in the
>> interactive curl call has disappeared after the change.
>> I think CDash should check the error code of curl_exec in this case,
>> this would have made
>> finding the cause easier.
>>
>> I will send another mail, when the next submission has occurred to
>> report, if this really did the
>> trick.
>>
>> Martin
>>
>> On 07/01/14 12:37, Julien Jomier wrote:
>>> Hi Martin,
>>>
>>> Could you try to hack one of the main CDash php pages and add the
>>> following line:
>>>
>>> echo get_server_URI(true);
>>>
>>> to see which URL you are getting.
>>>
>>> Then on the server try to do a "curl <the URL you got>" and see if that
>>> works.
>>>
>>> Julien
>>>
>>> On 07/01/2014 12:31, Martin Apel wrote:
>>>> Hi Julien,
>>>>
>>>> we have modified config.php instead of using config.local.php, which is
>>>> probably still a relict
>>>> from old CDash times.
>>>> The variable values are as follows:
>>>>
>>>> $CDASH_SERVER_NAME = 'simdev-server';
>>>> $CDASH_SERVER_PORT = '';
>>>> $CDASH_CURL_REQUEST_LOCALHOST='1';
>>>> $CDASH_CURL_LOCALHOST_PREFIX='';
>>>> $CDASH_BASE_URL='';
>>>>
>>>> Martin
>>>>
>>>> On 07/01/14 12:27, Julien Jomier wrote:
>>>>> Hi Martin,
>>>>>
>>>>> I have seen this issue in the past. It's cleary the asynchronous call
>>>>> using CURL which is the issue. Are you using https?
>>>>>
>>>>> Also can you send me the values of the following variables in your
>>>>> config.local.php:
>>>>>
>>>>> $CDASH_SERVER_NAME
>>>>> $CDASH_SERVER_PORT
>>>>> $CDASH_CURL_REQUEST_LOCALHOST
>>>>> $CDASH_CURL_LOCALHOST_PREFIX
>>>>> $CDASH_BASE_URL
>>>>>
>>>>> Julien
>>>>>
>>>>> On 07/01/2014 12:18, Martin Apel wrote:
>>>>>> Hi Julien,
>>>>>>
>>>>>> that's what we did the last days (well, the days before Christmas and
>>>>>> today, the rest was vacation ;-)
>>>>>> and this works. But you probably did not read my last update, when you
>>>>>> wrote your mail,
>>>>>> the processsubmissions is not called automatically.
>>>>>>
>>>>>> Martin
>>>>>>
>>>>>> On 07/01/14 12:15, Julien Jomier wrote:
>>>>>>> Hi Martin,
>>>>>>>
>>>>>>> Can you confirm that when you manually go to
>>>>>>>
>>>>>>> .../cdash/processsubmissions.php?projectid=1
>>>>>>>
>>>>>>> then the processing is done correctly?
>>>>>>>
>>>>>>> Julien
>>>>>>>
>>>>>>> On 07/01/2014 12:05, Martin Apel wrote:
>>>>>>>> Hi Julien,
>>>>>>>>
>>>>>>>> thanks for the quick reply.
>>>>>>>> Configuration changes were my initial idea as well. The latest
>>>>>>>> installation changes were in October and Apache was
>>>>>>>> restarted several times between October and December, so this rules
>>>>>>>> out
>>>>>>>> installation changes.
>>>>>>>> The disk is definitely not full, but I cannot rule out, that it
>>>>>>>> once was
>>>>>>>> full sometime in December.
>>>>>>>> It might be the case, that CDash could not correctly write at that
>>>>>>>> time,
>>>>>>>> and that we still see the symptom of this.
>>>>>>>> I have checked the access.log file of Apache and the accesses to the
>>>>>>>> processsubmissions.php script are there,
>>>>>>>> so it is called, but somehow does not process anything. Can I check
>>>>>>>> the
>>>>>>>> database somehow?
>>>>>>>> Any other ideas?
>>>>>>>>
>>>>>>>> Martin
>>>>>>>>
>>>>>>>> On 07/01/14 11:51, Julien Jomier wrote:
>>>>>>>>> Hi Martin,
>>>>>>>>>
>>>>>>>>> Did something change in the current configuration of the server (an
>>>>>>>>> Apache update?). Otherwise can you check that the disk is not full?
>>>>>>>>>
>>>>>>>>> Just ideas,
>>>>>>>>> Julien
>>>>>>>>>
>>>>>>>>> On 07/01/2014 11:26, Martin Apel wrote:
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> we have a CDash server 2.0.2 running for years now, but shortly
>>>>>>>>>> before
>>>>>>>>>> Christmas it stopped processing
>>>>>>>>>> its asynchronous submissions. The processing can be triggered by
>>>>>>>>>> hand by
>>>>>>>>>> visiting the URL .../cdash/processsubmissions.php?projectid=1,
>>>>>>>>>> but this used to happen automatically. I have no idea, why this
>>>>>>>>>> doesn't
>>>>>>>>>> happen automatically anymore, the log file does not reveal
>>>>>>>>>> anything useful.
>>>>>>>>>>
>>>>>>>>>> The monitoring page outputs the relevant statistics:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Currently Processing Submissions as of 2014-01-07
>>>>>>>>>> 10:18:13 UTC
>>>>>>>>>>
>>>>>>>>>> Nothing is currently processing...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Pending Submissions
>>>>>>>>>>
>>>>>>>>>> project.name, project.id, count of pending queued submissions
>>>>>>>>>>
>>>>>>>>>> SIMPACK NA, 1, 4
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Average Wait Times
>>>>>>>>>>
>>>>>>>>>> projectid 1 wait time
>>>>>>>>>> hours_ago, time_local, num_files, avg_hours_delay, mean, shortest,
>>>>>>>>>> longest
>>>>>>>>>> 0, 5:00 AM, 11, 0.2, 1.5455, 1, 3
>>>>>>>>>> 1, 5:00 AM, 8, 0.1, 1.5000, 1, 3
>>>>>>>>>> 2, 4:00 AM, 6, 0.8, 2.0000, 1, 4
>>>>>>>>>> 3, 2:00 AM, 3, 2.1, 15.3333, 1, 44
>>>>>>>>>> 4, 1:00 AM, 3, 3.6, 1.0000, 1, 1
>>>>>>>>>> 13, 4:00 PM, 3, 12.5, 1.3333, 1, 2
>>>>>>>>>> 14, 4:00 PM, 3, 12.6, 15.0000, 1, 43
>>>>>>>>>> 27, 2:00 AM, 5, 26.3, 8.6000, 1, 38
>>>>>>>>>> 30, 11:00 PM, 3, 29.5, 13.0000, 1, 37
>>>>>>>>>> 38, 4:00 PM, 3, 36.6, 14.6667, 1, 41
>>>>>>>>>> 46, 7:00 AM, 3, 45.2, 13.3333, 1, 38
>>>>>>>>>> 50, 4:00 AM, 3, 48.8, 2.0000, 1, 4
>>>>>>>>>> 51, 3:00 AM, 3, 49.7, 13.3333, 1, 38
>>>>>>>>>> 53, 12:00 AM, 3, 52.4, 18.3333, 1, 53
>>>>>>>>>> 74, 3:00 AM, 3, 73.4, 1.6667, 1, 3
>>>>>>>>>> 75, 2:00 AM, 3, 74.0, 13.3333, 1, 38
>>>>>>>>>> 78, 11:00 PM, 3, 77.1, 13.0000, 1, 37
>>>>>>>>>> 86, 4:00 PM, 3, 84.5, 15.0000, 1, 43
>>>>>>>>>> 98, 3:00 AM, 3, 97.4, 2.0000, 1, 3
>>>>>>>>>> 99, 2:00 AM, 3, 97.9, 13.3333, 1, 38
>>>>>>>>>> 102, 11:00 PM, 3, 101.2, 13.0000, 1, 37
>>>>>>>>>> 110, 4:00 PM, 3, 108.5, 15.6667, 1, 45
>>>>>>>>>> 122, 3:00 AM, 3, 121.4, 1.6667, 1, 3
>>>>>>>>>> 123, 2:00 AM, 3, 122.0, 13.3333, 1, 37
>>>>>>>>>> 126, 11:00 PM, 3, 125.3, 12.3333, 1, 35
>>>>>>>>>> 134, 4:00 PM, 3, 132.4, 15.6667, 1, 45
>>>>>>>>>> 146, 3:00 AM, 3, 145.3, 1.6667, 1, 3
>>>>>>>>>> 147, 2:00 AM, 3, 145.9, 13.0000, 1, 37
>>>>>>>>>> 150, 11:00 PM, 3, 149.2, 13.0000, 1, 37
>>>>>>>>>> 158, 4:00 PM, 3, 156.4, 15.3333, 1, 44
>>>>>>>>>> 170, 3:00 AM, 3, 169.3, 1.6667, 1, 3
>>>>>>>>>> 171, 2:00 AM, 3, 169.9, 13.6667, 1, 38
>>>>>>>>>> 174, 11:00 PM, 3, 173.2, 13.3333, 1, 38
>>>>>>>>>> 182, 4:00 PM, 3, 180.4, 16.3333, 1, 47
>>>>>>>>>> 194, 3:00 AM, 3, 193.2, 2.0000, 1, 4
>>>>>>>>>> 195, 2:00 AM, 3, 193.9, 13.0000, 1, 37
>>>>>>>>>> 198, 11:00 PM, 3, 197.0, 12.6667, 1, 36
>>>>>>>>>> 206, 4:00 PM, 3, 204.3, 15.0000, 1, 42
>>>>>>>>>> 218, 4:00 AM, 3, 216.3, 1.6667, 1, 3
>>>>>>>>>> 219, 3:00 AM, 3, 217.3, 13.6667, 1, 39
>>>>>>>>>> 221, 12:00 AM, 3, 219.7, 18.6667, 1, 54
>>>>>>>>>> 242, 3:00 AM, 3, 241.2, 2.0000, 1, 4
>>>>>>>>>> 243, 2:00 AM, 3, 241.8, 13.0000, 1, 37
>>>>>>>>>> 246, 11:00 PM, 3, 245.0, 12.6667, 1, 36
>>>>>>>>>> 254, 4:00 PM, 3, 252.3, 15.0000, 1, 43
>>>>>>>>>> 266, 3:00 AM, 3, 265.1, 1.6667, 1, 3
>>>>>>>>>> 267, 2:00 AM, 3, 265.7, 14.0000, 1, 40
>>>>>>>>>> 270, 11:00 PM, 3, 269.1, 13.6667, 1, 39
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Table `submissionprocessor` (one row per project)
>>>>>>>>>>
>>>>>>>>>> project.name, projectid, pid, lastupdated, locked
>>>>>>>>>>
>>>>>>>>>> SIMPACK NA, 1, 0, 2014-01-07 10:09:04, 1980-01-01 00:00:00
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Table `submission` (most recently queued 25)
>>>>>>>>>>
>>>>>>>>>> id, filename, projectid, status, attempts, filesize, filemd5sum,
>>>>>>>>>> lastupdated, created, started, finished
>>>>>>>>>>
>>>>>>>>>> 226639, /var/www/CDash-2-0-2/backup/502859723.xml, 1, 0, 0, 3528,
>>>>>>>>>> 54e89efcd9908c4db29d64a912a2c2ff, 1980-01-01 00:00:00, 2014-01-07
>>>>>>>>>> 10:14:27, 1980-01-01 00:00:00, 1980-01-01 00:00:00
>>>>>>>>>> 226638, /var/www/CDash-2-0-2/backup/1420765816.xml, 1, 0, 0,
>>>>>>>>>> 5192294,
>>>>>>>>>> 4a6794bd2953a2cc255ddb1fa1dcdf40, 1980-01-01 00:00:00, 2014-01-07
>>>>>>>>>> 10:14:26, 1980-01-01 00:00:00, 1980-01-01 00:00:00
>>>>>>>>>> 226637, /var/www/CDash-2-0-2/backup/555091309.xml, 1, 0, 0, 1699,
>>>>>>>>>> 7abedb2437a6c6b963ece3f977304cd9, 1980-01-01 00:00:00, 2014-01-07
>>>>>>>>>> 10:14:25, 1980-01-01 00:00:00, 1980-01-01 00:00:00
>>>>>>>>>> 226636, /var/www/CDash-2-0-2/backup/1075704639.xml, 1, 0, 0, 1063,
>>>>>>>>>> 1d45170765e7380653bcc3f8f9a38846, 1980-01-01 00:00:00, 2014-01-07
>>>>>>>>>> 10:14:24, 1980-01-01 00:00:00, 1980-01-01 00:00:00
>>>>>>>>>> 226635, /var/www/CDash-2-0-2/backup/1722771811.xml, 1, 2, 1, 1584,
>>>>>>>>>> 3f1768c96932eb14c69404d69f0d55d0, 2014-01-07 10:09:04, 2014-01-07
>>>>>>>>>> 09:54:51, 2014-01-07 10:09:03, 2014-01-07 10:09:04
>>>>>>>>>> 226634, /var/www/CDash-2-0-2/backup/758486451.xml, 1, 2, 1,
>>>>>>>>>> 5162754,
>>>>>>>>>> feb09272338e599884934f0937921084, 2014-01-07 10:09:03, 2014-01-07
>>>>>>>>>> 09:54:50, 2014-01-07 10:09:00, 2014-01-07 10:09:03
>>>>>>>>>> 226633, /var/www/CDash-2-0-2/backup/242524917.xml, 1, 2, 1, 1643,
>>>>>>>>>> df12695150ba0a9b6326069eca3617a9, 2014-01-07 10:09:00, 2014-01-07
>>>>>>>>>> 09:54:49, 2014-01-07 10:08:59, 2014-01-07 10:09:00
>>>>>>>>>> 226632, /var/www/CDash-2-0-2/backup/328287145.xml, 1, 2, 1, 1000,
>>>>>>>>>> 9e46d671ef9b1c17658f1235e14d9203, 2014-01-07 10:08:59, 2014-01-07
>>>>>>>>>> 09:54:48, 2014-01-07 10:08:58, 2014-01-07 10:08:59
>>>>>>>>>> 226631, /var/www/CDash-2-0-2/backup/1413398585.xml, 1, 2, 1, 1584,
>>>>>>>>>> 0a2471cf0471b5a9f0c9f9fafe4c5e6a, 2014-01-07 10:08:58, 2014-01-07
>>>>>>>>>> 09:53:53, 2014-01-07 10:08:57, 2014-01-07 10:08:58
>>>>>>>>>> 226630, /var/www/CDash-2-0-2/backup/1643613376.xml, 1, 2, 1,
>>>>>>>>>> 5165054,
>>>>>>>>>> 701d41e1b8a2e0756b1a30138c86f88c, 2014-01-07 10:08:57, 2014-01-07
>>>>>>>>>> 09:53:52, 2014-01-07 10:08:54, 2014-01-07 10:08:57
>>>>>>>>>> 226629, /var/www/CDash-2-0-2/backup/1580744380.xml, 1, 2, 1, 1643,
>>>>>>>>>> afd6ecbf0bca78d9700f7052f514bb6e, 2014-01-07 10:08:54, 2014-01-07
>>>>>>>>>> 09:53:51, 2014-01-07 10:08:53, 2014-01-07 10:08:54
>>>>>>>>>> 226628, /var/www/CDash-2-0-2/backup/516354354.xml, 1, 2, 1, 1000,
>>>>>>>>>> f2f3ce4da201305b25304acb45281c73, 2014-01-07 10:08:53, 2014-01-07
>>>>>>>>>> 09:53:50, 2014-01-07 10:08:52, 2014-01-07 10:08:53
>>>>>>>>>> 226627, /var/www/CDash-2-0-2/backup/347351203.xml, 1, 2, 1,
>>>>>>>>>> 5190173,
>>>>>>>>>> b96b79efcb356f7cb699ff382542aaa8, 2014-01-07 09:52:50, 2014-01-07
>>>>>>>>>> 09:48:11, 2014-01-07 09:52:47, 2014-01-07 09:52:50
>>>>>>>>>> 226626, /var/www/CDash-2-0-2/backup/196135980.xml, 1, 2, 1, 7980,
>>>>>>>>>> b9d8fb37323efdd946042f7b9ea5d645, 2014-01-07 09:52:47, 2014-01-07
>>>>>>>>>> 09:48:10, 2014-01-07 09:52:46, 2014-01-07 09:52:47
>>>>>>>>>> 226625, /var/www/CDash-2-0-2/backup/540120385.xml, 1, 2, 1, 1064,
>>>>>>>>>> 856656e9f9f09a1ea80ad2265b692823, 2014-01-07 09:52:46, 2014-01-07
>>>>>>>>>> 09:48:09, 2014-01-07 09:52:45, 2014-01-07 09:52:46
>>>>>>>>>> 226624, /var/www/CDash-2-0-2/backup/822820666.xml, 1, 2, 1, 2495,
>>>>>>>>>> 19107c45a119eb4b32cad6f3f8819908, 2014-01-07 09:14:30, 2014-01-07
>>>>>>>>>> 09:10:30, 2014-01-07 09:14:29, 2014-01-07 09:14:30
>>>>>>>>>> 226623, /var/www/CDash-2-0-2/backup/1436949172.xml, 1, 2, 1,
>>>>>>>>>> 5161118,
>>>>>>>>>> d809e82fb32f1f6e1b88b19821b5e5f0, 2014-01-07 09:14:29, 2014-01-07
>>>>>>>>>> 09:10:29, 2014-01-07 09:14:26, 2014-01-07 09:14:29
>>>>>>>>>> 226622, /var/www/CDash-2-0-2/backup/868711020.xml, 1, 2, 1, 1643,
>>>>>>>>>> 85d1592d4082faf2963727e6b4ed2bc2, 2014-01-07 09:14:26, 2014-01-07
>>>>>>>>>> 09:10:28, 2014-01-07 09:14:25, 2014-01-07 09:14:26
>>>>>>>>>> 226621, /var/www/CDash-2-0-2/backup/1531771759.xml, 1, 2, 1, 1000,
>>>>>>>>>> 096731f87d4d4f826a199410c0f280f8, 2014-01-07 09:14:25, 2014-01-07
>>>>>>>>>> 09:10:27, 2014-01-07 09:14:24, 2014-01-07 09:14:25
>>>>>>>>>> 226620, /var/www/CDash-2-0-2/backup/1002205360.xml, 1, 2, 1, 2495,
>>>>>>>>>> 3d9698df2114a13199e2947cf2b91366, 2014-01-07 09:14:24, 2014-01-07
>>>>>>>>>> 09:07:10, 2014-01-07 09:14:23, 2014-01-07 09:14:24
>>>>>>>>>> 226619, /var/www/CDash-2-0-2/backup/1082455045.xml, 1, 2, 1,
>>>>>>>>>> 5164845,
>>>>>>>>>> c96cfcc43fb580ca7c099117c370d57f, 2014-01-07 09:14:23, 2014-01-07
>>>>>>>>>> 09:07:09, 2014-01-07 09:14:20, 2014-01-07 09:14:23
>>>>>>>>>> 226618, /var/www/CDash-2-0-2/backup/1147021750.xml, 1, 2, 1, 1643,
>>>>>>>>>> 371946b5a5e948744d64966d8da5ef69, 2014-01-07 09:14:20, 2014-01-07
>>>>>>>>>> 09:07:08, 2014-01-07 09:14:19, 2014-01-07 09:14:20
>>>>>>>>>> 226617, /var/www/CDash-2-0-2/backup/1238403471.xml, 1, 2, 1, 1000,
>>>>>>>>>> 522474f1cbfc4259b29d090f96bc84e1, 2014-01-07 09:14:19, 2014-01-07
>>>>>>>>>> 09:07:07, 2014-01-07 09:14:18, 2014-01-07 09:14:19
>>>>>>>>>> 226616, /var/www/CDash-2-0-2/backup/620749966.xml, 1, 2, 1,
>>>>>>>>>> 5163405,
>>>>>>>>>> 1dae92592f807e250f0dc9fa5a7ec793, 2014-01-07 08:54:50, 2014-01-07
>>>>>>>>>> 08:08:30, 2014-01-07 08:54:46, 2014-01-07 08:54:50
>>>>>>>>>> 226615, /var/www/CDash-2-0-2/backup/849392602.xml, 1, 2, 1, 8002,
>>>>>>>>>> 960e2e3f8fd7a79144c316c8b7e2d897, 2014-01-07 08:54:46, 2014-01-07
>>>>>>>>>> 08:08:29, 2014-01-07 08:54:45, 2014-01-07 08:54:46
>>>>>>>>>>
>>>>>>>>>> Any ideas what goes wrong here would be very helpful.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Martin
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Cdash mailing list
>>>>>>>>>> Cdash at public.kitware.com
>>>>>>>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>>>>>>>>>>
>>
--
Martin Apel
Software Architect
Phone: + 49 8105 77266-53
E-Mail: martin.apel at simpack.de
SIMPACK AG
Friedrichshafener Strasse 1, 82205 Gilching, Germany
info at simpack.de, www.simpack.com
Phone: + 49 8105 77266-0
Fax: + 49 8105 77266-11
Executive Board: Dr. Alexander Eichberger, Dr. Lutz Mauer
Chair of Supervisory Board: Silvia Förster (CPA)
Commercial Register München HRB 181 229
More information about the Cdash
mailing list