[Cdash] Asynchronous submissions not processed anymore

Julien Jomier julien.jomier at kitware.com
Tue Jan 7 11:27:30 UTC 2014 

Hi Martin,

I have seen this issue in the past. It's cleary the asynchronous call 
using CURL which is the issue. Are you using https?

Also can you send me the values of the following variables in your 
config.local.php:

$CDASH_SERVER_NAME
$CDASH_SERVER_PORT
$CDASH_CURL_REQUEST_LOCALHOST
$CDASH_CURL_LOCALHOST_PREFIX
$CDASH_BASE_URL

Julien

On 07/01/2014 12:18, Martin Apel wrote:
> Hi Julien,
>
> that's what we did the last days (well, the days before Christmas and
> today, the rest was vacation ;-)
> and this works. But you probably did not read my last update, when you
> wrote your mail,
> the processsubmissions is not called automatically.
>
> Martin
>
> On 07/01/14 12:15, Julien Jomier wrote:
>> Hi Martin,
>>
>> Can you confirm that when you manually go to
>>
>>     .../cdash/processsubmissions.php?projectid=1
>>
>> then the processing is done correctly?
>>
>> Julien
>>
>> On 07/01/2014 12:05, Martin Apel wrote:
>>> Hi Julien,
>>>
>>> thanks for the quick reply.
>>> Configuration changes were my initial idea as well. The latest
>>> installation changes were in October and Apache was
>>> restarted several times between October and December, so this rules out
>>> installation changes.
>>> The disk is definitely not full, but I cannot rule out, that it once was
>>> full sometime in December.
>>> It might be the case, that CDash could not correctly write at that time,
>>> and that we still see the symptom of this.
>>> I have checked the access.log file of Apache and the accesses to the
>>> processsubmissions.php script are there,
>>> so it is called, but somehow does not process anything. Can I check the
>>> database somehow?
>>> Any other ideas?
>>>
>>> Martin
>>>
>>> On 07/01/14 11:51, Julien Jomier wrote:
>>>> Hi Martin,
>>>>
>>>> Did something change in the current configuration of the server (an
>>>> Apache update?). Otherwise can you check that the disk is not full?
>>>>
>>>> Just ideas,
>>>> Julien
>>>>
>>>> On 07/01/2014 11:26, Martin Apel wrote:
>>>>> Hi all,
>>>>>
>>>>> we have a CDash server 2.0.2 running for years now, but shortly before
>>>>> Christmas it stopped processing
>>>>> its asynchronous submissions. The processing can be triggered by
>>>>> hand by
>>>>> visiting the URL .../cdash/processsubmissions.php?projectid=1,
>>>>> but this used to happen automatically. I have no idea, why this
>>>>> doesn't
>>>>> happen automatically anymore, the log file does not reveal
>>>>> anything useful.
>>>>>
>>>>> The monitoring page outputs the relevant statistics:
>>>>>
>>>>>
>>>>>         Currently Processing Submissions as of 2014-01-07 10:18:13 UTC
>>>>>
>>>>> Nothing is currently processing...
>>>>>
>>>>>
>>>>>         Pending Submissions
>>>>>
>>>>> project.name, project.id, count of pending queued submissions
>>>>>
>>>>> SIMPACK NA, 1, 4
>>>>>
>>>>>
>>>>>         Average Wait Times
>>>>>
>>>>> projectid 1 wait time
>>>>> hours_ago, time_local, num_files, avg_hours_delay, mean, shortest,
>>>>> longest
>>>>>        0, 5:00 AM, 11, 0.2, 1.5455, 1, 3
>>>>>        1, 5:00 AM, 8, 0.1, 1.5000, 1, 3
>>>>>        2, 4:00 AM, 6, 0.8, 2.0000, 1, 4
>>>>>        3, 2:00 AM, 3, 2.1, 15.3333, 1, 44
>>>>>        4, 1:00 AM, 3, 3.6, 1.0000, 1, 1
>>>>>        13, 4:00 PM, 3, 12.5, 1.3333, 1, 2
>>>>>        14, 4:00 PM, 3, 12.6, 15.0000, 1, 43
>>>>>        27, 2:00 AM, 5, 26.3, 8.6000, 1, 38
>>>>>        30, 11:00 PM, 3, 29.5, 13.0000, 1, 37
>>>>>        38, 4:00 PM, 3, 36.6, 14.6667, 1, 41
>>>>>        46, 7:00 AM, 3, 45.2, 13.3333, 1, 38
>>>>>        50, 4:00 AM, 3, 48.8, 2.0000, 1, 4
>>>>>        51, 3:00 AM, 3, 49.7, 13.3333, 1, 38
>>>>>        53, 12:00 AM, 3, 52.4, 18.3333, 1, 53
>>>>>        74, 3:00 AM, 3, 73.4, 1.6667, 1, 3
>>>>>        75, 2:00 AM, 3, 74.0, 13.3333, 1, 38
>>>>>        78, 11:00 PM, 3, 77.1, 13.0000, 1, 37
>>>>>        86, 4:00 PM, 3, 84.5, 15.0000, 1, 43
>>>>>        98, 3:00 AM, 3, 97.4, 2.0000, 1, 3
>>>>>        99, 2:00 AM, 3, 97.9, 13.3333, 1, 38
>>>>>        102, 11:00 PM, 3, 101.2, 13.0000, 1, 37
>>>>>        110, 4:00 PM, 3, 108.5, 15.6667, 1, 45
>>>>>        122, 3:00 AM, 3, 121.4, 1.6667, 1, 3
>>>>>        123, 2:00 AM, 3, 122.0, 13.3333, 1, 37
>>>>>        126, 11:00 PM, 3, 125.3, 12.3333, 1, 35
>>>>>        134, 4:00 PM, 3, 132.4, 15.6667, 1, 45
>>>>>        146, 3:00 AM, 3, 145.3, 1.6667, 1, 3
>>>>>        147, 2:00 AM, 3, 145.9, 13.0000, 1, 37
>>>>>        150, 11:00 PM, 3, 149.2, 13.0000, 1, 37
>>>>>        158, 4:00 PM, 3, 156.4, 15.3333, 1, 44
>>>>>        170, 3:00 AM, 3, 169.3, 1.6667, 1, 3
>>>>>        171, 2:00 AM, 3, 169.9, 13.6667, 1, 38
>>>>>        174, 11:00 PM, 3, 173.2, 13.3333, 1, 38
>>>>>        182, 4:00 PM, 3, 180.4, 16.3333, 1, 47
>>>>>        194, 3:00 AM, 3, 193.2, 2.0000, 1, 4
>>>>>        195, 2:00 AM, 3, 193.9, 13.0000, 1, 37
>>>>>        198, 11:00 PM, 3, 197.0, 12.6667, 1, 36
>>>>>        206, 4:00 PM, 3, 204.3, 15.0000, 1, 42
>>>>>        218, 4:00 AM, 3, 216.3, 1.6667, 1, 3
>>>>>        219, 3:00 AM, 3, 217.3, 13.6667, 1, 39
>>>>>        221, 12:00 AM, 3, 219.7, 18.6667, 1, 54
>>>>>        242, 3:00 AM, 3, 241.2, 2.0000, 1, 4
>>>>>        243, 2:00 AM, 3, 241.8, 13.0000, 1, 37
>>>>>        246, 11:00 PM, 3, 245.0, 12.6667, 1, 36
>>>>>        254, 4:00 PM, 3, 252.3, 15.0000, 1, 43
>>>>>        266, 3:00 AM, 3, 265.1, 1.6667, 1, 3
>>>>>        267, 2:00 AM, 3, 265.7, 14.0000, 1, 40
>>>>>        270, 11:00 PM, 3, 269.1, 13.6667, 1, 39
>>>>>
>>>>>
>>>>>         Table `submissionprocessor` (one row per project)
>>>>>
>>>>> project.name, projectid, pid, lastupdated, locked
>>>>>
>>>>> SIMPACK NA, 1, 0, 2014-01-07 10:09:04, 1980-01-01 00:00:00
>>>>>
>>>>>
>>>>>         Table `submission` (most recently queued 25)
>>>>>
>>>>> id, filename, projectid, status, attempts, filesize, filemd5sum,
>>>>> lastupdated, created, started, finished
>>>>>
>>>>> 226639, /var/www/CDash-2-0-2/backup/502859723.xml, 1, 0, 0, 3528,
>>>>> 54e89efcd9908c4db29d64a912a2c2ff, 1980-01-01 00:00:00, 2014-01-07
>>>>> 10:14:27, 1980-01-01 00:00:00, 1980-01-01 00:00:00
>>>>> 226638, /var/www/CDash-2-0-2/backup/1420765816.xml, 1, 0, 0, 5192294,
>>>>> 4a6794bd2953a2cc255ddb1fa1dcdf40, 1980-01-01 00:00:00, 2014-01-07
>>>>> 10:14:26, 1980-01-01 00:00:00, 1980-01-01 00:00:00
>>>>> 226637, /var/www/CDash-2-0-2/backup/555091309.xml, 1, 0, 0, 1699,
>>>>> 7abedb2437a6c6b963ece3f977304cd9, 1980-01-01 00:00:00, 2014-01-07
>>>>> 10:14:25, 1980-01-01 00:00:00, 1980-01-01 00:00:00
>>>>> 226636, /var/www/CDash-2-0-2/backup/1075704639.xml, 1, 0, 0, 1063,
>>>>> 1d45170765e7380653bcc3f8f9a38846, 1980-01-01 00:00:00, 2014-01-07
>>>>> 10:14:24, 1980-01-01 00:00:00, 1980-01-01 00:00:00
>>>>> 226635, /var/www/CDash-2-0-2/backup/1722771811.xml, 1, 2, 1, 1584,
>>>>> 3f1768c96932eb14c69404d69f0d55d0, 2014-01-07 10:09:04, 2014-01-07
>>>>> 09:54:51, 2014-01-07 10:09:03, 2014-01-07 10:09:04
>>>>> 226634, /var/www/CDash-2-0-2/backup/758486451.xml, 1, 2, 1, 5162754,
>>>>> feb09272338e599884934f0937921084, 2014-01-07 10:09:03, 2014-01-07
>>>>> 09:54:50, 2014-01-07 10:09:00, 2014-01-07 10:09:03
>>>>> 226633, /var/www/CDash-2-0-2/backup/242524917.xml, 1, 2, 1, 1643,
>>>>> df12695150ba0a9b6326069eca3617a9, 2014-01-07 10:09:00, 2014-01-07
>>>>> 09:54:49, 2014-01-07 10:08:59, 2014-01-07 10:09:00
>>>>> 226632, /var/www/CDash-2-0-2/backup/328287145.xml, 1, 2, 1, 1000,
>>>>> 9e46d671ef9b1c17658f1235e14d9203, 2014-01-07 10:08:59, 2014-01-07
>>>>> 09:54:48, 2014-01-07 10:08:58, 2014-01-07 10:08:59
>>>>> 226631, /var/www/CDash-2-0-2/backup/1413398585.xml, 1, 2, 1, 1584,
>>>>> 0a2471cf0471b5a9f0c9f9fafe4c5e6a, 2014-01-07 10:08:58, 2014-01-07
>>>>> 09:53:53, 2014-01-07 10:08:57, 2014-01-07 10:08:58
>>>>> 226630, /var/www/CDash-2-0-2/backup/1643613376.xml, 1, 2, 1, 5165054,
>>>>> 701d41e1b8a2e0756b1a30138c86f88c, 2014-01-07 10:08:57, 2014-01-07
>>>>> 09:53:52, 2014-01-07 10:08:54, 2014-01-07 10:08:57
>>>>> 226629, /var/www/CDash-2-0-2/backup/1580744380.xml, 1, 2, 1, 1643,
>>>>> afd6ecbf0bca78d9700f7052f514bb6e, 2014-01-07 10:08:54, 2014-01-07
>>>>> 09:53:51, 2014-01-07 10:08:53, 2014-01-07 10:08:54
>>>>> 226628, /var/www/CDash-2-0-2/backup/516354354.xml, 1, 2, 1, 1000,
>>>>> f2f3ce4da201305b25304acb45281c73, 2014-01-07 10:08:53, 2014-01-07
>>>>> 09:53:50, 2014-01-07 10:08:52, 2014-01-07 10:08:53
>>>>> 226627, /var/www/CDash-2-0-2/backup/347351203.xml, 1, 2, 1, 5190173,
>>>>> b96b79efcb356f7cb699ff382542aaa8, 2014-01-07 09:52:50, 2014-01-07
>>>>> 09:48:11, 2014-01-07 09:52:47, 2014-01-07 09:52:50
>>>>> 226626, /var/www/CDash-2-0-2/backup/196135980.xml, 1, 2, 1, 7980,
>>>>> b9d8fb37323efdd946042f7b9ea5d645, 2014-01-07 09:52:47, 2014-01-07
>>>>> 09:48:10, 2014-01-07 09:52:46, 2014-01-07 09:52:47
>>>>> 226625, /var/www/CDash-2-0-2/backup/540120385.xml, 1, 2, 1, 1064,
>>>>> 856656e9f9f09a1ea80ad2265b692823, 2014-01-07 09:52:46, 2014-01-07
>>>>> 09:48:09, 2014-01-07 09:52:45, 2014-01-07 09:52:46
>>>>> 226624, /var/www/CDash-2-0-2/backup/822820666.xml, 1, 2, 1, 2495,
>>>>> 19107c45a119eb4b32cad6f3f8819908, 2014-01-07 09:14:30, 2014-01-07
>>>>> 09:10:30, 2014-01-07 09:14:29, 2014-01-07 09:14:30
>>>>> 226623, /var/www/CDash-2-0-2/backup/1436949172.xml, 1, 2, 1, 5161118,
>>>>> d809e82fb32f1f6e1b88b19821b5e5f0, 2014-01-07 09:14:29, 2014-01-07
>>>>> 09:10:29, 2014-01-07 09:14:26, 2014-01-07 09:14:29
>>>>> 226622, /var/www/CDash-2-0-2/backup/868711020.xml, 1, 2, 1, 1643,
>>>>> 85d1592d4082faf2963727e6b4ed2bc2, 2014-01-07 09:14:26, 2014-01-07
>>>>> 09:10:28, 2014-01-07 09:14:25, 2014-01-07 09:14:26
>>>>> 226621, /var/www/CDash-2-0-2/backup/1531771759.xml, 1, 2, 1, 1000,
>>>>> 096731f87d4d4f826a199410c0f280f8, 2014-01-07 09:14:25, 2014-01-07
>>>>> 09:10:27, 2014-01-07 09:14:24, 2014-01-07 09:14:25
>>>>> 226620, /var/www/CDash-2-0-2/backup/1002205360.xml, 1, 2, 1, 2495,
>>>>> 3d9698df2114a13199e2947cf2b91366, 2014-01-07 09:14:24, 2014-01-07
>>>>> 09:07:10, 2014-01-07 09:14:23, 2014-01-07 09:14:24
>>>>> 226619, /var/www/CDash-2-0-2/backup/1082455045.xml, 1, 2, 1, 5164845,
>>>>> c96cfcc43fb580ca7c099117c370d57f, 2014-01-07 09:14:23, 2014-01-07
>>>>> 09:07:09, 2014-01-07 09:14:20, 2014-01-07 09:14:23
>>>>> 226618, /var/www/CDash-2-0-2/backup/1147021750.xml, 1, 2, 1, 1643,
>>>>> 371946b5a5e948744d64966d8da5ef69, 2014-01-07 09:14:20, 2014-01-07
>>>>> 09:07:08, 2014-01-07 09:14:19, 2014-01-07 09:14:20
>>>>> 226617, /var/www/CDash-2-0-2/backup/1238403471.xml, 1, 2, 1, 1000,
>>>>> 522474f1cbfc4259b29d090f96bc84e1, 2014-01-07 09:14:19, 2014-01-07
>>>>> 09:07:07, 2014-01-07 09:14:18, 2014-01-07 09:14:19
>>>>> 226616, /var/www/CDash-2-0-2/backup/620749966.xml, 1, 2, 1, 5163405,
>>>>> 1dae92592f807e250f0dc9fa5a7ec793, 2014-01-07 08:54:50, 2014-01-07
>>>>> 08:08:30, 2014-01-07 08:54:46, 2014-01-07 08:54:50
>>>>> 226615, /var/www/CDash-2-0-2/backup/849392602.xml, 1, 2, 1, 8002,
>>>>> 960e2e3f8fd7a79144c316c8b7e2d897, 2014-01-07 08:54:46, 2014-01-07
>>>>> 08:08:29, 2014-01-07 08:54:45, 2014-01-07 08:54:46
>>>>>
>>>>> Any ideas what goes wrong here would be very helpful.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Martin
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Cdash mailing list
>>>>> Cdash at public.kitware.com
>>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>>>>>
>>>
>
>

More information about the CDash mailing list