[Cdash] Asynchronous submissions not processed anymore
Martin Apel
martin.apel at simpack.de
Tue Jan 7 11:05:53 UTC 2014
Hi Julien,
thanks for the quick reply.
Configuration changes were my initial idea as well. The latest
installation changes were in October and Apache was
restarted several times between October and December, so this rules out
installation changes.
The disk is definitely not full, but I cannot rule out, that it once was
full sometime in December.
It might be the case, that CDash could not correctly write at that time,
and that we still see the symptom of this.
I have checked the access.log file of Apache and the accesses to the
processsubmissions.php script are there,
so it is called, but somehow does not process anything. Can I check the
database somehow?
Any other ideas?
Martin
On 07/01/14 11:51, Julien Jomier wrote:
> Hi Martin,
>
> Did something change in the current configuration of the server (an
> Apache update?). Otherwise can you check that the disk is not full?
>
> Just ideas,
> Julien
>
> On 07/01/2014 11:26, Martin Apel wrote:
>> Hi all,
>>
>> we have a CDash server 2.0.2 running for years now, but shortly before
>> Christmas it stopped processing
>> its asynchronous submissions. The processing can be triggered by hand by
>> visiting the URL .../cdash/processsubmissions.php?projectid=1,
>> but this used to happen automatically. I have no idea, why this doesn't
>> happen automatically anymore, the log file does not reveal
>> anything useful.
>>
>> The monitoring page outputs the relevant statistics:
>>
>>
>> Currently Processing Submissions as of 2014-01-07 10:18:13 UTC
>>
>> Nothing is currently processing...
>>
>>
>> Pending Submissions
>>
>> project.name, project.id, count of pending queued submissions
>>
>> SIMPACK NA, 1, 4
>>
>>
>> Average Wait Times
>>
>> projectid 1 wait time
>> hours_ago, time_local, num_files, avg_hours_delay, mean, shortest, longest
>> 0, 5:00 AM, 11, 0.2, 1.5455, 1, 3
>> 1, 5:00 AM, 8, 0.1, 1.5000, 1, 3
>> 2, 4:00 AM, 6, 0.8, 2.0000, 1, 4
>> 3, 2:00 AM, 3, 2.1, 15.3333, 1, 44
>> 4, 1:00 AM, 3, 3.6, 1.0000, 1, 1
>> 13, 4:00 PM, 3, 12.5, 1.3333, 1, 2
>> 14, 4:00 PM, 3, 12.6, 15.0000, 1, 43
>> 27, 2:00 AM, 5, 26.3, 8.6000, 1, 38
>> 30, 11:00 PM, 3, 29.5, 13.0000, 1, 37
>> 38, 4:00 PM, 3, 36.6, 14.6667, 1, 41
>> 46, 7:00 AM, 3, 45.2, 13.3333, 1, 38
>> 50, 4:00 AM, 3, 48.8, 2.0000, 1, 4
>> 51, 3:00 AM, 3, 49.7, 13.3333, 1, 38
>> 53, 12:00 AM, 3, 52.4, 18.3333, 1, 53
>> 74, 3:00 AM, 3, 73.4, 1.6667, 1, 3
>> 75, 2:00 AM, 3, 74.0, 13.3333, 1, 38
>> 78, 11:00 PM, 3, 77.1, 13.0000, 1, 37
>> 86, 4:00 PM, 3, 84.5, 15.0000, 1, 43
>> 98, 3:00 AM, 3, 97.4, 2.0000, 1, 3
>> 99, 2:00 AM, 3, 97.9, 13.3333, 1, 38
>> 102, 11:00 PM, 3, 101.2, 13.0000, 1, 37
>> 110, 4:00 PM, 3, 108.5, 15.6667, 1, 45
>> 122, 3:00 AM, 3, 121.4, 1.6667, 1, 3
>> 123, 2:00 AM, 3, 122.0, 13.3333, 1, 37
>> 126, 11:00 PM, 3, 125.3, 12.3333, 1, 35
>> 134, 4:00 PM, 3, 132.4, 15.6667, 1, 45
>> 146, 3:00 AM, 3, 145.3, 1.6667, 1, 3
>> 147, 2:00 AM, 3, 145.9, 13.0000, 1, 37
>> 150, 11:00 PM, 3, 149.2, 13.0000, 1, 37
>> 158, 4:00 PM, 3, 156.4, 15.3333, 1, 44
>> 170, 3:00 AM, 3, 169.3, 1.6667, 1, 3
>> 171, 2:00 AM, 3, 169.9, 13.6667, 1, 38
>> 174, 11:00 PM, 3, 173.2, 13.3333, 1, 38
>> 182, 4:00 PM, 3, 180.4, 16.3333, 1, 47
>> 194, 3:00 AM, 3, 193.2, 2.0000, 1, 4
>> 195, 2:00 AM, 3, 193.9, 13.0000, 1, 37
>> 198, 11:00 PM, 3, 197.0, 12.6667, 1, 36
>> 206, 4:00 PM, 3, 204.3, 15.0000, 1, 42
>> 218, 4:00 AM, 3, 216.3, 1.6667, 1, 3
>> 219, 3:00 AM, 3, 217.3, 13.6667, 1, 39
>> 221, 12:00 AM, 3, 219.7, 18.6667, 1, 54
>> 242, 3:00 AM, 3, 241.2, 2.0000, 1, 4
>> 243, 2:00 AM, 3, 241.8, 13.0000, 1, 37
>> 246, 11:00 PM, 3, 245.0, 12.6667, 1, 36
>> 254, 4:00 PM, 3, 252.3, 15.0000, 1, 43
>> 266, 3:00 AM, 3, 265.1, 1.6667, 1, 3
>> 267, 2:00 AM, 3, 265.7, 14.0000, 1, 40
>> 270, 11:00 PM, 3, 269.1, 13.6667, 1, 39
>>
>>
>> Table `submissionprocessor` (one row per project)
>>
>> project.name, projectid, pid, lastupdated, locked
>>
>> SIMPACK NA, 1, 0, 2014-01-07 10:09:04, 1980-01-01 00:00:00
>>
>>
>> Table `submission` (most recently queued 25)
>>
>> id, filename, projectid, status, attempts, filesize, filemd5sum, lastupdated, created, started, finished
>>
>> 226639, /var/www/CDash-2-0-2/backup/502859723.xml, 1, 0, 0, 3528, 54e89efcd9908c4db29d64a912a2c2ff, 1980-01-01 00:00:00, 2014-01-07 10:14:27, 1980-01-01 00:00:00, 1980-01-01 00:00:00
>> 226638, /var/www/CDash-2-0-2/backup/1420765816.xml, 1, 0, 0, 5192294, 4a6794bd2953a2cc255ddb1fa1dcdf40, 1980-01-01 00:00:00, 2014-01-07 10:14:26, 1980-01-01 00:00:00, 1980-01-01 00:00:00
>> 226637, /var/www/CDash-2-0-2/backup/555091309.xml, 1, 0, 0, 1699, 7abedb2437a6c6b963ece3f977304cd9, 1980-01-01 00:00:00, 2014-01-07 10:14:25, 1980-01-01 00:00:00, 1980-01-01 00:00:00
>> 226636, /var/www/CDash-2-0-2/backup/1075704639.xml, 1, 0, 0, 1063, 1d45170765e7380653bcc3f8f9a38846, 1980-01-01 00:00:00, 2014-01-07 10:14:24, 1980-01-01 00:00:00, 1980-01-01 00:00:00
>> 226635, /var/www/CDash-2-0-2/backup/1722771811.xml, 1, 2, 1, 1584, 3f1768c96932eb14c69404d69f0d55d0, 2014-01-07 10:09:04, 2014-01-07 09:54:51, 2014-01-07 10:09:03, 2014-01-07 10:09:04
>> 226634, /var/www/CDash-2-0-2/backup/758486451.xml, 1, 2, 1, 5162754, feb09272338e599884934f0937921084, 2014-01-07 10:09:03, 2014-01-07 09:54:50, 2014-01-07 10:09:00, 2014-01-07 10:09:03
>> 226633, /var/www/CDash-2-0-2/backup/242524917.xml, 1, 2, 1, 1643, df12695150ba0a9b6326069eca3617a9, 2014-01-07 10:09:00, 2014-01-07 09:54:49, 2014-01-07 10:08:59, 2014-01-07 10:09:00
>> 226632, /var/www/CDash-2-0-2/backup/328287145.xml, 1, 2, 1, 1000, 9e46d671ef9b1c17658f1235e14d9203, 2014-01-07 10:08:59, 2014-01-07 09:54:48, 2014-01-07 10:08:58, 2014-01-07 10:08:59
>> 226631, /var/www/CDash-2-0-2/backup/1413398585.xml, 1, 2, 1, 1584, 0a2471cf0471b5a9f0c9f9fafe4c5e6a, 2014-01-07 10:08:58, 2014-01-07 09:53:53, 2014-01-07 10:08:57, 2014-01-07 10:08:58
>> 226630, /var/www/CDash-2-0-2/backup/1643613376.xml, 1, 2, 1, 5165054, 701d41e1b8a2e0756b1a30138c86f88c, 2014-01-07 10:08:57, 2014-01-07 09:53:52, 2014-01-07 10:08:54, 2014-01-07 10:08:57
>> 226629, /var/www/CDash-2-0-2/backup/1580744380.xml, 1, 2, 1, 1643, afd6ecbf0bca78d9700f7052f514bb6e, 2014-01-07 10:08:54, 2014-01-07 09:53:51, 2014-01-07 10:08:53, 2014-01-07 10:08:54
>> 226628, /var/www/CDash-2-0-2/backup/516354354.xml, 1, 2, 1, 1000, f2f3ce4da201305b25304acb45281c73, 2014-01-07 10:08:53, 2014-01-07 09:53:50, 2014-01-07 10:08:52, 2014-01-07 10:08:53
>> 226627, /var/www/CDash-2-0-2/backup/347351203.xml, 1, 2, 1, 5190173, b96b79efcb356f7cb699ff382542aaa8, 2014-01-07 09:52:50, 2014-01-07 09:48:11, 2014-01-07 09:52:47, 2014-01-07 09:52:50
>> 226626, /var/www/CDash-2-0-2/backup/196135980.xml, 1, 2, 1, 7980, b9d8fb37323efdd946042f7b9ea5d645, 2014-01-07 09:52:47, 2014-01-07 09:48:10, 2014-01-07 09:52:46, 2014-01-07 09:52:47
>> 226625, /var/www/CDash-2-0-2/backup/540120385.xml, 1, 2, 1, 1064, 856656e9f9f09a1ea80ad2265b692823, 2014-01-07 09:52:46, 2014-01-07 09:48:09, 2014-01-07 09:52:45, 2014-01-07 09:52:46
>> 226624, /var/www/CDash-2-0-2/backup/822820666.xml, 1, 2, 1, 2495, 19107c45a119eb4b32cad6f3f8819908, 2014-01-07 09:14:30, 2014-01-07 09:10:30, 2014-01-07 09:14:29, 2014-01-07 09:14:30
>> 226623, /var/www/CDash-2-0-2/backup/1436949172.xml, 1, 2, 1, 5161118, d809e82fb32f1f6e1b88b19821b5e5f0, 2014-01-07 09:14:29, 2014-01-07 09:10:29, 2014-01-07 09:14:26, 2014-01-07 09:14:29
>> 226622, /var/www/CDash-2-0-2/backup/868711020.xml, 1, 2, 1, 1643, 85d1592d4082faf2963727e6b4ed2bc2, 2014-01-07 09:14:26, 2014-01-07 09:10:28, 2014-01-07 09:14:25, 2014-01-07 09:14:26
>> 226621, /var/www/CDash-2-0-2/backup/1531771759.xml, 1, 2, 1, 1000, 096731f87d4d4f826a199410c0f280f8, 2014-01-07 09:14:25, 2014-01-07 09:10:27, 2014-01-07 09:14:24, 2014-01-07 09:14:25
>> 226620, /var/www/CDash-2-0-2/backup/1002205360.xml, 1, 2, 1, 2495, 3d9698df2114a13199e2947cf2b91366, 2014-01-07 09:14:24, 2014-01-07 09:07:10, 2014-01-07 09:14:23, 2014-01-07 09:14:24
>> 226619, /var/www/CDash-2-0-2/backup/1082455045.xml, 1, 2, 1, 5164845, c96cfcc43fb580ca7c099117c370d57f, 2014-01-07 09:14:23, 2014-01-07 09:07:09, 2014-01-07 09:14:20, 2014-01-07 09:14:23
>> 226618, /var/www/CDash-2-0-2/backup/1147021750.xml, 1, 2, 1, 1643, 371946b5a5e948744d64966d8da5ef69, 2014-01-07 09:14:20, 2014-01-07 09:07:08, 2014-01-07 09:14:19, 2014-01-07 09:14:20
>> 226617, /var/www/CDash-2-0-2/backup/1238403471.xml, 1, 2, 1, 1000, 522474f1cbfc4259b29d090f96bc84e1, 2014-01-07 09:14:19, 2014-01-07 09:07:07, 2014-01-07 09:14:18, 2014-01-07 09:14:19
>> 226616, /var/www/CDash-2-0-2/backup/620749966.xml, 1, 2, 1, 5163405, 1dae92592f807e250f0dc9fa5a7ec793, 2014-01-07 08:54:50, 2014-01-07 08:08:30, 2014-01-07 08:54:46, 2014-01-07 08:54:50
>> 226615, /var/www/CDash-2-0-2/backup/849392602.xml, 1, 2, 1, 8002, 960e2e3f8fd7a79144c316c8b7e2d897, 2014-01-07 08:54:46, 2014-01-07 08:08:29, 2014-01-07 08:54:45, 2014-01-07 08:54:46
>>
>> Any ideas what goes wrong here would be very helpful.
>>
>> Regards,
>>
>> Martin
>>
>>
>>
>>
>> _______________________________________________
>> Cdash mailing list
>> Cdash at public.kitware.com
>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>>
--
Martin Apel
Software Architect
Phone: + 49 8105 77266-53
E-Mail: martin.apel at simpack.de
SIMPACK AG
Friedrichshafener Strasse 1, 82205 Gilching, Germany
info at simpack.de, www.simpack.com
Phone: + 49 8105 77266-0
Fax: + 49 8105 77266-11
Executive Board: Dr. Alexander Eichberger, Dr. Lutz Mauer
Chair of Supervisory Board: Silvia Förster (CPA)
Commercial Register München HRB 181 229
More information about the CDash
mailing list