[CDash] Slow CDash submission processing leading to backlog

[Sitsofe Wheeler]

Hi,

We’ve been having issues with CDash 2.6.0 and the MySQL 5.7.26
database backing it with asynchronous uploads becoming tied up/stuck
running queries like this:

SELECT COUNT(*) FROM
      (SELECT name FROM build2test,test WHERE build2test.buildid=‘2602946’ AND
       build2test.testid=test.id AND build2test.status=‘passed’ ) AS testa
    LEFT JOIN (SELECT name as name2 FROM build2test,test WHERE
build2test.buildid=‘2603291’
        AND build2test.testid=test.id AND build2test.status=‘passed’ )

These queries do not finish quickly and they seem to typically be in
the “sending” state:

select id,TIME_FORMAT(SEC_TO_TIME(time),‘%Hh %im’),state from
information_schema.processlist where time > 200;
+--------+------------------------------------------+--------------+
| id  | TIME_FORMAT(SEC_TO_TIME(time),‘%Hh %im’) | state    |
+--------+------------------------------------------+--------------+
| 253423 | 04h 54m                 | Sending data |
| 222453 | 38h 06m                 | Sending data |
[...]
| 283752 | 10h 37m                 | Sending data |
+--------+------------------------------------------+--------------+
145 rows in set (0.01 sec)

(I assume this is a query from CDash trying to calculate when certain
tests started failing/stopped failing)

These queries seem to finish so slowly they start accumulating until
there are no connections left. We have two seperate cdash+mysql
instances and both are exhibiting problematic behaviour. The end
result is that new test uploads become backed up for days and even the
CDash web server eventually becomes unresponsive until the database is
restarted. A single build is uploading upwards of 15000 tests (albeit
split across multiple uploads) and we have over a hundred such builds
per day (and this number is growing).

Are the bottlenecks we’re seeing to be expected and are there any
expected limits of CDash when it comes to upload sizes for raw size
and test counts? Also, is there a way to turn off the checking for
when test failure started/ended?

-- 
Sitsofe