[Cdash] How to handle huge test output

[Martin Apel]

Hi Julien,

we are indeed using MySQL as the database backend. As I don't know the
internal table organisation of CDash, I am wondering, if compressing
whole rows wouldn't slow down accesses
to the tables very much. Wouldn't it be better to only compress the
column containing the text output? This could be done in CDash upon
access of this column only.

I already tried to put a feature request in the bug tracker, but the bug
tracker, which is linked from www.cdash.org does only contain category
entries for CMake, CTest etc., but not for CDash.
So I thought I'd ask on the mailing list first.

Regards,

Martin

Julien Jomier wrote:
>
>
> Hi Martin,
>
> This is an interesting point and, as you said, we have two options:
>
> 1) Provide a way to store the body of the tests outside the database on
> disk. We would have to create a a directory structure to avoid writing
> all the files in the same place.
>
> 2) Compress the data in the database. I assume you are using MySQL (and
> MyISAM as your storage engine). One temporary option is to use InnoDB
> which allows for compression of the entire row. PostGreSQL already
> compresses the rows so we should be OK for this one.
>
> I think we should try option 2) first and see how this goes, since
> option 1) is a little bit more complex to implement. Could you log a
> feature request in the bug tracker?
>
> Thanks,
> Julien
>
> Martin Apel wrote:
>   
>> Hi all,
>>
>> we currently run lots of tests using CTest/CDash every night (about
>> 4000), which produce a huge amount of output. For reasons difficult to
>> explain here, we'd like to keep the output, even if the tests succeeded.
>> For this reason our database has grown a lot recently, so I am looking
>> for ways how to handle this. I thought about two ways:
>> 1. Is it possible to embed links into the test output, so we could store
>> the test output in separate files, and simply link to these files in
>> CDash? We have tried outputting links in the tests,
>>     but ctest seems to replace any special characters such as ampersand,
>> so we did not succeed until now.
>> 2. Is it possible to store the test output inside the database in a
>> compressed way? As it is ASCII output, it would probably save a lot of
>> disk space.
>>
>> Any comments are welcome.
>>
>> Best Regards,
>>
>> Martin
>>
>> _______________________________________________
>> Cdash mailing list
>> Cdash at public.kitware.com
>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>>
>>     
> .
>
>   


-- 

Martin Apel                                    
Software Architect                              
Phone:   + 49 8105 77266-53
E-Mail:  martin.apel at simpack.de

SIMPACK AG
Friedrichshafener Strasse 1, 82205 Gilching, Germany
info at simpack.de, www.simpack.com
Phone: + 49 8105 77266-0
Fax:   + 49 8105 77266-11


Executive Board: Dr. Alexander Eichberger, Dr. Lutz Mauer
Chair of Supervisory Board: Silvia Förster (CPA)
Commercial Register München HRB 181 229

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cdash/attachments/20091210/8ff59c23/attachment-0002.htm>