[Cdash] [cmake-developers] RFC: pretty/HTML output in test logs in ctest/cdash

[Julien Jomier]

Alex,

I'm very sorry for the late reply.

I like this idea very much. I'm a little bit ambivalent about adopting 
HTML as the output as it could be tricky to standardize the output on 
the CDash side, but at the same time it gives flexibility. I'm wondering 
if CDash should do this automatically based on some pre-defined tags.

Anyone has an opinion on this?

Regarding the short lines, I would think that the test should be 
modified to produce the right output. We could implement something in 
CDash but it might be difficult to make it generic enough.

Julien

>> We are using cdash a lot, and for many of our tests the log is quite
>> verbose and somewhat hard to read. When looking for the reason why a test
>> failed, it can be somewhat hard to find the line which contains "ERROR".
>> Also, a lot of stuff is repeated every line, which also makes the log
>> harder to read, e.g. the directory part of the file where the log came
>> from:
>>
>> <timestamp> /some/where/deep/in/a/subdir/test.py: Something...
>> <timestamp> /some/where/deep/in/a/subdir/test.py: happend...
>> <timestamp> /some/where/deep/in/a/subdir/test.py: here...
>> <timestamp> /some/where/deep/in/a/subdir/test.py: ...
>>
>> The directory part is actually so wide that we have to scroll to see actual
>> text.
>>
>> So, to help with this, we came up with the idea to use HTML to format the
>> log so that it becomes easier to read.
>>
>> The attached patch for ctest adds a new test property GENERATE_HTML, and if
>> this is enabled, ctest generates HTML. It searches the lines which made the
>> test fail and colors them red, and the lines which make a test pass are
>> colored green.
>> This makes it easier to find the fail/succeed lines.
>>
>> (In the patch the fail strings are still hardcoded, this would have to use
>> the respective test property.)
>>
>> On the cdash side, if "</span>" is found in the log, it is considered HTML
>> and not escaped. I guess this should be done in a different way, which
>> does not involve parsing the log text.
>> For making our log lines shorter, we are not sure yet how to do that.
>> We could add support for squish XML to ctest, so that ctest reads the
>> squish XML and converts it to HTML which is in some way shorter, maybe a
>> HTML table, or something.
>> Or we could add a test property with a regular expression, if that regexp
>> matches, ctest replaces it with a shorter version or something.
>> Or we could modify our test scripts so that they already generate the
>> pretty HTML, which ctest then simply sends to cdash.
>> That's the ideas we have so far, but it doesn't feel quite right yet.
>>
>> So, what do you think ?
>
> Any comments ?
>
> Alex
>