[CDash] Fwd: filtering and comparing test results on a CDash server

[Zack Galbreath]

Hi John,

Thanks for reaching out, and sorry for the slow reply.

On Thu, Jan 7, 2021 at 5:13 PM John Verdicchio <
john.verdicchio at cd-adapco.com> wrote:

> [image: cdash.png]
> For each test we run (via ctest) we are sending statistical data about our
> program via the <DartMeasurement></DartMeasurement> capability. As you can
> see from the attached picture the statistics we gather is a mix of float
> and integer data.
>
> Question 1:
> Is it possible OOTB to filter this list of statistical data? e.g. only
> show the _count_ datasets? This screenshot is a subset of the complete list
> around 160 entries. This list is most likely going to grow as we add
> additional statistical elements from other parts of the code. If it isn't
> possible OOTB, how much coding effort would be involved?
>

I don't think this is currently possible with CDash OOTB. That being said,
we do have an existing filtering system in CDash, so it's definitely
possible. For someone already familiar with the ins and outs of CDash (ie,
me) I would expect this to take hours, but not days. I could give you some
pointers if you want to try to add it yourself.


As we change and test our program the statistics for any given test may
> change e.g. SurfaceRemesher_cpu_cadProjectionTimer_total_ may change from
> 21.027 (seconds) to 19.01 seconds.
>
> What we would like to do is be able to select two sets of data from the
> same test and "compare" the statistical data. In my example, be able to
> difference the SurfaceRemesher_cpu_cadProjectionTimer_total_ = 21.027 -
> 19.01 = 2.017. Ideally, these comparisons could be more complex that
> add/subtract, but relatively arbitary expressions involving +,-,/,*,min,max
> and numbers
>

I might not be 100% following you here, but we do have the capability to
chart how test measurements change from one run to the next. I know we also
have capability to download test data as CSV for offline parsing, but I'd
have to take a closer look to see if that suits your use case or not.



> Question  2:
> We could just download the data from the server and do the difference
> locally via a python script (this script does exist). However, I'm
> wondering if this is possible within the CDash server environment. If not
> possible OOTB, how much coding effort would be involved? Could this be
> achieved with some .php programming on the CDash side? I can see this as
> several problems 1) Presenting and Selecting the two sets of results from
> any given test 2) Coding a comparison engine within the CDash server that
> takes the two results and compares them 3) presenting the comparison back
> to the user.
>

Happy to discuss this further. Almost anything is possible, but I would
need to gain a better understanding of your use case to see if it's general
enough to be a good fit for CDash or not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cdash/attachments/20210217/41690201/attachment-0002.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cdash.png
Type: image/png
Size: 191062 bytes
Desc: not available
URL: <http://public.kitware.com/pipermail/cdash/attachments/20210217/41690201/attachment-0002.png>