[CMake] Benchmarking with CMake

[Nagy-Egri Máté Ferenc]

Thank you all for your resonses,

My use case is that I have a set of executables that function as unit tests. Currently I use the return value of the .exe as a means of indicating success, though I know there is way to match std::cout against a regex for more complex tests. However, the timings of the tests are not always suitable for benchmarking, because I only want the timings of a particular part of the tests. Their initialization times are comparable to their runtimes. While I could tune a parameter so that the init phase is negligable, but then benchmarking would take for ages. I want to write benchmark executables (similar to unit tests) that write a single number to the console and nothing else. Run some command (make bench?, cbench?) which runs, these tests, fetches the sole number from the std::cout and prints those numbers as output instead of the total run time presented by ctest.

@Peter: I also encountered Google Benchmark. I have already seen it’s output and is perfectly fine. I am however a bit reluctant to grab another dependency that all my consumers would be advised to install. CMake is a fairly full blown suite and I was hoping to find something that could provide similar results.

@Matt: Seems nice. Python is not much a dependency, but VTK is less evident to be present on all machines.

@Alexander: I still have to look into this, because information seems a little scarce on doing it. It seems to me, that CDash has to be installed somewhere to be able visualize the data. Is there a way to generate the HTML-r for local consumption instead of submitting somewhere? (Ideally bypassing a local installation of CDash)

I know, I have a lot of problems. Ultimately I will choose one of the above, I was just looking for the one that is the least painful. I am very reluctant on obtaining dependencies. CMake is such a good tool, it seems strange to me that no direct benchmarking support has been added yet.

Any ideas?


Feladó: Nagy-Egri Máté Ferenc via CMake
Elküldve: 2015. november 9., hétfő 23:27
Címzett: CMake felhasználók
Tárgy: [CMake] Benchmarking with CMake


Hi!

Can anyone tell me if CMake (or CTest) can be customized in a way to produce decent benchmark output? An .xlsx perhaps, or something GnuPlot friendly? I have not found any examples of CTest being (ab)used in this way.

Is my feeling correct, that this should be something like CBench?

Cheers,
Máté



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake/attachments/20151113/7bdc1dc0/attachment.html>