[Smtk-developers] Attribute callback benchmarks

[David Thompson]

Hi all,

In a continuing series of followups, I've been able to reduce the initial overhead of a callback to within a few nanoseconds of nothing by eliminating overhead from the construction of the event object and (following the SimpleSignal design) moving the trigger method to the event object (which then passes itself by reference to each event responder).

Without any changes to SMTK:
a. Baseline:                     131 +/-  7 nsec per setValue() call.
With callback code added:
b. No callbacks registered:      134 +/-  3 nsec per setValue() call.
c. With 1000 empty callbacks:   2317 +/- 17 nsec per setValue() call.
d. With 1 trivial callback:      142 +/-  7 nsec per setValue() call.

Given that I've only instrumented 3 or 4 of the 61 different events so far, there is still plenty of work to do. However, this seems to be a reasonable design.

The only issues left to address before working on the remaining events are

1. Should we investigate a signal-slot library? This seems redundant given the low overhead of the current design.

2. What is required to provide callback access to Python? Ben suggested writing a special callback that invokes a Python function (I'm guessing we would need one of these for each event type), but it's not clear to me how to integrate this with shiboken.

	David

> On Aug 11, 2015, at 2:29 PM, David Thompson <david.thompson at kitware.com> wrote:
> 
> To follow up,
> 
> 1. The timings in my original message were from a debug build (doh!). For a release build, the ratio of time spent is similar but the times are smaller (nanoseconds not microseconds):
> 
> Without any changes to SMTK:
> a. Baseline:                     131 +/-  7 nsec per setValue() call.
> With callback code added:
> b. No callbacks registered:      340 +/-  6 nsec per setValue() call.
> c. With 1000 empty callbacks:   2533 +/- 22 nsec per setValue() call.
> d. With 1 trivial callback:      342 +/-  7 nsec per setValue() call.
> 
> So, subtracting the apparent per-setValue cost (340 nsec) from the 1000-empty-callback version and dividing by 1000, it looks like 2.2 nsec per callback.
> 
> 2. At least one signal-slot library I wanted to consider was SimpleSignal:
>    https://testbit.eu/cpp11-signal-system-performance/
> 
> 	David
> 
> On Aug 11, 2015, at 11:38 AM, David Thompson <david.thompson at kitware.com> wrote:
> 
>> Hi all,
>> 
>> I am working on adding callbacks to notify applications when attributes are changed. One of the concerns has been the overhead of callbacks on value change events, since these could conceivably happen frequently. I've done some preliminary benchmarking to quantify the cost of calling C++ lambdas each time an attribute value changes. The results are below.
>> 
>> To benchmark the timing, I create an attribute with a single DoubleItem and repeatedly change the value of the DoubleItem to a random number. The timings were done on my MacBookPro running Mac OS X 10.9.
>> 
>> Without any changes to SMTK:
>> 1. Baseline:                   0.355 +/- 0.01 usec per setValue() call.
>> With callback code added:
>> 2. No callbacks registered:    1.238 +/- 0.01 usec per setValue() call.
>> 3. With 1000 empty callbacks: 21.32  +/- 0.20 usec per setValue() call.
>> 4. With 1 trivial callback:    1.281 +/- 0.02 usec per setValue() call.
>> 
>> These times were averaged over 5-6 runs, with each run consisting of 1-million setValue calls (except for the version with 1000 callbacks registered, which used 100,000 setValue calls per run).
>> 
>> I have not evaluated signal/slot libraries; the callback code uses smtk::function<void(const EventType&)> to hold references to callbacks.
>> 
>> 	David
>