[vtk-developers] Massive lag in a Pileline. Too complex pipeline ?

[Jonathan Borduas]

Hi all,



VTK pipelines are great, they allow to dynamically create a flow of algorithms. However, we recently encountered a massive lag in our pipeline related to the pipeline mechanisms (as opposed to the algorithms' calculation time).  We are using a derivative of ParaView with homemade filters.



We profiled our application in such a way that the profiling session started when all of the algorithms present in the pipeline had already been executed once and were up to date. The profiler pointed out two methods that were called a very significant amount of times while no updates were required on any of the algorithms of the pipeline. Those methods were:


1) vtkInformation::GetAsObjectBase(...)
2) vtkDemandDrivenPipeline::ComputePipelineMTime(...)



This lag doesn't appear when only filters at the beginning of the pipeline are shown, but as we go down in the pipeline and show the outputs of filters, the lag become unbearable:



We think this lag is caused by the number of inputs our filters have combined with the length and complexity of the pipeline. In the following, I will explain our line of thought and propose a way to solve our pipeline scalability problem.



Demand Driven Pipelines for Representation:

Our understanding is that a representation pipeline usually involves that the multiplicity of outputs is greater than the multiplicity of inputs. The pipeline fans out.  This mean that the Update() process is rather fast and scalable.



[cid:image001.jpg at 01D3B474.DFE725B0]
In the previous picture, if filter 3 needs an Update(), only filter 0,1 and 3 will have a RequestData().



Demand Driven Pipeline for Modeling:

However, in our case, we use VTK as a visualisation library as well as a mesh processing tool. To process mesh, you often need multiple inputs (boolean operations, points projections on a mesh...): the multiplicity of inputs is equivalent to the multiplicity of outputs.

[cid:image002.jpg at 01D3B474.DFE725B0]

If filter 8 needs an Update(), all filters Upstream will require to Update() except (6,3,1)

With high input multiplicity, the Update() process, even with a distributed executive pipeline, will impact performance, creating a significant overhead.



Demand Driven Pipeline with Event Driven Subsets:

Here we suggest an improvement over the DemandDrivenPipeline by introducing Event Driven (push) subsets:



[cid:image003.jpg at 01D3B474.DFE725B0]
If Filter 8 needs an Update(), the signal will be passed to the subset and then to Filter0. Filter 2,4,5 and 7 are force to calculate in a Event-driven fashion.



The overhead of the representation pipeline Update() when you have a high input multiplicity could be reduced with the creation of subsets of a pipeline. The subsets should be handled as Event Triggered Pipelines (Push Pipelines).

A push subset could be considered like a filter that contains a small part of the pipeline and where all outputs execute a RequestData() using a Push Pipeline.



We are aware of the vtkThreadedStreamingPipeline that had PUSH/PULL capability, however it is now legacy. Also, we do not require the threaded/scheduler capability of this pipeline model.  An extended version of the vtkDemandDrivenPipeline with a Event Driven (pull) pipeline that would apply to a subset would be ideal.



Please share your thoughts  on creating such a pipeline. Please keep in mind that this suggestion has the goal to keep the pipeline's dynamism, we are aware that we could package the subset into a single filter.

Thank you,

Jonathan Borduas

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://vtk.org/pipermail/vtk-developers/attachments/20180305/9efc4d2f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 5915 bytes
Desc: image001.jpg
URL: <https://vtk.org/pipermail/vtk-developers/attachments/20180305/9efc4d2f/attachment-0003.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 10479 bytes
Desc: image002.jpg
URL: <https://vtk.org/pipermail/vtk-developers/attachments/20180305/9efc4d2f/attachment-0004.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.jpg
Type: image/jpeg
Size: 14261 bytes
Desc: image003.jpg
URL: <https://vtk.org/pipermail/vtk-developers/attachments/20180305/9efc4d2f/attachment-0005.jpg>