[vtk-developers] Massive lag in a Pileline. Too complex pipeline ?

Mon Mar 5 14:56:05 EST 2018

Hi all,

Thank you greatly for your responses.

Well I do agree ParaView is a different beast, but I have seen no problem in scaling with the number of actors (except maybe when loading a StateFile).

I’m will take a look at mapper->StaticOn(), but I’m afraid the problem will now be of determining when a mapper should be static and when it should be dynamic.

A more automated principle would be welcome.

Jonathan Borduas

From: Ken Martin [mailto:ken.martin at kitware.com]
Sent: March 5, 2018 12:56 PM
To: Will Schroeder <will.schroeder at kitware.com>
Cc: Jonathan Borduas <jonathan.borduas at caboma.com>; vtk-developers at vtk.org
Subject: Re: [vtk-developers] Massive lag in a Pileline. Too complex pipeline ?

For VTK there is no need to disconnect. In fact mapper->StaticOn() is faster than disconnecting once you have rendered once I believe.  Also if you know your data is opaque setting ForceOpaqueOn() helps as well. With just those two settings on VTK master I can get 10000 actors rendering at a rate of 11 fps on a laptop.

For VTK I really do believe some of the core information object pipeline stuff needs to be reworked. Simple calls like GetInputObject take a huge % of time and create multiple objects on the heap per call etc.

ParaView is a different beast as it has GUI etc.

On Mon, Mar 5, 2018 at 12:04 PM, Will Schroeder <will.schroeder at kitware.com<mailto:will.schroeder at kitware.com>> wrote:
Just a quick response off of the top of my head:

vtkMapper::Static (i.e, SetStaticOn()) prevents the mapper from propagating updates down the pipeline.

I have seen pipelines hacked as follows: executing them once, and then perform a local shallow copy of the resulting data into data object(s) (e.g., polydata) which is disconnected from the pipeline hence the updates are fast.

I suspect there are more elegant ways but maybe this will get the conversation going...

On Mon, Mar 5, 2018 at 11:56 AM, Jonathan Borduas <jonathan.borduas at caboma.com<mailto:jonathan.borduas at caboma.com>> wrote:

Hi all,

VTK pipelines are great, they allow to dynamically create a flow of algorithms. However, we recently encountered a massive lag in our pipeline related to the pipeline mechanisms (as opposed to the algorithms’ calculation time).  We are using a derivative of ParaView with homemade filters.

We profiled our application in such a way that the profiling session started when all of the algorithms present in the pipeline had already been executed once and were up to date. The profiler pointed out two methods that were called a very significant amount of times while no updates were required on any of the algorithms of the pipeline. Those methods were:

1) vtkInformation::GetAsObjectBase(...)
2) vtkDemandDrivenPipeline::ComputePipelineMTime(...)

This lag doesn’t appear when only filters at the beginning of the pipeline are shown, but as we go down in the pipeline and show the outputs of filters, the lag become unbearable:

We think this lag is caused by the number of inputs our filters have combined with the length and complexity of the pipeline. In the following, I will explain our line of thought and propose a way to solve our pipeline scalability problem.

Demand Driven Pipelines for Representation:

Our understanding is that a representation pipeline usually involves that the multiplicity of outputs is greater than the multiplicity of inputs. The pipeline fans out.  This mean that the Update() process is rather fast and scalable.

[cid:image001.jpg at 01D3B489.338F1290]
In the previous picture, if filter 3 needs an Update(), only filter 0,1 and 3 will have a RequestData().

Demand Driven Pipeline for Modeling:

However, in our case, we use VTK as a visualisation library as well as a mesh processing tool. To process mesh, you often need multiple inputs (boolean operations, points projections on a mesh…): the multiplicity of inputs is equivalent to the multiplicity of outputs.

[cid:image002.jpg at 01D3B489.338F1290]

If filter 8 needs an Update(), all filters Upstream will require to Update() except (6,3,1)

With high input multiplicity, the Update() process, even with a distributed executive pipeline, will impact performance, creating a significant overhead.

Demand Driven Pipeline with Event Driven Subsets:

Here we suggest an improvement over the DemandDrivenPipeline by introducing Event Driven (push) subsets:

[cid:image003.jpg at 01D3B489.338F1290]
If Filter 8 needs an Update(), the signal will be passed to the subset and then to Filter0. Filter 2,4,5 and 7 are force to calculate in a Event-driven fashion.

The overhead of the representation pipeline Update() when you have a high input multiplicity could be reduced with the creation of subsets of a pipeline. The subsets should be handled as Event Triggered Pipelines (Push Pipelines).

A push subset could be considered like a filter that contains a small part of the pipeline and where all outputs execute a RequestData() using a Push Pipeline.

We are aware of the vtkThreadedStreamingPipeline that had PUSH/PULL capability, however it is now legacy. Also, we do not require the threaded/scheduler capability of this pipeline model.  An extended version of the vtkDemandDrivenPipeline with a Event Driven (pull) pipeline that would apply to a subset would be ideal.

Please share your thoughts  on creating such a pipeline. Please keep in mind that this suggestion has the goal to keep the pipeline’s dynamism, we are aware that we could package the subset into a single filter.

Thank you,

Jonathan Borduas

_______________________________________________
Powered by www.kitware.com<http://www.kitware.com>

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

--
William J. Schroeder, PhD
Kitware, Inc. - Building the World's Technical Computing Software
28 Corporate Drive
Clifton Park, NY 12065
will.schroeder at kitware.com<mailto:will.schroeder at kitware.com>
http://www.kitware.com
(518) 881-4902<tel:(518)%20881-4902>

_______________________________________________
Powered by www.kitware.com<http://www.kitware.com>

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Search the list archives at: http://markmail.org/search/?q=vtk-developers

Follow this link to subscribe/unsubscribe:
https://vtk.org/mailman/listinfo/vtk-developers

--
Ken Martin PhD
Distinguished Engineer
Kitware Inc.
28 Corporate Drive
Clifton Park NY 12065

This communication, including all attachments, contains confidential and legally privileged information, and it is intended only for the use of the addressee.  Access to this email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken in reliance on it is prohibited and may be unlawful. If you received this communication in error please notify us immediately and destroy the original message.  Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://vtk.org/pipermail/vtk-developers/attachments/20180305/81567e00/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 5915 bytes
Desc: image001.jpg
URL: <https://vtk.org/pipermail/vtk-developers/attachments/20180305/81567e00/attachment-0003.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 10479 bytes
Desc: image002.jpg
URL: <https://vtk.org/pipermail/vtk-developers/attachments/20180305/81567e00/attachment-0004.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.jpg
Type: image/jpeg
Size: 14261 bytes
Desc: image003.jpg
URL: <https://vtk.org/pipermail/vtk-developers/attachments/20180305/81567e00/attachment-0005.jpg>