[Paraview] Paraview starts to lag extremely after adding too much complexity in the pipeline

[Jonathan Borduas]

Hi Utkarsh,

At first I though so, then we converted all our small statefiles to .cpd and used those custom filters into a "master" statefile. The result was much less "GeometryRepresentation". It did reduce the loading time by minutes but the bulk of the loading time was still there. At that moment we decided to profile the load state mechanism with Vtune and the vtkDemandDrivenPipeline::ComputePipelineMTime was definitely the culprit (99.6% of cputime). 

This scenario might be related to  :
https://public.kitware.com/pipermail/vtkusers/2013-November/082003.html

After searching more, I think we simply reached the VTK limit in terms of Demand-Driven Pipeline. I understand that a Demand-Driven pipeline is perfect with low pipeline complexity, expansive computation where we want to maximize interactivity. But for complex pipeline with fast algorithms an Event-Driven pipeline might be better. However in our case, we have a complex pipeline, that we want to be as interactive as possible while having expansive computation. So we need an hybrid of the two type of pipelines.

I know we could convert all our smaller pipelines to filters, and reduce the overall pipeline complexity. However, at the speed we are making changes and prototyping, using the interface of ParaView to create our pipeline is our goal. I have two solutions: 

1 - A way to keep the flexibility and prototyping capability while having complex subset of filters could be to use programmable filters. However the inexistent error passing of a programmable filter makes it impossible to use in a production context. This issue would help in solving our problem: https://gitlab.kitware.com/paraview/paraview/issues/17591 

2 - A good way might be to keep the main statefile as a Demand-Driven Pipeline, while having smaller subset execute as Event-Driven Pipeline. The way to define those subset could be by using .cpd file. 

For #2, I have an implementation in mind, which might be difficult to implement (and a bit crazy too) : create a pvserver for each custom filter that is instantiate in the pipeline.  This way, I keep the interactivity of all 3dwidget in the .cpd while keeping all filters contains in the .cpd outside of the main pipeline. The main pipeline stays a fast and interactive Demand-Driven pipeline while .cpd behave like small Event-Driven pipeline.   This mechanics would be handle by the custom filter itself and would act as a wrapper to a external pvserver. For passing the information between the main pipeline and slaves pvserver, I though about Catalyst. 

Of course, any simpler way to tell Paraview to handle a custom filter as an event-driven pipeline is welcome.

Best regards,

Jonathan Borduas


-----Original Message-----
From: Utkarsh Ayachit [mailto:utkarsh.ayachit at kitware.com] 
Sent: February 28, 2018 10:02 AM
To: Jonathan Borduas <jonathan.borduas at caboma.com>
Cc: paraview at paraview.org
Subject: Re: [Paraview] Paraview starts to lag extremely after adding too much complexity in the pipeline

Jonathan,

Do you think this is related to the same issue or something else?

https://gitlab.kitware.com/paraview/paraview/issues/17957

Is there a sample state file that you can share?

Utkarsh


On Tue, Feb 27, 2018 at 3:39 PM, Jonathan Borduas <jonathan.borduas at caboma.com> wrote:
> Hi all,
>
> We use a fairly complex pipeline with more than 300 filters.
>
> When divided across 20 .pvsm , everything is fast. We started to merge 
> those .pvsm into a single one.
>
> For the first 17 .pvsm everything was still fast. No .pvsm load time 
> and no lag.
>
> Once we started integrating the 18th .pvsm the performance started to 
> degrade. We used VTune to diagnose paraview:
> Seems that the performance drop is purely related to 
> vtkInformation::GetAsObjectBase and 
> vtkDemandDrivenPipeline::ComputePipelineMTime which is call 
> recursively  37 times in a single call stack sample !
>
>
>
> It makes no difference if we integrate the 20 small .pvsm into a 
> single large .pvsm or if we integrate thoses .pvsm as custom filters 
> (.cpd) in the main .pvsm.
>
>
>
> This is true when we are loading the .pvsm, but this is also true when 
> we are modifying values in the statefile.
>
> We are using:
> Win10
> Paraview 5.3 derivative.
>
> Memory consumption is 1Go on a 16 Go machine.
>
>
>
> Any idea on how to improve the performance ? We though about dividing 
> the pipeline into a multitude of pvserver…
>
>
>
> Jonathan Borduas
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at 
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at:
> http://paraview.org/Wiki/ParaView
>
> Search the list archives at: http://markmail.org/search/?q=ParaView
>
> Follow this link to subscribe/unsubscribe:
> https://public.kitware.com/mailman/listinfo/paraview
>