[Paraview-developers] MAXIMUM_NUMBER_OF_PIECES (was suspicious behaviour ...)

Thu Sep 8 10:55:20 EDT 2011

David, et al,

I need some advice on pipeline re-execution.

I've removed all the spurious pipeline re-execution problems I had, by simply stopping the NeedToExecute in the executives from ever saying yes based on ghost cells. However, I can't do the same for number of pieces.

I suspect that one reason my filters are crashing on massive data and causing issues on others is because paraview constantly triggers unnecessary executions when stuff hasn't changed.

My pipeline is essentially as follows (please view using wide window and courier font)

H5PartReader (reads N pieces, exports polydata)

   |

   V

ParticleRepartitionFilter (UG/polydata, redistributes particles across processes

and replaces ExtentTranslator with BoundsExtentTranslator, redistribution

is spatial (KdTree) and irregular - also adds scalar array to

denote ghost particles which are duplicated around process boundaries)

   |

   |___________________

___                   V

|                    ImageGridSampler (creates a vtkImageData grid and uses the

|                    BoundsExtentTranslator to get the right extents for the piece

|                    this means we can get irregular subdivisions of space)

|           ________________|

V          V

SPHResampler (any datatype, resamples the particles using an SPH kernel onto

the vtkImageData provided by the image sampler - the use of the BoundsExtentTranslator

in the earlier stages ensures we have the data we want to reample on the right process

for the sampling geometry)

This bit works fine (on the desktop, but sometimes crashes on the cluster when running N=64 processes), ... but prompted by your suggestion, I threw in a volume renderer after the SPHResampler to view the resampled data as a volume.

This triggers a whole bunch of filter reexecutions, caused by updateNumberOfPieces(1) != dataNumberOfPieces(4) (4 being what I'd expect it to be not 1 which is just silly)

I have not yet identified who is setting the update pieces to 1, when all the rest of the pipeline is using 4, but I experimented with using wavelet sources and creating pipelines which were similar and I found that the problem was not unique to my setup.

I'd like to know if there is any rule to

NOT SET Maximum_number_of_pieces

NOT SET Any kind of piece information

YES SET Extent information etc

in particular kinds of filters.

By which I mean, image data generators should NEVER set maximum number of pieces? and piece extent translators etc etc. I'm looking for clues as to why my pipeline might misbehave, because debugging the executives is hard work and time consuming (particularly so since David completely buggered them up with all his fancy streaming stuff :) )

Any advice welcome.

JB

PS. I see that paraview has updated it's number scheme to 3.12 RCxxx : noting that Paraview only really works properly in serial, is this wise?

-----Original Message-----
From: paraview-developers-bounces at paraview.org [mailto:paraview-developers-bounces at paraview.org] On Behalf Of Biddiscombe, John A.
Sent: 07 September 2011 09:57
To: David E DeMarle
Cc: paraview-developers at paraview.org
Subject: Re: [Paraview-developers] suspicious behaviour with structured Grids (in partuicular)

David

>Was volume rendering involved?

Not involved. The visual errors appear when drawing just a surface representation.

I tried a large run on a 64 pvserver job yesterday using the new imagedata version of the resampler, after the resampling completed, pvservers shut down as before and I lost connection. I am trying to debug it on the cluster today but my parallel debugging skills on linux are sadly quite poor.

JB

David E DeMarle

Kitware, Inc.

R&D Engineer

21 Corporate Drive

Clifton Park, NY 12065-8662

Phone: 518-881-4909

On Thu, Sep 1, 2011 at 2:32 AM, Biddiscombe, John A. <biddisco at cscs.ch> wrote:

> I've had a lot of problems with a particular set of filters of mine which resample SPH data onto grids.

>

> I have a Grid Generator class which is used as the sample location. This class outputs a vtkStructuredGrid.

> When running in Parallel, I see very odd behaviour - which appears similar to a post I saw some months ago but can't find now, where the data displayed appears to be incorrect. Almsot as though the data from process 0 is being drawn on all processes and something is just not right, but I can't quite pin it down.

>

> Because I have custom extent translators, dodgy extent manipulations and all parallel features, I can never be quite sure that it's paraview that's fundamentally broken or my stuff.

>

> However, in the case where the resampling grid is regular and axis aligned, the output could be vtkImageData, which I believe does work quite well in parallel, so I made a new stripped down resampler which outputs only image data. This works fine. The extent translation/manipulation is the same, the resampling the same, everything the same, except it produces image data.

>

> This one doesn't crash or misbehave when run in parallel, however the image looks wrong 'first time' it is done, if I delete the filter, recreate a second one and then reexecute using identical settings, I get a completely different image and it looks perfect.

>

> I really don't understand what is going on, but my feeling now is that paraview is broken somewhere deep inside (and is essentially unfit for purpose)

>

> The problem is slightly less bad on 3.11 than on 3.10, but I still don't know what is actually wrong. I still don't know if I've messed up, or if ParaView is messing me up. (I have a complicated scenario where a multi-input filter passes update extent information from the second input).

>

> I'm only posting this because I have some vague memory that someone else reported duplicated colours from process 0 on some gridded data in parallel and it's just possible, I'm seeing aspects of the same bug. I can't file a proper bug report because I'm still not completely sure a bug really exists. I'm just hoping someone reading this might mention something which triggers other thoughts, comments etc.

>

> yours

>

> JB

>

>

> --

> John Biddiscombe,                            email:biddisco @ cscs.ch

> http://www.cscs.ch/

> CSCS, Swiss National Supercomputing Centre  | Tel:  +41 (91) 610.82.07

> Via Cantonale, 6928 Manno, Switzerland      | Fax:  +41 (91) 610.82.82

>

>

> _______________________________________________

> Paraview-developers mailing list

> Paraview-developers at paraview.org

> http://public.kitware.com/mailman/listinfo/paraview-developers

>

_______________________________________________

Paraview-developers mailing list

Paraview-developers at paraview.org

http://public.kitware.com/mailman/listinfo/paraview-developers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/paraview-developers/attachments/20110908/c6ac3bf2/attachment.htm>