[Paraview-developers] MAXIMUM_NUMBER_OF_PIECES (was suspicious behaviour ...)

Thu Sep 8 11:01:18 EDT 2011

Just in case (would be luck but who knows...), I just committed a fix for
review:
http://review.source.kitware.com/#change,2719
I had issues with combined update extents due to
UPDATE_NUMBER_OF_PIECES forcing
a needtoexecute in the update_extent but not in the REQUEST_DATA.

Julien.

On Thu, Sep 8, 2011 at 10:55 AM, Biddiscombe, John A. <biddisco at cscs.ch>wrote:

>  David, et al,****
>
> ** **
>
> I need some advice on pipeline re-execution.****
>
> I've removed all the spurious pipeline re-execution problems I had, by
> simply stopping the NeedToExecute in the executives from ever saying yes
> based on ghost cells. However, I can't do the same for number of pieces.**
> **
>
> ** **
>
> I suspect that one reason my filters are crashing on massive data and
> causing issues on others is because paraview constantly triggers unnecessary
> executions when stuff hasn't changed.****
>
> ** **
>
> My pipeline is essentially as follows (please view using wide window and
> courier font)****
>
> ** **
>
> H5PartReader (reads N pieces, exports polydata)****
>
>    |****
>
>    V****
>
> ParticleRepartitionFilter (UG/polydata, redistributes particles across
> processes****
>
> and replaces ExtentTranslator with BoundsExtentTranslator, redistribution*
> ***
>
> is spatial (KdTree) and irregular – also adds scalar array to ****
>
> denote ghost particles which are duplicated around process boundaries)****
>
>    |****
>
>    |___________________****
>
> ___                   V****
>
> |                    ImageGridSampler (creates a vtkImageData grid and uses
> the****
>
> |                    BoundsExtentTranslator to get the right extents for
> the piece****
>
> |                    this means we can get irregular subdivisions of space)
> ****
>
> |           ________________|****
>
> V          V****
>
> SPHResampler (any datatype, resamples the particles using an SPH kernel
> onto****
>
> the vtkImageData provided by the image sampler – the use of the
> BoundsExtentTranslator****
>
> in the earlier stages ensures we have the data we want to reample on the
> right process****
>
> for the sampling geometry)****
>
> ** **
>
> This bit works fine (on the desktop, but sometimes crashes on the cluster
> when running N=64 processes), ... but prompted by your suggestion, I threw
> in a volume renderer after the SPHResampler to view the resampled data as a
> volume.****
>
> ** **
>
> This triggers a whole bunch of filter reexecutions, caused by
> updateNumberOfPieces(1) != dataNumberOfPieces(4) (4 being what I’d expect it
> to be not 1 which is just silly)****
>
> ** **
>
> I have not yet identified who is setting the update pieces to 1, when all
> the rest of the pipeline is using 4, but I experimented with using wavelet
> sources and creating pipelines which were similar and I found that the
> problem was not unique to my setup.****
>
> ** **
>
> I’d like to know if there is any rule to****
>
> NOT SET Maximum_number_of_pieces****
>
> NOT SET Any kind of piece information****
>
> YES SET Extent information etc****
>
> in particular kinds of filters.****
>
> ** **
>
> By which I mean, image data generators should NEVER set maximum number of
> pieces? and piece extent translators etc etc. I’m looking for clues as to
> why my pipeline might misbehave, because debugging the executives is hard
> work and time consuming (particularly so since David completely buggered
> them up with all his fancy streaming stuff J )****
>
> ** **
>
> Any advice welcome.****
>
> ** **
>
> JB****
>
> PS. I see that paraview has updated it’s number scheme to 3.12 RCxxx :
> noting that Paraview only really works properly in serial, is this wise?**
> **
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
> -----Original Message-----
> From: paraview-developers-bounces at paraview.org [mailto:
> paraview-developers-bounces at paraview.org] On Behalf Of Biddiscombe, John
> A.
> Sent: 07 September 2011 09:57
> To: David E DeMarle
> Cc: paraview-developers at paraview.org
> Subject: Re: [Paraview-developers] suspicious behaviour with structured
> Grids (in partuicular)
>
> ** **
>
> David****
>
> ** **
>
> >Was volume rendering involved?****
>
> ** **
>
> Not involved. The visual errors appear when drawing just a surface
> representation.****
>
> ** **
>
> I tried a large run on a 64 pvserver job yesterday using the new imagedata
> version of the resampler, after the resampling completed, pvservers shut
> down as before and I lost connection. I am trying to debug it on the cluster
> today but my parallel debugging skills on linux are sadly quite poor.****
>
> ** **
>
> JB****
>
> ** **
>
> ** **
>
> David E DeMarle****
>
> Kitware, Inc.****
>
> R&D Engineer****
>
> 21 Corporate Drive****
>
> Clifton Park, NY 12065-8662****
>
> Phone: 518-881-4909****
>
> ** **
>
> ** **
>
> ** **
>
> On Thu, Sep 1, 2011 at 2:32 AM, Biddiscombe, John A. <biddisco at cscs.ch>
> wrote:****
>
> > I've had a lot of problems with a particular set of filters of mine which
> resample SPH data onto grids.****
>
> >** **
>
> > I have a Grid Generator class which is used as the sample location. This
> class outputs a vtkStructuredGrid.****
>
> > When running in Parallel, I see very odd behaviour - which appears
> similar to a post I saw some months ago but can't find now, where the data
> displayed appears to be incorrect. Almsot as though the data from process 0
> is being drawn on all processes and something is just not right, but I can't
> quite pin it down.****
>
> >** **
>
> > Because I have custom extent translators, dodgy extent manipulations and
> all parallel features, I can never be quite sure that it's paraview that's
> fundamentally broken or my stuff.****
>
> >** **
>
> > However, in the case where the resampling grid is regular and axis
> aligned, the output could be vtkImageData, which I believe does work quite
> well in parallel, so I made a new stripped down resampler which outputs only
> image data. This works fine. The extent translation/manipulation is the
> same, the resampling the same, everything the same, except it produces image
> data.****
>
> >** **
>
> > This one doesn't crash or misbehave when run in parallel, however the
> image looks wrong 'first time' it is done, if I delete the filter, recreate
> a second one and then reexecute using identical settings, I get a completely
> different image and it looks perfect.****
>
> >** **
>
> > I really don't understand what is going on, but my feeling now is that
> paraview is broken somewhere deep inside (and is essentially unfit for
> purpose)****
>
> >** **
>
> > The problem is slightly less bad on 3.11 than on 3.10, but I still don't
> know what is actually wrong. I still don't know if I've messed up, or if
> ParaView is messing me up. (I have a complicated scenario where a
> multi-input filter passes update extent information from the second input).
> ****
>
> >** **
>
> > I'm only posting this because I have some vague memory that someone else
> reported duplicated colours from process 0 on some gridded data in parallel
> and it's just possible, I'm seeing aspects of the same bug. I can't file a
> proper bug report because I'm still not completely sure a bug really exists.
> I'm just hoping someone reading this might mention something which triggers
> other thoughts, comments etc.****
>
> >** **
>
> > yours****
>
> >** **
>
> > JB****
>
> >** **
>
> >** **
>
> > --****
>
> > John Biddiscombe,                            email:biddisco @ cscs.ch***
> *
>
> > http://www.cscs.ch/****
>
> > CSCS, Swiss National Supercomputing Centre  | Tel:  +41 (91) 610.82.07**
> **
>
> > Via Cantonale, 6928 Manno, Switzerland      | Fax:  +41 (91) 610.82.82**
> **
>
> >** **
>
> >** **
>
> > _______________________________________________****
>
> > Paraview-developers mailing list****
>
> > Paraview-developers at paraview.org****
>
> > http://public.kitware.com/mailman/listinfo/paraview-developers****
>
> >** **
>
> _______________________________________________****
>
> Paraview-developers mailing list****
>
> Paraview-developers at paraview.org****
>
> http://public.kitware.com/mailman/listinfo/paraview-developers****
>
> _______________________________________________
> Paraview-developers mailing list
> Paraview-developers at paraview.org
> http://public.kitware.com/mailman/listinfo/paraview-developers
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/paraview-developers/attachments/20110908/cad2bc27/attachment-0001.htm>