Just in case (would be luck but who knows...), I just committed a fix for review:<div><a href="http://review.source.kitware.com/#change,2719">http://review.source.kitware.com/#change,2719</a></div><div>I had issues with combined update extents due to <span class="Apple-style-span" style="background-color: rgb(255, 255, 255); "><font class="Apple-style-span" face="'Lucida Console', 'Lucida Sans Typewriter', Monaco, monospace"><span class="Apple-style-span" style="white-space: pre; font-size: 11px;">UPDATE_NUMBER_OF_PIECES </span></font>forcing a<font class="Apple-style-span" face="'Lucida Console', 'Lucida Sans Typewriter', Monaco, monospace"><span class="Apple-style-span" style="white-space: pre; font-size: 11px;"> needtoexecute </span></font>in the<font class="Apple-style-span" face="'Lucida Console', 'Lucida Sans Typewriter', Monaco, monospace"><span class="Apple-style-span" style="white-space: pre; font-size: 11px;"> update_extent </span></font>but not in the<font class="Apple-style-span" face="'Lucida Console', 'Lucida Sans Typewriter', Monaco, monospace"><span class="Apple-style-span" style="white-space: pre; font-size: 11px; "> REQUEST_DATA.</span></font></span></div>
<div><br></div><div>Julien.<br><br><div class="gmail_quote">On Thu, Sep 8, 2011 at 10:55 AM, Biddiscombe, John A. <span dir="ltr"><<a href="mailto:biddisco@cscs.ch">biddisco@cscs.ch</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div lang="EN-GB" link="blue" vlink="purple">
<div>
<p><span style="font-family:"Courier New"">David, et al,<u></u><u></u></span></p>
<p><span style="font-family:"Courier New""><u></u> <u></u></span></p>
<p><span style="font-family:"Courier New"">I need some advice on pipeline re-execution.<u></u><u></u></span></p>
<p><span style="font-family:"Courier New"">I've removed all the spurious pipeline re-execution problems I had, by simply stopping the NeedToExecute in the executives from ever saying yes based on ghost cells. However, I can't do the same
for number of pieces.<u></u><u></u></span></p>
<p><span style="font-family:"Courier New""><u></u> <u></u></span></p>
<p><span style="font-family:"Courier New"">I suspect that one reason my filters are crashing on massive data and causing issues on others is because paraview constantly triggers unnecessary executions when stuff hasn't changed.<u></u><u></u></span></p>
<p><span style="font-family:"Courier New""><u></u> <u></u></span></p>
<p><span style="font-family:"Courier New"">My pipeline is essentially as follows (please view using wide window and courier font)<u></u><u></u></span></p>
<p><span style="font-family:"Courier New""><u></u> <u></u></span></p>
<p><span style="font-family:"Courier New"">H5PartReader (reads N pieces, exports polydata)<u></u><u></u></span></p>
<p><span style="font-family:"Courier New""> |<u></u><u></u></span></p>
<p><span style="font-family:"Courier New""> V<u></u><u></u></span></p>
<p><span style="font-family:"Courier New"">ParticleRepartitionFilter (UG/polydata, redistributes particles across processes<u></u><u></u></span></p>
<p><span style="font-family:"Courier New"">and replaces ExtentTranslator with BoundsExtentTranslator, redistribution<u></u><u></u></span></p>
<p><span style="font-family:"Courier New"">is spatial (KdTree) and irregular – also adds scalar array to
<u></u><u></u></span></p>
<p><span style="font-family:"Courier New"">denote ghost particles which are duplicated around process boundaries)<u></u><u></u></span></p>
<p><span style="font-family:"Courier New""> |<u></u><u></u></span></p>
<p><span style="font-family:"Courier New""> |___________________<u></u><u></u></span></p>
<p><span style="font-family:"Courier New"">___ V<u></u><u></u></span></p>
<p><span style="font-family:"Courier New"">| ImageGridSampler (creates a vtkImageData grid and uses the<u></u><u></u></span></p>
<p><span style="font-family:"Courier New"">| BoundsExtentTranslator to get the right extents for the piece<u></u><u></u></span></p>
<p><span style="font-family:"Courier New"">| this means we can get irregular subdivisions of space)<u></u><u></u></span></p>
<p><span style="font-family:"Courier New"">| ________________|<u></u><u></u></span></p>
<p><span style="font-family:"Courier New"">V V<u></u><u></u></span></p>
<p><span style="font-family:"Courier New"">SPHResampler (any datatype, resamples the particles using an SPH kernel onto<u></u><u></u></span></p>
<p><span style="font-family:"Courier New"">the vtkImageData provided by the image sampler – the use of the BoundsExtentTranslator<u></u><u></u></span></p>
<p><span style="font-family:"Courier New"">in the earlier stages ensures we have the data we want to reample on the right process<u></u><u></u></span></p>
<p><span style="font-family:"Courier New"">for the sampling geometry)<u></u><u></u></span></p>
<p><u></u> <u></u></p>
<p>This bit works fine (on the desktop, but sometimes crashes on the cluster when running N=64 processes), ... but prompted by your suggestion, I threw in a volume renderer after the SPHResampler to view the resampled data as a volume.<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>This triggers a whole bunch of filter reexecutions, caused by updateNumberOfPieces(1) != dataNumberOfPieces(4) (4 being what I’d expect it to be not 1 which is just silly)<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>I have not yet identified who is setting the update pieces to 1, when all the rest of the pipeline is using 4, but I experimented with using wavelet sources and creating pipelines which were similar and I found that the problem was not
unique to my setup.<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>I’d like to know if there is any rule to<u></u><u></u></p>
<p>NOT SET Maximum_number_of_pieces<u></u><u></u></p>
<p>NOT SET Any kind of piece information<u></u><u></u></p>
<p>YES SET Extent information etc<u></u><u></u></p>
<p>in particular kinds of filters.<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>By which I mean, image data generators should NEVER set maximum number of pieces? and piece extent translators etc etc. I’m looking for clues as to why my pipeline might misbehave, because debugging the executives is hard work and time
consuming (particularly so since David completely buggered them up with all his fancy streaming stuff
<span style="font-family:Wingdings">J</span> )<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>Any advice welcome.<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>JB<u></u><u></u></p>
<p>PS. I see that paraview has updated it’s number scheme to 3.12 RCxxx : noting that Paraview only really works properly in serial, is this wise?<u></u><u></u></p>
<p><u></u> <u></u></p>
<p><u></u> <u></u></p>
<p><u></u> <u></u></p>
<p><u></u> <u></u></p>
<p><span lang="EN-US">-----Original Message-----<br>
From: <a href="mailto:paraview-developers-bounces@paraview.org" target="_blank">paraview-developers-bounces@paraview.org</a> [mailto:<a href="mailto:paraview-developers-bounces@paraview.org" target="_blank">paraview-developers-bounces@paraview.org</a>] On Behalf Of Biddiscombe, John A.<br>
Sent: 07 September 2011 09:57<br>
To: David E DeMarle<br>
Cc: <a href="mailto:paraview-developers@paraview.org" target="_blank">paraview-developers@paraview.org</a><br>
Subject: Re: [Paraview-developers] suspicious behaviour with structured Grids (in partuicular)</span></p>
<p><u></u> <u></u></p>
<p>David<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>>Was volume rendering involved?<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>Not involved. The visual errors appear when drawing just a surface representation.<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>I tried a large run on a 64 pvserver job yesterday using the new imagedata version of the resampler, after the resampling completed, pvservers shut down as before and I lost connection. I am trying to debug it on the cluster today but
my parallel debugging skills on linux are sadly quite poor.<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>JB<u></u><u></u></p>
<p><u></u> <u></u></p>
<p><u></u> <u></u></p>
<p>David E DeMarle<u></u><u></u></p>
<p>Kitware, Inc.<u></u><u></u></p>
<p>R&D Engineer<u></u><u></u></p>
<p>21 Corporate Drive<u></u><u></u></p>
<p>Clifton Park, NY 12065-8662<u></u><u></u></p>
<p>Phone: <a href="tel:518-881-4909" value="+15188814909" target="_blank">518-881-4909</a><u></u><u></u></p>
<p><u></u> <u></u></p>
<p><u></u> <u></u></p>
<p><u></u> <u></u></p>
<p>On Thu, Sep 1, 2011 at 2:32 AM, Biddiscombe, John A. <<a href="mailto:biddisco@cscs.ch" target="_blank">biddisco@cscs.ch</a>> wrote:<u></u><u></u></p>
<p>> I've had a lot of problems with a particular set of filters of mine which resample SPH data onto grids.<u></u><u></u></p>
<p>><u></u> <u></u></p>
<p>> I have a Grid Generator class which is used as the sample location. This class outputs a vtkStructuredGrid.<u></u><u></u></p>
<p>> When running in Parallel, I see very odd behaviour - which appears similar to a post I saw some months ago but can't find now, where the data displayed appears to be incorrect. Almsot as though the data from process 0 is being drawn
on all processes and something is just not right, but I can't quite pin it down.<u></u><u></u></p>
<p>><u></u> <u></u></p>
<p>> Because I have custom extent translators, dodgy extent manipulations and all parallel features, I can never be quite sure that it's paraview that's fundamentally broken or my stuff.<u></u><u></u></p>
<p>><u></u> <u></u></p>
<p>> However, in the case where the resampling grid is regular and axis aligned, the output could be vtkImageData, which I believe does work quite well in parallel, so I made a new stripped down resampler which outputs only image data.
This works fine. The extent translation/manipulation is the same, the resampling the same, everything the same, except it produces image data.<u></u><u></u></p>
<p>><u></u> <u></u></p>
<p>> This one doesn't crash or misbehave when run in parallel, however the image looks wrong 'first time' it is done, if I delete the filter, recreate a second one and then reexecute using identical settings, I get a completely different
image and it looks perfect.<u></u><u></u></p>
<p>><u></u> <u></u></p>
<p>> I really don't understand what is going on, but my feeling now is that paraview is broken somewhere deep inside (and is essentially unfit for purpose)<u></u><u></u></p>
<p>><u></u> <u></u></p>
<p>> The problem is slightly less bad on 3.11 than on 3.10, but I still don't know what is actually wrong. I still don't know if I've messed up, or if ParaView is messing me up. (I have a complicated scenario where a multi-input filter
passes update extent information from the second input).<u></u><u></u></p>
<p>><u></u> <u></u></p>
<p>> I'm only posting this because I have some vague memory that someone else reported duplicated colours from process 0 on some gridded data in parallel and it's just possible, I'm seeing aspects of the same bug. I can't file a proper
bug report because I'm still not completely sure a bug really exists. I'm just hoping someone reading this might mention something which triggers other thoughts, comments etc.<u></u><u></u></p>
<p>><u></u> <u></u></p>
<p>> yours<u></u><u></u></p>
<p>><u></u> <u></u></p>
<p>> JB<u></u><u></u></p>
<p>><u></u> <u></u></p>
<p>><u></u> <u></u></p>
<p>> --<u></u><u></u></p>
<p>> John Biddiscombe, email:biddisco @ <a href="http://cscs.ch" target="_blank">cscs.ch</a><u></u><u></u></p>
<p>> <a href="http://www.cscs.ch/" target="_blank">http://www.cscs.ch/</a><u></u><u></u></p>
<p>> CSCS, Swiss National Supercomputing Centre | Tel: <a href="tel:%2B41%20%2891%29%20610.82.07" value="+41916108207" target="_blank">+41 (91) 610.82.07</a><u></u><u></u></p>
<p>> Via Cantonale, 6928 Manno, Switzerland | Fax: <a href="tel:%2B41%20%2891%29%20610.82.82" value="+41916108282" target="_blank">+41 (91) 610.82.82</a><u></u><u></u></p>
<p>><u></u> <u></u></p>
<p>><u></u> <u></u></p>
<p>> _______________________________________________<u></u><u></u></p>
<p>> Paraview-developers mailing list<u></u><u></u></p>
<p>> <a href="mailto:Paraview-developers@paraview.org" target="_blank">Paraview-developers@paraview.org</a><u></u><u></u></p>
<p>> <a href="http://public.kitware.com/mailman/listinfo/paraview-developers" target="_blank">http://public.kitware.com/mailman/listinfo/paraview-developers</a><u></u><u></u></p>
<p>><u></u> <u></u></p>
<p>_______________________________________________<u></u><u></u></p>
<p>Paraview-developers mailing list<u></u><u></u></p>
<p><a href="mailto:Paraview-developers@paraview.org" target="_blank">Paraview-developers@paraview.org</a><u></u><u></u></p>
<p><a href="http://public.kitware.com/mailman/listinfo/paraview-developers" target="_blank">http://public.kitware.com/mailman/listinfo/paraview-developers</a><u></u><u></u></p>
</div>
</div>
<br>_______________________________________________<br>
Paraview-developers mailing list<br>
<a href="mailto:Paraview-developers@paraview.org">Paraview-developers@paraview.org</a><br>
<a href="http://public.kitware.com/mailman/listinfo/paraview-developers" target="_blank">http://public.kitware.com/mailman/listinfo/paraview-developers</a><br>
<br></blockquote></div><br></div>