[Paraview] Newbie question on Python Programmable Filters

Berk Geveci berk.geveci at kitware.com
Thu Sep 6 13:47:45 EDT 2012 

Just to make sure that I understand this: you have some code that reads
some files in a proprietary format and spits out 1 vtu file per component?
What is that format like? What is the script written in? It should be
pretty easy to change that script to combine components together and write
them out as a vector. You could also calculate derived quantities there and
write them to the vtu file. Or you could still calculate that using the
Programmable Filter.

On Thu, Sep 6, 2012 at 1:19 PM, Frank Horowitz
<frank.horowitz at cornell.edu>wrote:

> Thanks for the replies!
>
> At the moment, I need to finish a poster for a presentation on Saturday,
> but promise to get to this Sunday or Monday. (I'll need to re-build a
> debugging version of paraview. Any hints on how to accomplish this within
> ccmake/cmake will be gratefully received, since that system is new to me
> too. Is it as simple as setting a CCFLAGS environment variable before
> invoking make -- or something similar??)
>
> I also found that the 200**3 wavelet worked fine on my
> paraview-compiled-from-source Linux installation. Let's ignore the problem
> I found on the OSX box with the binary installer for the moment. (One
> problem at a time!)
>
> Because of that, I'm now suspicious of the upstream code that writes .vtu
> files.
>
> Also, responding here to your later comment from your next email:
>
> > I am going to disagree with Andy about this being a better solution. I
> > don't even think that it addresses what Frank is trying to do. But beyond
> > that Array Calculator is cumbersome and slow compared to the Python
> > Calculator or Python Programmable Filter.
> >
> > As far as I understand, Frank is not trying to combine components from
> > different files. He is trying to combine components written as different
> > arrays in the same file hence the need for sqrt(x*x + y*y + z*z). I think
> > that Andy is confused by inputs[0] - that's simple the way to access the
> > input (even if there is only one) of the Python Programmable Filter.
>
> Actually, there is one file per component per timeslice.
>
> I've had a quick look at the file format written by my upstream simulator.
> It's very simple.  Maybe I should just write the .vtu files from within my
> own script that reads the upstream meshes to avoid any other potential
> problems with the upstream codes.
>
> Does paraview.simple have an easy way to construct a field vector at the
> location of arbitrary points? If so, I think the Python script to do *that*
> will be faster than me trying to chase all of this down…
>
> Thanks again for the advice!
>
> Cheers,
>         Frank Horowitz
>
>
>
> On 06/09/2012, at 1:00 PM, Berk Geveci wrote:
>
> > I just ran this with a 500^3 wavelet on my machine without any problems.
> > The whole thing (including a slice filter) takes 20-30 seconds but the
> > programmable filter is actually pretty much instantaneous (<0.1 seconds)
> > using a debug build from Git master. Is there any way you can attach to
> the
> > stuck ParaView with a debugger and look at the stack trace?
> >
> > On Wed, Sep 5, 2012 at 9:13 PM, Frank Horowitz
> > <frank.horowitz at cornell.edu>wrote:
> >
> >> Still more checking things out. A problem (THE problem???) can be
> >> demonstrated with your code snippet from a few messages back if you
> >> construct the Wavelet like so rather than using the default constructor:
> >>
> >> Wavelet(WholeExtent=(-100,100,-100,100,-100,100))
> >>
> >> N.B. this is on a different machine, with a stock Mac OSX  binary
> version
> >> 3.14.1 installed.
> >>
> >>
> >> On 05/09/2012, at 5:49 PM, Frank Horowitz wrote:
> >>
> >>> OK, more tracking the problem:
> >>>
> >>> The assumption underlying my "close up" description below is not
> >> correct.  The problem appears to be the actual _size_ of the dataset I'm
> >> feeding in. Even a single time step fed in to the filter (with some
> 866,000
> >> points) hangs at the x*x computation, even though that appeared to work
> >> perfectly fine with your 3D Wavelet as a source (only some 9,000
> points).
> >>>
> >>> I'm guessing a memory leak somewhere?  When was the last time paraview
> >> was run through valgrind?
> >>>
> >>> Cheers (and sorry about the red herring earlier),
> >>>      Frank
> >>>
> >>>
> >>>
> >>> On 05/09/2012, at 4:42 PM, Frank Horowitz wrote:
> >>>
> >>>> OK, the 30,000 foot view:
> >>>>
> >>>> I have a time series of about 20 slices (currently) of values of each
> >> component (3 components total) of a vector displacement 3d spatial field
> >> from an elastic wave propagation simulation. At the moment, I'm simply
> >> trying to visualise the vector length to see the propagating phases of
> the
> >> wavefield. Because my upstream codes and scripts only write individual
> >> components of the vector field, I need to combine the components and
> then
> >> calculate the vector lengths *somewhere*.  The programmable filter
> seemed
> >> like a good place to try that, since once I am past the learning curve
> for
> >> it, I'll have the full flexibilty of numpy for doing almost anything
> else I
> >> want with these and similar data from my upstream simulation codes.
> >>>>
> >>>> Now the 10,000 foot view:
> >>>> I've been trying to simply calculate the sqrt(x*x + y*y + z*z) scalar
> >> field (where x, y, and z are the respective components as a function of
> >> position)  and let that flow on down the pipeline as a new spatial
> field to
> >> be visualised. I assumed (correctly if I'm interpreting your answer
> >> correctly) that if I built the filter to deal with the spatial aspects
> of
> >> the calculation, the time series would be taken care of by paraview
> >> iterating over all files pointed to by inputs[0] (a set of files
> containing
> >> timesteps of the x components), inputs[1] (ditto for the y components),
> and
> >> inputs[2] (ditto for the z components).  The set of *.vtu files for each
> >> component have filenames specifying component and time step (e.g.: x001,
> >> x002, x003, etc., y001, y002, y003, etc., and z001, z002, z003 etc.).
> >>>>
> >>>> Now the close-up:
> >>>> As far as I can tell, after compiling from source from the git
> >> "release" branch (on an up-to-date Linux Mint distro -- which tracks
> Ubuntu
> >> Precise), my approach will work for a set of 3 single timestep arrays
> fed
> >> into the "ports" (or whatever paraview's terminology is for the inputs
> to a
> >> filter in the pipeline). In other words, a single timeslice
> >> (x001,y001,z001) appears on inputs[0:3] (numpy slice notation) and is
> >> processed correctly. (I'll go away and verify that shortly.)  I *think*
> the
> >> "hangs/race-conditions/whatever" start when there are multiple timesteps
> >> presented to each input port.  I was unsure whether the 'inputs[0],
> >> inputs[1], inputs[2]' syntax needed to be extended to deal with time
> series
> >> as above, and could not find documentation on the inputs[] array after
> >> extensive searching -- just a few snippets of example code that were all
> >> consistent with paraview dealing with the time series on its own.
> >>>>
> >>>> I'm inexperienced enough with paraview to not know if I have a
> >> conceptual error with the way I'm approaching this problem, or if I have
> >> simply stumbled over a bug. That's why I've posted to the mailing list!
> >>>>
> >>>> As always, your (and the rest of the list's) help is greatly
> >> appreciated!
> >>>>
> >>>> Cheers,
> >>>>     Frank
> >>>>
> >>>>
> >>>> On 05/09/2012, at 4:01 PM, Andy Bauer wrote:
> >>>>
> >>>>> I'm not sure I completely understand about your inputs[0]. As for
> >> iterating
> >>>>> over time with the python programmable filter, inside the filter it
> >> will
> >>>>> only have access to values from a single time step at each invocation
> >> of
> >>>>> RequestData(). You can think of the order of operations if you have 5
> >> time
> >>>>> steps as:
> >>>>> 1) iterate over each of the 5 time steps (i.e. load a file and update
> >> the
> >>>>> pipeline with that data set)
> >>>>> 2) update the python programmable filter output for the current data
> >> set
> >>>>> resulting from the loaded file
> >>>>>
> >>>>> If you want time statistics for your variables I think you probably
> >> want to
> >>>>> use the calculator filter to square the value at each point and then
> >> use
> >>>>> the temporal statistics filter to get an average of those values.
> After
> >>>>> that you can just multiply the results by the number of time steps.
> >>>>>
> >>>>> If I'm misunderstanding what you're trying to do, I'd suggest giving
> a
> >>>>> description of what you're trying to do and maybe there's a simpler
> >> way of
> >>>>> doing it.
> >>>>>
> >>>>> Andy
> >>>>>
> >>>>> On Wed, Sep 5, 2012 at 2:56 PM, Frank Horowitz
> >>>>> <frank.horowitz at cornell.edu>wrote:
> >>>>>
> >>>>>> OK Andy,
> >>>>>>
> >>>>>> Thanks for that!
> >>>>>>
> >>>>>> I've compiled and installed Paraview from git's release branch, and
> (a
> >>>>>> slight modification) of your code runs correctly. I think that means
> >> that
> >>>>>> my paraview/python integration is semi-sane.
> >>>>>>
> >>>>>> However, my problem still occurs. Recall that I have a time series
> of
> >> .vtu
> >>>>>> files (indexed by integers built-in to their filenames) being fed
> into
> >>>>>> inputs[0] (at least as I understand it).
> >>>>>>
> >>>>>> In my original code snippet, I assumed that the "m = x*x" statement
> >> would
> >>>>>> iterate not only over the point values of the spatial array, but
> also
> >> over
> >>>>>> all timeslices.  Is that a correct assumption? Is that the source of
> >> my
> >>>>>> problems???
> >>>>>>
> >>>>>> Thanks again for your help,
> >>>>>> Frank Horowitz
> >>>>>>
> >>>>>> ------------------------------
> >>>>>> *From:* Andy Bauer [andy.bauer at kitware.com]
> >>>>>> *Sent:* Tuesday, September 04, 2012 7:57 PM
> >>>>>> *To:* Frank Horowitz
> >>>>>> *Cc:* paraview at paraview.org
> >>>>>> *Subject:* Re: [Paraview] Newbie question on Python Programmable
> >> Filters
> >>>>>>
> >>>>>> Hi Frank,
> >>>>>>
> >>>>>> I didn't have a problem with the following on my windows machine
> >> using the
> >>>>>> ParaView 3.14.1 installer.
> >>>>>> =======
> >>>>>> try: paraview.simple
> >>>>>>
> >>>>>> except: from paraview.simple import *
> >>>>>>
> >>>>>> paraview.simple._DisableFirstRenderCameraReset()
> >>>>>>
> >>>>>> Wavelet1 = Wavelet()
> >>>>>>
> >>>>>> RenderView1 = GetRenderView()
> >>>>>>
> >>>>>> ProgrammableFilter1 = ProgrammableFilter()
> >>>>>>
> >>>>>> ProgrammableFilter1.PythonPath = ''
> >>>>>>
> >>>>>> ProgrammableFilter1.RequestInformationScript = ''
> >>>>>>
> >>>>>> ProgrammableFilter1.Script = 'x =
> >> inputs[0].PointData[\'RTData\']\nprint
> >>>>>> "x shape =",x.shape\n\nm = x*x # I _thought_ this should run at
> numpy
> >>>>>> speeds?\nprint "m shape=",m.shape\n\n# more debugging code to finish
> >>>>>> computation of the norm
> >> omitted\n\noutput.PointData.append(m,"Displacement
> >>>>>> Norm")'
> >>>>>>
> >>>>>> Show()
> >>>>>>
> >>>>>> Render()
> >>>>>>
> >>>>>> =============
> >>>>>>
> >>>>>> Does this work for you?  If it doesn't, I wonder if it's an issue
> with
> >>>>>> having multiple pythons and numpy or system environments causing
> >> problems.
> >>>>>> What version of paraview are you using and how did you get it
> >>>>>> built/installed on your machine?
> >>>>>>
> >>>>>> Andy
> >>>>>>
> >>>>>> On Tue, Sep 4, 2012 at 6:04 PM, Frank Horowitz <
> >> frank.horowitz at cornell.edu
> >>>>>>> wrote:
> >>>>>>
> >>>>>>> Hal Canary wrote on Tue Sep 4 15:22:01 EDT 2012:
> >>>>>>>> On 09/04/2012 02:20 PM, Frank Horowitz wrote:
> >>>>>>>>>
> >>>>>>>> x = inputs[0].PointData['Scalars_']
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I thought one needs to convert a vtkarray to a numpy array with
> >>>>>>>>
> >>>>>>>>   x = numpy.array(inputs[0].PointData['Scalars_'])
> >>>>>>>>
> >>>>>>>> before doing anything with it.
> >>>>>>>
> >>>>>>>
> >>>>>>> Computations after that style of construction hang too.  To my eye,
> >> the
> >>>>>>> results of such an expression should be 100% standard numpy,
> obeying
> >> 100%
> >>>>>>> standard numpy semantics.
> >>>>>>>
> >>>>>>> It appears that there is a bug in numpy integration to the Python
> >>>>>>> Programmable Filter.
> >>>>>>>
> >>>>>>> I'll re-compile from source and report back on the results.  Is the
> >> bug
> >>>>>>> tracking system evident from the homepage?
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>>     Frank Horowitz
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> Powered by www.kitware.com
> >>>>>>>
> >>>>>>> Visit other Kitware open-source projects at
> >>>>>>> http://www.kitware.com/opensource/opensource.html
> >>>>>>>
> >>>>>>> Please keep messages on-topic and check the ParaView Wiki at:
> >>>>>>> http://paraview.org/Wiki/ParaView
> >>>>>>>
> >>>>>>> Follow this link to subscribe/unsubscribe:
> >>>>>>> http://www.paraview.org/mailman/listinfo/paraview
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>
> >>
> >> _______________________________________________
> >> Powered by www.kitware.com
> >>
> >> Visit other Kitware open-source projects at
> >> http://www.kitware.com/opensource/opensource.html
> >>
> >> Please keep messages on-topic and check the ParaView Wiki at:
> >> http://paraview.org/Wiki/ParaView
> >>
> >> Follow this link to subscribe/unsubscribe:
> >> http://www.paraview.org/mailman/listinfo/paraview
> >>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.paraview.org/pipermail/paraview/attachments/20120906/e9edb01c/attachment-0001.htm>

More information about the ParaView mailing list