[Amr] yt inside paraview

Fri Jun 10 14:35:57 EDT 2011

HI Matt

Looks nice.
In the 'stream' frontends, when the stream.StreamHandler(...) is create,
one parameter is the parent_ids right? It's really mandatory to send it?
because the vtkHierarchicalBoxDataSet doesn't save this information, but we
can compute the parent/child relations.

Regards,

On Fri, Jun 10, 2011 at 2:19 PM, Matthew Turk <matthewturk at gmail.com> wrote:

> Hi Jorge,
>
> I thought this might be of interest to you; I've started keeping track
> of how to interface with external codes such as ParaView here:
>
> http://hg.enzotools.org/yt/wiki/InterfacingWithYT
>
> I'll be updating this as need be with information about the in-memory
> datasets and so on.
>
> Best,
>
> Matt
>
> On Fri, May 27, 2011 at 2:08 PM, Matthew Turk <matthewturk at gmail.com>
> wrote:
> > Hi all,
> >
> > I've created a new data frontend in yt entitled "Stream" (the name is
> > up for discussion, if someone has a better idea -- Charles pointed out
> > today that it might get confused with time series data.)  The idea
> > behind this is that the user can use it to construct an arbitrary
> > dataset in memory.  I've tested this by proxying an existing Enzo
> > dataset:
> >
> >
> http://matthewturk.bitbucket.org/html/ba3fd37b-842c-4641-b21e-2d3f5268eefe-stream_proxy-py.html
> >
> > (downloadable at the top level domain, or by "yt_lodgeit.py
> > --download=1672 if you have a yt stack installed.)
> >
> > You have to construct both a StreamHandler that knows how to pass
> > around the grid geometry and which, although not instantiated with
> > them, requires certain attributes to be set.  In addition, a fields
> > handler has to be created.  This script executes to completion and
> > passes its tests; projecting and slicing require slightly more effort,
> > because I wanted to pose to the Kitware group how the data should be
> > evaluated in terms of units -- should it be assumed to have CGS units
> > already?  How should the domain size be handled -- in yt right now
> > units are distinct from the code definitions, but I don't know how
> > Paraview handles this distinction.  (If you want to try setting up a
> > PlotCollection, you can, but you will have to set spf.units["cm"] =
> > spf.units["unitary"] = 1.0).
> >
> > Hopefully this can be a start, and we can continue to iterate on the
> > API necessary to load data in yt from that already loaded in Paraview.
> >  This is in the tip of the development yt repository, 6d4a5588a550.
> >
> > Best,
> >
> > Matt
> >
> > On Thu, May 26, 2011 at 3:54 PM, Jorge Poco <jorge.poco at kitware.com>
> wrote:
> >> In the  meantime I will be working to import yt's data structures inside
> >> vtkHierarchicalBoxDataset using ProgrammableSources inside Python.
> >> Regards,
> >>
> >> On Thu, May 26, 2011 at 3:43 PM, Jorge Poco <jorge.poco at kitware.com>
> wrote:
> >>>
> >>> Hi
> >>> YT has some function to access the arrays data in numpy.array objects (
> >>> pf.h.grids[0]['Density']), using them  we can instantiate a
> >>> vtkHierarchicalDataset to be process into paraview.
> >>> On Thu, May 26, 2011 at 2:53 PM, Berk Geveci <berk.geveci at kitware.com>
> >>> wrote:
> >>>>
> >>>> Awesome.
> >>>>
> >>>> What about from the other side? If we wanted to pass data structures
> >>>> that yt loaded (from an Enzo file for example) to VTK/ParaView for
> >>>> filtering/rendering?
> >>>>
> >>>> -berk
> >>>>
> >>>> On Thu, May 26, 2011 at 2:04 PM, Matthew Turk <matthewturk at gmail.com>
> >>>> wrote:
> >>>> > Hi Berk and Jorge,
> >>>> >
> >>>> > Okay, this makes sense.
> >>>> >
> >>>> > I will write up a new code frontend for yt that will act as a
> >>>> > generalized system; this will draw upon the current in situ Enzo
> >>>> > frontend, and will be designed to be instantiated with a list of
> >>>> > arrays and a set of callback functions.  We can them implement the
> >>>> > paraview-specific code inside this, or perhaps inside a
> >>>> > ProgrammableFilter inside paraview itself.  I'll ping the list back
> >>>> > when I have something up and running.
> >>>> >
> >>>> > Best,
> >>>> >
> >>>> > Matt
> >>>> >
> >>>> > On Thu, May 26, 2011 at 9:15 AM, Berk Geveci <
> berk.geveci at kitware.com>
> >>>> > wrote:
> >>>> >> In VTK, an AMR mesh is represented as a collection of
> vtkUniformGrid
> >>>> >> objects. Each vtkUniformGrid has a set of Fortran-ordered data
> arrays
> >>>> >> that represent the variables. One exception is how vectors are
> stored.
> >>>> >> They are stored in hybrid Fortran - C ordering, i.e. vector
> component
> >>>> >> increases fastest then i, then j, then k. It is straightforward to
> >>>> >> pass these to Python and back using NumPy arrays or buffer objects
> at
> >>>> >> the lowest level. So we can write Python code to bundle these
> arrays
> >>>> >> into any form that yt (and Enzo when we start doing in-situ) wants.
> We
> >>>> >> may have to deep-copy the vectors because of the issue I described.
> >>>> >>
> >>>> >> On Wed, May 25, 2011 at 7:59 PM, Matthew Turk <
> matthewturk at gmail.com>
> >>>> >> wrote:
> >>>> >>> Hi Jorge,
> >>>> >>>
> >>>> >>> This is really cool!
> >>>> >>>
> >>>> >>> I've been thinking about how to invert this process, and I think
> we
> >>>> >>> could address it by creating a new hierarchy of objects, similar
> to
> >>>> >>> the EnzoStaticOutputInMemory family of objects, that enables a
> >>>> >>> bi-directional transfer of information.  All that's really
> necessary
> >>>> >>> to get data in is to describe the mesh, describe the parameters of
> >>>> >>> the
> >>>> >>> simulation, and then communicate to yt how to access data.
> >>>> >>>
> >>>> >>> What do you think the best mechanism for passing data back and
> forth
> >>>> >>> is?  For in situ viz with Enzo, usually we do this by constructing
> a
> >>>> >>> new module object in the C code, and then insert into that a set
> of
> >>>> >>> arrays describing the mesh and a dictionary of dictionaries of
> >>>> >>> fields,
> >>>> >>> with keys that correspond to grid id and field name.  But really,
> it
> >>>> >>> could be any system, and I'm happy to work on constructing the
> >>>> >>> methods
> >>>> >>> to set this up.
> >>>> >>>
> >>>> >>> Best,
> >>>> >>>
> >>>> >>> Matt
> >>>> >>>
> >>>> >>> On Wed, May 25, 2011 at 5:54 PM, Jorge Poco <
> jorge.poco at kitware.com>
> >>>> >>> wrote:
> >>>> >>>> Hi Matthew and Berk
> >>>> >>>>
> >>>> >>>> the full stack is here http://paste.enzotools.org/show/1668/ I'm
> >>>> >>>> trying to
> >>>> >>>> compile paraview with the VTK_USE_SYSTEM_HDF5 but It's giving me
> >>>> >>>> some
> >>>> >>>> errors, I will try again tomorrow.
> >>>> >>>>
> >>>> >>>> Matthew: Right now, I have written a 'programmable source' script
> to
> >>>> >>>> read a
> >>>> >>>> file from the system using the yt function 'load'. Then, create a
> >>>> >>>> projection
> >>>> >>>> using yt and then the ProjectionPlot object is converted into a
> >>>> >>>> vtkImageData
> >>>> >>>> to be displayed into paraview.
> >>>> >>>> What I want to do now, is to read a Dataset using paraview, then
> >>>> >>>> create a
> >>>> >>>> 'programmable filter' script to convert the data set into  yt's
> data
> >>>> >>>> structure to create a visualization and finally display the image
> >>>> >>>> inside
> >>>> >>>> paraview. What would be the correct way to create yt objects
> >>>> >>>> (EnzoHierarchy
> >>>> >>>> and EnzoGrid)?
> >>>> >>>>
> >>>> >>>> Berk: I was trying to find a way to get access to camera
> information
> >>>> >>>> from
> >>>> >>>> the 'programmable filter' to send this information to yt  but I
> >>>> >>>> didn't found
> >>>> >>>> anything useful. Is it possible?
> >>>> >>>>
> >>>> >>>>
> >>>> >>>> Regards,
> >>>> >>>>
> >>>> >>>> On Wed, May 25, 2011 at 3:10 PM, Berk Geveci
> >>>> >>>> <berk.geveci at kitware.com>
> >>>> >>>> wrote:
> >>>> >>>>>
> >>>> >>>>> If I understood what Matt is saying, the problem may be fixed by
> >>>> >>>>> having ParaView use an external HDF5, the one that yt uses. I
> >>>> >>>>> believe
> >>>> >>>>> that the variable is something like PARAVIEW_USE_SYSTEM_HDF5 (or
> >>>> >>>>> VTK_,
> >>>> >>>>> not sure) in CMake.
> >>>> >>>>>
> >>>> >>>>> On Wed, May 25, 2011 at 10:49 AM, Matthew Turk
> >>>> >>>>> <matthewturk at gmail.com>
> >>>> >>>>> wrote:
> >>>> >>>>> > Hi Jorge,
> >>>> >>>>> >
> >>>> >>>>> > Wow, I didn't know that YT-embedding had taken off quite so
> much!
> >>>> >>>>> > This is great news.
> >>>> >>>>> >
> >>>> >>>>> > On Wed, May 25, 2011 at 9:48 AM, Jorge Poco
> >>>> >>>>> > <jorge.poco at kitware.com>
> >>>> >>>>> > wrote:
> >>>> >>>>> >> Hi
> >>>> >>>>> >> sorry my last email was sent incomplete
> >>>> >>>>> >> I have been trying to use YT inside the Paraview's python
> shell.
> >>>> >>>>> >> Initially I
> >>>> >>>>> >> have had some problems which are now identified. It is not
> the
> >>>> >>>>> >> best
> >>>> >>>>> >> solution but I at least solve temporally these problems:
> >>>> >>>>> >>
> >>>> >>>>> >> argv problem:
> >>>> >>>>> >>
> >>>> >>>>> >> Sentence: 'from yt.mods import *'
> >>>> >>>>> >> Error: for any reason paraview's python shell has not defined
> >>>> >>>>> >> the
> >>>> >>>>> >> variable
> >>>> >>>>> >> 'sys.argv'
> >>>> >>>>> >> Temporal solution: comment yt code where the sys.argv
> variable
> >>>> >>>>> >> is used:
> >>>> >>>>> >>
> >>>> >>>>> >> yt/funcs.py: lines 439-453
> >>>> >>>>> >> yt/utilities/parallel_tools/parallel_analysis_interface.py:
> >>>> >>>>> >> lines 47
> >>>> >>>>> >> and 48
> >>>> >>>>> >
> >>>> >>>>> > We can likely get around this on the yt side, although when
> >>>> >>>>> > embedding
> >>>> >>>>> > yt in Enzo we've escaped this by using the command
> >>>> >>>>> > PySys_SetArgv(argc,
> >>>> >>>>> > argv) immediately after calling Py_Initialize.  Does Partiview
> >>>> >>>>> > report
> >>>> >>>>> > a different executable name, by any chance?  If so we can use
> >>>> >>>>> > that as
> >>>> >>>>> > a signal to avoid requesting argv.
> >>>> >>>>> >
> >>>> >>>>> >>
> >>>> >>>>> >> :
> >>>> >>>>> >>
> >>>> >>>>> >> Sentence: "pf = load('tests/DD0010/moving7_0010')"
> >>>> >>>>> >> Error: python shell does not complete the load function and
> >>>> >>>>> >> paraview
> >>>> >>>>> >> keeps
> >>>> >>>>> >> waiting.
> >>>> >>>>> >> Temporal solution: When I debug paraview and interrupt its
> >>>> >>>>> >> execution
> >>>> >>>>> >> the
> >>>> >>>>> >> backtrace shows sem_wait(). It happens inside the load
> function
> >>>> >>>>> >> when
> >>>> >>>>> >> tries
> >>>> >>>>> >> to identify the candidates for input file (convenience.py
> line
> >>>> >>>>> >> 74).
> >>>> >>>>> >> There
> >>>> >>>>> >> are 11 possible candidates and gives error for 2 of them:
> >>>> >>>>> >> ChomboStaticOutput
> >>>> >>>>> >> and FlashStaticOutput. Skipping those candidates solve
> >>>> >>>>> >> temporally the
> >>>> >>>>> >> problem: (yt/convenienve.py line 75) inside the loop add if n
> ==
> >>>> >>>>> >> 'ChomboStaticOutput' or n == 'FlashStaticOutput': continue;
> >>>> >>>>> >
> >>>> >>>>> > This is interesting.  What those two routines have in common
> is
> >>>> >>>>> > that
> >>>> >>>>> > they are the only two that try opening the file using h5py.
>  HDF5
> >>>> >>>>> > has
> >>>> >>>>> > a number of error handling routines, which in the past I've
> seen
> >>>> >>>>> > issues with when embedding one app in another -- h5py for
> >>>> >>>>> > instance
> >>>> >>>>> > handles error handling, but that can conflict with other
> >>>> >>>>> > mechanisms
> >>>> >>>>> > that are attempting to handle those errors.
> >>>> >>>>> >
> >>>> >>>>> > Another solution that would avoid digging into the source code
> >>>> >>>>> > would
> >>>> >>>>> > be to remove the types from the actual type registry:
> >>>> >>>>> >
> >>>> >>>>> > from yt.convenience import output_type_registry
> >>>> >>>>> > del output_type_registry["ChomboStaticOutput"]
> >>>> >>>>> > del output_type_registry["FlashStaticOutput"]
> >>>> >>>>> >
> >>>> >>>>> > I'm more interested in figuring out how to fix the interplay
> >>>> >>>>> > between
> >>>> >>>>> > hdf5-paraview and h5py, though.  Do you happen to have a full
> >>>> >>>>> > stack
> >>>> >>>>> > trace?
> >>>> >>>>> >
> >>>> >>>>> > Thanks very much for these reports -- this is very exciting,
> and
> >>>> >>>>> > I'm
> >>>> >>>>> > eager to help out!
> >>>> >>>>> >
> >>>> >>>>> > Best,
> >>>> >>>>> >
> >>>> >>>>> > Matt
> >>>> >>>>> >
> >>>> >>>>> >>
> >>>> >>>>> >> Any of you have a better way to solve the problems above?
> >>>> >>>>> >> Regards,
> >>>> >>>>> >> On Wed, May 25, 2011 at 9:43 AM, Jorge Poco
> >>>> >>>>> >> <jorge.poco at kitware.com>
> >>>> >>>>> >> wrote:
> >>>> >>>>> >>>
> >>>> >>>>> >>> Hi
> >>>> >>>>> >>> I have been trying to use YT inside the Paraview's python
> >>>> >>>>> >>> shell.
> >>>> >>>>> >>> Initially
> >>>> >>>>> >>> I have had some problems which are now identified. It is not
> >>>> >>>>> >>> the
> >>>> >>>>> >>> smart solution but I at least solve temporally these
> problems:
> >>>> >>>>> >>>
> >>>> >>>>> >>> argv problem:
> >>>> >>>>> >>>
> >>>> >>>>> >>> Sentence: 'from yt.mods import *'
> >>>> >>>>> >>> Error: for any reason paraview's python shell has not
> defined
> >>>> >>>>> >>> the
> >>>> >>>>> >>> variable
> >>>> >>>>> >>> 'sys.argv'
> >>>> >>>>> >>> Temporal solution: comment yt code where the sys.argv
> variable
> >>>> >>>>> >>> is
> >>>> >>>>> >>> used:
> >>>> >>>>> >>>
> >>>> >>>>> >>> yt/funcs.py: lines 439-453
> >>>> >>>>> >>> yt/utilities/parallel_tools/parallel_analysis_interface.py:
> >>>> >>>>> >>> lines 47
> >>>> >>>>> >>> and
> >>>> >>>>> >>> 48
> >>>> >>>>> >>>
> >>>> >>>>> >>> pa:
> >>>> >>>>> >>>
> >>>> >>>>> >>> Sentence: "pf = load('tests/DD0010/moving7_0010')"
> >>>> >>>>> >>> Error: python shell does not complete the load function and
> >>>> >>>>> >>> paraview
> >>>> >>>>> >>> keeps
> >>>> >>>>> >>> waiting.
> >>>> >>>>> >>> Temporal solution: When I debug paraview and interrupt its
> >>>> >>>>> >>> execution
> >>>> >>>>> >>> the
> >>>> >>>>> >>> backtrace shows sem_wait(). It happens inside the load
> function
> >>>> >>>>> >>> when
> >>>> >>>>> >>> tries
> >>>> >>>>> >>> to identify the candidates for input file (convenience.py
> line
> >>>> >>>>> >>> 74).
> >>>> >>>>> >>> There
> >>>> >>>>> >>> are 11 possible candidates and gives error for 2 of them:
> >>>> >>>>> >>> ChomboStaticOutput
> >>>> >>>>> >>> and FlashStaticOutput. Skipping those candidates solve
> >>>> >>>>> >>> temporally the
> >>>> >>>>> >>> problem:
> >>>> >>>>> >>
> >>>> >>>>> >>
> >>>> >>>>> >> _______________________________________________
> >>>> >>>>> >> Amr mailing list
> >>>> >>>>> >> Amr at public.kitware.com
> >>>> >>>>> >> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
> >>>> >>>>> >>
> >>>> >>>>> >>
> >>>> >>>>> > _______________________________________________
> >>>> >>>>> > Amr mailing list
> >>>> >>>>> > Amr at public.kitware.com
> >>>> >>>>> > http://public.kitware.com/cgi-bin/mailman/listinfo/amr
> >>>> >>>>> >
> >>>> >>>>> _______________________________________________
> >>>> >>>>> Amr mailing list
> >>>> >>>>> Amr at public.kitware.com
> >>>> >>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
> >>>> >>>>
> >>>> >>>>
> >>>> >>>> _______________________________________________
> >>>> >>>> Amr mailing list
> >>>> >>>> Amr at public.kitware.com
> >>>> >>>> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
> >>>> >>>>
> >>>> >>>>
> >>>> >>> _______________________________________________
> >>>> >>> Amr mailing list
> >>>> >>> Amr at public.kitware.com
> >>>> >>> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
> >>>> >>>
> >>>> >>
> >>>> > _______________________________________________
> >>>> > Amr mailing list
> >>>> > Amr at public.kitware.com
> >>>> > http://public.kitware.com/cgi-bin/mailman/listinfo/amr
> >>>> >
> >>>> _______________________________________________
> >>>> Amr mailing list
> >>>> Amr at public.kitware.com
> >>>> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
> >>>
> >>
> >>
> >
> _______________________________________________
> Amr mailing list
> Amr at public.kitware.com
> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/amr/attachments/20110610/7bec7412/attachment-0001.htm>