[Amr] yt inside paraview

Fri Jun 10 14:19:57 EDT 2011

Hi Jorge,

I thought this might be of interest to you; I've started keeping track
of how to interface with external codes such as ParaView here:

http://hg.enzotools.org/yt/wiki/InterfacingWithYT

I'll be updating this as need be with information about the in-memory
datasets and so on.

Best,

Matt

On Fri, May 27, 2011 at 2:08 PM, Matthew Turk <matthewturk at gmail.com> wrote:
> Hi all,
>
> I've created a new data frontend in yt entitled "Stream" (the name is
> up for discussion, if someone has a better idea -- Charles pointed out
> today that it might get confused with time series data.)  The idea
> behind this is that the user can use it to construct an arbitrary
> dataset in memory.  I've tested this by proxying an existing Enzo
> dataset:
>
> http://matthewturk.bitbucket.org/html/ba3fd37b-842c-4641-b21e-2d3f5268eefe-stream_proxy-py.html
>
> (downloadable at the top level domain, or by "yt_lodgeit.py
> --download=1672 if you have a yt stack installed.)
>
> You have to construct both a StreamHandler that knows how to pass
> around the grid geometry and which, although not instantiated with
> them, requires certain attributes to be set.  In addition, a fields
> handler has to be created.  This script executes to completion and
> passes its tests; projecting and slicing require slightly more effort,
> because I wanted to pose to the Kitware group how the data should be
> evaluated in terms of units -- should it be assumed to have CGS units
> already?  How should the domain size be handled -- in yt right now
> units are distinct from the code definitions, but I don't know how
> Paraview handles this distinction.  (If you want to try setting up a
> PlotCollection, you can, but you will have to set spf.units["cm"] =
> spf.units["unitary"] = 1.0).
>
> Hopefully this can be a start, and we can continue to iterate on the
> API necessary to load data in yt from that already loaded in Paraview.
>  This is in the tip of the development yt repository, 6d4a5588a550.
>
> Best,
>
> Matt
>
> On Thu, May 26, 2011 at 3:54 PM, Jorge Poco <jorge.poco at kitware.com> wrote:
>> In the  meantime I will be working to import yt's data structures inside
>> vtkHierarchicalBoxDataset using ProgrammableSources inside Python.
>> Regards,
>>
>> On Thu, May 26, 2011 at 3:43 PM, Jorge Poco <jorge.poco at kitware.com> wrote:
>>>
>>> Hi
>>> YT has some function to access the arrays data in numpy.array objects (
>>> pf.h.grids[0]['Density']), using them  we can instantiate a
>>> vtkHierarchicalDataset to be process into paraview.
>>> On Thu, May 26, 2011 at 2:53 PM, Berk Geveci <berk.geveci at kitware.com>
>>> wrote:
>>>>
>>>> Awesome.
>>>>
>>>> What about from the other side? If we wanted to pass data structures
>>>> that yt loaded (from an Enzo file for example) to VTK/ParaView for
>>>> filtering/rendering?
>>>>
>>>> -berk
>>>>
>>>> On Thu, May 26, 2011 at 2:04 PM, Matthew Turk <matthewturk at gmail.com>
>>>> wrote:
>>>> > Hi Berk and Jorge,
>>>> >
>>>> > Okay, this makes sense.
>>>> >
>>>> > I will write up a new code frontend for yt that will act as a
>>>> > generalized system; this will draw upon the current in situ Enzo
>>>> > frontend, and will be designed to be instantiated with a list of
>>>> > arrays and a set of callback functions.  We can them implement the
>>>> > paraview-specific code inside this, or perhaps inside a
>>>> > ProgrammableFilter inside paraview itself.  I'll ping the list back
>>>> > when I have something up and running.
>>>> >
>>>> > Best,
>>>> >
>>>> > Matt
>>>> >
>>>> > On Thu, May 26, 2011 at 9:15 AM, Berk Geveci <berk.geveci at kitware.com>
>>>> > wrote:
>>>> >> In VTK, an AMR mesh is represented as a collection of vtkUniformGrid
>>>> >> objects. Each vtkUniformGrid has a set of Fortran-ordered data arrays
>>>> >> that represent the variables. One exception is how vectors are stored.
>>>> >> They are stored in hybrid Fortran - C ordering, i.e. vector component
>>>> >> increases fastest then i, then j, then k. It is straightforward to
>>>> >> pass these to Python and back using NumPy arrays or buffer objects at
>>>> >> the lowest level. So we can write Python code to bundle these arrays
>>>> >> into any form that yt (and Enzo when we start doing in-situ) wants. We
>>>> >> may have to deep-copy the vectors because of the issue I described.
>>>> >>
>>>> >> On Wed, May 25, 2011 at 7:59 PM, Matthew Turk <matthewturk at gmail.com>
>>>> >> wrote:
>>>> >>> Hi Jorge,
>>>> >>>
>>>> >>> This is really cool!
>>>> >>>
>>>> >>> I've been thinking about how to invert this process, and I think we
>>>> >>> could address it by creating a new hierarchy of objects, similar to
>>>> >>> the EnzoStaticOutputInMemory family of objects, that enables a
>>>> >>> bi-directional transfer of information.  All that's really necessary
>>>> >>> to get data in is to describe the mesh, describe the parameters of
>>>> >>> the
>>>> >>> simulation, and then communicate to yt how to access data.
>>>> >>>
>>>> >>> What do you think the best mechanism for passing data back and forth
>>>> >>> is?  For in situ viz with Enzo, usually we do this by constructing a
>>>> >>> new module object in the C code, and then insert into that a set of
>>>> >>> arrays describing the mesh and a dictionary of dictionaries of
>>>> >>> fields,
>>>> >>> with keys that correspond to grid id and field name.  But really, it
>>>> >>> could be any system, and I'm happy to work on constructing the
>>>> >>> methods
>>>> >>> to set this up.
>>>> >>>
>>>> >>> Best,
>>>> >>>
>>>> >>> Matt
>>>> >>>
>>>> >>> On Wed, May 25, 2011 at 5:54 PM, Jorge Poco <jorge.poco at kitware.com>
>>>> >>> wrote:
>>>> >>>> Hi Matthew and Berk
>>>> >>>>
>>>> >>>> the full stack is here http://paste.enzotools.org/show/1668/ I'm
>>>> >>>> trying to
>>>> >>>> compile paraview with the VTK_USE_SYSTEM_HDF5 but It's giving me
>>>> >>>> some
>>>> >>>> errors, I will try again tomorrow.
>>>> >>>>
>>>> >>>> Matthew: Right now, I have written a 'programmable source' script to
>>>> >>>> read a
>>>> >>>> file from the system using the yt function 'load'. Then, create a
>>>> >>>> projection
>>>> >>>> using yt and then the ProjectionPlot object is converted into a
>>>> >>>> vtkImageData
>>>> >>>> to be displayed into paraview.
>>>> >>>> What I want to do now, is to read a Dataset using paraview, then
>>>> >>>> create a
>>>> >>>> 'programmable filter' script to convert the data set into  yt's data
>>>> >>>> structure to create a visualization and finally display the image
>>>> >>>> inside
>>>> >>>> paraview. What would be the correct way to create yt objects
>>>> >>>> (EnzoHierarchy
>>>> >>>> and EnzoGrid)?
>>>> >>>>
>>>> >>>> Berk: I was trying to find a way to get access to camera information
>>>> >>>> from
>>>> >>>> the 'programmable filter' to send this information to yt  but I
>>>> >>>> didn't found
>>>> >>>> anything useful. Is it possible?
>>>> >>>>
>>>> >>>>
>>>> >>>> Regards,
>>>> >>>>
>>>> >>>> On Wed, May 25, 2011 at 3:10 PM, Berk Geveci
>>>> >>>> <berk.geveci at kitware.com>
>>>> >>>> wrote:
>>>> >>>>>
>>>> >>>>> If I understood what Matt is saying, the problem may be fixed by
>>>> >>>>> having ParaView use an external HDF5, the one that yt uses. I
>>>> >>>>> believe
>>>> >>>>> that the variable is something like PARAVIEW_USE_SYSTEM_HDF5 (or
>>>> >>>>> VTK_,
>>>> >>>>> not sure) in CMake.
>>>> >>>>>
>>>> >>>>> On Wed, May 25, 2011 at 10:49 AM, Matthew Turk
>>>> >>>>> <matthewturk at gmail.com>
>>>> >>>>> wrote:
>>>> >>>>> > Hi Jorge,
>>>> >>>>> >
>>>> >>>>> > Wow, I didn't know that YT-embedding had taken off quite so much!
>>>> >>>>> > This is great news.
>>>> >>>>> >
>>>> >>>>> > On Wed, May 25, 2011 at 9:48 AM, Jorge Poco
>>>> >>>>> > <jorge.poco at kitware.com>
>>>> >>>>> > wrote:
>>>> >>>>> >> Hi
>>>> >>>>> >> sorry my last email was sent incomplete
>>>> >>>>> >> I have been trying to use YT inside the Paraview's python shell.
>>>> >>>>> >> Initially I
>>>> >>>>> >> have had some problems which are now identified. It is not the
>>>> >>>>> >> best
>>>> >>>>> >> solution but I at least solve temporally these problems:
>>>> >>>>> >>
>>>> >>>>> >> argv problem:
>>>> >>>>> >>
>>>> >>>>> >> Sentence: 'from yt.mods import *'
>>>> >>>>> >> Error: for any reason paraview's python shell has not defined
>>>> >>>>> >> the
>>>> >>>>> >> variable
>>>> >>>>> >> 'sys.argv'
>>>> >>>>> >> Temporal solution: comment yt code where the sys.argv variable
>>>> >>>>> >> is used:
>>>> >>>>> >>
>>>> >>>>> >> yt/funcs.py: lines 439-453
>>>> >>>>> >> yt/utilities/parallel_tools/parallel_analysis_interface.py:
>>>> >>>>> >> lines 47
>>>> >>>>> >> and 48
>>>> >>>>> >
>>>> >>>>> > We can likely get around this on the yt side, although when
>>>> >>>>> > embedding
>>>> >>>>> > yt in Enzo we've escaped this by using the command
>>>> >>>>> > PySys_SetArgv(argc,
>>>> >>>>> > argv) immediately after calling Py_Initialize.  Does Partiview
>>>> >>>>> > report
>>>> >>>>> > a different executable name, by any chance?  If so we can use
>>>> >>>>> > that as
>>>> >>>>> > a signal to avoid requesting argv.
>>>> >>>>> >
>>>> >>>>> >>
>>>> >>>>> >> :
>>>> >>>>> >>
>>>> >>>>> >> Sentence: "pf = load('tests/DD0010/moving7_0010')"
>>>> >>>>> >> Error: python shell does not complete the load function and
>>>> >>>>> >> paraview
>>>> >>>>> >> keeps
>>>> >>>>> >> waiting.
>>>> >>>>> >> Temporal solution: When I debug paraview and interrupt its
>>>> >>>>> >> execution
>>>> >>>>> >> the
>>>> >>>>> >> backtrace shows sem_wait(). It happens inside the load function
>>>> >>>>> >> when
>>>> >>>>> >> tries
>>>> >>>>> >> to identify the candidates for input file (convenience.py line
>>>> >>>>> >> 74).
>>>> >>>>> >> There
>>>> >>>>> >> are 11 possible candidates and gives error for 2 of them:
>>>> >>>>> >> ChomboStaticOutput
>>>> >>>>> >> and FlashStaticOutput. Skipping those candidates solve
>>>> >>>>> >> temporally the
>>>> >>>>> >> problem: (yt/convenienve.py line 75) inside the loop add if n ==
>>>> >>>>> >> 'ChomboStaticOutput' or n == 'FlashStaticOutput': continue;
>>>> >>>>> >
>>>> >>>>> > This is interesting.  What those two routines have in common is
>>>> >>>>> > that
>>>> >>>>> > they are the only two that try opening the file using h5py.  HDF5
>>>> >>>>> > has
>>>> >>>>> > a number of error handling routines, which in the past I've seen
>>>> >>>>> > issues with when embedding one app in another -- h5py for
>>>> >>>>> > instance
>>>> >>>>> > handles error handling, but that can conflict with other
>>>> >>>>> > mechanisms
>>>> >>>>> > that are attempting to handle those errors.
>>>> >>>>> >
>>>> >>>>> > Another solution that would avoid digging into the source code
>>>> >>>>> > would
>>>> >>>>> > be to remove the types from the actual type registry:
>>>> >>>>> >
>>>> >>>>> > from yt.convenience import output_type_registry
>>>> >>>>> > del output_type_registry["ChomboStaticOutput"]
>>>> >>>>> > del output_type_registry["FlashStaticOutput"]
>>>> >>>>> >
>>>> >>>>> > I'm more interested in figuring out how to fix the interplay
>>>> >>>>> > between
>>>> >>>>> > hdf5-paraview and h5py, though.  Do you happen to have a full
>>>> >>>>> > stack
>>>> >>>>> > trace?
>>>> >>>>> >
>>>> >>>>> > Thanks very much for these reports -- this is very exciting, and
>>>> >>>>> > I'm
>>>> >>>>> > eager to help out!
>>>> >>>>> >
>>>> >>>>> > Best,
>>>> >>>>> >
>>>> >>>>> > Matt
>>>> >>>>> >
>>>> >>>>> >>
>>>> >>>>> >> Any of you have a better way to solve the problems above?
>>>> >>>>> >> Regards,
>>>> >>>>> >> On Wed, May 25, 2011 at 9:43 AM, Jorge Poco
>>>> >>>>> >> <jorge.poco at kitware.com>
>>>> >>>>> >> wrote:
>>>> >>>>> >>>
>>>> >>>>> >>> Hi
>>>> >>>>> >>> I have been trying to use YT inside the Paraview's python
>>>> >>>>> >>> shell.
>>>> >>>>> >>> Initially
>>>> >>>>> >>> I have had some problems which are now identified. It is not
>>>> >>>>> >>> the
>>>> >>>>> >>> smart solution but I at least solve temporally these problems:
>>>> >>>>> >>>
>>>> >>>>> >>> argv problem:
>>>> >>>>> >>>
>>>> >>>>> >>> Sentence: 'from yt.mods import *'
>>>> >>>>> >>> Error: for any reason paraview's python shell has not defined
>>>> >>>>> >>> the
>>>> >>>>> >>> variable
>>>> >>>>> >>> 'sys.argv'
>>>> >>>>> >>> Temporal solution: comment yt code where the sys.argv variable
>>>> >>>>> >>> is
>>>> >>>>> >>> used:
>>>> >>>>> >>>
>>>> >>>>> >>> yt/funcs.py: lines 439-453
>>>> >>>>> >>> yt/utilities/parallel_tools/parallel_analysis_interface.py:
>>>> >>>>> >>> lines 47
>>>> >>>>> >>> and
>>>> >>>>> >>> 48
>>>> >>>>> >>>
>>>> >>>>> >>> pa:
>>>> >>>>> >>>
>>>> >>>>> >>> Sentence: "pf = load('tests/DD0010/moving7_0010')"
>>>> >>>>> >>> Error: python shell does not complete the load function and
>>>> >>>>> >>> paraview
>>>> >>>>> >>> keeps
>>>> >>>>> >>> waiting.
>>>> >>>>> >>> Temporal solution: When I debug paraview and interrupt its
>>>> >>>>> >>> execution
>>>> >>>>> >>> the
>>>> >>>>> >>> backtrace shows sem_wait(). It happens inside the load function
>>>> >>>>> >>> when
>>>> >>>>> >>> tries
>>>> >>>>> >>> to identify the candidates for input file (convenience.py line
>>>> >>>>> >>> 74).
>>>> >>>>> >>> There
>>>> >>>>> >>> are 11 possible candidates and gives error for 2 of them:
>>>> >>>>> >>> ChomboStaticOutput
>>>> >>>>> >>> and FlashStaticOutput. Skipping those candidates solve
>>>> >>>>> >>> temporally the
>>>> >>>>> >>> problem:
>>>> >>>>> >>
>>>> >>>>> >>
>>>> >>>>> >> _______________________________________________
>>>> >>>>> >> Amr mailing list
>>>> >>>>> >> Amr at public.kitware.com
>>>> >>>>> >> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>>>> >>>>> >>
>>>> >>>>> >>
>>>> >>>>> > _______________________________________________
>>>> >>>>> > Amr mailing list
>>>> >>>>> > Amr at public.kitware.com
>>>> >>>>> > http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>>>> >>>>> >
>>>> >>>>> _______________________________________________
>>>> >>>>> Amr mailing list
>>>> >>>>> Amr at public.kitware.com
>>>> >>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>>>> >>>>
>>>> >>>>
>>>> >>>> _______________________________________________
>>>> >>>> Amr mailing list
>>>> >>>> Amr at public.kitware.com
>>>> >>>> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>>>> >>>>
>>>> >>>>
>>>> >>> _______________________________________________
>>>> >>> Amr mailing list
>>>> >>> Amr at public.kitware.com
>>>> >>> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>>>> >>>
>>>> >>
>>>> > _______________________________________________
>>>> > Amr mailing list
>>>> > Amr at public.kitware.com
>>>> > http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>>>> >
>>>> _______________________________________________
>>>> Amr mailing list
>>>> Amr at public.kitware.com
>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>>>
>>
>>
>