[Amr] yt inside paraview

Fri Jun 10 14:37:22 EDT 2011

Hi Jorge,

Nope, it should not be mandatory, although it currently is.  It is
currently only used to calculate child masking, which I believe we can
also get from vtkHierarchicalBoxDataSet?  I'll make this change later
today.

-Matt

On Fri, Jun 10, 2011 at 11:35 AM, Jorge Poco <jorge.poco at kitware.com> wrote:
> HI Matt
> Looks nice.
> In the 'stream' frontends, when the stream.StreamHandler(...) is create,
> one parameter is the parent_ids right? It's really mandatory to send it?
> because the vtkHierarchicalBoxDataSet doesn't save this information, but we
> can compute the parent/child relations.
> Regards,
>
> On Fri, Jun 10, 2011 at 2:19 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>
>> Hi Jorge,
>>
>> I thought this might be of interest to you; I've started keeping track
>> of how to interface with external codes such as ParaView here:
>>
>> http://hg.enzotools.org/yt/wiki/InterfacingWithYT
>>
>> I'll be updating this as need be with information about the in-memory
>> datasets and so on.
>>
>> Best,
>>
>> Matt
>>
>> On Fri, May 27, 2011 at 2:08 PM, Matthew Turk <matthewturk at gmail.com>
>> wrote:
>> > Hi all,
>> >
>> > I've created a new data frontend in yt entitled "Stream" (the name is
>> > up for discussion, if someone has a better idea -- Charles pointed out
>> > today that it might get confused with time series data.)  The idea
>> > behind this is that the user can use it to construct an arbitrary
>> > dataset in memory.  I've tested this by proxying an existing Enzo
>> > dataset:
>> >
>> >
>> > http://matthewturk.bitbucket.org/html/ba3fd37b-842c-4641-b21e-2d3f5268eefe-stream_proxy-py.html
>> >
>> > (downloadable at the top level domain, or by "yt_lodgeit.py
>> > --download=1672 if you have a yt stack installed.)
>> >
>> > You have to construct both a StreamHandler that knows how to pass
>> > around the grid geometry and which, although not instantiated with
>> > them, requires certain attributes to be set.  In addition, a fields
>> > handler has to be created.  This script executes to completion and
>> > passes its tests; projecting and slicing require slightly more effort,
>> > because I wanted to pose to the Kitware group how the data should be
>> > evaluated in terms of units -- should it be assumed to have CGS units
>> > already?  How should the domain size be handled -- in yt right now
>> > units are distinct from the code definitions, but I don't know how
>> > Paraview handles this distinction.  (If you want to try setting up a
>> > PlotCollection, you can, but you will have to set spf.units["cm"] =
>> > spf.units["unitary"] = 1.0).
>> >
>> > Hopefully this can be a start, and we can continue to iterate on the
>> > API necessary to load data in yt from that already loaded in Paraview.
>> >  This is in the tip of the development yt repository, 6d4a5588a550.
>> >
>> > Best,
>> >
>> > Matt
>> >
>> > On Thu, May 26, 2011 at 3:54 PM, Jorge Poco <jorge.poco at kitware.com>
>> > wrote:
>> >> In the  meantime I will be working to import yt's data structures
>> >> inside
>> >> vtkHierarchicalBoxDataset using ProgrammableSources inside Python.
>> >> Regards,
>> >>
>> >> On Thu, May 26, 2011 at 3:43 PM, Jorge Poco <jorge.poco at kitware.com>
>> >> wrote:
>> >>>
>> >>> Hi
>> >>> YT has some function to access the arrays data in numpy.array objects
>> >>> (
>> >>> pf.h.grids[0]['Density']), using them  we can instantiate a
>> >>> vtkHierarchicalDataset to be process into paraview.
>> >>> On Thu, May 26, 2011 at 2:53 PM, Berk Geveci <berk.geveci at kitware.com>
>> >>> wrote:
>> >>>>
>> >>>> Awesome.
>> >>>>
>> >>>> What about from the other side? If we wanted to pass data structures
>> >>>> that yt loaded (from an Enzo file for example) to VTK/ParaView for
>> >>>> filtering/rendering?
>> >>>>
>> >>>> -berk
>> >>>>
>> >>>> On Thu, May 26, 2011 at 2:04 PM, Matthew Turk <matthewturk at gmail.com>
>> >>>> wrote:
>> >>>> > Hi Berk and Jorge,
>> >>>> >
>> >>>> > Okay, this makes sense.
>> >>>> >
>> >>>> > I will write up a new code frontend for yt that will act as a
>> >>>> > generalized system; this will draw upon the current in situ Enzo
>> >>>> > frontend, and will be designed to be instantiated with a list of
>> >>>> > arrays and a set of callback functions.  We can them implement the
>> >>>> > paraview-specific code inside this, or perhaps inside a
>> >>>> > ProgrammableFilter inside paraview itself.  I'll ping the list back
>> >>>> > when I have something up and running.
>> >>>> >
>> >>>> > Best,
>> >>>> >
>> >>>> > Matt
>> >>>> >
>> >>>> > On Thu, May 26, 2011 at 9:15 AM, Berk Geveci
>> >>>> > <berk.geveci at kitware.com>
>> >>>> > wrote:
>> >>>> >> In VTK, an AMR mesh is represented as a collection of
>> >>>> >> vtkUniformGrid
>> >>>> >> objects. Each vtkUniformGrid has a set of Fortran-ordered data
>> >>>> >> arrays
>> >>>> >> that represent the variables. One exception is how vectors are
>> >>>> >> stored.
>> >>>> >> They are stored in hybrid Fortran - C ordering, i.e. vector
>> >>>> >> component
>> >>>> >> increases fastest then i, then j, then k. It is straightforward to
>> >>>> >> pass these to Python and back using NumPy arrays or buffer objects
>> >>>> >> at
>> >>>> >> the lowest level. So we can write Python code to bundle these
>> >>>> >> arrays
>> >>>> >> into any form that yt (and Enzo when we start doing in-situ)
>> >>>> >> wants. We
>> >>>> >> may have to deep-copy the vectors because of the issue I
>> >>>> >> described.
>> >>>> >>
>> >>>> >> On Wed, May 25, 2011 at 7:59 PM, Matthew Turk
>> >>>> >> <matthewturk at gmail.com>
>> >>>> >> wrote:
>> >>>> >>> Hi Jorge,
>> >>>> >>>
>> >>>> >>> This is really cool!
>> >>>> >>>
>> >>>> >>> I've been thinking about how to invert this process, and I think
>> >>>> >>> we
>> >>>> >>> could address it by creating a new hierarchy of objects, similar
>> >>>> >>> to
>> >>>> >>> the EnzoStaticOutputInMemory family of objects, that enables a
>> >>>> >>> bi-directional transfer of information.  All that's really
>> >>>> >>> necessary
>> >>>> >>> to get data in is to describe the mesh, describe the parameters
>> >>>> >>> of
>> >>>> >>> the
>> >>>> >>> simulation, and then communicate to yt how to access data.
>> >>>> >>>
>> >>>> >>> What do you think the best mechanism for passing data back and
>> >>>> >>> forth
>> >>>> >>> is?  For in situ viz with Enzo, usually we do this by
>> >>>> >>> constructing a
>> >>>> >>> new module object in the C code, and then insert into that a set
>> >>>> >>> of
>> >>>> >>> arrays describing the mesh and a dictionary of dictionaries of
>> >>>> >>> fields,
>> >>>> >>> with keys that correspond to grid id and field name.  But really,
>> >>>> >>> it
>> >>>> >>> could be any system, and I'm happy to work on constructing the
>> >>>> >>> methods
>> >>>> >>> to set this up.
>> >>>> >>>
>> >>>> >>> Best,
>> >>>> >>>
>> >>>> >>> Matt
>> >>>> >>>
>> >>>> >>> On Wed, May 25, 2011 at 5:54 PM, Jorge Poco
>> >>>> >>> <jorge.poco at kitware.com>
>> >>>> >>> wrote:
>> >>>> >>>> Hi Matthew and Berk
>> >>>> >>>>
>> >>>> >>>> the full stack is here http://paste.enzotools.org/show/1668/ I'm
>> >>>> >>>> trying to
>> >>>> >>>> compile paraview with the VTK_USE_SYSTEM_HDF5 but It's giving me
>> >>>> >>>> some
>> >>>> >>>> errors, I will try again tomorrow.
>> >>>> >>>>
>> >>>> >>>> Matthew: Right now, I have written a 'programmable source'
>> >>>> >>>> script to
>> >>>> >>>> read a
>> >>>> >>>> file from the system using the yt function 'load'. Then, create
>> >>>> >>>> a
>> >>>> >>>> projection
>> >>>> >>>> using yt and then the ProjectionPlot object is converted into a
>> >>>> >>>> vtkImageData
>> >>>> >>>> to be displayed into paraview.
>> >>>> >>>> What I want to do now, is to read a Dataset using paraview, then
>> >>>> >>>> create a
>> >>>> >>>> 'programmable filter' script to convert the data set into  yt's
>> >>>> >>>> data
>> >>>> >>>> structure to create a visualization and finally display the
>> >>>> >>>> image
>> >>>> >>>> inside
>> >>>> >>>> paraview. What would be the correct way to create yt objects
>> >>>> >>>> (EnzoHierarchy
>> >>>> >>>> and EnzoGrid)?
>> >>>> >>>>
>> >>>> >>>> Berk: I was trying to find a way to get access to camera
>> >>>> >>>> information
>> >>>> >>>> from
>> >>>> >>>> the 'programmable filter' to send this information to yt  but I
>> >>>> >>>> didn't found
>> >>>> >>>> anything useful. Is it possible?
>> >>>> >>>>
>> >>>> >>>>
>> >>>> >>>> Regards,
>> >>>> >>>>
>> >>>> >>>> On Wed, May 25, 2011 at 3:10 PM, Berk Geveci
>> >>>> >>>> <berk.geveci at kitware.com>
>> >>>> >>>> wrote:
>> >>>> >>>>>
>> >>>> >>>>> If I understood what Matt is saying, the problem may be fixed
>> >>>> >>>>> by
>> >>>> >>>>> having ParaView use an external HDF5, the one that yt uses. I
>> >>>> >>>>> believe
>> >>>> >>>>> that the variable is something like PARAVIEW_USE_SYSTEM_HDF5
>> >>>> >>>>> (or
>> >>>> >>>>> VTK_,
>> >>>> >>>>> not sure) in CMake.
>> >>>> >>>>>
>> >>>> >>>>> On Wed, May 25, 2011 at 10:49 AM, Matthew Turk
>> >>>> >>>>> <matthewturk at gmail.com>
>> >>>> >>>>> wrote:
>> >>>> >>>>> > Hi Jorge,
>> >>>> >>>>> >
>> >>>> >>>>> > Wow, I didn't know that YT-embedding had taken off quite so
>> >>>> >>>>> > much!
>> >>>> >>>>> > This is great news.
>> >>>> >>>>> >
>> >>>> >>>>> > On Wed, May 25, 2011 at 9:48 AM, Jorge Poco
>> >>>> >>>>> > <jorge.poco at kitware.com>
>> >>>> >>>>> > wrote:
>> >>>> >>>>> >> Hi
>> >>>> >>>>> >> sorry my last email was sent incomplete
>> >>>> >>>>> >> I have been trying to use YT inside the Paraview's python
>> >>>> >>>>> >> shell.
>> >>>> >>>>> >> Initially I
>> >>>> >>>>> >> have had some problems which are now identified. It is not
>> >>>> >>>>> >> the
>> >>>> >>>>> >> best
>> >>>> >>>>> >> solution but I at least solve temporally these problems:
>> >>>> >>>>> >>
>> >>>> >>>>> >> argv problem:
>> >>>> >>>>> >>
>> >>>> >>>>> >> Sentence: 'from yt.mods import *'
>> >>>> >>>>> >> Error: for any reason paraview's python shell has not
>> >>>> >>>>> >> defined
>> >>>> >>>>> >> the
>> >>>> >>>>> >> variable
>> >>>> >>>>> >> 'sys.argv'
>> >>>> >>>>> >> Temporal solution: comment yt code where the sys.argv
>> >>>> >>>>> >> variable
>> >>>> >>>>> >> is used:
>> >>>> >>>>> >>
>> >>>> >>>>> >> yt/funcs.py: lines 439-453
>> >>>> >>>>> >> yt/utilities/parallel_tools/parallel_analysis_interface.py:
>> >>>> >>>>> >> lines 47
>> >>>> >>>>> >> and 48
>> >>>> >>>>> >
>> >>>> >>>>> > We can likely get around this on the yt side, although when
>> >>>> >>>>> > embedding
>> >>>> >>>>> > yt in Enzo we've escaped this by using the command
>> >>>> >>>>> > PySys_SetArgv(argc,
>> >>>> >>>>> > argv) immediately after calling Py_Initialize.  Does
>> >>>> >>>>> > Partiview
>> >>>> >>>>> > report
>> >>>> >>>>> > a different executable name, by any chance?  If so we can use
>> >>>> >>>>> > that as
>> >>>> >>>>> > a signal to avoid requesting argv.
>> >>>> >>>>> >
>> >>>> >>>>> >>
>> >>>> >>>>> >> :
>> >>>> >>>>> >>
>> >>>> >>>>> >> Sentence: "pf = load('tests/DD0010/moving7_0010')"
>> >>>> >>>>> >> Error: python shell does not complete the load function and
>> >>>> >>>>> >> paraview
>> >>>> >>>>> >> keeps
>> >>>> >>>>> >> waiting.
>> >>>> >>>>> >> Temporal solution: When I debug paraview and interrupt its
>> >>>> >>>>> >> execution
>> >>>> >>>>> >> the
>> >>>> >>>>> >> backtrace shows sem_wait(). It happens inside the load
>> >>>> >>>>> >> function
>> >>>> >>>>> >> when
>> >>>> >>>>> >> tries
>> >>>> >>>>> >> to identify the candidates for input file (convenience.py
>> >>>> >>>>> >> line
>> >>>> >>>>> >> 74).
>> >>>> >>>>> >> There
>> >>>> >>>>> >> are 11 possible candidates and gives error for 2 of them:
>> >>>> >>>>> >> ChomboStaticOutput
>> >>>> >>>>> >> and FlashStaticOutput. Skipping those candidates solve
>> >>>> >>>>> >> temporally the
>> >>>> >>>>> >> problem: (yt/convenienve.py line 75) inside the loop add if
>> >>>> >>>>> >> n ==
>> >>>> >>>>> >> 'ChomboStaticOutput' or n == 'FlashStaticOutput': continue;
>> >>>> >>>>> >
>> >>>> >>>>> > This is interesting.  What those two routines have in common
>> >>>> >>>>> > is
>> >>>> >>>>> > that
>> >>>> >>>>> > they are the only two that try opening the file using h5py.
>> >>>> >>>>> >  HDF5
>> >>>> >>>>> > has
>> >>>> >>>>> > a number of error handling routines, which in the past I've
>> >>>> >>>>> > seen
>> >>>> >>>>> > issues with when embedding one app in another -- h5py for
>> >>>> >>>>> > instance
>> >>>> >>>>> > handles error handling, but that can conflict with other
>> >>>> >>>>> > mechanisms
>> >>>> >>>>> > that are attempting to handle those errors.
>> >>>> >>>>> >
>> >>>> >>>>> > Another solution that would avoid digging into the source
>> >>>> >>>>> > code
>> >>>> >>>>> > would
>> >>>> >>>>> > be to remove the types from the actual type registry:
>> >>>> >>>>> >
>> >>>> >>>>> > from yt.convenience import output_type_registry
>> >>>> >>>>> > del output_type_registry["ChomboStaticOutput"]
>> >>>> >>>>> > del output_type_registry["FlashStaticOutput"]
>> >>>> >>>>> >
>> >>>> >>>>> > I'm more interested in figuring out how to fix the interplay
>> >>>> >>>>> > between
>> >>>> >>>>> > hdf5-paraview and h5py, though.  Do you happen to have a full
>> >>>> >>>>> > stack
>> >>>> >>>>> > trace?
>> >>>> >>>>> >
>> >>>> >>>>> > Thanks very much for these reports -- this is very exciting,
>> >>>> >>>>> > and
>> >>>> >>>>> > I'm
>> >>>> >>>>> > eager to help out!
>> >>>> >>>>> >
>> >>>> >>>>> > Best,
>> >>>> >>>>> >
>> >>>> >>>>> > Matt
>> >>>> >>>>> >
>> >>>> >>>>> >>
>> >>>> >>>>> >> Any of you have a better way to solve the problems above?
>> >>>> >>>>> >> Regards,
>> >>>> >>>>> >> On Wed, May 25, 2011 at 9:43 AM, Jorge Poco
>> >>>> >>>>> >> <jorge.poco at kitware.com>
>> >>>> >>>>> >> wrote:
>> >>>> >>>>> >>>
>> >>>> >>>>> >>> Hi
>> >>>> >>>>> >>> I have been trying to use YT inside the Paraview's python
>> >>>> >>>>> >>> shell.
>> >>>> >>>>> >>> Initially
>> >>>> >>>>> >>> I have had some problems which are now identified. It is
>> >>>> >>>>> >>> not
>> >>>> >>>>> >>> the
>> >>>> >>>>> >>> smart solution but I at least solve temporally these
>> >>>> >>>>> >>> problems:
>> >>>> >>>>> >>>
>> >>>> >>>>> >>> argv problem:
>> >>>> >>>>> >>>
>> >>>> >>>>> >>> Sentence: 'from yt.mods import *'
>> >>>> >>>>> >>> Error: for any reason paraview's python shell has not
>> >>>> >>>>> >>> defined
>> >>>> >>>>> >>> the
>> >>>> >>>>> >>> variable
>> >>>> >>>>> >>> 'sys.argv'
>> >>>> >>>>> >>> Temporal solution: comment yt code where the sys.argv
>> >>>> >>>>> >>> variable
>> >>>> >>>>> >>> is
>> >>>> >>>>> >>> used:
>> >>>> >>>>> >>>
>> >>>> >>>>> >>> yt/funcs.py: lines 439-453
>> >>>> >>>>> >>> yt/utilities/parallel_tools/parallel_analysis_interface.py:
>> >>>> >>>>> >>> lines 47
>> >>>> >>>>> >>> and
>> >>>> >>>>> >>> 48
>> >>>> >>>>> >>>
>> >>>> >>>>> >>> pa:
>> >>>> >>>>> >>>
>> >>>> >>>>> >>> Sentence: "pf = load('tests/DD0010/moving7_0010')"
>> >>>> >>>>> >>> Error: python shell does not complete the load function and
>> >>>> >>>>> >>> paraview
>> >>>> >>>>> >>> keeps
>> >>>> >>>>> >>> waiting.
>> >>>> >>>>> >>> Temporal solution: When I debug paraview and interrupt its
>> >>>> >>>>> >>> execution
>> >>>> >>>>> >>> the
>> >>>> >>>>> >>> backtrace shows sem_wait(). It happens inside the load
>> >>>> >>>>> >>> function
>> >>>> >>>>> >>> when
>> >>>> >>>>> >>> tries
>> >>>> >>>>> >>> to identify the candidates for input file (convenience.py
>> >>>> >>>>> >>> line
>> >>>> >>>>> >>> 74).
>> >>>> >>>>> >>> There
>> >>>> >>>>> >>> are 11 possible candidates and gives error for 2 of them:
>> >>>> >>>>> >>> ChomboStaticOutput
>> >>>> >>>>> >>> and FlashStaticOutput. Skipping those candidates solve
>> >>>> >>>>> >>> temporally the
>> >>>> >>>>> >>> problem:
>> >>>> >>>>> >>
>> >>>> >>>>> >>
>> >>>> >>>>> >> _______________________________________________
>> >>>> >>>>> >> Amr mailing list
>> >>>> >>>>> >> Amr at public.kitware.com
>> >>>> >>>>> >> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>> >>>> >>>>> >>
>> >>>> >>>>> >>
>> >>>> >>>>> > _______________________________________________
>> >>>> >>>>> > Amr mailing list
>> >>>> >>>>> > Amr at public.kitware.com
>> >>>> >>>>> > http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>> >>>> >>>>> >
>> >>>> >>>>> _______________________________________________
>> >>>> >>>>> Amr mailing list
>> >>>> >>>>> Amr at public.kitware.com
>> >>>> >>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>> >>>> >>>>
>> >>>> >>>>
>> >>>> >>>> _______________________________________________
>> >>>> >>>> Amr mailing list
>> >>>> >>>> Amr at public.kitware.com
>> >>>> >>>> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>> >>>> >>>>
>> >>>> >>>>
>> >>>> >>> _______________________________________________
>> >>>> >>> Amr mailing list
>> >>>> >>> Amr at public.kitware.com
>> >>>> >>> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>> >>>> >>>
>> >>>> >>
>> >>>> > _______________________________________________
>> >>>> > Amr mailing list
>> >>>> > Amr at public.kitware.com
>> >>>> > http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>> >>>> >
>> >>>> _______________________________________________
>> >>>> Amr mailing list
>> >>>> Amr at public.kitware.com
>> >>>> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>> >>>
>> >>
>> >>
>> >
>> _______________________________________________
>> Amr mailing list
>> Amr at public.kitware.com
>> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>
>