[Amr] yt inside paraview

George Zagaris george.zagaris at kitware.com
Fri Jun 10 14:40:04 EDT 2011 

That's great!

Yes the vtkHierarchicalBoxDataSet has IBLANK information that
can be supplied directly to yt.

Best,
George

On Fri, Jun 10, 2011 at 2:37 PM, Matthew Turk <matthewturk at gmail.com> wrote:
> Hi Jorge,
>
> Nope, it should not be mandatory, although it currently is.  It is
> currently only used to calculate child masking, which I believe we can
> also get from vtkHierarchicalBoxDataSet?  I'll make this change later
> today.
>
> -Matt
>
>
>
> On Fri, Jun 10, 2011 at 11:35 AM, Jorge Poco <jorge.poco at kitware.com> wrote:
>> HI Matt
>> Looks nice.
>> In the 'stream' frontends, when the stream.StreamHandler(...) is create,
>> one parameter is the parent_ids right? It's really mandatory to send it?
>> because the vtkHierarchicalBoxDataSet doesn't save this information, but we
>> can compute the parent/child relations.
>> Regards,
>>
>> On Fri, Jun 10, 2011 at 2:19 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>>
>>> Hi Jorge,
>>>
>>> I thought this might be of interest to you; I've started keeping track
>>> of how to interface with external codes such as ParaView here:
>>>
>>> http://hg.enzotools.org/yt/wiki/InterfacingWithYT
>>>
>>> I'll be updating this as need be with information about the in-memory
>>> datasets and so on.
>>>
>>> Best,
>>>
>>> Matt
>>>
>>> On Fri, May 27, 2011 at 2:08 PM, Matthew Turk <matthewturk at gmail.com>
>>> wrote:
>>> > Hi all,
>>> >
>>> > I've created a new data frontend in yt entitled "Stream" (the name is
>>> > up for discussion, if someone has a better idea -- Charles pointed out
>>> > today that it might get confused with time series data.)  The idea
>>> > behind this is that the user can use it to construct an arbitrary
>>> > dataset in memory.  I've tested this by proxying an existing Enzo
>>> > dataset:
>>> >
>>> >
>>> > http://matthewturk.bitbucket.org/html/ba3fd37b-842c-4641-b21e-2d3f5268eefe-stream_proxy-py.html
>>> >
>>> > (downloadable at the top level domain, or by "yt_lodgeit.py
>>> > --download=1672 if you have a yt stack installed.)
>>> >
>>> > You have to construct both a StreamHandler that knows how to pass
>>> > around the grid geometry and which, although not instantiated with
>>> > them, requires certain attributes to be set.  In addition, a fields
>>> > handler has to be created.  This script executes to completion and
>>> > passes its tests; projecting and slicing require slightly more effort,
>>> > because I wanted to pose to the Kitware group how the data should be
>>> > evaluated in terms of units -- should it be assumed to have CGS units
>>> > already?  How should the domain size be handled -- in yt right now
>>> > units are distinct from the code definitions, but I don't know how
>>> > Paraview handles this distinction.  (If you want to try setting up a
>>> > PlotCollection, you can, but you will have to set spf.units["cm"] =
>>> > spf.units["unitary"] = 1.0).
>>> >
>>> > Hopefully this can be a start, and we can continue to iterate on the
>>> > API necessary to load data in yt from that already loaded in Paraview.
>>> >  This is in the tip of the development yt repository, 6d4a5588a550.
>>> >
>>> > Best,
>>> >
>>> > Matt
>>> >
>>> > On Thu, May 26, 2011 at 3:54 PM, Jorge Poco <jorge.poco at kitware.com>
>>> > wrote:
>>> >> In the  meantime I will be working to import yt's data structures
>>> >> inside
>>> >> vtkHierarchicalBoxDataset using ProgrammableSources inside Python.
>>> >> Regards,
>>> >>
>>> >> On Thu, May 26, 2011 at 3:43 PM, Jorge Poco <jorge.poco at kitware.com>
>>> >> wrote:
>>> >>>
>>> >>> Hi
>>> >>> YT has some function to access the arrays data in numpy.array objects
>>> >>> (
>>> >>> pf.h.grids[0]['Density']), using them  we can instantiate a
>>> >>> vtkHierarchicalDataset to be process into paraview.
>>> >>> On Thu, May 26, 2011 at 2:53 PM, Berk Geveci <berk.geveci at kitware.com>
>>> >>> wrote:
>>> >>>>
>>> >>>> Awesome.
>>> >>>>
>>> >>>> What about from the other side? If we wanted to pass data structures
>>> >>>> that yt loaded (from an Enzo file for example) to VTK/ParaView for
>>> >>>> filtering/rendering?
>>> >>>>
>>> >>>> -berk
>>> >>>>
>>> >>>> On Thu, May 26, 2011 at 2:04 PM, Matthew Turk <matthewturk at gmail.com>
>>> >>>> wrote:
>>> >>>> > Hi Berk and Jorge,
>>> >>>> >
>>> >>>> > Okay, this makes sense.
>>> >>>> >
>>> >>>> > I will write up a new code frontend for yt that will act as a
>>> >>>> > generalized system; this will draw upon the current in situ Enzo
>>> >>>> > frontend, and will be designed to be instantiated with a list of
>>> >>>> > arrays and a set of callback functions.  We can them implement the
>>> >>>> > paraview-specific code inside this, or perhaps inside a
>>> >>>> > ProgrammableFilter inside paraview itself.  I'll ping the list back
>>> >>>> > when I have something up and running.
>>> >>>> >
>>> >>>> > Best,
>>> >>>> >
>>> >>>> > Matt
>>> >>>> >
>>> >>>> > On Thu, May 26, 2011 at 9:15 AM, Berk Geveci
>>> >>>> > <berk.geveci at kitware.com>
>>> >>>> > wrote:
>>> >>>> >> In VTK, an AMR mesh is represented as a collection of
>>> >>>> >> vtkUniformGrid
>>> >>>> >> objects. Each vtkUniformGrid has a set of Fortran-ordered data
>>> >>>> >> arrays
>>> >>>> >> that represent the variables. One exception is how vectors are
>>> >>>> >> stored.
>>> >>>> >> They are stored in hybrid Fortran - C ordering, i.e. vector
>>> >>>> >> component
>>> >>>> >> increases fastest then i, then j, then k. It is straightforward to
>>> >>>> >> pass these to Python and back using NumPy arrays or buffer objects
>>> >>>> >> at
>>> >>>> >> the lowest level. So we can write Python code to bundle these
>>> >>>> >> arrays
>>> >>>> >> into any form that yt (and Enzo when we start doing in-situ)
>>> >>>> >> wants. We
>>> >>>> >> may have to deep-copy the vectors because of the issue I
>>> >>>> >> described.
>>> >>>> >>
>>> >>>> >> On Wed, May 25, 2011 at 7:59 PM, Matthew Turk
>>> >>>> >> <matthewturk at gmail.com>
>>> >>>> >> wrote:
>>> >>>> >>> Hi Jorge,
>>> >>>> >>>
>>> >>>> >>> This is really cool!
>>> >>>> >>>
>>> >>>> >>> I've been thinking about how to invert this process, and I think
>>> >>>> >>> we
>>> >>>> >>> could address it by creating a new hierarchy of objects, similar
>>> >>>> >>> to
>>> >>>> >>> the EnzoStaticOutputInMemory family of objects, that enables a
>>> >>>> >>> bi-directional transfer of information.  All that's really
>>> >>>> >>> necessary
>>> >>>> >>> to get data in is to describe the mesh, describe the parameters
>>> >>>> >>> of
>>> >>>> >>> the
>>> >>>> >>> simulation, and then communicate to yt how to access data.
>>> >>>> >>>
>>> >>>> >>> What do you think the best mechanism for passing data back and
>>> >>>> >>> forth
>>> >>>> >>> is?  For in situ viz with Enzo, usually we do this by
>>> >>>> >>> constructing a
>>> >>>> >>> new module object in the C code, and then insert into that a set
>>> >>>> >>> of
>>> >>>> >>> arrays describing the mesh and a dictionary of dictionaries of
>>> >>>> >>> fields,
>>> >>>> >>> with keys that correspond to grid id and field name.  But really,
>>> >>>> >>> it
>>> >>>> >>> could be any system, and I'm happy to work on constructing the
>>> >>>> >>> methods
>>> >>>> >>> to set this up.
>>> >>>> >>>
>>> >>>> >>> Best,
>>> >>>> >>>
>>> >>>> >>> Matt
>>> >>>> >>>
>>> >>>> >>> On Wed, May 25, 2011 at 5:54 PM, Jorge Poco
>>> >>>> >>> <jorge.poco at kitware.com>
>>> >>>> >>> wrote:
>>> >>>> >>>> Hi Matthew and Berk
>>> >>>> >>>>
>>> >>>> >>>> the full stack is here http://paste.enzotools.org/show/1668/ I'm
>>> >>>> >>>> trying to
>>> >>>> >>>> compile paraview with the VTK_USE_SYSTEM_HDF5 but It's giving me
>>> >>>> >>>> some
>>> >>>> >>>> errors, I will try again tomorrow.
>>> >>>> >>>>
>>> >>>> >>>> Matthew: Right now, I have written a 'programmable source'
>>> >>>> >>>> script to
>>> >>>> >>>> read a
>>> >>>> >>>> file from the system using the yt function 'load'. Then, create
>>> >>>> >>>> a
>>> >>>> >>>> projection
>>> >>>> >>>> using yt and then the ProjectionPlot object is converted into a
>>> >>>> >>>> vtkImageData
>>> >>>> >>>> to be displayed into paraview.
>>> >>>> >>>> What I want to do now, is to read a Dataset using paraview, then
>>> >>>> >>>> create a
>>> >>>> >>>> 'programmable filter' script to convert the data set into  yt's
>>> >>>> >>>> data
>>> >>>> >>>> structure to create a visualization and finally display the
>>> >>>> >>>> image
>>> >>>> >>>> inside
>>> >>>> >>>> paraview. What would be the correct way to create yt objects
>>> >>>> >>>> (EnzoHierarchy
>>> >>>> >>>> and EnzoGrid)?
>>> >>>> >>>>
>>> >>>> >>>> Berk: I was trying to find a way to get access to camera
>>> >>>> >>>> information
>>> >>>> >>>> from
>>> >>>> >>>> the 'programmable filter' to send this information to yt  but I
>>> >>>> >>>> didn't found
>>> >>>> >>>> anything useful. Is it possible?
>>> >>>> >>>>
>>> >>>> >>>>
>>> >>>> >>>> Regards,
>>> >>>> >>>>
>>> >>>> >>>> On Wed, May 25, 2011 at 3:10 PM, Berk Geveci
>>> >>>> >>>> <berk.geveci at kitware.com>
>>> >>>> >>>> wrote:
>>> >>>> >>>>>
>>> >>>> >>>>> If I understood what Matt is saying, the problem may be fixed
>>> >>>> >>>>> by
>>> >>>> >>>>> having ParaView use an external HDF5, the one that yt uses. I
>>> >>>> >>>>> believe
>>> >>>> >>>>> that the variable is something like PARAVIEW_USE_SYSTEM_HDF5
>>> >>>> >>>>> (or
>>> >>>> >>>>> VTK_,
>>> >>>> >>>>> not sure) in CMake.
>>> >>>> >>>>>
>>> >>>> >>>>> On Wed, May 25, 2011 at 10:49 AM, Matthew Turk
>>> >>>> >>>>> <matthewturk at gmail.com>
>>> >>>> >>>>> wrote:
>>> >>>> >>>>> > Hi Jorge,
>>> >>>> >>>>> >
>>> >>>> >>>>> > Wow, I didn't know that YT-embedding had taken off quite so
>>> >>>> >>>>> > much!
>>> >>>> >>>>> > This is great news.
>>> >>>> >>>>> >
>>> >>>> >>>>> > On Wed, May 25, 2011 at 9:48 AM, Jorge Poco
>>> >>>> >>>>> > <jorge.poco at kitware.com>
>>> >>>> >>>>> > wrote:
>>> >>>> >>>>> >> Hi
>>> >>>> >>>>> >> sorry my last email was sent incomplete
>>> >>>> >>>>> >> I have been trying to use YT inside the Paraview's python
>>> >>>> >>>>> >> shell.
>>> >>>> >>>>> >> Initially I
>>> >>>> >>>>> >> have had some problems which are now identified. It is not
>>> >>>> >>>>> >> the
>>> >>>> >>>>> >> best
>>> >>>> >>>>> >> solution but I at least solve temporally these problems:
>>> >>>> >>>>> >>
>>> >>>> >>>>> >> argv problem:
>>> >>>> >>>>> >>
>>> >>>> >>>>> >> Sentence: 'from yt.mods import *'
>>> >>>> >>>>> >> Error: for any reason paraview's python shell has not
>>> >>>> >>>>> >> defined
>>> >>>> >>>>> >> the
>>> >>>> >>>>> >> variable
>>> >>>> >>>>> >> 'sys.argv'
>>> >>>> >>>>> >> Temporal solution: comment yt code where the sys.argv
>>> >>>> >>>>> >> variable
>>> >>>> >>>>> >> is used:
>>> >>>> >>>>> >>
>>> >>>> >>>>> >> yt/funcs.py: lines 439-453
>>> >>>> >>>>> >> yt/utilities/parallel_tools/parallel_analysis_interface.py:
>>> >>>> >>>>> >> lines 47
>>> >>>> >>>>> >> and 48
>>> >>>> >>>>> >
>>> >>>> >>>>> > We can likely get around this on the yt side, although when
>>> >>>> >>>>> > embedding
>>> >>>> >>>>> > yt in Enzo we've escaped this by using the command
>>> >>>> >>>>> > PySys_SetArgv(argc,
>>> >>>> >>>>> > argv) immediately after calling Py_Initialize.  Does
>>> >>>> >>>>> > Partiview
>>> >>>> >>>>> > report
>>> >>>> >>>>> > a different executable name, by any chance?  If so we can use
>>> >>>> >>>>> > that as
>>> >>>> >>>>> > a signal to avoid requesting argv.
>>> >>>> >>>>> >
>>> >>>> >>>>> >>
>>> >>>> >>>>> >> :
>>> >>>> >>>>> >>
>>> >>>> >>>>> >> Sentence: "pf = load('tests/DD0010/moving7_0010')"
>>> >>>> >>>>> >> Error: python shell does not complete the load function and
>>> >>>> >>>>> >> paraview
>>> >>>> >>>>> >> keeps
>>> >>>> >>>>> >> waiting.
>>> >>>> >>>>> >> Temporal solution: When I debug paraview and interrupt its
>>> >>>> >>>>> >> execution
>>> >>>> >>>>> >> the
>>> >>>> >>>>> >> backtrace shows sem_wait(). It happens inside the load
>>> >>>> >>>>> >> function
>>> >>>> >>>>> >> when
>>> >>>> >>>>> >> tries
>>> >>>> >>>>> >> to identify the candidates for input file (convenience.py
>>> >>>> >>>>> >> line
>>> >>>> >>>>> >> 74).
>>> >>>> >>>>> >> There
>>> >>>> >>>>> >> are 11 possible candidates and gives error for 2 of them:
>>> >>>> >>>>> >> ChomboStaticOutput
>>> >>>> >>>>> >> and FlashStaticOutput. Skipping those candidates solve
>>> >>>> >>>>> >> temporally the
>>> >>>> >>>>> >> problem: (yt/convenienve.py line 75) inside the loop add if
>>> >>>> >>>>> >> n ==
>>> >>>> >>>>> >> 'ChomboStaticOutput' or n == 'FlashStaticOutput': continue;
>>> >>>> >>>>> >
>>> >>>> >>>>> > This is interesting.  What those two routines have in common
>>> >>>> >>>>> > is
>>> >>>> >>>>> > that
>>> >>>> >>>>> > they are the only two that try opening the file using h5py.
>>> >>>> >>>>> >  HDF5
>>> >>>> >>>>> > has
>>> >>>> >>>>> > a number of error handling routines, which in the past I've
>>> >>>> >>>>> > seen
>>> >>>> >>>>> > issues with when embedding one app in another -- h5py for
>>> >>>> >>>>> > instance
>>> >>>> >>>>> > handles error handling, but that can conflict with other
>>> >>>> >>>>> > mechanisms
>>> >>>> >>>>> > that are attempting to handle those errors.
>>> >>>> >>>>> >
>>> >>>> >>>>> > Another solution that would avoid digging into the source
>>> >>>> >>>>> > code
>>> >>>> >>>>> > would
>>> >>>> >>>>> > be to remove the types from the actual type registry:
>>> >>>> >>>>> >
>>> >>>> >>>>> > from yt.convenience import output_type_registry
>>> >>>> >>>>> > del output_type_registry["ChomboStaticOutput"]
>>> >>>> >>>>> > del output_type_registry["FlashStaticOutput"]
>>> >>>> >>>>> >
>>> >>>> >>>>> > I'm more interested in figuring out how to fix the interplay
>>> >>>> >>>>> > between
>>> >>>> >>>>> > hdf5-paraview and h5py, though.  Do you happen to have a full
>>> >>>> >>>>> > stack
>>> >>>> >>>>> > trace?
>>> >>>> >>>>> >
>>> >>>> >>>>> > Thanks very much for these reports -- this is very exciting,
>>> >>>> >>>>> > and
>>> >>>> >>>>> > I'm
>>> >>>> >>>>> > eager to help out!
>>> >>>> >>>>> >
>>> >>>> >>>>> > Best,
>>> >>>> >>>>> >
>>> >>>> >>>>> > Matt
>>> >>>> >>>>> >
>>> >>>> >>>>> >>
>>> >>>> >>>>> >> Any of you have a better way to solve the problems above?
>>> >>>> >>>>> >> Regards,
>>> >>>> >>>>> >> On Wed, May 25, 2011 at 9:43 AM, Jorge Poco
>>> >>>> >>>>> >> <jorge.poco at kitware.com>
>>> >>>> >>>>> >> wrote:
>>> >>>> >>>>> >>>
>>> >>>> >>>>> >>> Hi
>>> >>>> >>>>> >>> I have been trying to use YT inside the Paraview's python
>>> >>>> >>>>> >>> shell.
>>> >>>> >>>>> >>> Initially
>>> >>>> >>>>> >>> I have had some problems which are now identified. It is
>>> >>>> >>>>> >>> not
>>> >>>> >>>>> >>> the
>>> >>>> >>>>> >>> smart solution but I at least solve temporally these
>>> >>>> >>>>> >>> problems:
>>> >>>> >>>>> >>>
>>> >>>> >>>>> >>> argv problem:
>>> >>>> >>>>> >>>
>>> >>>> >>>>> >>> Sentence: 'from yt.mods import *'
>>> >>>> >>>>> >>> Error: for any reason paraview's python shell has not
>>> >>>> >>>>> >>> defined
>>> >>>> >>>>> >>> the
>>> >>>> >>>>> >>> variable
>>> >>>> >>>>> >>> 'sys.argv'
>>> >>>> >>>>> >>> Temporal solution: comment yt code where the sys.argv
>>> >>>> >>>>> >>> variable
>>> >>>> >>>>> >>> is
>>> >>>> >>>>> >>> used:
>>> >>>> >>>>> >>>
>>> >>>> >>>>> >>> yt/funcs.py: lines 439-453
>>> >>>> >>>>> >>> yt/utilities/parallel_tools/parallel_analysis_interface.py:
>>> >>>> >>>>> >>> lines 47
>>> >>>> >>>>> >>> and
>>> >>>> >>>>> >>> 48
>>> >>>> >>>>> >>>
>>> >>>> >>>>> >>> pa:
>>> >>>> >>>>> >>>
>>> >>>> >>>>> >>> Sentence: "pf = load('tests/DD0010/moving7_0010')"
>>> >>>> >>>>> >>> Error: python shell does not complete the load function and
>>> >>>> >>>>> >>> paraview
>>> >>>> >>>>> >>> keeps
>>> >>>> >>>>> >>> waiting.
>>> >>>> >>>>> >>> Temporal solution: When I debug paraview and interrupt its
>>> >>>> >>>>> >>> execution
>>> >>>> >>>>> >>> the
>>> >>>> >>>>> >>> backtrace shows sem_wait(). It happens inside the load
>>> >>>> >>>>> >>> function
>>> >>>> >>>>> >>> when
>>> >>>> >>>>> >>> tries
>>> >>>> >>>>> >>> to identify the candidates for input file (convenience.py
>>> >>>> >>>>> >>> line
>>> >>>> >>>>> >>> 74).
>>> >>>> >>>>> >>> There
>>> >>>> >>>>> >>> are 11 possible candidates and gives error for 2 of them:
>>> >>>> >>>>> >>> ChomboStaticOutput
>>> >>>> >>>>> >>> and FlashStaticOutput. Skipping those candidates solve
>>> >>>> >>>>> >>> temporally the
>>> >>>> >>>>> >>> problem:
>>> >>>> >>>>> >>
>>> >>>> >>>>> >>
>>> >>>> >>>>> >> _______________________________________________
>>> >>>> >>>>> >> Amr mailing list
>>> >>>> >>>>> >> Amr at public.kitware.com
>>> >>>> >>>>> >> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>>> >>>> >>>>> >>
>>> >>>> >>>>> >>
>>> >>>> >>>>> > _______________________________________________
>>> >>>> >>>>> > Amr mailing list
>>> >>>> >>>>> > Amr at public.kitware.com
>>> >>>> >>>>> > http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>>> >>>> >>>>> >
>>> >>>> >>>>> _______________________________________________
>>> >>>> >>>>> Amr mailing list
>>> >>>> >>>>> Amr at public.kitware.com
>>> >>>> >>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>>> >>>> >>>>
>>> >>>> >>>>
>>> >>>> >>>> _______________________________________________
>>> >>>> >>>> Amr mailing list
>>> >>>> >>>> Amr at public.kitware.com
>>> >>>> >>>> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>>> >>>> >>>>
>>> >>>> >>>>
>>> >>>> >>> _______________________________________________
>>> >>>> >>> Amr mailing list
>>> >>>> >>> Amr at public.kitware.com
>>> >>>> >>> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>>> >>>> >>>
>>> >>>> >>
>>> >>>> > _______________________________________________
>>> >>>> > Amr mailing list
>>> >>>> > Amr at public.kitware.com
>>> >>>> > http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>>> >>>> >
>>> >>>> _______________________________________________
>>> >>>> Amr mailing list
>>> >>>> Amr at public.kitware.com
>>> >>>> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>>> >>>
>>> >>
>>> >>
>>> >
>>> _______________________________________________
>>> Amr mailing list
>>> Amr at public.kitware.com
>>> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>>
>>
> _______________________________________________
> Amr mailing list
> Amr at public.kitware.com
> http://public.kitware.com/cgi-bin/mailman/listinfo/amr
>

More information about the Amr mailing list