HI Matt<div><br></div><div>Looks nice.</div><div>In the 'stream' frontends, when the stream.StreamHandler(...) is create, one parameter is the parent_ids right? It's really mandatory to send it?</div><div>because the vtkHierarchicalBoxDataSet doesn't save this information, but we can compute the parent/child relations. </div>
<div><br></div><div>Regards,<br><br><div class="gmail_quote">On Fri, Jun 10, 2011 at 2:19 PM, Matthew Turk <span dir="ltr"><<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi Jorge,<br>
<br>
I thought this might be of interest to you; I've started keeping track<br>
of how to interface with external codes such as ParaView here:<br>
<br>
<a href="http://hg.enzotools.org/yt/wiki/InterfacingWithYT" target="_blank">http://hg.enzotools.org/yt/wiki/InterfacingWithYT</a><br>
<br>
I'll be updating this as need be with information about the in-memory<br>
datasets and so on.<br>
<br>
Best,<br>
<br>
Matt<br>
<div><div></div><div class="h5"><br>
On Fri, May 27, 2011 at 2:08 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>> wrote:<br>
> Hi all,<br>
><br>
> I've created a new data frontend in yt entitled "Stream" (the name is<br>
> up for discussion, if someone has a better idea -- Charles pointed out<br>
> today that it might get confused with time series data.) The idea<br>
> behind this is that the user can use it to construct an arbitrary<br>
> dataset in memory. I've tested this by proxying an existing Enzo<br>
> dataset:<br>
><br>
> <a href="http://matthewturk.bitbucket.org/html/ba3fd37b-842c-4641-b21e-2d3f5268eefe-stream_proxy-py.html" target="_blank">http://matthewturk.bitbucket.org/html/ba3fd37b-842c-4641-b21e-2d3f5268eefe-stream_proxy-py.html</a><br>
><br>
> (downloadable at the top level domain, or by "yt_lodgeit.py<br>
> --download=1672 if you have a yt stack installed.)<br>
><br>
> You have to construct both a StreamHandler that knows how to pass<br>
> around the grid geometry and which, although not instantiated with<br>
> them, requires certain attributes to be set. In addition, a fields<br>
> handler has to be created. This script executes to completion and<br>
> passes its tests; projecting and slicing require slightly more effort,<br>
> because I wanted to pose to the Kitware group how the data should be<br>
> evaluated in terms of units -- should it be assumed to have CGS units<br>
> already? How should the domain size be handled -- in yt right now<br>
> units are distinct from the code definitions, but I don't know how<br>
> Paraview handles this distinction. (If you want to try setting up a<br>
> PlotCollection, you can, but you will have to set spf.units["cm"] =<br>
> spf.units["unitary"] = 1.0).<br>
><br>
> Hopefully this can be a start, and we can continue to iterate on the<br>
> API necessary to load data in yt from that already loaded in Paraview.<br>
> This is in the tip of the development yt repository, 6d4a5588a550.<br>
><br>
> Best,<br>
><br>
> Matt<br>
><br>
> On Thu, May 26, 2011 at 3:54 PM, Jorge Poco <<a href="mailto:jorge.poco@kitware.com">jorge.poco@kitware.com</a>> wrote:<br>
>> In the meantime I will be working to import yt's data structures inside<br>
>> vtkHierarchicalBoxDataset using ProgrammableSources inside Python.<br>
>> Regards,<br>
>><br>
>> On Thu, May 26, 2011 at 3:43 PM, Jorge Poco <<a href="mailto:jorge.poco@kitware.com">jorge.poco@kitware.com</a>> wrote:<br>
>>><br>
>>> Hi<br>
>>> YT has some function to access the arrays data in numpy.array objects (<br>
>>> pf.h.grids[0]['Density']), using them we can instantiate a<br>
>>> vtkHierarchicalDataset to be process into paraview.<br>
>>> On Thu, May 26, 2011 at 2:53 PM, Berk Geveci <<a href="mailto:berk.geveci@kitware.com">berk.geveci@kitware.com</a>><br>
>>> wrote:<br>
>>>><br>
>>>> Awesome.<br>
>>>><br>
>>>> What about from the other side? If we wanted to pass data structures<br>
>>>> that yt loaded (from an Enzo file for example) to VTK/ParaView for<br>
>>>> filtering/rendering?<br>
>>>><br>
>>>> -berk<br>
>>>><br>
>>>> On Thu, May 26, 2011 at 2:04 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>>>> wrote:<br>
>>>> > Hi Berk and Jorge,<br>
>>>> ><br>
>>>> > Okay, this makes sense.<br>
>>>> ><br>
>>>> > I will write up a new code frontend for yt that will act as a<br>
>>>> > generalized system; this will draw upon the current in situ Enzo<br>
>>>> > frontend, and will be designed to be instantiated with a list of<br>
>>>> > arrays and a set of callback functions. We can them implement the<br>
>>>> > paraview-specific code inside this, or perhaps inside a<br>
>>>> > ProgrammableFilter inside paraview itself. I'll ping the list back<br>
>>>> > when I have something up and running.<br>
>>>> ><br>
>>>> > Best,<br>
>>>> ><br>
>>>> > Matt<br>
>>>> ><br>
>>>> > On Thu, May 26, 2011 at 9:15 AM, Berk Geveci <<a href="mailto:berk.geveci@kitware.com">berk.geveci@kitware.com</a>><br>
>>>> > wrote:<br>
>>>> >> In VTK, an AMR mesh is represented as a collection of vtkUniformGrid<br>
>>>> >> objects. Each vtkUniformGrid has a set of Fortran-ordered data arrays<br>
>>>> >> that represent the variables. One exception is how vectors are stored.<br>
>>>> >> They are stored in hybrid Fortran - C ordering, i.e. vector component<br>
>>>> >> increases fastest then i, then j, then k. It is straightforward to<br>
>>>> >> pass these to Python and back using NumPy arrays or buffer objects at<br>
>>>> >> the lowest level. So we can write Python code to bundle these arrays<br>
>>>> >> into any form that yt (and Enzo when we start doing in-situ) wants. We<br>
>>>> >> may have to deep-copy the vectors because of the issue I described.<br>
>>>> >><br>
>>>> >> On Wed, May 25, 2011 at 7:59 PM, Matthew Turk <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>>>> >> wrote:<br>
>>>> >>> Hi Jorge,<br>
>>>> >>><br>
>>>> >>> This is really cool!<br>
>>>> >>><br>
>>>> >>> I've been thinking about how to invert this process, and I think we<br>
>>>> >>> could address it by creating a new hierarchy of objects, similar to<br>
>>>> >>> the EnzoStaticOutputInMemory family of objects, that enables a<br>
>>>> >>> bi-directional transfer of information. All that's really necessary<br>
>>>> >>> to get data in is to describe the mesh, describe the parameters of<br>
>>>> >>> the<br>
>>>> >>> simulation, and then communicate to yt how to access data.<br>
>>>> >>><br>
>>>> >>> What do you think the best mechanism for passing data back and forth<br>
>>>> >>> is? For in situ viz with Enzo, usually we do this by constructing a<br>
>>>> >>> new module object in the C code, and then insert into that a set of<br>
>>>> >>> arrays describing the mesh and a dictionary of dictionaries of<br>
>>>> >>> fields,<br>
>>>> >>> with keys that correspond to grid id and field name. But really, it<br>
>>>> >>> could be any system, and I'm happy to work on constructing the<br>
>>>> >>> methods<br>
>>>> >>> to set this up.<br>
>>>> >>><br>
>>>> >>> Best,<br>
>>>> >>><br>
>>>> >>> Matt<br>
>>>> >>><br>
>>>> >>> On Wed, May 25, 2011 at 5:54 PM, Jorge Poco <<a href="mailto:jorge.poco@kitware.com">jorge.poco@kitware.com</a>><br>
>>>> >>> wrote:<br>
>>>> >>>> Hi Matthew and Berk<br>
>>>> >>>><br>
>>>> >>>> the full stack is here <a href="http://paste.enzotools.org/show/1668/" target="_blank">http://paste.enzotools.org/show/1668/</a> I'm<br>
>>>> >>>> trying to<br>
>>>> >>>> compile paraview with the VTK_USE_SYSTEM_HDF5 but It's giving me<br>
>>>> >>>> some<br>
>>>> >>>> errors, I will try again tomorrow.<br>
>>>> >>>><br>
>>>> >>>> Matthew: Right now, I have written a 'programmable source' script to<br>
>>>> >>>> read a<br>
>>>> >>>> file from the system using the yt function 'load'. Then, create a<br>
>>>> >>>> projection<br>
>>>> >>>> using yt and then the ProjectionPlot object is converted into a<br>
>>>> >>>> vtkImageData<br>
>>>> >>>> to be displayed into paraview.<br>
>>>> >>>> What I want to do now, is to read a Dataset using paraview, then<br>
>>>> >>>> create a<br>
>>>> >>>> 'programmable filter' script to convert the data set into yt's data<br>
>>>> >>>> structure to create a visualization and finally display the image<br>
>>>> >>>> inside<br>
>>>> >>>> paraview. What would be the correct way to create yt objects<br>
>>>> >>>> (EnzoHierarchy<br>
>>>> >>>> and EnzoGrid)?<br>
>>>> >>>><br>
>>>> >>>> Berk: I was trying to find a way to get access to camera information<br>
>>>> >>>> from<br>
>>>> >>>> the 'programmable filter' to send this information to yt but I<br>
>>>> >>>> didn't found<br>
>>>> >>>> anything useful. Is it possible?<br>
>>>> >>>><br>
>>>> >>>><br>
>>>> >>>> Regards,<br>
>>>> >>>><br>
>>>> >>>> On Wed, May 25, 2011 at 3:10 PM, Berk Geveci<br>
>>>> >>>> <<a href="mailto:berk.geveci@kitware.com">berk.geveci@kitware.com</a>><br>
>>>> >>>> wrote:<br>
>>>> >>>>><br>
>>>> >>>>> If I understood what Matt is saying, the problem may be fixed by<br>
>>>> >>>>> having ParaView use an external HDF5, the one that yt uses. I<br>
>>>> >>>>> believe<br>
>>>> >>>>> that the variable is something like PARAVIEW_USE_SYSTEM_HDF5 (or<br>
>>>> >>>>> VTK_,<br>
>>>> >>>>> not sure) in CMake.<br>
>>>> >>>>><br>
>>>> >>>>> On Wed, May 25, 2011 at 10:49 AM, Matthew Turk<br>
>>>> >>>>> <<a href="mailto:matthewturk@gmail.com">matthewturk@gmail.com</a>><br>
>>>> >>>>> wrote:<br>
>>>> >>>>> > Hi Jorge,<br>
>>>> >>>>> ><br>
>>>> >>>>> > Wow, I didn't know that YT-embedding had taken off quite so much!<br>
>>>> >>>>> > This is great news.<br>
>>>> >>>>> ><br>
>>>> >>>>> > On Wed, May 25, 2011 at 9:48 AM, Jorge Poco<br>
>>>> >>>>> > <<a href="mailto:jorge.poco@kitware.com">jorge.poco@kitware.com</a>><br>
>>>> >>>>> > wrote:<br>
>>>> >>>>> >> Hi<br>
>>>> >>>>> >> sorry my last email was sent incomplete<br>
>>>> >>>>> >> I have been trying to use YT inside the Paraview's python shell.<br>
>>>> >>>>> >> Initially I<br>
>>>> >>>>> >> have had some problems which are now identified. It is not the<br>
>>>> >>>>> >> best<br>
>>>> >>>>> >> solution but I at least solve temporally these problems:<br>
>>>> >>>>> >><br>
>>>> >>>>> >> argv problem:<br>
>>>> >>>>> >><br>
>>>> >>>>> >> Sentence: 'from yt.mods import *'<br>
>>>> >>>>> >> Error: for any reason paraview's python shell has not defined<br>
>>>> >>>>> >> the<br>
>>>> >>>>> >> variable<br>
>>>> >>>>> >> 'sys.argv'<br>
>>>> >>>>> >> Temporal solution: comment yt code where the sys.argv variable<br>
>>>> >>>>> >> is used:<br>
>>>> >>>>> >><br>
>>>> >>>>> >> yt/funcs.py: lines 439-453<br>
>>>> >>>>> >> yt/utilities/parallel_tools/parallel_analysis_interface.py:<br>
>>>> >>>>> >> lines 47<br>
>>>> >>>>> >> and 48<br>
>>>> >>>>> ><br>
>>>> >>>>> > We can likely get around this on the yt side, although when<br>
>>>> >>>>> > embedding<br>
>>>> >>>>> > yt in Enzo we've escaped this by using the command<br>
>>>> >>>>> > PySys_SetArgv(argc,<br>
>>>> >>>>> > argv) immediately after calling Py_Initialize. Does Partiview<br>
>>>> >>>>> > report<br>
>>>> >>>>> > a different executable name, by any chance? If so we can use<br>
>>>> >>>>> > that as<br>
>>>> >>>>> > a signal to avoid requesting argv.<br>
>>>> >>>>> ><br>
>>>> >>>>> >><br>
>>>> >>>>> >> :<br>
>>>> >>>>> >><br>
>>>> >>>>> >> Sentence: "pf = load('tests/DD0010/moving7_0010')"<br>
>>>> >>>>> >> Error: python shell does not complete the load function and<br>
>>>> >>>>> >> paraview<br>
>>>> >>>>> >> keeps<br>
>>>> >>>>> >> waiting.<br>
>>>> >>>>> >> Temporal solution: When I debug paraview and interrupt its<br>
>>>> >>>>> >> execution<br>
>>>> >>>>> >> the<br>
>>>> >>>>> >> backtrace shows sem_wait(). It happens inside the load function<br>
>>>> >>>>> >> when<br>
>>>> >>>>> >> tries<br>
>>>> >>>>> >> to identify the candidates for input file (convenience.py line<br>
>>>> >>>>> >> 74).<br>
>>>> >>>>> >> There<br>
>>>> >>>>> >> are 11 possible candidates and gives error for 2 of them:<br>
>>>> >>>>> >> ChomboStaticOutput<br>
>>>> >>>>> >> and FlashStaticOutput. Skipping those candidates solve<br>
>>>> >>>>> >> temporally the<br>
>>>> >>>>> >> problem: (yt/convenienve.py line 75) inside the loop add if n ==<br>
>>>> >>>>> >> 'ChomboStaticOutput' or n == 'FlashStaticOutput': continue;<br>
>>>> >>>>> ><br>
>>>> >>>>> > This is interesting. What those two routines have in common is<br>
>>>> >>>>> > that<br>
>>>> >>>>> > they are the only two that try opening the file using h5py. HDF5<br>
>>>> >>>>> > has<br>
>>>> >>>>> > a number of error handling routines, which in the past I've seen<br>
>>>> >>>>> > issues with when embedding one app in another -- h5py for<br>
>>>> >>>>> > instance<br>
>>>> >>>>> > handles error handling, but that can conflict with other<br>
>>>> >>>>> > mechanisms<br>
>>>> >>>>> > that are attempting to handle those errors.<br>
>>>> >>>>> ><br>
>>>> >>>>> > Another solution that would avoid digging into the source code<br>
>>>> >>>>> > would<br>
>>>> >>>>> > be to remove the types from the actual type registry:<br>
>>>> >>>>> ><br>
>>>> >>>>> > from yt.convenience import output_type_registry<br>
>>>> >>>>> > del output_type_registry["ChomboStaticOutput"]<br>
>>>> >>>>> > del output_type_registry["FlashStaticOutput"]<br>
>>>> >>>>> ><br>
>>>> >>>>> > I'm more interested in figuring out how to fix the interplay<br>
>>>> >>>>> > between<br>
>>>> >>>>> > hdf5-paraview and h5py, though. Do you happen to have a full<br>
>>>> >>>>> > stack<br>
>>>> >>>>> > trace?<br>
>>>> >>>>> ><br>
>>>> >>>>> > Thanks very much for these reports -- this is very exciting, and<br>
>>>> >>>>> > I'm<br>
>>>> >>>>> > eager to help out!<br>
>>>> >>>>> ><br>
>>>> >>>>> > Best,<br>
>>>> >>>>> ><br>
>>>> >>>>> > Matt<br>
>>>> >>>>> ><br>
>>>> >>>>> >><br>
>>>> >>>>> >> Any of you have a better way to solve the problems above?<br>
>>>> >>>>> >> Regards,<br>
>>>> >>>>> >> On Wed, May 25, 2011 at 9:43 AM, Jorge Poco<br>
>>>> >>>>> >> <<a href="mailto:jorge.poco@kitware.com">jorge.poco@kitware.com</a>><br>
>>>> >>>>> >> wrote:<br>
>>>> >>>>> >>><br>
>>>> >>>>> >>> Hi<br>
>>>> >>>>> >>> I have been trying to use YT inside the Paraview's python<br>
>>>> >>>>> >>> shell.<br>
>>>> >>>>> >>> Initially<br>
>>>> >>>>> >>> I have had some problems which are now identified. It is not<br>
>>>> >>>>> >>> the<br>
>>>> >>>>> >>> smart solution but I at least solve temporally these problems:<br>
>>>> >>>>> >>><br>
>>>> >>>>> >>> argv problem:<br>
>>>> >>>>> >>><br>
>>>> >>>>> >>> Sentence: 'from yt.mods import *'<br>
>>>> >>>>> >>> Error: for any reason paraview's python shell has not defined<br>
>>>> >>>>> >>> the<br>
>>>> >>>>> >>> variable<br>
>>>> >>>>> >>> 'sys.argv'<br>
>>>> >>>>> >>> Temporal solution: comment yt code where the sys.argv variable<br>
>>>> >>>>> >>> is<br>
>>>> >>>>> >>> used:<br>
>>>> >>>>> >>><br>
>>>> >>>>> >>> yt/funcs.py: lines 439-453<br>
>>>> >>>>> >>> yt/utilities/parallel_tools/parallel_analysis_interface.py:<br>
>>>> >>>>> >>> lines 47<br>
>>>> >>>>> >>> and<br>
>>>> >>>>> >>> 48<br>
>>>> >>>>> >>><br>
>>>> >>>>> >>> pa:<br>
>>>> >>>>> >>><br>
>>>> >>>>> >>> Sentence: "pf = load('tests/DD0010/moving7_0010')"<br>
>>>> >>>>> >>> Error: python shell does not complete the load function and<br>
>>>> >>>>> >>> paraview<br>
>>>> >>>>> >>> keeps<br>
>>>> >>>>> >>> waiting.<br>
>>>> >>>>> >>> Temporal solution: When I debug paraview and interrupt its<br>
>>>> >>>>> >>> execution<br>
>>>> >>>>> >>> the<br>
>>>> >>>>> >>> backtrace shows sem_wait(). It happens inside the load function<br>
>>>> >>>>> >>> when<br>
>>>> >>>>> >>> tries<br>
>>>> >>>>> >>> to identify the candidates for input file (convenience.py line<br>
>>>> >>>>> >>> 74).<br>
>>>> >>>>> >>> There<br>
>>>> >>>>> >>> are 11 possible candidates and gives error for 2 of them:<br>
>>>> >>>>> >>> ChomboStaticOutput<br>
>>>> >>>>> >>> and FlashStaticOutput. Skipping those candidates solve<br>
>>>> >>>>> >>> temporally the<br>
>>>> >>>>> >>> problem:<br>
>>>> >>>>> >><br>
>>>> >>>>> >><br>
>>>> >>>>> >> _______________________________________________<br>
>>>> >>>>> >> Amr mailing list<br>
>>>> >>>>> >> <a href="mailto:Amr@public.kitware.com">Amr@public.kitware.com</a><br>
>>>> >>>>> >> <a href="http://public.kitware.com/cgi-bin/mailman/listinfo/amr" target="_blank">http://public.kitware.com/cgi-bin/mailman/listinfo/amr</a><br>
>>>> >>>>> >><br>
>>>> >>>>> >><br>
>>>> >>>>> > _______________________________________________<br>
>>>> >>>>> > Amr mailing list<br>
>>>> >>>>> > <a href="mailto:Amr@public.kitware.com">Amr@public.kitware.com</a><br>
>>>> >>>>> > <a href="http://public.kitware.com/cgi-bin/mailman/listinfo/amr" target="_blank">http://public.kitware.com/cgi-bin/mailman/listinfo/amr</a><br>
>>>> >>>>> ><br>
>>>> >>>>> _______________________________________________<br>
>>>> >>>>> Amr mailing list<br>
>>>> >>>>> <a href="mailto:Amr@public.kitware.com">Amr@public.kitware.com</a><br>
>>>> >>>>> <a href="http://public.kitware.com/cgi-bin/mailman/listinfo/amr" target="_blank">http://public.kitware.com/cgi-bin/mailman/listinfo/amr</a><br>
>>>> >>>><br>
>>>> >>>><br>
>>>> >>>> _______________________________________________<br>
>>>> >>>> Amr mailing list<br>
>>>> >>>> <a href="mailto:Amr@public.kitware.com">Amr@public.kitware.com</a><br>
>>>> >>>> <a href="http://public.kitware.com/cgi-bin/mailman/listinfo/amr" target="_blank">http://public.kitware.com/cgi-bin/mailman/listinfo/amr</a><br>
>>>> >>>><br>
>>>> >>>><br>
>>>> >>> _______________________________________________<br>
>>>> >>> Amr mailing list<br>
>>>> >>> <a href="mailto:Amr@public.kitware.com">Amr@public.kitware.com</a><br>
>>>> >>> <a href="http://public.kitware.com/cgi-bin/mailman/listinfo/amr" target="_blank">http://public.kitware.com/cgi-bin/mailman/listinfo/amr</a><br>
>>>> >>><br>
>>>> >><br>
>>>> > _______________________________________________<br>
>>>> > Amr mailing list<br>
>>>> > <a href="mailto:Amr@public.kitware.com">Amr@public.kitware.com</a><br>
>>>> > <a href="http://public.kitware.com/cgi-bin/mailman/listinfo/amr" target="_blank">http://public.kitware.com/cgi-bin/mailman/listinfo/amr</a><br>
>>>> ><br>
>>>> _______________________________________________<br>
>>>> Amr mailing list<br>
>>>> <a href="mailto:Amr@public.kitware.com">Amr@public.kitware.com</a><br>
>>>> <a href="http://public.kitware.com/cgi-bin/mailman/listinfo/amr" target="_blank">http://public.kitware.com/cgi-bin/mailman/listinfo/amr</a><br>
>>><br>
>><br>
>><br>
><br>
_______________________________________________<br>
Amr mailing list<br>
<a href="mailto:Amr@public.kitware.com">Amr@public.kitware.com</a><br>
<a href="http://public.kitware.com/cgi-bin/mailman/listinfo/amr" target="_blank">http://public.kitware.com/cgi-bin/mailman/listinfo/amr</a><br>
</div></div></blockquote></div><br></div>