<div dir="ltr">Okay, I found what I was looking for. Thanks for your help Andy! Your comments and suggestions helped me along in the process.<div><br></div><div>I've pasted the code for anyone else who is interested. I hope it will be useful to have this documented since similar questions have been asked in the past. (<a href="http://vtk.1045678.n5.nabble.com/quot-manually-quot-specify-extents-for-vtkXMLPRectilinearGridWriter-td5714844.html">http://vtk.1045678.n5.nabble.com/quot-manually-quot-specify-extents-for-vtkXMLPRectilinearGridWriter-td5714844.html</a>) The key was to specify the WholeExtent when defining the grid, but then have a filter that gets called during the write process which changes the extent to the local extent for each processor.<div><br></div><div>I still have a lot more to learn about all of this. It would be nice if you guys at kitware would put some more specific examples and documentation for working in a parallel environment. I have been working on this problem off an on for several months, gave up on it for a while and then came back to it when I started this email chain. </div><div><div><br></div><div>The code below is as bare bones as I could get it for specifying local and whole extents. I also got it to work with the vtkExtentTranslator, which might prove useful in the future. However, I have my own partitioning schemes for this current problem.</div><div><br></div><div>--------------------------CODE-------------------------------</div><div><div>'''</div><div>This is intended to write data peices that are unique to each</div><div>processor and have it be combined in the pvts file.</div><div>'''</div><div>import vtk</div><div>import numpy as np</div><div>from vtk.numpy_interface import dataset_adapter as dsa</div><div>from vtk.util import numpy_support</div><div>debug=True</div><div>contr=vtk.vtkMultiProcessController.GetGlobalController()</div><div>if not contr:</div><div> nranks=1</div><div> rank=0</div><div>else:</div><div> nranks=contr.GetNumberOfProcesses()</div><div> rank =contr.GetLocalProcessId()</div><div>if debug:</div><div> print 'Hello from rank {}'.format(rank)</div><div>pnts=vtk.vtkPoints()</div><div>#Global size/extent</div><div>sizeGlobal=np.array([16,33,32],dtype=int)</div><div>extentGlobal=np.array([0,15,0,32,0,31],dtype=int)</div><div>#cylindrical coordinates</div><div>r=np.linspace(0,1,sizeGlobal[0])</div><div>theta=np.linspace(0,2*np.pi,sizeGlobal[1])</div><div>z=np.linspace(-0.5,0.5,sizeGlobal[2])</div><div>#rank specific extents</div><div>if rank==0:</div><div> extentLocal=np.array([0,7,0,32,0,15],dtype=int)</div><div>if rank==1:</div><div> extentLocal=np.array([7,15,0,32,0,15],dtype=int)</div><div>if rank==2:</div><div> extentLocal=np.array([0,7,0,32,15,31],dtype=int)</div><div>if rank==3:</div><div> extentLocal=np.array([7,15,0,32,15,31],dtype=int)</div><div>#define points specific to each rank</div><div>for i in range(extentLocal[4],extentLocal[5]+1):</div><div> for j in range(extentLocal[2],extentLocal[3]+1):</div><div> for k in range(extentLocal[0],extentLocal[1]+1):</div><div> xl=r[k]*np.cos(theta[j])</div><div> yl=r[k]*np.sin(theta[j])</div><div> zl=z[i]</div><div> pnts.InsertNextPoint(xl,yl,zl)</div><div>#dump data</div><div><span style="background-color:rgb(255,255,0)">pf=vtk.vtkProgrammableFilter()</span></div><div><span style="background-color:rgb(255,255,0)">def execute():</span></div><div><span style="background-color:rgb(255,255,0)"> info = pf.GetOutputInformation(0)</span></div><div><span style="background-color:rgb(255,255,0)"> output = pf.GetOutput()</span></div><div><span style="background-color:rgb(255,255,0)"> input = pf.GetInput()</span></div><div><span style="background-color:rgb(255,255,0)"> output.ShallowCopy(input)</span></div><div><span style="background-color:rgb(255,255,0)"> output.SetExtent(extentLocal)</span></div><div><span style="background-color:rgb(255,255,0)">pf.SetExecuteMethod(execute)</span></div><div>sg=vtk.vtkStructuredGrid()</div><div><span style="background-color:rgb(255,255,0)">sg.SetExtent(extentGlobal)</span></div><div>pf.SetInputData(sg)</div><div><div>sg.SetPoints(pnts)</div><div>writer=vtk.vtkXMLPStructuredGridWriter()</div><div><span style="background-color:rgb(255,255,0)">writer.SetInputConnection(pf.GetOutputPort())</span></div><div>writer.SetController(contr)</div><div>writer.SetFileName('testgrid.pvts')</div><div>writer.SetNumberOfPieces(nranks)</div><div>writer.SetStartPiece(rank)</div><div>writer.SetEndPiece(rank)</div><div>writer.Update()</div><div>writer.Write()</div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Dec 28, 2016 at 4:44 PM, Philip Sakievich <span dir="ltr"><<a href="mailto:psakievich@gmail.com" target="_blank">psakievich@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I'm not really looking to do in-situ at this point. I'm probably not making sense. I've put together a test problem in python that illustrates what I'm trying to do (it should be run with <=4 processors). I've attached the code and I'll copy it at the bottom for anyone on the mailing list. <div><br></div><div>Basically, each rank declares a portion of the grid in memory, and then calls the writer. They all have access to the WholeExtent, but I don't know how to get it working in the pipeline. All I need to do to get it working in paraview and visit is manually change the WholeExtent in the *pvts file to "0, 15, 0, 33, 0, 15".</div><div><br></div><div>-----------CODE---------------<wbr>-----</div><div><div>'''</div><div>This is intended to write data peices that are unique to each</div><div>processor and have it be combined in the pvts file.</div><div>'''</div><div>import vtk</div><div>import numpy as np</div><div>from vtk.numpy_interface import dataset_adapter as dsa</div><div>from vtk.util import numpy_support</div><div>debug=True</div><div><br></div><div>contr=vtk.<wbr>vtkMultiProcessController.<wbr>GetGlobalController()</div><div>if not contr:</div><div> nranks=1</div><div> rank=0</div><div>else:</div><div> nranks=contr.<wbr>GetNumberOfProcesses()</div><div> rank =contr.GetLocalProcessId()</div><div><br></div><div>if debug:</div><div> print 'Hello from rank {}'.format(rank)</div><div><br></div><div>pnts=vtk.vtkPoints()</div><div>#Global size/extent</div><div>sizeGlobal=np.array([16,33,16]<wbr>,dtype=int)</div><div>extentGlobal=np.array([0,15,0,<wbr>33,0,15],dtype=int)</div><div>#cylindrical coordinates</div><div>r=np.linspace(0,1,sizeGlobal[<wbr>0])</div><div>theta=np.linspace(0,2*np.pi,<wbr>sizeGlobal[1])</div><div>z=np.linspace(-0.5,0.5,<wbr>sizeGlobal[2])</div><div>#rank specific extents</div><div>if rank==0:</div><div> extentLocal=np.array([0,7,0,<wbr>32,0,7],dtype=int)</div><div>if rank==1:</div><div> extentLocal=np.array([7,15,0,<wbr>32,0,7],dtype=int)</div><div>if rank==2:</div><div> extentLocal=np.array([0,7,0,<wbr>32,7,15],dtype=int)</div><div>if rank==3:</div><div> extentLocal=np.array([7,15,0,<wbr>32,7,15],dtype=int)</div><div>#define points specific to each rank</div><div>for i in range(extentLocal[4],<wbr>extentLocal[5]+1):</div><div> for j in range(extentLocal[2],<wbr>extentLocal[3]+1):</div><div> for k in range(extentLocal[0],<wbr>extentLocal[1]+1):</div><div> xl=r[k]*np.cos(theta[j])</div><div> yl=r[k]*np.sin(theta[j])</div><div> zl=z[i]</div><div> pnts.InsertNextPoint(xl,yl,<wbr>zl)</div><div>#dump data</div><div>sg=vtk.vtkStructuredGrid()</div><div>sg.SetExtent(extentLocal)</div><div>sg.SetPoints(pnts)</div><div>writer=vtk.<wbr>vtkXMLPStructuredGridWriter()</div><div>writer.SetInputData(sg)</div><div>writer.SetController(contr)</div><span class=""><div>writer.SetFileName('testgrid.<wbr>pvts')</div><div>writer.SetNumberOfPieces(<wbr>nranks)</div><div>writer.SetStartPiece(rank)</div><div>writer.SetEndPiece(rank)</div></span><div>'''</div><div>This is what I wish I had. I need some way to set the WholeExtent</div><div>that only affects the *pvts file. Everything else works.</div><div> |</div><div> |</div><div> \|/</div><div> V</div><div>writer.SetWholeExtent(<wbr>extentGlobal)</div><div>'''</div><div>writer.Update()</div><div>writer.Write()</div></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 22, 2016 at 3:35 PM, Andy Bauer <span dir="ltr"><<a href="mailto:andy.bauer@kitware.com" target="_blank">andy.bauer@kitware.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">If you want to use VTK as an in situ library instead of reading data in from disk I strongly recommended looking at ParaView Catalyst (<a href="http://www.paraview.org/in-situ/" target="_blank">http://www.paraview.org/in-si<wbr>tu/</a>). You can use a vtkTrivialProducer to have the data layout you want but if I understand what you're trying to do you'll probably want to use a VTK composite data set with each process having its own data set.<br></div><div class="m_7847961624686500605HOEnZb"><div class="m_7847961624686500605h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 22, 2016 at 12:07 AM, Philip Sakievich <span dir="ltr"><<a href="mailto:psakievich@gmail.com" target="_blank">psakievich@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I'm unclear on how to work with this if I don't specify the local extent. In my code I am subdividing my computational domain so the data is unique on each processor, doing work and then dumping the results. It is completely distributed and the processors don't share any data except on the boundaries. I already know the global and local extents and now I need the local extent in the pvts to line up with the other portions of my code. I don't want the pipeline to give me the local extent unless I can get it to match the extent I've already specified.<div><br></div><div> At this point, it is almost seeming easier to write a script that writes the *pvts for me instead of using the pipeline, but it frustrates me that I can't get the pipeline to do what I want. I'm trying to learn VTK in a broad sense in my spare time. However, for this specific problem the only thing I need is for the PVTS file to have the LocalExtents match my datasets on each of the processors. If I can just solve this all of my problems will be over. I am still wrapping my head around the pipeline concept, but it has been a struggle since there aren't many examples for the problem I'm trying to work with. I'm actually not doing any rendering. I'm just managing IO.<div><br></div><div>I looked at the source you mentioned, but I'm not sure where I am supposed to interact with the RequestInformation request. I'm assuming it will be in the filter that is the input to vtkXMLPStructuredGridWriter, but I haven't really worked with any filters since I've only been doing IO. With the programmable filter <a href="http://www.vtk.org/gitweb?p=VTK.git;a=blob;f=IO/ParallelXML/Testing/Python/testParallelXMLWriters.py" target="_blank">example</a>, I can get it to write local extents that differ from the WholeExtent but they do not match my datasets. (In this case I just copied the execute function).</div><div><div><br></div></div><div>Also, I don't really see why it is necessary to specify the WholeExtent in RequestInformation. I can specify the WholeExtent via vtkStructuredGrid.SetDimension<wbr>s(), or SetExtent and it works fine. It's just that the LocalExtents aren't correct. I think I'm really hitting a roadblock because I don't understand how the parallel writer and the partitioning works with the pipeline.</div><div><br></div><div>Can you provide some specifics on how the partitioning is supposed to work and/or help me modify the code in my previous example to do what I'm looking for? Either that or specify an open source code that has implemented vtkXMLPStructuredGridWriter th<wbr>at I can review. My application is a CFD solver and it's post processing routines.</div><div><div><br></div><div><br></div></div></div></div><div class="m_7847961624686500605m_7441687988187502967HOEnZb"><div class="m_7847961624686500605m_7441687988187502967h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 19, 2016 at 7:18 AM, Andy Bauer <span dir="ltr"><<a href="mailto:andy.bauer@kitware.com" target="_blank">andy.bauer@kitware.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">You need to specify the WholeExtent in the RequestInformation request.You actually don't specify the local extent, the pipeline will give you that for a source. I'd recommend looking at the Imaging/Core/vtkRTAnalyticSour<wbr>ce.cxx class in VTK to see how it's done as a source.<br><br></div><div class="m_7847961624686500605m_7441687988187502967m_3400359959090317814HOEnZb"><div class="m_7847961624686500605m_7441687988187502967m_3400359959090317814h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Dec 18, 2016 at 1:16 AM, Philip Sakievich <span dir="ltr"><<a href="mailto:psakievich@gmail.com" target="_blank">psakievich@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Andy and community,<div><br></div><div>I have read about the concept of whole extent vs extent, but the one thing I don't seem to be able to determine is how to set the whole extent to be different. I tried using vtkStructuredGrid.Crop() but nothing is happening.</div><div><br></div><div>In the code snippet below, I have previously declared points local to each processor. I now want to populate a structured grid and write a *pvts that ties them all together. When I run this code the pvts writes itself and the files, but they all have the same whole extent and extent. Nothing is cropped to the desired local extent. What am I doing wrong? How do I specify a whole extent that is uniform across all processors, and a local extent that is specific to each one? </div><div><br></div><div><div>#create grid and filter for processing to writer</div><div>pf=vtk.vtkProgrammableFilter()</div><div>sg=vtk.vtkStructuredGrid()</div><div><br></div><div>#set extent global</div><div>sg.SetExtent(0,15,0,32,0,15)</div><div><br></div><div>#set extent local</div><div>if rank==0:</div><div> lE=np.array([0,8,0,32,0,8],dt<wbr>ype=int)</div><div> sg.Crop(lE)</div><div>if rank==1:</div><div> lE=np.array([0,8,0,32,7,15],d<wbr>type=int)</div><div> sg.Crop(lE)</div><div>if rank==2:</div><div> lE=np.array([7,15,0,32,0,8],d<wbr>type=int)</div><div> sg.Crop(lE)</div><div>if rank==3:</div><div> lE=np.array([7,15,0,32,7,15],<wbr>dtype=int)</div><div> sg.Crop(lE)</div><div><br></div><div>sg.SetPoints(pnts)</div><div>pf.SetInputData(sg)</div><div><br></div><div>writer=vtk.vtkXMLPStructuredGr<wbr>idWriter()</div><div>writer.SetInputConnection(pf.G<wbr>etOutputPort())</div><div>writer.SetController(contr)</div><div>writer.SetDataModeToAscii()</div><div>writer.SetFileName('testgrid.p<wbr>vts')</div><div>writer.SetNumberOfPieces(nrank<wbr>s)</div><div>writer.SetStartPiece(rank)</div><div>writer.SetEndPiece(rank)</div><div>writer.Write()</div></div><div><br></div><div>Thanks,</div><div><br></div><div>Phil</div></div><div class="m_7847961624686500605m_7441687988187502967m_3400359959090317814m_-8653393100804381657HOEnZb"><div class="m_7847961624686500605m_7441687988187502967m_3400359959090317814m_-8653393100804381657h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Dec 13, 2016 at 7:31 AM, Andy Bauer <span dir="ltr"><<a href="mailto:andy.bauer@kitware.com" target="_blank">andy.bauer@kitware.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>Hi,<br><br></div>I would recommend using the pvts format since that is the best format for storing structured grids. If you read it back in it will know how to properly partition the data set for different amounts of processes as well as do things like ghost cells, extract surfaces, etc.<br><br></div>For topologically structured grids like vtkStructuredGrid there are two types of extents, "whole extent" describes the beginning and ending node (inclusive) in each direction for the entire grid while "extent" refers to each process's (or pieces if you're serial but doing streaming) partition of the grid. I believe this should be explained in the VTK User's Guide which is now available for free as a pdf download. <br><br></div>Cheers,<br></div>Andy<br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_7847961624686500605m_7441687988187502967m_3400359959090317814m_-8653393100804381657m_-6206546010799080445h5">On Tue, Dec 13, 2016 at 10:21 AM, Philip Sakievich <span dir="ltr"><<a href="mailto:psakievich@gmail.com" target="_blank">psakievich@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_7847961624686500605m_7441687988187502967m_3400359959090317814m_-8653393100804381657m_-6206546010799080445h5"><div dir="ltr"><div>Greetings,</div><div><br></div><div>I am reasonably new to vtk and I am mainly using it to manage datasets on structured grids.</div><div><br></div>I am trying to write data for a structured grid in parallel python via MPI. Each process has a separate portion of the grid, and I'm trying to figure out how to set up the write process. I was following this example:<div><br></div><div><a href="http://www.vtk.org/gitweb?p=VTK.git;a=blob;f=IO/ParallelXML/Testing/Python/testParallelXMLWriters.py" target="_blank">http://www.vtk.org/gitweb?p=VT<wbr>K.git;a=blob;f=IO/ParallelXML/<wbr>Testing/Python/testParallelXML<wbr>Writers.py</a></div><div><br></div><div>But then I realized that in this case each process has the entire grid, and each processor is just writing a portion of the data it contains. So do I need to use a multiblock data set? Can someone please provide a simple example of how to write a structured grid in parallel provided each process has the local extent correctly specified? </div><div><br></div><div>Thanks<span class="m_7847961624686500605m_7441687988187502967m_3400359959090317814m_-8653393100804381657m_-6206546010799080445m_3999332338089669960HOEnZb"><font color="#888888"><br clear="all"><div><br></div>-- <br><div class="m_7847961624686500605m_7441687988187502967m_3400359959090317814m_-8653393100804381657m_-6206546010799080445m_3999332338089669960m_3902007003167008255gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Phil Sakievich</div><div><br></div><div><div>PhD Candidate - Mechanical Engineering</div><div>Arizona State University - Ira A. Fulton School for Engineering of Matter Transport and Energy<br>Tempe, Arizona</div></div></div></div></div></div>
</font></span></div></div>
<br></div></div>______________________________<wbr>_________________<br>
Powered by <a href="http://www.kitware.com" rel="noreferrer" target="_blank">www.kitware.com</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" rel="noreferrer" target="_blank">http://www.kitware.com/opensou<wbr>rce/opensource.html</a><br>
<br>
Please keep messages on-topic and check the VTK FAQ at: <a href="http://www.vtk.org/Wiki/VTK_FAQ" rel="noreferrer" target="_blank">http://www.vtk.org/Wiki/VTK_FA<wbr>Q</a><br>
<br>
Search the list archives at: <a href="http://markmail.org/search/?q=vtkusers" rel="noreferrer" target="_blank">http://markmail.org/search/?q=<wbr>vtkusers</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="http://public.kitware.com/mailman/listinfo/vtkusers" rel="noreferrer" target="_blank">http://public.kitware.com/mail<wbr>man/listinfo/vtkusers</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="m_7847961624686500605m_7441687988187502967m_3400359959090317814m_-8653393100804381657m_-6206546010799080445gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Phil Sakievich</div><div><br></div><div><div>PhD Candidate - Mechanical Engineering</div><div>Arizona State University - Ira A. Fulton School for Engineering of Matter Transport and Energy<br>Tempe, Arizona</div></div></div></div></div></div>
</div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="m_7847961624686500605m_7441687988187502967m_3400359959090317814gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Phil Sakievich</div><div><br></div><div><div>PhD Candidate - Mechanical Engineering</div><div>Arizona State University - Ira A. Fulton School for Engineering of Matter Transport and Energy<br>Tempe, Arizona</div></div></div></div></div></div>
</div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="m_7847961624686500605gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Phil Sakievich</div><div><br></div><div><div>PhD Candidate - Mechanical Engineering</div><div>Arizona State University - Ira A. Fulton School for Engineering of Matter Transport and Energy<br>Tempe, Arizona</div></div></div></div></div></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Phil Sakievich</div><div><br></div><div><div>PhD Candidate - Mechanical Engineering</div><div>Arizona State University - Ira A. Fulton School for Engineering of Matter Transport and Energy<br>Tempe, Arizona</div></div></div></div></div></div>
</div>