<div dir="ltr"><div><div>Hi Mark,<br><br></div>You're very brave to try and play around with vtkCellArray and how it's used inside of VTK/PV :)<br><br></div>One thing to keep in mind is that both the reader and the Catalyst adaptor are expected to provide a separate data sets for each time step in a time varying data set. If you're trying to modify the data you could get into trouble with things like particle paths that needs the data set at more than one time step at a time (e.g. particle paths uses the current and the previous time step to update the particle locations and for in situ it caches the pointer to the previous time step). Other filters like temporal statistics will be fine since that is done in a single pass through the data and with only computing results from a single time step at a time. Since your data set topology will often change over time you would want a separate vtkDataSet for each "static" topology data set that you could then shallow copy. <div><br></div><div>You may want to look at the CxxSOADataArray example as this is a computationally efficient improvement on the vtkMappedArray stuff (<a href="https://gitlab.kitware.com/paraview/paraview/tree/master/Examples/Catalyst/CxxSOADataArrayExample">https://gitlab.kitware.com/paraview/paraview/tree/master/Examples/Catalyst/CxxSOADataArrayExample</a>).<br></div><div><br></div><div>I inlined some more responses below trying to address some specific questions. <br><div class="gmail_extra"><br></div><div class="gmail_extra">Cheers,</div><div class="gmail_extra">Andy</div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, May 12, 2017 at 1:07 PM, Mark Olesen <span dir="ltr"><<a href="mailto:Mark.Olesen@esi-group.com" target="_blank">Mark.Olesen@esi-group.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I am reworking a reader module to see how I can improve memory usage and performance by using internal caching and I would like to see if my concept has major flaws or other things to worry about. The final target will be in-situ usage, but I'll practicing a bit with the paraview reader module.<br>
<br>
I have concluded that since the basic simulation data do not lend themselves to normal zero-copy techniques, and I'm not completely convinced that the vtkMappedUnstructuredGrid approach will work either (I need to deal with possible on-the-fly polyhedral decomposition too), my approach like an inside-out version of vtkMappedUnstructuredGrid.<br>
<br>
1) Query my simulation to obtain all the sizing data.<br>
2) Allocate a vtkUnstructuredGrid with the appropriate sizing and pass the underlying VTK data contents back for filling in. This easiest to done by the simulation, since it knows its own data structures and minimizes the ABI connection. Only the size of vtkIdType is needed by the simulation itself and the interface code is templated on variants of that.<br>
3) Finally update the vtkUnstructuredGrid<br>
<br>
Eg,<br>
... query simulation for sizing<br>
// get pointers for/from unstructured grid:<br>
vtkSmartPointer<vtkCellArray> cells = vtkmesh->GetCells();<br>
cells->GetData()-><wbr>SetNumberOfTuples(<wbr>nConnectivity);<br>
.... other arrays and sizing<br>
<br>
// wrap the WritePointer with a pass-through list-container (no allocation)<br>
UList<vtkIdType> ul_cells<br>
(<br>
cells->WritePointer(sizing.<wbr>nFieldCells(), sizing.nConnectivity()),<br>
sizing.nConnectivity()<br>
);<br>
<br>
// fill with contents in a form that VTK would expect<br>
sizing.populateInternal(<wbr>myMesh, ul_cellTypes,ul_cells, ul_cellLocations,ul_faces, ul_faceLocations, myMapping);<br>
<br>
// update VTK side of things:<br>
vtkmesh->SetCells(cellTypes, cellLocations, cells, faceLocations, faces);<br>
<br>
This seems to work OK and is more-or-less an en bloc alternative to vtkMappedUnstructuredGrid.<br>
<br>
The next stage is where it gets interesting, or at least where I get quite confused. Assuming that the mesh doesn't change very often during a simulation, I would like to re-use the VTK grid entirely. For this, I'm using hashtable with a (std::string, vtkSmartPointer<<wbr>vtkUnstructuredGrid>) key/value pair that handles deletion nicely.<br>
<br>
On the first time through, I create the vtk-mesh and store as a SmartPointer in the hash before attaching it to the ParaView output:<br>
vtkMultiBlockDataSet* output = vtkMultiBlockDataSet::<wbr>SafeDownCast(outputVector-><wbr>GetInformationObject(0)->Get(<wbr>vtkMultiBlockDataSet::DATA_<wbr>OBJECT());<br>
...<br>
block->SetBlock(datasetNo, dataset)<br>
<br>
I assume that this simply increases the RefCount, or does paraview make a deep-copy of the data returned on the output vector?<br>
On further calls, I can simply do the same type of thing (adding to the output object), but with the grid retrieved from the hashtable cache.<br></blockquote><div><br></div><div>Yes, for this it would just increase the reference count of dataset and NOT do a deep-copy of the data.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
So what happens when I start modify the cached values (presuming that paraview has a shallow copy of the data)?<br>
For example, there are topology changes that I would like to handle. If I access my pointer from the cache, make the modifications and then add it back to the output vector, what state is the data in? Am I modifying (corrupting) data that is currently in use by paraview? Or does the paraview hold off until I complete the action? During this modification time, do I have two copied of the data, or are they always shallow copies referencing the same data?<br></blockquote><div><br></div><div>Shallow-copy will allow some of the data to be different. For example, shallow-copying a vtkUnstructuredGrid would initially have them looking identical. The source and target unstructured grids would be sharing references to the same arrays but some of the data members (e.g. PointData, CellData, FieldData) would be separate. Thus, after the shallow copy you could add in another point data array to the target unstructured grid and this would not affect the source unstructured grid. Similarly, you could replace an array in the target grid and not affect the source grid but if you went and got a specific array from the target grid and changed some values in there that change would also be reflected in the source grid.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Any good pointers to understanding this would be very greatly appreciated! I've scoured the wiki and other locations, but without much success.<br>
<br>
Cheers,<br>
/mark<br>
______________________________<wbr>_________________<br>
Powered by <a href="http://www.kitware.com" rel="noreferrer" target="_blank">www.kitware.com</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" rel="noreferrer" target="_blank">http://www.kitware.com/<wbr>opensource/opensource.html</a><br>
<br>
Search the list archives at: <a href="http://markmail.org/search/?q=Paraview-developers" rel="noreferrer" target="_blank">http://markmail.org/search/?q=<wbr>Paraview-developers</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="http://public.kitware.com/mailman/listinfo/paraview-developers" rel="noreferrer" target="_blank">http://public.kitware.com/<wbr>mailman/listinfo/paraview-<wbr>developers</a><br>
</blockquote></div><br></div></div></div>