Hello all,<br><br>Thanks for all the good comments.<br><br>Bill: because when shallow copying the structure, then the vector holding the pointers would be duplicated. This could be a problem with the data size that I am aiming for.<br>
<br>For context: we want to represent O(10^10) or more AMR meshes. Typically, one "root" can represent in an implicit (tree) structure O(10^4) or more AMR cells. At the root level, we thus need to be able to hold O(10^6) pointers to these trees. If we store these in a STD vector, a shallow copy of the entire data set will mean a deep copy of the vector holding the pointers (and yet a shallow copy of the pointers themselves). The cost is prohibitive. So, we need an efficient structure for shared ownership of the collection of pointers.<br>
<br>I think I am going to try again the vtkCollection route, but with the following acceleration:<br>> I think you are going to want to allocate a slab of memory the exact nx * ny<br>
> * nz * sizeof pointer. Than you can encode just the number of pointers and<br>
> initial memory position in each instance of the collection.<br>A question though: will that work if I use vtkSmartPointers instead of raw pointers? Can I predict the size of a SmartPointer? I have not done this before.<br>
<br>Thanks<br>Philippe<br><br><br><div class="gmail_quote">On Wed, Nov 21, 2012 at 3:37 AM, Bill Lorensen <span dir="ltr"><<a href="mailto:bill.lorensen@gmail.com" target="_blank">bill.lorensen@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Why not an std::vector of vtk smart pointers?<br>
<div><div class="h5"><br>
On Tue, Nov 20, 2012 at 6:27 PM, Robert Maynard<br>
<<a href="mailto:robert.maynard@kitware.com">robert.maynard@kitware.com</a>> wrote:<br>
> I think you are going to want to allocate a slab of memory the exact nx * ny<br>
> * nz * sizeof pointer. Than you can encode just the number of pointers and<br>
> initial memory position in each instance of the collection.<br>
><br>
> Lookup of exact elements is constant time, and iteration over the entire set<br>
> can be done very efficiently as you will have cache coherency on the<br>
> pointers(slab allocating the actual objects is also recommended).<br>
><br>
> Sent from my iPhone<br>
><br>
> On Nov 20, 2012, at 5:45 PM, Philippe Pébay <<a href="mailto:philippe.pebay@kitware.com">philippe.pebay@kitware.com</a>><br>
> wrote:<br>
><br>
> Hello Dave<br>
><br>
> Thanks for the comments. So, this might be the better option: rather than<br>
> implementing a new concrete array type, using a possibly improved version of<br>
> the vtkCollection.<br>
><br>
> A little more context: no random access is necessary. All that is needed is<br>
> the ability to iterate over the elements rapidly and in a scalable fashion.<br>
> The collection size is set once for all at the time of the creation of the<br>
> hyper tree grid object. The collection is used to collect a serialized list<br>
> of the nx * ny * nz grid of individual trees. It is, also, necessary that<br>
> shallow copies be done efficiently, that is, by not only shallow copying all<br>
> individual trees, but also the collection itself, for we do not want to have<br>
> to replicate O(10^6) pointers.<br>
><br>
> Thanks<br>
> Philippe<br>
><br>
> On Tue, Nov 20, 2012 at 11:34 PM, David Cole <<a href="mailto:david.cole@kitware.com">david.cole@kitware.com</a>> wrote:<br>
>><br>
>> Its implementation is just a linked list. It will be as fast as you<br>
>> can possibly be for:<br>
>><br>
>> - iterating the whole collection and doing something on each element<br>
>> - adding a new element to the end of the list<br>
>><br>
>> For all else, including finding or removing specific items (by index<br>
>> or identity), it will be as ridiculously slow as you'd expect a linked<br>
>> list to be...<br>
>><br>
>> After just a glance at the implementation, I'm not surprised it doesn't<br>
>> "scale".<br>
>><br>
>> If you need scalability, perhaps a better implementation could be<br>
>> dropped in to the same API.<br>
>><br>
>><br>
>> On Tue, Nov 20, 2012 at 5:25 PM, Philippe Pébay<br>
>> <<a href="mailto:philippe.pebay@kitware.com">philippe.pebay@kitware.com</a>> wrote:<br>
>> > Hello David<br>
>> ><br>
>> > Thanks for the suggestion. I looked into vtkCollection but the code was<br>
>> > running slow with large collections. But it is possible that I will not<br>
>> > get<br>
>> > any better results with the vtkObjectArray. Are you surprised that the<br>
>> > vtkCollection did not scale well (my observation on only one platform)?<br>
>> ><br>
>> > Thanks<br>
>> > Philippe<br>
>> ><br>
>> ><br>
>> ><br>
>> > On Tue, Nov 20, 2012 at 6:41 PM, David Thompson<br>
>> > <<a href="mailto:david.thompson@kitware.com">david.thompson@kitware.com</a>><br>
>> > wrote:<br>
>> >><br>
>> >> Hi Philippe,<br>
>> >><br>
>> >> I think the vtkObjectArray class sounds like a good idea, however be<br>
>> >> aware<br>
>> >> that vtkCollection already exists to fulfill a similar purpose.<br>
>> >><br>
>> >> David<br>
>> >><br>
>> >> On Nov 20, 2012, at 12:28 PM, Philippe Pébay<br>
>> >> <<a href="mailto:philippe.pebay@kitware.com">philippe.pebay@kitware.com</a>><br>
>> >> wrote:<br>
>> >><br>
>> >> > Hello David<br>
>> >> ><br>
>> >> > Yes, I think this is the best option. I am thinking that I could<br>
>> >> > create<br>
>> >> > this as a new concrete subclass of vtkAbstractArray available to all,<br>
>> >> > and<br>
>> >> > not only as a helper class internal to my hyper tree implementation.<br>
>> >> > What do<br>
>> >> > you think? Would this be useful?<br>
>> >> ><br>
>> >> > Thanks<br>
>> >> > Philippe<br>
>> >> ><br>
>> >> > On Tue, Nov 20, 2012 at 6:25 PM, David Gobbi <<a href="mailto:david.gobbi@gmail.com">david.gobbi@gmail.com</a>><br>
>> >> > wrote:<br>
>> >> > Hi Phillippe,<br>
>> >> ><br>
>> >> > Actually I was wrong about vtkVariant not having any size overhead.<br>
>> >> > It<br>
>> >> > needs<br>
>> >> > to store the type along with the value. So vtkVariantArray is no<br>
>> >> > good<br>
>> >> > for this.<br>
>> >> ><br>
>> >> > A helper class might be the best option. You could make a<br>
>> >> > vtkObjectArray<br>
>> >> > class which would be just like vtkVoidArray except that it would<br>
>> >> > objects.<br>
>> >> ><br>
>> >> > - David<br>
>> >> ><br>
>> >> ><br>
>> >> > On Tue, Nov 20, 2012 at 9:54 AM, Philippe Pébay<br>
>> >> > <<a href="mailto:philippe.pebay@kitware.com">philippe.pebay@kitware.com</a>> wrote:<br>
>> >> > > Hello David<br>
>> >> > ><br>
>> >> > > Thanks for the summary about the absence of performance hit when<br>
>> >> > > using<br>
>> >> > > SPs<br>
>> >> > > as opposed to RPs. I will remember it and use them more often as a<br>
>> >> > > result.<br>
>> >> > ><br>
>> >> > > Regarding the variant array, yes, the conversion to vtkObject is a<br>
>> >> > > concern.<br>
>> >> > > I think that the most efficient option would be to create a small<br>
>> >> > > helper<br>
>> >> > > class that would internally store a raw array of pointers. That<br>
>> >> > > helper<br>
>> >> > > would<br>
>> >> > > derive from vtkObjectBase. That should avoid any perfomance hit.<br>
>> >> > ><br>
>> >> > > Philippe<br>
>> >> > ><br>
>> >> > ><br>
>> >> > > On Tue, Nov 20, 2012 at 5:49 PM, David Gobbi<br>
>> >> > > <<a href="mailto:david.gobbi@gmail.com">david.gobbi@gmail.com</a>><br>
>> >> > > wrote:<br>
>> >> > >><br>
>> >> > >> Hi Phillippe,<br>
>> >> > >><br>
>> >> > >> A vtkVariant is 64 bits, it should be no larger than a pointer.<br>
>> >> > >> There<br>
>> >> > >> is a function call overhead in ToVTKObject(), which might be a<br>
>> >> > >> concern to<br>
>> >> > >> you.<br>
>> >> > >><br>
>> >> > >> The vtkSmartPointer is exactly the same size as a pointer, so<br>
>> >> > >> there<br>
>> >> > >> is no<br>
>> >> > >> size<br>
>> >> > >> overhead. It has no computational overhead, either, except for<br>
>> >> > >> the<br>
>> >> > >> "operator=",<br>
>> >> > >> the copy constructor, and the destructor (I wouldn't even count<br>
>> >> > >> the<br>
>> >> > >> overhead in<br>
>> >> > >> the destructor, because you have to call ->Delete() eventually<br>
>> >> > >> anyway).<br>
>> >> > >> So do<br>
>> >> > >> not avoid smart pointers because you are worried about efficiency.<br>
>> >> > >> The<br>
>> >> > >> VTK<br>
>> >> > >> smart pointers are very efficient because vtkObjectBase already<br>
>> >> > >> has a<br>
>> >> > >> built-in<br>
>> >> > >> reference count.<br>
>> >> > >><br>
>> >> > >> - David<br>
>> >> > >><br>
>> >> > >><br>
>> >> > >> On Tue, Nov 20, 2012 at 9:35 AM, Philippe Pébay<br>
>> >> > >> <<a href="mailto:philippe.pebay@kitware.com">philippe.pebay@kitware.com</a>> wrote:<br>
>> >> > >> > Hello David<br>
>> >> > >> ><br>
>> >> > >> > Thanks for the suggestion: I am concerned by the overhead that<br>
>> >> > >> > comes<br>
>> >> > >> > with<br>
>> >> > >> > variants. Wouldn't it be more lightweight to use a vtkVoidArray<br>
>> >> > >> > instead?<br>
>> >> > >> > The<br>
>> >> > >> > array can potentially contain O(10⁶) pointers. By the way this<br>
>> >> > >> > is<br>
>> >> > >> > for<br>
>> >> > >> > the<br>
>> >> > >> > same reason that I have, so far, refrained from using<br>
>> >> > >> > vtkSmartPointers<br>
>> >> > >> > as<br>
>> >> > >> > opposed to raw pointers, by fear of the overhead involved. Is<br>
>> >> > >> > this<br>
>> >> > >> > extra<br>
>> >> > >> > caution excessive?<br>
>> >> > >> ><br>
>> >> > >> > Thanks<br>
>> >> > >> > Philippe<br>
>> >> > >> ><br>
>> >> > >> > On Tue, Nov 20, 2012 at 5:31 PM, David Gobbi<br>
>> >> > >> > <<a href="mailto:david.gobbi@gmail.com">david.gobbi@gmail.com</a>><br>
>> >> > >> > wrote:<br>
>> >> > >> >><br>
>> >> > >> >> Hi Philippe,<br>
>> >> > >> >><br>
>> >> > >> >> If the pointers are pointers to VTK objects, then you can use<br>
>> >> > >> >> vtkVariantArray.<br>
>> >> > >> >><br>
>> >> > >> >> - David<br>
>> >> > >> >><br>
>> >> > >> >><br>
>> >> > >> >> On Tue, Nov 20, 2012 at 9:29 AM, Philippe Pébay<br>
>> >> > >> >> <<a href="mailto:philippe.pebay@kitware.com">philippe.pebay@kitware.com</a>> wrote:<br>
>> >> > >> >> > Just an idea: couldn't a vtkVoidArray be used (or abused) as<br>
>> >> > >> >> > container<br>
>> >> > >> >> > of<br>
>> >> > >> >> > raw pointers, that would be appropriately downcasted<br>
>> >> > >> >> > internally<br>
>> >> > >> >> > to<br>
>> >> > >> >> > the<br>
>> >> > >> >> > concrete type to which they are supposed to point? For<br>
>> >> > >> >> > instance,<br>
>> >> > >> >> > instead<br>
>> >> > >> >> > of<br>
>> >> > >> >> > having a<br>
>> >> > >> >> > vtkMyInternalObject** obj<br>
>> >> > >> >> > instance variable, couldn't I use a<br>
>> >> > >> >> > vtkVoidArray* obj<br>
>> >> > >> >> > instead, where each entry in obj would be the raw pointer to<br>
>> >> > >> >> > a<br>
>> >> > >> >> > vtkSmartPointer<vtkMyInternalObject>?<br>
>> >> > >> >> ><br>
>> >> > >> >> > Thanks<br>
>> >> > >> >> > Philippe<br>
>> >> > >> >> ><br>
>> >> > >> >> ><br>
>> >> > >> >> > On Tue, Nov 20, 2012 at 5:14 PM, Philippe Pébay<br>
>> >> > >> >> > <<a href="mailto:philippe.pebay@kitware.com">philippe.pebay@kitware.com</a>><br>
>> >> > >> >> > wrote:<br>
>> >> > >> >> >><br>
>> >> > >> >> >> Hello Kyle<br>
>> >> > >> >> >><br>
>> >> > >> >> >> Yes, I would want to avoid a dependency on Boost. But this<br>
>> >> > >> >> >> is<br>
>> >> > >> >> >> basically<br>
>> >> > >> >> >> what I am tried to reproduce (the Boost shared array of<br>
>> >> > >> >> >> pointers).<br>
>> >> > >> >> >><br>
>> >> > >> >> >> Thanks<br>
>> >> > >> >> >> P<br>
>> >> > >> >> >><br>
>> >> > >> >> >><br>
>> >> > >> >> >> On Tue, Nov 20, 2012 at 5:13 PM, Kyle Lutz<br>
>> >> > >> >> >> <<a href="mailto:kyle.lutz@kitware.com">kyle.lutz@kitware.com</a>><br>
>> >> > >> >> >> wrote:<br>
>> >> > >> >> >>><br>
>> >> > >> >> >>> Would using boost suffice? Or do you want a solution using<br>
>> >> > >> >> >>> just<br>
>> >> > >> >> >>> VTK?<br>
>> >> > >> >> >>><br>
>> >> > >> >> >>> -kyle<br>
>> >> > >> >> >>><br>
>> >> > >> >> >>> On Tue, Nov 20, 2012 at 7:58 AM, Philippe Pébay<br>
>> >> > >> >> >>> <<a href="mailto:philippe.pebay@kitware.com">philippe.pebay@kitware.com</a>> wrote:<br>
>> >> > >> >> >>> > Hello all,<br>
>> >> > >> >> >>> ><br>
>> >> > >> >> >>> > I need to keep track of shared ownership of an array of<br>
>> >> > >> >> >>> > pointers.<br>
>> >> > >> >> >>> > Can<br>
>> >> > >> >> >>> > someone point me towards an example of code that already<br>
>> >> > >> >> >>> > does<br>
>> >> > >> >> >>> > this?<br>
>> >> > >> >> >>> > This<br>
>> >> > >> >> >>> > should amount to a small helper class that contains the<br>
>> >> > >> >> >>> > array of<br>
>> >> > >> >> >>> > pointers<br>
>> >> > >> >> >>> > along with a reference count.<br>
>> >> > >> >> >>> ><br>
>> >> > >> >> >>> > Thank you!<br>
>> >> > >> >> >>> > Philippe<br>
>> >> > >> >> >>> ><br>
>> >> > >> >> >>> ><br>
>> >> > >> >> >>> > --<br>
>> >> > >> >> >>> > Philippe Pébay, PhD<br>
>> >> > >> >> >>> > Director of Visualization and High Performance Computing<br>
>> >> > >> >> >>> > /<br>
>> >> > >> >> >>> > Directeur de la Visualisation et du Calcul Haute<br>
>> >> > >> >> >>> > Performance<br>
>> >> > >> >> >>> > Kitware SAS<br>
>> >> > >> >> >>> > 26 rue Louis Guérin, 69100 Villeurbanne, France<br>
>> >> > >> >> >>> > <a href="tel:%2B33%20%280%29%206.83.61.55.70" value="+33683615570">+33 (0) 6.83.61.55.70</a> / 4.37.45.04.15<br>
>> >> > >> >> >>> > <a href="http://www.kitware.fr" target="_blank">http://www.kitware.fr</a><br>
>> >> > ><br>
>> >> > ><br>
>> >> > ><br>
>> >> > ><br>
>> >> > > --<br>
>> >> > > Philippe Pébay, PhD<br>
>> >> > > Director of Visualization and High Performance Computing /<br>
>> >> > > Directeur de la Visualisation et du Calcul Haute Performance<br>
>> >> > > Kitware SAS<br>
>> >> > > 26 rue Louis Guérin, 69100 Villeurbanne, France<br>
>> >> > > <a href="tel:%2B33%20%280%29%206.83.61.55.70" value="+33683615570">+33 (0) 6.83.61.55.70</a> / 4.37.45.04.15<br>
>> >> > > <a href="http://www.kitware.fr" target="_blank">http://www.kitware.fr</a><br>
>> >> ><br>
>> >> ><br>
>> >> ><br>
>> >> > --<br>
>> >> > Philippe Pébay, PhD<br>
>> >> > Director of Visualization and High Performance Computing /<br>
>> >> > Directeur de la Visualisation et du Calcul Haute Performance<br>
>> >> > Kitware SAS<br>
>> >> > 26 rue Louis Guérin, 69100 Villeurbanne, France<br>
>> >> > <a href="tel:%2B33%20%280%29%206.83.61.55.70" value="+33683615570">+33 (0) 6.83.61.55.70</a> / 4.37.45.04.15<br>
>> >> > <a href="http://www.kitware.fr" target="_blank">http://www.kitware.fr</a><br>
>> >> > _______________________________________________<br>
>> >> > Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
>> >> ><br>
>> >> > Visit other Kitware open-source projects at<br>
>> >> > <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
>> >> ><br>
>> >> > Follow this link to subscribe/unsubscribe:<br>
>> >> > <a href="http://www.vtk.org/mailman/listinfo/vtk-developers" target="_blank">http://www.vtk.org/mailman/listinfo/vtk-developers</a><br>
>> >> ><br>
>> >><br>
>> ><br>
>> ><br>
>> ><br>
>> > --<br>
>> > Philippe Pébay, PhD<br>
>> > Director of Visualization and High Performance Computing /<br>
>> > Directeur de la Visualisation et du Calcul Haute Performance<br>
>> > Kitware SAS<br>
>> > 26 rue Louis Guérin, 69100 Villeurbanne, France<br>
>> > <a href="tel:%2B33%20%280%29%206.83.61.55.70" value="+33683615570">+33 (0) 6.83.61.55.70</a> / 4.37.45.04.15<br>
>> > <a href="http://www.kitware.fr" target="_blank">http://www.kitware.fr</a><br>
>> ><br>
>> > _______________________________________________<br>
>> > Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
>> ><br>
>> > Visit other Kitware open-source projects at<br>
>> > <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
>> ><br>
>> > Follow this link to subscribe/unsubscribe:<br>
>> > <a href="http://www.vtk.org/mailman/listinfo/vtk-developers" target="_blank">http://www.vtk.org/mailman/listinfo/vtk-developers</a><br>
>> ><br>
>> ><br>
><br>
><br>
><br>
><br>
> --<br>
> Philippe Pébay, PhD<br>
> Director of Visualization and High Performance Computing /<br>
> Directeur de la Visualisation et du Calcul Haute Performance<br>
> Kitware SAS<br>
> 26 rue Louis Guérin, 69100 Villeurbanne, France<br>
> <a href="tel:%2B33%20%280%29%206.83.61.55.70" value="+33683615570">+33 (0) 6.83.61.55.70</a> / 4.37.45.04.15<br>
> <a href="http://www.kitware.fr" target="_blank">http://www.kitware.fr</a><br>
><br>
> _______________________________________________<br>
> Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
><br>
> Visit other Kitware open-source projects at<br>
> <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
><br>
> Follow this link to subscribe/unsubscribe:<br>
> <a href="http://www.vtk.org/mailman/listinfo/vtk-developers" target="_blank">http://www.vtk.org/mailman/listinfo/vtk-developers</a><br>
><br>
><br>
> _______________________________________________<br>
> Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
><br>
> Visit other Kitware open-source projects at<br>
> <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
><br>
> Follow this link to subscribe/unsubscribe:<br>
> <a href="http://www.vtk.org/mailman/listinfo/vtk-developers" target="_blank">http://www.vtk.org/mailman/listinfo/vtk-developers</a><br>
><br>
><br>
<br>
<br>
<br>
--<br>
</div></div>Unpaid intern in BillsBasement at noware dot com<br>
</blockquote></div><br><br clear="all"><br>-- <br><font color="#888888">Philippe Pébay, PhD<br></font><font color="#888888">Director of Visualization and High Performance Computing /<br>
</font><font color="#888888">Directeur de la Visualisation et du Calcul Haute Performance<br>
Kitware SAS<br>26 rue Louis Guérin, 69100 Villeurbanne, France</font><br>
<font color="#888888"><a value="+33426685003">+33 (0) 6.83.61.55.70 / 4.37.45.04.15</a></font><font color="#888888"><br><a href="http://www.kitware.fr/" target="_blank">http://www.kitware.fr</a></font><font color="#888888"><a href="http://www.kitware.fr/" target="_blank"></a></font><br>