[Paraview] Creating Large Datasets from CSV

[Cory Quammen]

David,

I'm currently working on a translator that sounds very similar to
yours. It uses HDF5 for the heavy data and that part works fine.

For what it's worth, my experience is that the Xdmf library is
painfully slow when serializing a large tree. For some of the data
sets that I work with, writing the XML description of the data takes
far longer than writing the HDF5 files.

Cory

On Wed, Aug 15, 2012 at 2:29 PM, David Zemon <david.zemon at mst.edu> wrote:
> Cory,
>
> Unfortunately no, it isn't. Reading CSV is just a small stepping-stone in
> the overall goal of this project. I'm trying to make a reader that will
> convert any text-delimited file of any size (we have professors on campus
> with Terabytes of data - necessitating that it run separately from
> ParaView). I also plan to give the user options like creating a difference
> field between a column in one file and another column in a different file.
>
> David
>
>
> On 08/15/2012 01:19 PM, Cory Quammen wrote:
>>
>> David,
>>
>> Just curious, is ParaView's CSV reader not sufficient for reading your
>> files?
>>
>> Cory
>>
>> On Wed, Aug 15, 2012 at 1:58 PM, David Zemon <david.zemon at mst.edu> wrote:
>>>
>>> Hello,
>>>
>>> I'm creating a reader to convert large dataset from CSV to a ParaView
>>> readable format. XDMF was chosen because it seems like a simple-to-use
>>> and
>>> understand format. This worked well while I was testing small datasets
>>> but
>>> when I scaled up to larger data, I ran across a problem where the XML
>>> node
>>> was too large (could not have 350,000 rows).
>>>
>>> I want to make sure now that I am on the right track. I've decided to
>>> start
>>> researching the HDF5 format and will place all of my data into an HDF5
>>> file
>>> and then include that in the XDMF file. Does this seem reasonable? Is
>>> there
>>> a better way to do it?
>>>
>>> Thank you,
>>> David Zemon
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Please keep messages on-topic and check the ParaView Wiki at:
>>> http://paraview.org/Wiki/ParaView
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://www.paraview.org/mailman/listinfo/paraview
>>
>>
>>
>



-- 
Cory Quammen
Research Associate
Department of Computer Science
The University of North Carolina at Chapel Hill