[Paraview] reducing load time?

[John Doe]

I ended up having to reverse engineer the file format.  Apparently
compression is required, and I had to reverse engineer the paraview
code to determine how to write the "header".  Apparently the zlib
compression is performed on blocks of 32768 bytes, and a header must
be written to separate them back out on read.


Regards,

Juan

On Wed, Oct 15, 2008 at 7:31 PM, John Doe <ufnoise at gmail.com> wrote:
> I have a vector of N points.  So the number of doubles in 3*N.  A
> double is 8 long.  Base 64 produces 4/3 the number of original bytes.
>
> As expected my base64 string is 3*N*8 * (4/3)
>
> So for 1417 points, I should expect a string 45344 characters long.
>
> Paraview claims my string is too short:
> ERROR: In /home/jsanchez/ParaView3/VTK/IO/vtkXMLUnstructuredDataReader.cxx,
> line 522
> vtkXMLUnstructuredGridReader (0x4dde7e0): Cannot read points array
> from Points in piece 0.  The data array in the element may be too
> short.
>
> Here is the header I am writing:
> <UnstructuredGrid>
> <Piece NumberOfPoints="1417" NumberOfCells="8781">
> <Points>
> <DataArray type="Float64" NumberOfComponents="3" format="binary">
> this is where my raw base64 encoding string goes
> </DataArray>
> </Points>
>
> What could be wrong with my string of characters?
>
> Thanks,
>
> Juan
>
>
> On Wed, Oct 15, 2008 at 6:05 PM, Berk Geveci <berk.geveci at kitware.com> wrote:
>> By the way, if you are willing to link against the VTK libraries, VTK
>> has a C/Fortran interface to write XML files.
>>
>> -berk
>>
>> On Wed, Oct 15, 2008 at 4:48 PM, John Doe <ufnoise at gmail.com> wrote:
>>> Hello,
>>>
>>> Thanks for the suggestion.  I assume that for double precision I am
>>> doing a base64 encoding for every number in the list?  Is compression
>>> worthwhile for floating point data?  Are there free compression
>>> libraries out there?
>>>
>>> Juan
>>>
>>> On Wed, Oct 15, 2008 at 3:26 PM, Berk Geveci <berk.geveci at kitware.com> wrote:
>>>> ASCII XML is very slow to load because it uses the XML parser. For
>>>> large data, you should switch to a binary format.
>>>>
>>>> XML binary
>>>> legacy VTK
>>>> Exodus
>>>> Xdmf
>>>>
>>>> are a few choices.
>>>>
>>>> To see the difference in load time, save the data out as a binary vtu
>>>> or vtk file and load it back in.
>>>>
>>>> -berk
>>>>
>>>> On Wed, Oct 15, 2008 at 3:25 PM, John Doe <ufnoise at gmail.com> wrote:
>>>>> Hi,
>>>>>
>>>>> I have a 220 MB vtu file that is being loaded and it is taking at
>>>>> least 15 min to load the data.  Is there any option to reduce the
>>>>> amount of time paraview would take to load the data.  Or is there some
>>>>> way I should be formatting my data.  The vtu file is in the ASCII XML
>>>>> VTK format.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Juan
>>>>> _______________________________________________
>>>>> ParaView mailing list
>>>>> ParaView at paraview.org
>>>>> http://www.paraview.org/mailman/listinfo/paraview
>>>>>
>>>>
>>>
>>
>