[Paraview] reducing load time?

[Berk Geveci]

http://www.vtk.org/Wiki/VTK_XML_Formats

As I said earlier compression and encoding are not required.

-berk

On Thu, Oct 16, 2008 at 4:37 AM, John Doe <ufnoise at gmail.com> wrote:
> I ended up having to reverse engineer the file format.  Apparently
> compression is required, and I had to reverse engineer the paraview
> code to determine how to write the "header".  Apparently the zlib
> compression is performed on blocks of 32768 bytes, and a header must
> be written to separate them back out on read.
>
>
> Regards,
>
> Juan
>
> On Wed, Oct 15, 2008 at 7:31 PM, John Doe <ufnoise at gmail.com> wrote:
>> I have a vector of N points.  So the number of doubles in 3*N.  A
>> double is 8 long.  Base 64 produces 4/3 the number of original bytes.
>>
>> As expected my base64 string is 3*N*8 * (4/3)
>>
>> So for 1417 points, I should expect a string 45344 characters long.
>>
>> Paraview claims my string is too short:
>> ERROR: In /home/jsanchez/ParaView3/VTK/IO/vtkXMLUnstructuredDataReader.cxx,
>> line 522
>> vtkXMLUnstructuredGridReader (0x4dde7e0): Cannot read points array
>> from Points in piece 0.  The data array in the element may be too
>> short.
>>
>> Here is the header I am writing:
>> <UnstructuredGrid>
>> <Piece NumberOfPoints="1417" NumberOfCells="8781">
>> <Points>
>> <DataArray type="Float64" NumberOfComponents="3" format="binary">
>> this is where my raw base64 encoding string goes
>> </DataArray>
>> </Points>
>>
>> What could be wrong with my string of characters?
>>
>> Thanks,
>>
>> Juan
>>
>>
>> On Wed, Oct 15, 2008 at 6:05 PM, Berk Geveci <berk.geveci at kitware.com> wrote:
>>> By the way, if you are willing to link against the VTK libraries, VTK
>>> has a C/Fortran interface to write XML files.
>>>
>>> -berk
>>>
>>> On Wed, Oct 15, 2008 at 4:48 PM, John Doe <ufnoise at gmail.com> wrote:
>>>> Hello,
>>>>
>>>> Thanks for the suggestion.  I assume that for double precision I am
>>>> doing a base64 encoding for every number in the list?  Is compression
>>>> worthwhile for floating point data?  Are there free compression
>>>> libraries out there?
>>>>
>>>> Juan
>>>>
>>>> On Wed, Oct 15, 2008 at 3:26 PM, Berk Geveci <berk.geveci at kitware.com> wrote:
>>>>> ASCII XML is very slow to load because it uses the XML parser. For
>>>>> large data, you should switch to a binary format.
>>>>>
>>>>> XML binary
>>>>> legacy VTK
>>>>> Exodus
>>>>> Xdmf
>>>>>
>>>>> are a few choices.
>>>>>
>>>>> To see the difference in load time, save the data out as a binary vtu
>>>>> or vtk file and load it back in.
>>>>>
>>>>> -berk
>>>>>
>>>>> On Wed, Oct 15, 2008 at 3:25 PM, John Doe <ufnoise at gmail.com> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I have a 220 MB vtu file that is being loaded and it is taking at
>>>>>> least 15 min to load the data.  Is there any option to reduce the
>>>>>> amount of time paraview would take to load the data.  Or is there some
>>>>>> way I should be formatting my data.  The vtu file is in the ASCII XML
>>>>>> VTK format.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Juan
>>>>>> _______________________________________________
>>>>>> ParaView mailing list
>>>>>> ParaView at paraview.org
>>>>>> http://www.paraview.org/mailman/listinfo/paraview
>>>>>>
>>>>>
>>>>
>>>
>>
>