[Paraview] reducing load time?

[John Doe]

Hello,

It appears to me that base64 encoding is necessary if you want the
file to be easily looked at in a texteditor, and to remove data easily
from the file.

Base64 encoding seems necessary to have the data in a normal section
and in not some huge block at the end of the file.

According, to:
http://www.geophysik.uni-muenchen.de/intranet/it-service/applications/paraview/vtk-file-formats/

an encoding appears to be required in order to have a valid xml file.

Rewriting my VTK file writer to use offsets into a raw array would
have taken much longer than trying to understand how to use the Base64
encoding and Zlib compression.

Please stipulate in your Wiki that the default block size for the
paraview writer is 32768 bytes.

Regards,

Juan

On Thu, Oct 16, 2008 at 6:43 AM, Berk Geveci <berk.geveci at kitware.com> wrote:
> http://www.vtk.org/Wiki/VTK_XML_Formats
>
> As I said earlier compression and encoding are not required.
>
> -berk
>
> On Thu, Oct 16, 2008 at 4:37 AM, John Doe <ufnoise at gmail.com> wrote:
>> I ended up having to reverse engineer the file format.  Apparently
>> compression is required, and I had to reverse engineer the paraview
>> code to determine how to write the "header".  Apparently the zlib
>> compression is performed on blocks of 32768 bytes, and a header must
>> be written to separate them back out on read.
>>
>>
>> Regards,
>>
>> Juan
>>
>> On Wed, Oct 15, 2008 at 7:31 PM, John Doe <ufnoise at gmail.com> wrote:
>>> I have a vector of N points.  So the number of doubles in 3*N.  A
>>> double is 8 long.  Base 64 produces 4/3 the number of original bytes.
>>>
>>> As expected my base64 string is 3*N*8 * (4/3)
>>>
>>> So for 1417 points, I should expect a string 45344 characters long.
>>>
>>> Paraview claims my string is too short:
>>> ERROR: In /home/jsanchez/ParaView3/VTK/IO/vtkXMLUnstructuredDataReader.cxx,
>>> line 522
>>> vtkXMLUnstructuredGridReader (0x4dde7e0): Cannot read points array
>>> from Points in piece 0.  The data array in the element may be too
>>> short.
>>>
>>> Here is the header I am writing:
>>> <UnstructuredGrid>
>>> <Piece NumberOfPoints="1417" NumberOfCells="8781">
>>> <Points>
>>> <DataArray type="Float64" NumberOfComponents="3" format="binary">
>>> this is where my raw base64 encoding string goes
>>> </DataArray>
>>> </Points>
>>>
>>> What could be wrong with my string of characters?
>>>
>>> Thanks,
>>>
>>> Juan
>>>
>>>
>>> On Wed, Oct 15, 2008 at 6:05 PM, Berk Geveci <berk.geveci at kitware.com> wrote:
>>>> By the way, if you are willing to link against the VTK libraries, VTK
>>>> has a C/Fortran interface to write XML files.
>>>>
>>>> -berk
>>>>
>>>> On Wed, Oct 15, 2008 at 4:48 PM, John Doe <ufnoise at gmail.com> wrote:
>>>>> Hello,
>>>>>
>>>>> Thanks for the suggestion.  I assume that for double precision I am
>>>>> doing a base64 encoding for every number in the list?  Is compression
>>>>> worthwhile for floating point data?  Are there free compression
>>>>> libraries out there?
>>>>>
>>>>> Juan
>>>>>
>>>>> On Wed, Oct 15, 2008 at 3:26 PM, Berk Geveci <berk.geveci at kitware.com> wrote:
>>>>>> ASCII XML is very slow to load because it uses the XML parser. For
>>>>>> large data, you should switch to a binary format.
>>>>>>
>>>>>> XML binary
>>>>>> legacy VTK
>>>>>> Exodus
>>>>>> Xdmf
>>>>>>
>>>>>> are a few choices.
>>>>>>
>>>>>> To see the difference in load time, save the data out as a binary vtu
>>>>>> or vtk file and load it back in.
>>>>>>
>>>>>> -berk
>>>>>>
>>>>>> On Wed, Oct 15, 2008 at 3:25 PM, John Doe <ufnoise at gmail.com> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have a 220 MB vtu file that is being loaded and it is taking at
>>>>>>> least 15 min to load the data.  Is there any option to reduce the
>>>>>>> amount of time paraview would take to load the data.  Or is there some
>>>>>>> way I should be formatting my data.  The vtu file is in the ASCII XML
>>>>>>> VTK format.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Juan
>>>>>>> _______________________________________________
>>>>>>> ParaView mailing list
>>>>>>> ParaView at paraview.org
>>>>>>> http://www.paraview.org/mailman/listinfo/paraview
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>