[Paraview] Error when reading large XMLUnstructuredData file

[Takuya OSHIMA]

Hi Guenter and all,

I also had the same issue with .vti (XML vtkImageData) files of
roughly the same sizes of around 3 GBytes. Currently I'm working
around the issue by building VTK and ParaView with VTK_USE_64BIT_IDS
set to ON, which will allow you to load up to 4 GBytes as compared to
2 GBytes when VTK_USE_64BIT_IDS is OFF.

However I'd be interested if there's more fundamental way of resolving
the issue because I'll be handling larger cases soon. Is there a
format (other than series of legacy VTK format files) that is
time-aware, parallelizable and allows more than 4 GBytes of
vtkImageData per partition?

Takuya

Takuya OSHIMA, Ph.D.
Faculty of Engineering, Niigata University
8050 Ikarashi-Ninocho, Nishi-ku, Niigata, 950-2181, JAPAN

From: Guenter Bartsch <guenter.bartsch at googlemail.com>
Subject: [Paraview] Error when reading large XMLUnstructuredData file
Date: Tue, 4 Jan 2011 11:45:18 +0100

> hi all,
> 
> we have run into a problem reading a pretty large (3.6 GB) .vtu file.
> the file was produced by a fortran program that writes vtk data in the
> old ascii format which results in a 17G ascii file which takes very
> long to load in paraview (and results in an error message). so, we
> have written a converter program that reads the ascii file and outputs
> xml:
> 
> 
> int main(int argc, char *argp[])
> {
>   if (argc != 2) {
>     cerr << "usage: " << argp[0] << " <vtk filename>\n\n";
>     return 1;
>   }
> 
>   char *filename = argp[1];
> 
>   cout << "Input filename: " << filename << "\n";
> 
>   vtkSmartPointer<vtkUnstructuredGridReader> reader =
> vtkSmartPointer<vtkUnstructuredGridReader>::New();
>   reader->SetFileName(filename);
> 
>   char *ofilename = "foo.vtu";
>   cout << "Output filename: " << ofilename << "\n";
>   vtkSmartPointer<vtkXMLUnstructuredGridWriter> writer =
> vtkSmartPointer<vtkXMLUnstructuredGridWriter>::New();
>   writer->SetFileName(ofilename);
>   writer->SetInputConnection(reader->GetOutputPort());
>   writer->Write();
> 
>   return EXIT_SUCCESS;
> }
> 
> this program runs fine. now, if i open the resulting file in paraview,
> i run into this error as soon as i hit apply:
> 
> ERROR: In /opt/paraview-3.8.1/src/ParaView-3.8.1/VTK/IO/vtkXMLUnstructuredDataReader.cxx,
> line 652
> vtkXMLUnstructuredGridReader (0x143f0c40): Cannot read cell
> connectivity from Cells in piece 0 because the "connectivity" array is
> not long enough.
> 
> the converter did work on a smaller test data set and paraview could
> open it - only now with the large data set paraview fails. here is the
> header of the failing file:
> 
> <?xml version="1.0"?>
> <VTKFile type="UnstructuredGrid" version="0.1"
> byte_order="LittleEndian" compressor="vtkZLibDataCompressor">
>   <UnstructuredGrid>
>     <Piece NumberOfPoints="106496000"
> NumberOfCells="105122745"           >
>       <PointData Vectors="vectors">
>         <DataArray type="Float32" Name="vectors"
> NumberOfComponents="3" format="appended" RangeMin="0.0021820231521"
>   RangeMax="24.459394872"         offset="0"                   />
>       </PointData>
>       <CellData>
>       </CellData>
>       <Points>
>         <DataArray type="Float32" Name="Points" NumberOfComponents="3"
> format="appended" RangeMin="0"
> RangeMax="5007.1818403"         offset="1580185620"          />
>       </Points>
>       <Cells>
>         <DataArray type="Int64" Name="connectivity" format="appended"
> RangeMin=""                     RangeMax=""
> offset="2045521660"          />
>         <DataArray type="Int64" Name="offsets" format="appended"
> RangeMin=""                     RangeMax=""
> offset="3504937336"          />
>         <DataArray type="UInt8" Name="types" format="appended"
> RangeMin=""                     RangeMax=""
> offset="3671971228"          />
>       </Cells>
>     </Piece>
>   </UnstructuredGrid>
>   <AppendedData encoding="base64">
>    _WJgAAACAAAAAAAAAvXYAALF2AADJdgAAUHcAA...
> 
> does anyone know what could be going on here? do we need to configure
> paraview to handle larger files? is this a bug? could the original
> ascii data set have been corrupt and the error reflects now in the xml
> file?
> 
> thanks in advance and best regards,
> 
>    guenter
> _______________________________________________
> Powered by www.kitware.com
> 
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
> 
> Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView
> 
> Follow this link to subscribe/unsubscribe:
> http://www.paraview.org/mailman/listinfo/paraview