[Paraview] Error when reading large XMLUnstructuredData file

[Guenter Bartsch]

hi all,

we have run into a problem reading a pretty large (3.6 GB) .vtu file.
the file was produced by a fortran program that writes vtk data in the
old ascii format which results in a 17G ascii file which takes very
long to load in paraview (and results in an error message). so, we
have written a converter program that reads the ascii file and outputs
xml:


int main(int argc, char *argp[])
{
  if (argc != 2) {
    cerr << "usage: " << argp[0] << " <vtk filename>\n\n";
    return 1;
  }

  char *filename = argp[1];

  cout << "Input filename: " << filename << "\n";

  vtkSmartPointer<vtkUnstructuredGridReader> reader =
vtkSmartPointer<vtkUnstructuredGridReader>::New();
  reader->SetFileName(filename);

  char *ofilename = "foo.vtu";
  cout << "Output filename: " << ofilename << "\n";
  vtkSmartPointer<vtkXMLUnstructuredGridWriter> writer =
vtkSmartPointer<vtkXMLUnstructuredGridWriter>::New();
  writer->SetFileName(ofilename);
  writer->SetInputConnection(reader->GetOutputPort());
  writer->Write();

  return EXIT_SUCCESS;
}

this program runs fine. now, if i open the resulting file in paraview,
i run into this error as soon as i hit apply:

ERROR: In /opt/paraview-3.8.1/src/ParaView-3.8.1/VTK/IO/vtkXMLUnstructuredDataReader.cxx,
line 652
vtkXMLUnstructuredGridReader (0x143f0c40): Cannot read cell
connectivity from Cells in piece 0 because the "connectivity" array is
not long enough.

the converter did work on a smaller test data set and paraview could
open it - only now with the large data set paraview fails. here is the
header of the failing file:

<?xml version="1.0"?>
<VTKFile type="UnstructuredGrid" version="0.1"
byte_order="LittleEndian" compressor="vtkZLibDataCompressor">
  <UnstructuredGrid>
    <Piece NumberOfPoints="106496000"
NumberOfCells="105122745"           >
      <PointData Vectors="vectors">
        <DataArray type="Float32" Name="vectors"
NumberOfComponents="3" format="appended" RangeMin="0.0021820231521"
  RangeMax="24.459394872"         offset="0"                   />
      </PointData>
      <CellData>
      </CellData>
      <Points>
        <DataArray type="Float32" Name="Points" NumberOfComponents="3"
format="appended" RangeMin="0"
RangeMax="5007.1818403"         offset="1580185620"          />
      </Points>
      <Cells>
        <DataArray type="Int64" Name="connectivity" format="appended"
RangeMin=""                     RangeMax=""
offset="2045521660"          />
        <DataArray type="Int64" Name="offsets" format="appended"
RangeMin=""                     RangeMax=""
offset="3504937336"          />
        <DataArray type="UInt8" Name="types" format="appended"
RangeMin=""                     RangeMax=""
offset="3671971228"          />
      </Cells>
    </Piece>
  </UnstructuredGrid>
  <AppendedData encoding="base64">
   _WJgAAACAAAAAAAAAvXYAALF2AADJdgAAUHcAA...

does anyone know what could be going on here? do we need to configure
paraview to handle larger files? is this a bug? could the original
ascii data set have been corrupt and the error reflects now in the xml
file?

thanks in advance and best regards,

   guenter