[Paraview] Error when reading large XMLUnstructuredData file

[Guenter Bartsch]

Takuya,

thanks for your quick reply. however, we do have set

VTK_USE_64BIT_IDS                ON

and this is a 64bit linux platform (CentOS/RHEL x86_64) in case that
matters - so unfortunately that isn't the solution here.

thanks anyway and kind regards,

   guenter

On Tue, Jan 4, 2011 at 12:06 PM, Takuya OSHIMA
<oshima at eng.niigata-u.ac.jp> wrote:
> Hi Guenter and all,
>
> I also had the same issue with .vti (XML vtkImageData) files of
> roughly the same sizes of around 3 GBytes. Currently I'm working
> around the issue by building VTK and ParaView with VTK_USE_64BIT_IDS
> set to ON, which will allow you to load up to 4 GBytes as compared to
> 2 GBytes when VTK_USE_64BIT_IDS is OFF.
>
> However I'd be interested if there's more fundamental way of resolving
> the issue because I'll be handling larger cases soon. Is there a
> format (other than series of legacy VTK format files) that is
> time-aware, parallelizable and allows more than 4 GBytes of
> vtkImageData per partition?
>
> Takuya
>
> Takuya OSHIMA, Ph.D.
> Faculty of Engineering, Niigata University
> 8050 Ikarashi-Ninocho, Nishi-ku, Niigata, 950-2181, JAPAN
>
> From: Guenter Bartsch <guenter.bartsch at googlemail.com>
> Subject: [Paraview] Error when reading large XMLUnstructuredData file
> Date: Tue, 4 Jan 2011 11:45:18 +0100
>
>> hi all,
>>
>> we have run into a problem reading a pretty large (3.6 GB) .vtu file.
>> the file was produced by a fortran program that writes vtk data in the
>> old ascii format which results in a 17G ascii file which takes very
>> long to load in paraview (and results in an error message). so, we
>> have written a converter program that reads the ascii file and outputs
>> xml:
>>
>>
>> int main(int argc, char *argp[])
>> {
>>   if (argc != 2) {
>>     cerr << "usage: " << argp[0] << " <vtk filename>\n\n";
>>     return 1;
>>   }
>>
>>   char *filename = argp[1];
>>
>>   cout << "Input filename: " << filename << "\n";
>>
>>   vtkSmartPointer<vtkUnstructuredGridReader> reader =
>> vtkSmartPointer<vtkUnstructuredGridReader>::New();
>>   reader->SetFileName(filename);
>>
>>   char *ofilename = "foo.vtu";
>>   cout << "Output filename: " << ofilename << "\n";
>>   vtkSmartPointer<vtkXMLUnstructuredGridWriter> writer =
>> vtkSmartPointer<vtkXMLUnstructuredGridWriter>::New();
>>   writer->SetFileName(ofilename);
>>   writer->SetInputConnection(reader->GetOutputPort());
>>   writer->Write();
>>
>>   return EXIT_SUCCESS;
>> }
>>
>> this program runs fine. now, if i open the resulting file in paraview,
>> i run into this error as soon as i hit apply:
>>
>> ERROR: In /opt/paraview-3.8.1/src/ParaView-3.8.1/VTK/IO/vtkXMLUnstructuredDataReader.cxx,
>> line 652
>> vtkXMLUnstructuredGridReader (0x143f0c40): Cannot read cell
>> connectivity from Cells in piece 0 because the "connectivity" array is
>> not long enough.
>>
>> the converter did work on a smaller test data set and paraview could
>> open it - only now with the large data set paraview fails. here is the
>> header of the failing file:
>>
>> <?xml version="1.0"?>
>> <VTKFile type="UnstructuredGrid" version="0.1"
>> byte_order="LittleEndian" compressor="vtkZLibDataCompressor">
>>   <UnstructuredGrid>
>>     <Piece NumberOfPoints="106496000"
>> NumberOfCells="105122745"           >
>>       <PointData Vectors="vectors">
>>         <DataArray type="Float32" Name="vectors"
>> NumberOfComponents="3" format="appended" RangeMin="0.0021820231521"
>>   RangeMax="24.459394872"         offset="0"                   />
>>       </PointData>
>>       <CellData>
>>       </CellData>
>>       <Points>
>>         <DataArray type="Float32" Name="Points" NumberOfComponents="3"
>> format="appended" RangeMin="0"
>> RangeMax="5007.1818403"         offset="1580185620"          />
>>       </Points>
>>       <Cells>
>>         <DataArray type="Int64" Name="connectivity" format="appended"
>> RangeMin=""                     RangeMax=""
>> offset="2045521660"          />
>>         <DataArray type="Int64" Name="offsets" format="appended"
>> RangeMin=""                     RangeMax=""
>> offset="3504937336"          />
>>         <DataArray type="UInt8" Name="types" format="appended"
>> RangeMin=""                     RangeMax=""
>> offset="3671971228"          />
>>       </Cells>
>>     </Piece>
>>   </UnstructuredGrid>
>>   <AppendedData encoding="base64">
>>    _WJgAAACAAAAAAAAAvXYAALF2AADJdgAAUHcAA...
>>
>> does anyone know what could be going on here? do we need to configure
>> paraview to handle larger files? is this a bug? could the original
>> ascii data set have been corrupt and the error reflects now in the xml
>> file?
>>
>> thanks in advance and best regards,
>>
>>    guenter
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>
>> Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.paraview.org/mailman/listinfo/paraview
>