[Paraview] Error when reading large XMLUnstructuredData file

[Takuya OSHIMA]

Hi Guenter, just for confirmation, did you build both VTK that your
converter uses and ParaView with VTK_USE_64BIT_IDS ON?

By the way I'm working with OS X 10.6, if it makes difference...

Takuya

Takuya OSHIMA, Ph.D.
Faculty of Engineering, Niigata University
8050 Ikarashi-Ninocho, Nishi-ku, Niigata, 950-2181, JAPAN

From: Guenter Bartsch <guenter.bartsch at googlemail.com>
Subject: Re: [Paraview] Error when reading large XMLUnstructuredData file
Date: Tue, 4 Jan 2011 12:10:27 +0100

> Takuya,
> 
> thanks for your quick reply. however, we do have set
> 
> VTK_USE_64BIT_IDS                ON
> 
> and this is a 64bit linux platform (CentOS/RHEL x86_64) in case that
> matters - so unfortunately that isn't the solution here.
> 
> thanks anyway and kind regards,
> 
>    guenter
> 
> On Tue, Jan 4, 2011 at 12:06 PM, Takuya OSHIMA
> <oshima at eng.niigata-u.ac.jp> wrote:
>> Hi Guenter and all,
>>
>> I also had the same issue with .vti (XML vtkImageData) files of
>> roughly the same sizes of around 3 GBytes. Currently I'm working
>> around the issue by building VTK and ParaView with VTK_USE_64BIT_IDS
>> set to ON, which will allow you to load up to 4 GBytes as compared to
>> 2 GBytes when VTK_USE_64BIT_IDS is OFF.
>>
>> However I'd be interested if there's more fundamental way of resolving
>> the issue because I'll be handling larger cases soon. Is there a
>> format (other than series of legacy VTK format files) that is
>> time-aware, parallelizable and allows more than 4 GBytes of
>> vtkImageData per partition?
>>
>> Takuya
>>
>> Takuya OSHIMA, Ph.D.
>> Faculty of Engineering, Niigata University
>> 8050 Ikarashi-Ninocho, Nishi-ku, Niigata, 950-2181, JAPAN
>>
>> From: Guenter Bartsch <guenter.bartsch at googlemail.com>
>> Subject: [Paraview] Error when reading large XMLUnstructuredData file
>> Date: Tue, 4 Jan 2011 11:45:18 +0100
>>
>>> hi all,
>>>
>>> we have run into a problem reading a pretty large (3.6 GB) .vtu file.
>>> the file was produced by a fortran program that writes vtk data in the
>>> old ascii format which results in a 17G ascii file which takes very
>>> long to load in paraview (and results in an error message). so, we
>>> have written a converter program that reads the ascii file and outputs
>>> xml:
>>>
>>>
>>> int main(int argc, char *argp[])
>>> {
>>>   if (argc != 2) {
>>>     cerr << "usage: " << argp[0] << " <vtk filename>\n\n";
>>>     return 1;
>>>   }
>>>
>>>   char *filename = argp[1];
>>>
>>>   cout << "Input filename: " << filename << "\n";
>>>
>>>   vtkSmartPointer<vtkUnstructuredGridReader> reader =
>>> vtkSmartPointer<vtkUnstructuredGridReader>::New();
>>>   reader->SetFileName(filename);
>>>
>>>   char *ofilename = "foo.vtu";
>>>   cout << "Output filename: " << ofilename << "\n";
>>>   vtkSmartPointer<vtkXMLUnstructuredGridWriter> writer =
>>> vtkSmartPointer<vtkXMLUnstructuredGridWriter>::New();
>>>   writer->SetFileName(ofilename);
>>>   writer->SetInputConnection(reader->GetOutputPort());
>>>   writer->Write();
>>>
>>>   return EXIT_SUCCESS;
>>> }
>>>
>>> this program runs fine. now, if i open the resulting file in paraview,
>>> i run into this error as soon as i hit apply:
>>>
>>> ERROR: In /opt/paraview-3.8.1/src/ParaView-3.8.1/VTK/IO/vtkXMLUnstructuredDataReader.cxx,
>>> line 652
>>> vtkXMLUnstructuredGridReader (0x143f0c40): Cannot read cell
>>> connectivity from Cells in piece 0 because the "connectivity" array is
>>> not long enough.
>>>
>>> the converter did work on a smaller test data set and paraview could
>>> open it - only now with the large data set paraview fails. here is the
>>> header of the failing file:
>>>
>>> <?xml version="1.0"?>
>>> <VTKFile type="UnstructuredGrid" version="0.1"
>>> byte_order="LittleEndian" compressor="vtkZLibDataCompressor">
>>>   <UnstructuredGrid>
>>>     <Piece NumberOfPoints="106496000"
>>> NumberOfCells="105122745"           >
>>>       <PointData Vectors="vectors">
>>>         <DataArray type="Float32" Name="vectors"
>>> NumberOfComponents="3" format="appended" RangeMin="0.0021820231521"
>>>   RangeMax="24.459394872"         offset="0"                   />
>>>       </PointData>
>>>       <CellData>
>>>       </CellData>
>>>       <Points>
>>>         <DataArray type="Float32" Name="Points" NumberOfComponents="3"
>>> format="appended" RangeMin="0"
>>> RangeMax="5007.1818403"         offset="1580185620"          />
>>>       </Points>
>>>       <Cells>
>>>         <DataArray type="Int64" Name="connectivity" format="appended"
>>> RangeMin=""                     RangeMax=""
>>> offset="2045521660"          />
>>>         <DataArray type="Int64" Name="offsets" format="appended"
>>> RangeMin=""                     RangeMax=""
>>> offset="3504937336"          />
>>>         <DataArray type="UInt8" Name="types" format="appended"
>>> RangeMin=""                     RangeMax=""
>>> offset="3671971228"          />
>>>       </Cells>
>>>     </Piece>
>>>   </UnstructuredGrid>
>>>   <AppendedData encoding="base64">
>>>    _WJgAAACAAAAAAAAAvXYAALF2AADJdgAAUHcAA...
>>>
>>> does anyone know what could be going on here? do we need to configure
>>> paraview to handle larger files? is this a bug? could the original
>>> ascii data set have been corrupt and the error reflects now in the xml
>>> file?
>>>
>>> thanks in advance and best regards,
>>>
>>>    guenter
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>>
>>> Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://www.paraview.org/mailman/listinfo/paraview
>>