[Paraview] Error when reading large XMLUnstructuredData file
Guenter Bartsch
guenter.bartsch at googlemail.com
Tue Jan 4 06:28:57 EST 2011
Takuya,
On Tue, Jan 4, 2011 at 12:21 PM, Takuya OSHIMA
<oshima at eng.niigata-u.ac.jp> wrote:
> Hi Guenter, just for confirmation, did you build both VTK that your
> converter uses and ParaView with VTK_USE_64BIT_IDS ON?
yes - at least that's what i thought i did ;) . actually we have built
paraview 3.8.1 from source which came with a statically embedded
version of VTK. i have then built my converter against that VTK that
came with paraview.
i have blogged about my little converter program:
https://sites.google.com/site/guenterbartsch/blog/myfirstvtkprogram
so others may benefit from it - but i don't think i have any
additional information in my blog post.
thanks again for your help and best regards,
guenter
>
> By the way I'm working with OS X 10.6, if it makes difference...
>
> Takuya
>
> Takuya OSHIMA, Ph.D.
> Faculty of Engineering, Niigata University
> 8050 Ikarashi-Ninocho, Nishi-ku, Niigata, 950-2181, JAPAN
>
> From: Guenter Bartsch <guenter.bartsch at googlemail.com>
> Subject: Re: [Paraview] Error when reading large XMLUnstructuredData file
> Date: Tue, 4 Jan 2011 12:10:27 +0100
>
>> Takuya,
>>
>> thanks for your quick reply. however, we do have set
>>
>> VTK_USE_64BIT_IDS ON
>>
>> and this is a 64bit linux platform (CentOS/RHEL x86_64) in case that
>> matters - so unfortunately that isn't the solution here.
>>
>> thanks anyway and kind regards,
>>
>> guenter
>>
>> On Tue, Jan 4, 2011 at 12:06 PM, Takuya OSHIMA
>> <oshima at eng.niigata-u.ac.jp> wrote:
>>> Hi Guenter and all,
>>>
>>> I also had the same issue with .vti (XML vtkImageData) files of
>>> roughly the same sizes of around 3 GBytes. Currently I'm working
>>> around the issue by building VTK and ParaView with VTK_USE_64BIT_IDS
>>> set to ON, which will allow you to load up to 4 GBytes as compared to
>>> 2 GBytes when VTK_USE_64BIT_IDS is OFF.
>>>
>>> However I'd be interested if there's more fundamental way of resolving
>>> the issue because I'll be handling larger cases soon. Is there a
>>> format (other than series of legacy VTK format files) that is
>>> time-aware, parallelizable and allows more than 4 GBytes of
>>> vtkImageData per partition?
>>>
>>> Takuya
>>>
>>> Takuya OSHIMA, Ph.D.
>>> Faculty of Engineering, Niigata University
>>> 8050 Ikarashi-Ninocho, Nishi-ku, Niigata, 950-2181, JAPAN
>>>
>>> From: Guenter Bartsch <guenter.bartsch at googlemail.com>
>>> Subject: [Paraview] Error when reading large XMLUnstructuredData file
>>> Date: Tue, 4 Jan 2011 11:45:18 +0100
>>>
>>>> hi all,
>>>>
>>>> we have run into a problem reading a pretty large (3.6 GB) .vtu file.
>>>> the file was produced by a fortran program that writes vtk data in the
>>>> old ascii format which results in a 17G ascii file which takes very
>>>> long to load in paraview (and results in an error message). so, we
>>>> have written a converter program that reads the ascii file and outputs
>>>> xml:
>>>>
>>>>
>>>> int main(int argc, char *argp[])
>>>> {
>>>> if (argc != 2) {
>>>> cerr << "usage: " << argp[0] << " <vtk filename>\n\n";
>>>> return 1;
>>>> }
>>>>
>>>> char *filename = argp[1];
>>>>
>>>> cout << "Input filename: " << filename << "\n";
>>>>
>>>> vtkSmartPointer<vtkUnstructuredGridReader> reader =
>>>> vtkSmartPointer<vtkUnstructuredGridReader>::New();
>>>> reader->SetFileName(filename);
>>>>
>>>> char *ofilename = "foo.vtu";
>>>> cout << "Output filename: " << ofilename << "\n";
>>>> vtkSmartPointer<vtkXMLUnstructuredGridWriter> writer =
>>>> vtkSmartPointer<vtkXMLUnstructuredGridWriter>::New();
>>>> writer->SetFileName(ofilename);
>>>> writer->SetInputConnection(reader->GetOutputPort());
>>>> writer->Write();
>>>>
>>>> return EXIT_SUCCESS;
>>>> }
>>>>
>>>> this program runs fine. now, if i open the resulting file in paraview,
>>>> i run into this error as soon as i hit apply:
>>>>
>>>> ERROR: In /opt/paraview-3.8.1/src/ParaView-3.8.1/VTK/IO/vtkXMLUnstructuredDataReader.cxx,
>>>> line 652
>>>> vtkXMLUnstructuredGridReader (0x143f0c40): Cannot read cell
>>>> connectivity from Cells in piece 0 because the "connectivity" array is
>>>> not long enough.
>>>>
>>>> the converter did work on a smaller test data set and paraview could
>>>> open it - only now with the large data set paraview fails. here is the
>>>> header of the failing file:
>>>>
>>>> <?xml version="1.0"?>
>>>> <VTKFile type="UnstructuredGrid" version="0.1"
>>>> byte_order="LittleEndian" compressor="vtkZLibDataCompressor">
>>>> <UnstructuredGrid>
>>>> <Piece NumberOfPoints="106496000"
>>>> NumberOfCells="105122745" >
>>>> <PointData Vectors="vectors">
>>>> <DataArray type="Float32" Name="vectors"
>>>> NumberOfComponents="3" format="appended" RangeMin="0.0021820231521"
>>>> RangeMax="24.459394872" offset="0" />
>>>> </PointData>
>>>> <CellData>
>>>> </CellData>
>>>> <Points>
>>>> <DataArray type="Float32" Name="Points" NumberOfComponents="3"
>>>> format="appended" RangeMin="0"
>>>> RangeMax="5007.1818403" offset="1580185620" />
>>>> </Points>
>>>> <Cells>
>>>> <DataArray type="Int64" Name="connectivity" format="appended"
>>>> RangeMin="" RangeMax=""
>>>> offset="2045521660" />
>>>> <DataArray type="Int64" Name="offsets" format="appended"
>>>> RangeMin="" RangeMax=""
>>>> offset="3504937336" />
>>>> <DataArray type="UInt8" Name="types" format="appended"
>>>> RangeMin="" RangeMax=""
>>>> offset="3671971228" />
>>>> </Cells>
>>>> </Piece>
>>>> </UnstructuredGrid>
>>>> <AppendedData encoding="base64">
>>>> _WJgAAACAAAAAAAAAvXYAALF2AADJdgAAUHcAA...
>>>>
>>>> does anyone know what could be going on here? do we need to configure
>>>> paraview to handle larger files? is this a bug? could the original
>>>> ascii data set have been corrupt and the error reflects now in the xml
>>>> file?
>>>>
>>>> thanks in advance and best regards,
>>>>
>>>> guenter
>>>> _______________________________________________
>>>> Powered by www.kitware.com
>>>>
>>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>>>
>>>> Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView
>>>>
>>>> Follow this link to subscribe/unsubscribe:
>>>> http://www.paraview.org/mailman/listinfo/paraview
>>>
>
More information about the ParaView
mailing list