[vtkusers] Issues handling very large files

[Williams, Kenneth (LLU)]

Hello,

I'm currently running into issues trying to analyze very large .vti data files (~110 GB and up) generated with VTK 5.8. Each file contains several arrays within the point data that describes properties of simulated particles (e.g., tuples of the particles' x, y, and z coordinates).

We generate one .vti file per simulation and then use VTK to combine these files into a single large .vti file.  We then use Python with VTK to generate e.g. histograms of particle position from the .vti files.

Here is some test code that produces the error I am seeing:


#!/usr/bin/env python

import vtk
from sys import argv

reader = vtk.vtkXMLImageDataReader()
reader.SetFileName(argv[1])
reader.Update()

data = reader.GetOutput()
ptdata = data.GetPointData()
array = ptdata.GetArray(0)
npts = array.GetNumberOfTuples()

for ii in range(npts):
  print array.GetTuple3(ii)


The error:


ERROR: In /build/buildd/vtk-5.8.0/IO/vtkXMLStructuredDataReader.cxx, line 325
vtkXMLImageDataReader (0x1caa160): Error reading extent 0 195310399 0 0 0 0 from piece 0

ERROR: In /build/buildd/vtk-5.8.0/IO/vtkXMLDataReader.cxx, line 510
vtkXMLImageDataReader (0x1caa160): Cannot read point data array "DistalAperturePlus0cm\PhasePosition" from PointData in piece 0.  The data array in the element may be too short.

Traceback (most recent call last):
  File "test.py", line 13, in <module>
    npts = array.GetNumberOfTuples()
AttributeError: 'NoneType' object has no attribute 'GetNumberOfTuples'


For testing purposes, I have generated files slightly smaller in size (~82 GB) that do work correctly, and the only difference is the number of particles generated. Is there a known issue or workaround for dealing with such large files, or am I possibly looking in the wrong place as to the cause of this error?

Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/vtkusers/attachments/20140925/5b7a0e0d/attachment.html>