[vtk-developers] vtkImageConvolve crashes when malloc fails

[Berk Geveci]

> I guess this is a common problem when dealing with large data sets
> (100Mbytes). I can reproduce the problem easily just by calling a series
> of malloc and free. The memory allocation is always bellow 150 MBytes in
> my app, but there is so many fragmentation that malloc fails. If I call
> malloc, I can detect such a failure and take appropriate measure (saving
> data, etc.).

It is surprising to me that you are experiencing this with a 100Mb
dataset (which not that big). Do you have a lot of 100Mb data sets
maybe? Assuming that you are compiling 32 bit, it would take quite a
few 100Mb objects to fragment the virtual memory (2 Gb)  enough to
cause problems. Also, this may be caused by a misbehaving algorithm,
if it grows an array slowly to get to 100Mb causing a lot of reallocs.

> However, if malloc is called inside VTK, I'd like the pipeline execution
> to stop before it reaches a crash.

I agree. VTK's handling of memory allocation failures is far from
optimal. How would you address it?

-berk


>> The parallel computing experts should address this, but basically the
>> solution to large data is to stream the data in pieces. While this is not
>> always possible, VTK provides good support for this especially for image
>> data. This means that your source object must be able to respond to data
>> requests that are less than the whole extent of the image.
>>
>> Also, make sure that you are setting the release data flags, etc. to
>> discard
>> memory in the pipeline when it is no longer needed.
>>
>> W
>>
>> On Tue, Jun 3, 2008 at 4:37 PM, Berk Geveci <berk.geveci at kitware.com>
>> wrote:
>>
>>> If memory allocation fails, you will run into a crash sooner or later
>>> unfortunately. Even if you were to add such a check, somewhere down
>>> the pipeline some other class is going to die. IMHO, the only
>>> "graceful" way of handling this is to throw an exception that the
>>> application catches. Then it would have to report an error and shut
>>> down. Adding bunch of if statements for an exceptional case is
>>> probably not a good idea.
>>>
>>> -berk
>>>
>>> 2008/6/3 Mathieu Coursolle <mcoursolle at rogue-research.com>:
>>> > Hi VTK developers,
>>> > I am performing a series of filters, where some of them are
>>> > vtkImageConvolve.
>>> > My input vtkImageData is usually close to 100Mbytes, and the execution
>>> of
>>> my
>>> > pipeline
>>> > seems to cause some memory fragmentation. Therefore, malloc will
>>> eventually
>>> > fail, causing
>>> > a crash in vtkImageConvolve.
>>> > If malloc fails during the execution of a pipeline, then the
>>> scalarPointer
>>> > of the resulting vtkImageData is NULL. If a connected vtkImageConvolve
>>> > filter receives such a vtkImageData
>>> > as its input, it tries to access the scalar pointer (which is NULL),
>>> causing
>>> > a crash.
>>> > A solution would be to check the pointers:
>>> >   // ADDED CODE BEGIN--------
>>> >   // Make sure the scalar pointers are valid.
>>> >   if (!inData[0][0]->GetScalarPointer() ||
>>> !outData[0]->GetScalarPointer())
>>> >     {
>>> > vtkErrorMacro(<< "Execute: input and/or output image does not have any
>>> > scalar value.");
>>> >     return;
>>> >     }
>>> >   // ADDED CODE END--------
>>> >
>>> >
>>> >
>>> >   switch (inData[0][0]->GetScalarType())
>>> >     {
>>> >     vtkTemplateMacro(
>>> >       vtkImageConvolveExecute(this, inData[0][0],
>>> >                               static_cast<VTK_TT *>(inPtr),
>>> outData[0],
>>> >                               static_cast<VTK_TT *>(outPtr),
>>> >                               outExt, id, inInfo));
>>> >     default:
>>> >       vtkErrorMacro(<< "Execute: Unknown ScalarType");
>>> >       return;
>>> >     }
>>> > So I was wondering,
>>> > 1) Is that the right way to avoid such crashes?
>>> > 2) I guess that problem might occur with other filters as well...
>>> > Thanks.
>>> > Mathieu
>>> >
>>> > _______________________________________________
>>> > vtk-developers mailing list
>>> > vtk-developers at vtk.org
>>> > http://www.vtk.org/mailman/listinfo/vtk-developers
>>> >
>>> >
>>> _______________________________________________
>>> vtk-developers mailing list
>>> vtk-developers at vtk.org
>>> http://www.vtk.org/mailman/listinfo/vtk-developers
>>>
>>
>>
>>
>> --
>> William J. Schroeder, PhD
>> Kitware, Inc.
>> 28 Corporate Drive
>> Clifton Park, NY 12065
>> will.schroeder at kitware.com
>> http://www.kitware.com
>> 518-371-3971 (phone and fax)
>>
>
>