[vtk-developers] vtkImageConvolve crashes when malloc fails

[Sean McBride]

On 6/4/08 4:55 PM, Mathieu Coursolle said:

>> It is surprising to me that you are experiencing this with a 100Mb
>> dataset (which not that big). Do you have a lot of 100Mb data sets
>> maybe? Assuming that you are compiling 32 bit, it would take quite a
>> few 100Mb objects to fragment the virtual memory (2 Gb)  enough to
>> cause problems. Also, this may be caused by a misbehaving algorithm,
>> if it grows an array slowly to get to 100Mb causing a lot of reallocs.
>
>I could reproduce the problem without VTK with a simple App:
>
>bool ok = true;
>unsigned long idx = 0;
>do
>{
>	char* x = malloc(100000000);  // 100 MBytes
>	if (x)
>	{
>		free(x);
>                 idx++;
>	}
>	else
>	{
>		ok = false;
>	}
>}
>while (ok);
>
>At the end, the loop is executed less than 50 times.
>
>I could reproduce the same behavior by applying vtkImageConvolve to a  
>100 MBytes image data in the loop.

For the record, the reason such a test case fragmented and exhausted
memory so easily was because we had the "MallocGuardEdges" debug
environment variable on.  It adds a protector page before and after the
pages actually allocated.  However, these protector pages are _never_
freed, even when the malloced memory is freed.  Hence the rapid
fragmentation.  With MallocGuardEdges off, it is harder to repro the
fragmentation and crashing that we saw in VTK.

-- 
____________________________________________________________
Sean McBride, B. Eng                 sean at rogue-research.com
Rogue Research                        www.rogue-research.com 
Mac Software Developer              Montréal, Québec, Canada