[vtk-developers] exodusII reader slowness

[David Thompson]

> I just spent the better part of a day trying to discover why the  
> exodusII reader was so slow. I was eventually pointed to the cache  
> capacity. I found that my Update() times reduced from 235 seconds to  
> 8 seconds when I changed SetCacheCapacity from 0.0 to 128 and  
> recompiled.
>
> I’ve heard that there was some discussion on the cache capacity  
> issue in April, so I’m broadcasting a gentle reminder that this  
> issue still needs a resolution.

Hi Warren,

I don't think anyone has filed a bug for this. Feel free to file one  
and assign it to Nathan Fabian. :-)

This has come up on the list in a couple of ways:

1. The cache was causing problems for someone who had many files to  
read in, each with many timesteps, on a single node (which caused  
memory exhaustion even though a single timestep would easily fit in  
memory). So caching was turned off. Nathan proposed exposing the cache  
setting in ParaView and defaulting the cache to "on" back in April  
(the mailing list thread was "change to  
vtkExodusIIReaderPrivate::ResetCache"). He hinted that he might submit  
a fix but hasn't yet, to my knowledge.

2. Totally separate from the Exodus reader, there's been a thread on  
freeing intermediate pipeline results in ParaView to deal with memory  
exhaustion (subject line "Ongoing Work to Release Unused Memory from  
Pipeline").

The underlying cause of both problems is that there isn't really a way  
to measure memory pressure on modern OSes... you can allocate memory  
and page it in until the kernel decides it's overcommitted and either  
swaps or kills processes or both. Linux has some limited memory  
pressure facilities as part of cpuset but as I understand it, you must  
opt in to constraints on cores your process will be scheduled on and  
it's not clear how to use the information that's available... it  
sounds like a lot of work that is very system-specific (even on  
systems with cpuset it may not be built or enabled). See

   http://www.kernel.org/doc/man-pages/online/pages/man7/cpuset.7.html

for some information. I haven't found anything for Darwin and haven't  
looked for Windows but even if they exist it's unlikely the  
information they provide could be used in the same way -- there would  
have to be some adaptor layer between the OS and VTK's decisions about  
memory.

	David