[CMake] slow CMake performance configuring builds on remote SMB servers

[Philip Lowman]

Bill Hoffman wrote:
>>> Most files cmake writes out are copy if different.  So, cmake writes a
>>> file, then diff's it with the
>>> file that was already there, and if they are different it gets rid of
>>> the original and moves the new
>>> file into place.   This is to avoid too many reloads and other side
>>> effects of changing all the
>>> files all the time.   It would be a lot of work, but I suppose you might
>>> be able to build the whole
>>> tree in some staging area, then move it later.   But, that sounds very
>>> difficult.   It may just be
>>> that running cmake over network drives on windows is slow....
>>>     
>>
>> Hmm.. so CMake writes out a temporary file and then diffs it against the
>> existing file using the operating system and only moves it back into
>> place if it's out-of-date?
>>
>> If I understand this correctly there's a simple optimization that could
>> be used which might dramatically improve performance.  Simply do the
>> diffing within CMake between the "new" file (in memory) and the "old"
>> file (already in memory because it's been read in).  Then only write the
>> file out if it's changed.
>>   
> Sure, and that has been on the todo list for some time, but code wise it 
> is not that easy.  Involves creating
> sub-classes of c++ streams and such.   See the class 
> cmGeneratedFileStream for how it is currently
> done.  If anyone can get a clean implementation that works, I would be 
> happy to put it in CMake.   It should
> only involve "fixing" the class cmGeneratedFileStream.

Bill,

I've started working on this.  My plan is to turn cmGeneratedFileStream 
into an ostringstream and then have the Close() method effectively do 
the comparison and create the ofstream and output if necessary (and if 
"CopyIfDifferent" is true, to maintain existing behavior).

This will only yield a performance improvement on generation of 
VCprojfiles/Makefiles, however.  I'd like to look through the code to 
try to see if I can optimize the "Configure" case, as well, which seems 
to be fairly slow when on remote file shares.

I know part of the problem is that object files are being generated as 
the result of compilation checks which I'm guessing are being generated 
in the build directory.  These could likely be generated in temp space 
on the local hard drive instead.  I know this is only part of the 
Configure phase, however, and it usually only gets run once.. I guess 
I'm just looking for any other ideas people might have for optimizing 
the Configure phase.  It seems to vary between 30 and 60 seconds on our 
fileservers (even after the cache has been fully populated).

-- 
Philip Lowman
Simulation Development Engineer, Modeling and Simulation Technology
General Dynamics Land Systems
http://www.gdls.com