[CMake] Memory usage and configuration time for large project

Tue May 8 16:24:32 EDT 2018

I manage a very large project using CMake that I have done significant restructuring to lower the CMake configure times.  I will say the one thing that surprised me the most is that for one piece of it where we have repeated work done say “for each folder in this subdir build 11 executables with a ton of code generation.”  If I structure the build as such:

One top level CMakeLists.txt and one single line CMakeLists.txt in each subdir containing do_work(${SUBDIR}):
foreach( SUBDIR ${LIST_OF_SUBDIRS} )
   add_subdirectory(${SUBDIR})
endforeach()

Versus:

One top level CMakeLists.txt:
foreach( SUBDIR ${LIST_OF_SUBDIRS} )
   do_work(${SUBDIR})
endforeach()

In short, the decision to do all the work in the same CMakeLists.txt file or do I enter and leave a CMakeLists.txt for each directory causes a dramatic performance and memory difference.

The add_subdirectory method saves many Gigs of RAM and reduces our configure time from 30 minutes to 5 minutes.  Please note I am writing this from memory so my numbers may be a bit off but scale is accurate.

I don’t know if this is applicable to your case but it was the biggest saving in my project and took a long time to figure out.

-Kris

From: CMake [mailto:cmake-bounces at cmake.org] On Behalf Of Isaiah Norton
Sent: Friday, May 04, 2018 7:51 PM
To: Patrick E Gartung <gartung at fnal.gov>
Cc: cmake at cmake.org
Subject: Re: [CMake] Memory usage and configuration time for large project

As one ballpark datapoint: a "superbuild" of 3D Slicer (slicer.org<http://slicer.org>) has a similar object and library count:

macbook-pro:s5nj inorton$ find ./ -name *.o | wc -l
   14127
macbook-pro:s5nj inorton$ find ./ -name *.dylib -or -name *.so | wc -l
    2158

Based on a few quick tests, the aggregate cmake time is probably about 6-8 minutes for the ninja generator, over the multi-hour build (each dependency is configured and built as a separate sub-project via CMake's ExternalProject mechanism) -- with the caveat that each of those cmake runs is doing lengthy checks that should only be done once if your project strictly uses `add_subdirectory`.

As a more concrete point of comparison, building VTK generates 5747 object files, and a clean configure on my 2-core macbook takes about 90s.

Isaiah

On Fri, May 4, 2018 at 2:16 PM, Patrick E Gartung <gartung at fnal.gov<mailto:gartung at fnal.gov>> wrote:
Just to be clear, the memory and time used are just to configure and generate the makefiles or Ninja file. The build itself can take several hours.

On 4/30/18, 4:55 PM, "R0b0t1" <r030t1 at gmail.com<mailto:r030t1 at gmail.com>> wrote:

    On Mon, Apr 30, 2018 at 4:44 PM, Patrick E Gartung <gartung at fnal.gov<mailto:gartung at fnal.gov>> wrote:
    >  Hi,
    >
    > We have a large c/c++/fortran project, CMSSW, that is built with a custom tool, scram.
    >
    > https://github.com/cms-sw/cmssw
    >
    > The project might move to a cmake based build in the future. The scripts to convert to CmakeLists.tx has been written
    >
    > https://github.com/cms-sw/cmssw2cmake
    >
    > Tests show that with the cmake files generated with this script, configuring the project uses up to 1.5GB of ram and takes 11 minutes when using -GMakefiles. Using -GNinja and using the latest cmake, this time can be reduced to 8 minutes.
    >
    > The project builds 14k object files, 2.2k libraries, ~600 binaries, 500 generated source files with links to ~100 external libraries.
    >
    > Is this amount of memory usage and time typical for a project of this size?
    >

    I'm inclined to say "yes" as many builds such as Firefox, its
    supporting libraries, and Chrome all take lots of time and memory.
    Chrome uses Ninja, I might add. But the issue is not CMake itself.
    CMake tends to produce better builds.

    As I am not intimately familiar with your project, I can't make good
    concrete suggestions. You may enjoy reading
    https://news.ycombinator.com/item?id=14733829 and searching for build
    optimization strategies.

    Keep in mind a lot of the blame falls on C++ and Windows, should you
    be using Windows. If you aren't using Windows, then the advice in the
    comments above should still be relevant, and give you something to go
    on.

    Cheers,
         R0b0t1

    > Patrick Gartung
    > Fermilab
    >

--

Powered by www.kitware.com<http://www.kitware.com>

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake

________________________________

IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://cmake.org/pipermail/cmake/attachments/20180508/faab4967/attachment-0001.html>