[CMake] Poor performance of copy_if_different
Robert Dailey
rcdailey.lists at gmail.com
Tue Nov 6 15:06:38 EST 2012
Well I found the problem -- it's quite silly:
add_custom_target( ${target_name}
DEPENDS "${output_bins}"
)
You can't have quotes around ${output_bins}... if I remove the quotes
it works -_-
Now I get the following output (the percentages don't seem to work
though, but at least the copies happen):
Scanning dependencies of target copy_dlls_Debug
[ 0%] Processing changed debug binaries in epixmlrpc
[ 0%] Processing changed debug binaries in ewsmanagedapi
[ 0%] Processing changed debug binaries in icu
[ 0%] Processing changed debug binaries in json
[100%] Processing changed debug binaries in msredist
[100%] Processing changed debug binaries in mssdkutils
[100%] Processing changed debug binaries in notesapi
[100%] Processing changed debug binaries in nspr
[100%] Processing changed debug binaries in nss
[100%] Processing changed debug binaries in openssl
[100%] Processing changed debug binaries in sqlapi
[100%] Processing changed debug binaries in stlport
[100%] Processing changed debug binaries in we70
[100%] Processing changed debug binaries in zlib
[100%] Built target copy_dlls_debug
On Tue, Nov 6, 2012 at 2:02 PM, Robert Dailey <rcdailey.lists at gmail.com> wrote:
> I did some more testing.
>
> If I only make 1 command and depend on that command's output in 1
> target, it executes the command just fine.
>
> If I create 2 commands and add both commands output files as one big
> list of depends in 1 target, the commands fail.
>
> Is this a bug or am I setting up my dependencies incorrectly?
>
> On Tue, Nov 6, 2012 at 1:21 PM, Robert Dailey <rcdailey.lists at gmail.com> wrote:
>> You're right Clinton, I want to leave them alone but for a while they
>> were not really used a lot. I went ahead and removed my logic to set
>> CMAKE_CONFIGURATION_TYPES and it works great now. I'm able to generate
>> my custom targets and commands without any errors. When I execute the
>> target from nmake now, I get a failure:
>>
>> Scanning dependencies of target copy_dlls_Debug
>> NMAKE : fatal error U1073: don't know how to make 'output\bin\.\libdb47d.dll'
>> Stop.
>> NMAKE : fatal error U1077: '"c:\Program Files (x86)\Microsoft Visual
>> Studio 9.0\VC\BIN\nmake.exe"' : return code '0x2'
>> Stop.
>> NMAKE : fatal error U1077: '"c:\Program Files (x86)\Microsoft Visual
>> Studio 9.0\VC\BIN\nmake.exe"' : return code '0x2'
>> Stop.
>> NMAKE : fatal error U1077: '"c:\Program Files (x86)\Microsoft Visual
>> Studio 9.0\VC\BIN\nmake.exe"' : return code '0x2'
>> Stop.
>>
>> Any idea what it is trying to say? Isn't it just supposed to run the
>> copy command to generate the output file? Here is my updated code
>> (below). For each of my third party libraries, I create 1 custom
>> command to copy its DLLs (each third party library can have more than
>> one dll to copy). I then combine all of these custom commands into 1
>> custom target for each configuration (copy_dlls_debug,
>> copy_dlls_release, etc).
>>
>>
>> set( copycmd "${CMAKE_COMMAND}" ARGS "-E" "copy" )
>>
>> if( CMAKE_CONFIGURATION_TYPES )
>> set( configurations ${CMAKE_CONFIGURATION_TYPES} )
>> else()
>> set( configurations ${CMAKE_BUILD_TYPE} )
>> endif()
>>
>> status( 0 "Generating copy_dlls for configurations: ${configurations}" )
>>
>> foreach( config ${configurations} )
>> string( TOUPPER ${config} upper_config )
>>
>> get_property( binarydirs GLOBAL PROPERTY THIRD_PARTY_DIRS_${upper_config} )
>> foreach( dir ${binarydirs} )
>> file( GLOB binaries "${dir}/*${CMAKE_SHARED_LIBRARY_SUFFIX}" )
>>
>> if( WIN32 )
>> file( GLOB pdbs "${dir}/*.pdb" )
>> list( APPEND binaries ${pdbs} )
>> endif()
>>
>> unset( tp_source_bins )
>> unset( tp_output_bins )
>> unset( copy_commands )
>>
>> foreach( bin ${binaries} )
>> get_filename_component( bin_file ${bin} NAME )
>> set( output_bin
>> "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${CMAKE_CFG_INTDIR}/${bin_file}" )
>>
>> list( APPEND tp_source_bins ${bin} )
>> list( APPEND tp_output_bins ${output_bin} )
>>
>> list( APPEND copy_commands
>> COMMAND ${copycmd} "${bin}" "${output_bin}"
>> )
>> endforeach()
>>
>> file( RELATIVE_PATH relative_tp_dir ${THIRD_PARTY_DIR} ${dir} )
>> string( FIND ${relative_tp_dir} "/" slash_pos )
>> string( SUBSTRING ${relative_tp_dir} 0 ${slash_pos} tp_name )
>>
>> add_custom_command(
>> COMMENT "Processing changed binaries in ${tp_name}"
>> OUTPUT ${tp_output_bins}
>> DEPENDS ${tp_source_bins}
>> ${copy_commands}
>> )
>>
>> list( APPEND output_bins ${tp_output_bins} )
>> endforeach()
>>
>> set( target_name copy_dlls_${config} )
>> add_custom_target( ${target_name}
>> COMMENT "Copying ${config} binaries..."
>> DEPENDS "${output_bins}"
>> )
>>
>> list( APPEND copy_dlls_targets ${target_name} )
>> endforeach()
>>
>> On Tue, Nov 6, 2012 at 1:07 PM, Clinton Stimpson <clinton at elemtech.com> wrote:
>>> On Tuesday, November 06, 2012 12:46:01 PM Robert Dailey wrote:
>>>> Is there an assumption here that CMAKE_CONFIGURATION_TYPES will not be
>>>> set for single-configuration generators?
>>>
>>> Yes, that's the assumption. That's how I normally check for single vs. multi
>>> config generators.
>>>
>>>>
>>>> I have some code in my root CMake script that forces only 'debug' and
>>>> 'release' configurations to exist (I didn't want the MinRelSize and
>>>> friends in there). I think this will force the logic that depends on
>>>> CMAKE_CONFIGURATION_TYPES to fail. What would you do in this case?
>>>>
>>>> Here is the code that sets CMAKE_CONFIGURATION_TYPES:
>>>>
>>>> cmake_minimum_required( VERSION 2.8.8 FATAL_ERROR )
>>>>
>>>> set( CMAKE_BUILD_TYPE debug )
>>>> set( CMAKE_CONFIGURATION_TYPES debug release CACHE INTERNAL "" FORCE )
>>>>
>>>> project( gmms )
>>>>
>>>> ... etc ...
>>>
>>>
>>> It doesn't make much sense to me why one could only do a release build with a
>>> mulit-config generator... I normally leave those variables alone and let the
>>> user set them. I find RelWithDebInfo quite useful sometimes.
>>>
>>> But if you had to, perhaps you could do this, after your project(gmms):
>>>
>>> if(CMAKE_CONFIGURATION_TYPES)
>>> set( CMAKE_CONFIGURATION_TYPES debug release CACHE INTERNAL "" FORCE )
>>> endif()
>>>
>>> Or some code to strip out types from CMAKE_CONFIGURATION_TYPES except for
>>> debug and release. So if it was empty to begin with, it could stay empty.
>>>
>>>>
>>>> On Tue, Nov 6, 2012 at 12:41 PM, Clinton Stimpson <clinton at elemtech.com>
>>> wrote:
>>>> > You could base your logic on CMAKE_CONFIGURATION_TYPES and CMAKE_BUILD
>>>> > TYPE.
>>>> >
>>>> > For example:
>>>> >
>>>> > if(CMAKE_CONFIGURATION_TYPES)
>>>> >
>>>> > foreach(type ${CMAKE_CONFIGURATION_TYPES})
>>>> >
>>>> > make_copy_rules(${type})
>>>> >
>>>> > endforeach()
>>>> >
>>>> > else()
>>>> >
>>>> > make_copy_rules(${CMAKE_BUILD_TYPE})
>>>> >
>>>> > endif()
>>>> >
>>>> > On Tuesday, November 06, 2012 12:34:22 PM Robert Dailey wrote:
>>>> >> I figured out the problem.
>>>> >>
>>>> >> On single-configuration generators (like NMake, which I'm testing now)
>>>> >> both release and debug configurations have the same binary output
>>>> >> directory. Since I'm creating 1 custom target (and corresponding
>>>> >> commands) per configuration, the 2nd configuration has the same
>>>> >> outputs as the 1st, so the 2nd target fails because I"m adding
>>>> >> different rules to the same output file. This isn't a problem with
>>>> >> Visual Studio generators because my outputs go to Debug and Release.
>>>> >>
>>>> >> Know any good workarounds for this?
>>>> >>
>>>> >> On Tue, Nov 6, 2012 at 11:54 AM, Clinton Stimpson <clinton at elemtech.com>
>>>> >
>>>> > wrote:
>>>> >> > Try this:
>>>> >> > add_custom_command(
>>>> >> >
>>>> >> > OUTPUT ${bin_outputs}
>>>> >> > COMMAND ${copy_commands}
>>>> >> > DEPENDS ....
>>>> >> >
>>>> >> > )
>>>> >> >
>>>> >> > On Tuesday, November 06, 2012 11:50:54 AM Robert Dailey wrote:
>>>> >> >> Thanks Clinton,
>>>> >> >>
>>>> >> >> Here is what I have so far:
>>>> >> >>
>>>> >> >>
>>>> >> >> project( copy_dlls )
>>>> >> >>
>>>> >> >> set( copycmd "${CMAKE_COMMAND}" ARGS "-E" "copy_if_different" )
>>>> >> >>
>>>> >> >> set( configurations debug release )
>>>> >> >> foreach( config ${configurations} )
>>>> >> >>
>>>> >> >> string( TOUPPER ${config} upper_config )
>>>> >> >>
>>>> >> >> get_property( binarydirs GLOBAL PROPERTY
>>>> >> >>
>>>> >> >> THIRD_PARTY_DIRS_${upper_config} ) foreach( dir ${binarydirs} )
>>>> >> >>
>>>> >> >> file( GLOB binaries "${dir}/*${CMAKE_SHARED_LIBRARY_SUFFIX}" )
>>>> >> >>
>>>> >> >> if( WIN32 AND config STREQUAL "debug" )
>>>> >> >>
>>>> >> >> file( GLOB pdbs "${dir}/*.pdb" )
>>>> >> >> list( APPEND binaries ${pdbs} )
>>>> >> >>
>>>> >> >> endif()
>>>> >> >>
>>>> >> >> foreach( bin ${binaries} )
>>>> >> >>
>>>> >> >> get_filename_component( bin_file ${bin} NAME )
>>>> >> >>
>>>> >> >> list( APPEND bin_outputs ${bin} )
>>>> >> >> list( APPEND copy_commands
>>>> >> >>
>>>> >> >> COMMAND ${copycmd} "${bin}"
>>>> >> >>
>>>> >> >> "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${CMAKE_CFG_INTDIR}/${bin_file}"
>>>> >> >>
>>>> >> >> )
>>>> >> >>
>>>> >> >> endforeach()
>>>> >> >>
>>>> >> >> endforeach()
>>>> >> >>
>>>> >> >> add_custom_command(
>>>> >> >>
>>>> >> >> OUTPUT ${bin_outputs}
>>>> >> >> ${copy_commands}
>>>> >> >>
>>>> >> >> )
>>>> >> >>
>>>> >> >> add_custom_target( copy_dlls_${config}
>>>> >> >>
>>>> >> >> COMMENT "Copying ${config} binaries..."
>>>> >> >> DEPENDS ${bin_outputs}
>>>> >> >>
>>>> >> >> )
>>>> >> >>
>>>> >> >> endforeach()
>>>> >> >>
>>>> >> >>
>>>> >> >> This doesn't seem to work though, CMake tells me:
>>>> >> >>
>>>> >> >> CMake Error: Attempt to add a custom rule to output
>>>> >> >> "C:/Work/rdailey-hp/dpd-cmake/build-nmake-vc9/third_party/bdb/4.7.25/v
>>>> >> >> c9s
>>>> >> >> p1/ debug/bin/libdb47d.dll.rule" which already has a custom rule.
>>>> >> >>
>>>> >> >> I'm trying to stuff all the copy commands and output files into 1
>>>> >> >> custom command, and wrap that with a custom target, so I can make
>>>> >> >> other targets depend on the custom target.
>>>> >> >>
>>>> >> >> On Tue, Nov 6, 2012 at 11:44 AM, Clinton Stimpson
>>>> >> >> <clinton at elemtech.com>
>>>> >> >
>>>> >> > wrote:
>>>> >> >> > add_custom_command(
>>>> >> >> >
>>>> >> >> > OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/my.dll
>>>> >> >> > COMMAND cmake -E copy ${CMAKE_CURRENT_SOURCE_DIR}/my.dll
>>>> >> >> >
>>>> >> >> > ${CMAKE_CURRENT_BINARY_DIR}/my.dll
>>>> >> >> >
>>>> >> >> > DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/my.dll
>>>> >> >> > )
>>>> >> >> >
>>>> >> >> > add_custom_target(copy_dll ALL DEPENDS
>>>> >> >> > ${CMAKE_CURRENT_BINARY_DIR}/my.dll)
>>>> >> >> >
>>>> >> >> > Whether you use 1 custom command per dll, or 1 for all dlls, is up
>>>> >> >> > to
>>>> >> >> > you.
>>>> >> >> >
>>>> >> >> > If they usually change together, then you could have fewer custom
>>>> >> >> >
>>>> >> >> > commands. In that case, the OUTPUT would have multiple files and
>>>> >> >> > both
>>>> >> >> > DEPENDS would have multiple files.
>>>> >> >> >
>>>> >> >> > Another alternative is to use add_custom_command(TARGET POST_BUILD
>>>> >> >> > ...)
>>>> >> >> > if
>>>> >> >> > these are dlls that you compile yourself and need to copy after the
>>>> >> >> > build
>>>> >> >> > step.
>>>> >> >> >
>>>> >> >> > On Tuesday, November 06, 2012 11:23:03 AM Robert Dailey wrote:
>>>> >> >> >> Also, is it safe to add 1 custom command for each DLL copy? Or
>>>> >> >> >> would
>>>> >> >> >> you somehow only use 1 custom command and copy them all with it?
>>>> >> >> >> Also
>>>> >> >> >> would I add the custom commands to the custom target?
>>>> >> >> >>
>>>> >> >> >> On Tue, Nov 6, 2012 at 11:11 AM, Clinton Stimpson
>>>> >> >> >> <clinton at elemtech.com>
>>>> >> >> >
>>>> >> >> > wrote:
>>>> >> >> >> > On Tuesday, November 06, 2012 11:06:45 AM Robert Dailey wrote:
>>>> >> >> >> >> I use ${CMAKE_COMMAND} -E copy_if_different to copy DLL files to
>>>> >> >> >> >> my
>>>> >> >> >> >> binary output directory. The custom target runs this command
>>>> >> >> >> >> about
>>>> >> >> >> >> 50-100 times (for that many files).
>>>> >> >> >> >>
>>>> >> >> >> >> I notice that when all files are already copied, the commands
>>>> >> >> >> >> still
>>>> >> >> >> >> run extremely slowly. It takes just as long to copy all the
>>>> >> >> >> >> files
>>>> >> >> >> >> as
>>>> >> >> >> >> it does to copy none of them.
>>>> >> >> >> >>
>>>> >> >> >> >> I used the Sysinternals tool called Process Monitor to see what
>>>> >> >> >> >> CMake
>>>> >> >> >> >> is doing as it runs copy_if_different. It's opening the full
>>>> >> >> >> >> file
>>>> >> >> >> >> and
>>>> >> >> >> >> seems to be comparing actual contents instead of something
>>>> >> >> >> >> simple,
>>>> >> >> >> >> such as timestamp.
>>>> >> >> >> >>
>>>> >> >> >> >> I do not need such a thorough check, I simply want the check to
>>>> >> >> >> >> see
>>>> >> >> >> >> which timestamp is higher and copy if the source is newer than
>>>> >> >> >> >> the
>>>> >> >> >> >> target.
>>>> >> >> >> >>
>>>> >> >> >> >> Any reason why copy_if_different is so slow? Is my assumption
>>>> >> >> >> >> correct?
>>>> >> >> >> >> How can I make it faster?
>>>> >> >> >> >
>>>> >> >> >> > How about using plain "cmake -E copy ..." and rely on the
>>>> >> >> >> > timestamp
>>>> >> >> >> > check
>>>> >> >> >> > done by a custom command (add_custom_command()).
>>>> >> >> >> > You need to make sure the input/ouput parts of the custom command
>>>> >> >> >> > are
>>>> >> >> >> > set
>>>> >> >> >> > correctly so it can do a timestamp check.
>>>> >> >> >> >
>>>> >> >> >> > --
>>>> >> >> >> > Clinton Stimpson
>>>> >> >> >> > Elemental Technologies, Inc
>>>> >> >> >> > Computational Simulation Software, LLC
>>>> >> >> >> > www.csimsoft.com
>>>> >> >> >> > --
>>>> >> >> >> >
>>>> >> >> >> > Powered by www.kitware.com
>>>> >> >> >> >
>>>> >> >> >> > Visit other Kitware open-source projects at
>>>> >> >> >> > http://www.kitware.com/opensource/opensource.html
>>>> >> >> >> >
>>>> >> >> >> > Please keep messages on-topic and check the CMake FAQ at:
>>>> >> >> >> > http://www.cmake.org/Wiki/CMake_FAQ
>>>> >> >> >> >
>>>> >> >> >> > Follow this link to subscribe/unsubscribe:
>>>> >> >> >> > http://www.cmake.org/mailman/listinfo/cmake
>>>> >> >> >
>>>> >> >> > --
>>>> >> >> > Clinton Stimpson
>>>> >> >> > Elemental Technologies, Inc
>>>> >> >> > Computational Simulation Software, LLC
>>>> >> >> > www.csimsoft.com
>>>> >> >> > --
>>>> >> >> >
>>>> >> >> > Powered by www.kitware.com
>>>> >> >> >
>>>> >> >> > Visit other Kitware open-source projects at
>>>> >> >> > http://www.kitware.com/opensource/opensource.html
>>>> >> >> >
>>>> >> >> > Please keep messages on-topic and check the CMake FAQ at:
>>>> >> >> > http://www.cmake.org/Wiki/CMake_FAQ
>>>> >> >> >
>>>> >> >> > Follow this link to subscribe/unsubscribe:
>>>> >> >> > http://www.cmake.org/mailman/listinfo/cmake
>>>> >> >
>>>> >> > --
>>>> >> > Clinton Stimpson
>>>> >> > Elemental Technologies, Inc
>>>> >> > Computational Simulation Software, LLC
>>>> >> > www.csimsoft.com
>>>> >> > --
>>>> >> >
>>>> >> > Powered by www.kitware.com
>>>> >> >
>>>> >> > Visit other Kitware open-source projects at
>>>> >> > http://www.kitware.com/opensource/opensource.html
>>>> >> >
>>>> >> > Please keep messages on-topic and check the CMake FAQ at:
>>>> >> > http://www.cmake.org/Wiki/CMake_FAQ
>>>> >> >
>>>> >> > Follow this link to subscribe/unsubscribe:
>>>> >> > http://www.cmake.org/mailman/listinfo/cmake
>>>> >
>>>> > --
>>>> > Clinton Stimpson
>>>> > Elemental Technologies, Inc
>>>> > Computational Simulation Software, LLC
>>>> > www.csimsoft.com
>>>> > --
>>>> >
>>>> > Powered by www.kitware.com
>>>> >
>>>> > Visit other Kitware open-source projects at
>>>> > http://www.kitware.com/opensource/opensource.html
>>>> >
>>>> > Please keep messages on-topic and check the CMake FAQ at:
>>>> > http://www.cmake.org/Wiki/CMake_FAQ
>>>> >
>>>> > Follow this link to subscribe/unsubscribe:
>>>> > http://www.cmake.org/mailman/listinfo/cmake
>>> --
>>> Clinton Stimpson
>>> Elemental Technologies, Inc
>>> Computational Simulation Software, LLC
>>> www.csimsoft.com
>>> --
>>>
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>>
>>> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://www.cmake.org/mailman/listinfo/cmake
More information about the CMake
mailing list