<div dir="ltr">2. It looks as though CUDA_LINK_SEPARABLE_COMPILATION_OBJECTS is only looking at the configuration specific flags. You can add the flag specifically to all your configs (e.g. CMAKE_CXX_FLAGS_DEBUG) or you could try adding these lines of code in your FindCUDA.cmake file somewhere in CUDA_LINK_SEPARABLE_COMPILATION_OBJECTS after set(flags). It looks as though this was overlooked (sorry).<div><br></div><div><div> set(important_host_flags)</div><div> _cuda_get_important_host_flags(important_host_flags ${CMAKE_${CUDA_C_OR_CXX}_FLAGS})</div><div> foreach(f ${important_host_flags})</div><div> list(APPEND flags -Xcompiler ${f})</div><div> endforeach()</div></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 5, 2015 at 3:43 PM, Irwin Zaid <span dir="ltr"><<a href="mailto:irwin.zaid@physics.ox.ac.uk" target="_blank">irwin.zaid@physics.ox.ac.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Alright, this is a lot of progress!<br>
<br>
1) We are using Makefiles. I agree with you about the dependency graph, so I'll try and sort that out. I'll let you know what the result is.<br>
<br>
2) I just checked and, indeed, the *_intermediate_link.o file is not being passed -fPIC. Is this our problem? What is the correct fix?<br>
<br>
Irwin<br>
<br>
James Bigler wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
<br>
<br>
On Mon, Jan 5, 2015 at 1:57 PM, Irwin Zaid <<a href="mailto:irwin.zaid@physics.ox.ac.uk" target="_blank">irwin.zaid@physics.ox.ac.uk</a><br></span><span class="">
<mailto:<a href="mailto:irwin.zaid@physics.ox.ac.uk" target="_blank">irwin.zaid@physics.ox.<u></u>ac.uk</a>>> wrote:<br>
<br>
Hi James,<br>
<br>
Thanks for the quick reply! As I mentioned, we've hit two issues.<br>
The first is the project dependencies one, which I'll try and<br>
describe more a bit below. I'm not a CMake expert, so please bear<br>
with me.<br>
<br>
The second is what I've put under 2).<br>
<br>
The only CMake build dependency changes when doing separable<br>
compilation<br></span>
are found in CUDA_LINK_SEPARABLE___<u></u>COMPILATION_OBJECTS.<span class=""><br>
Basically what<br>
this does is create a new rule to build an intermediate link<br>
file. For<br>
everything but some versions of MSVC generators it adds a custom<br>
command<br>
to generate this intermediate link file. The other case adds a<br>
custom<br>
command that runs as a PRE_LINK command to generate the object<br>
file (the<br>
reasons for this is a bug in VS custom command dependency<br>
scanning), but<br>
this should happen during link phase and not compile phase.<br>
<br>
Nothing in here should change what happens before the target is<br>
built,<br>
though.<br>
<br>
<br>
1) Okay, I understand that, but I do think we saw a different<br>
behaviour when we switched to separable compilation. Let me describe<br>
what we are doing.<br>
<br>
We generate part of our library from a simple program (call the<br>
simple program 'gen', which generates a source file 'gen.hpp') that<br>
we want to execute before compiling our library (call our library<br>
'main'). We set this up with the following:<br>
<br>
- add_executable(...) is called for 'gen'<br>
- add_custom_command(...) sets up a command that executes 'gen'<br></span>
- set_source_files_properties(..<u></u>__.) is called to set 'gen.hpp' as<span class=""><br>
having the PROPERTY of GENERATED<br>
- add_dependencies(main gen) is called to establish 'main' depends<br>
on 'gen'<br>
<br>
So far, this has only failed for CUDA with separable compilation.<br>
(It has worked for all of our other configurations. including CUDA<br>
without separable compilation.)<br>
<br>
Have we done something wrong? Is there some additional information<br>
we can look at to figure out what's going on?<br>
<br>
<br></span><span class="">
What kind of generator are you using (e.g. makefile)?<br>
<br>
Here's what I'm thinking might be the problem, though I'm not sure why<br>
it would have worked without CUDA_SEPARABLE_COMPILATION.<br>
<br>
There's a dependency between gen and gen.hpp (encoded in the call to<br>
add_custom_command(OUTPUTS gen.hpp COMMAND gen)).<br>
There's a dependency between main and gen (can't start building main<br>
until gen is built).<br>
There's a dependency between gen.hpp and main (gen.hpp is an input<br>
source to main, so it needs to be built as part of main).<br>
<br>
What I don't see is a dependency between gen.hpp and all the cuda<br>
sources that might use it as input. So from a dependency graph<br>
standpoint a makefile (if one is being used in this case) is entirely<br>
free to start compiling the CUDA code once the gen target has been<br>
satisfied.<br>
<br>
What you need is another target that builds the gen.hpp file which can<br>
be forced to run before main starts to build. There are more than one<br>
way to do this, and I'm not sure what the best option is, but you might<br>
try this:<br>
<br>
add_custom_command(OUTPUTS gen.hpp ....)<br>
add_custom_target(make_gen_hpp DEPENDS gen.hpp)<br>
add_dependency(main make_gen_hpp)<br>
<br></span><span class="">
2) A second problem we've run into is an error when trying to link a<br>
CUDA shared library with separable compilation. This is specifically<br>
a Linux problem, on Mac it is fine. A static library is also fine,<br>
working for both Linux and Mac.<br>
<br>
The particular error is "relocation R_X86_64_32S against<br>
`__nv_module_id' can not be used when making a shared object;<br>
recompile with -fPIC". However, we are already compiling with -fPIC.<br>
I can confirm that -fPIC appears to be passed to both the host<br>
compiler for non-CUDA source and via -Xcompiler -fPIC for CUDA source.<br>
<br>
This error occurs when trying to link all the different object files<br>
together of our library.<br>
<br>
Do you have any idea of what this could be?<br>
<br>
<br></span><span class="">
I'm not sure which object file wasn't compiled with -fPIC, but I would<br>
suspect it might be the intermediate link object file. FindCUDA is<br>
supposed to pass this flag along (see<br>
function(_cuda_get_important_<u></u>host_flags)), but you might want to verify<br>
this with a 'make VERBOSE=1' and look for<br>
<target_name>_intermediate_<u></u>link.o (substitute your target name in or<br>
leave it out of the search string).<br>
<br>
James<br>
</span></blockquote>
</blockquote></div><br></div>