[Paraview] Building on Titan using ParaViewSuperbuild

[David E DeMarle]

Probably easiest to set them in
CMake/cross_compile/bgp_xlc/ParaViewDefaults.cmake.

That is what defined the initial CMakeCache for ParaView itself (as opposed
to the CMakeCache of the outer super build project).


David E DeMarle
Kitware, Inc.
R&D Engineer
21 Corporate Drive
Clifton Park, NY 12065-8662
Phone: 518-881-4909


On Thu, Sep 5, 2013 at 12:12 PM, Hong Yi <hongyi at renci.org> wrote:

>  Thanks very much for the info, David. Turning off MPI4PY for version
> 3.98 as you suggested works for me to build version 3.98 on Titan also with
> SuperBuild. ****
>
> ** **
>
> I do want to stick with 4.0, but built 3.98 also just in case since my new
> plugin works well on version 3.98. After some further examination, it looks
> like my plugin should work on version 4.0 also. Have put my plugin code
> into 4.0 source directory under both tools and cross subdirectories and
> hope it can be built and work for version 4.0 also.****
>
> ** **
>
> Have spent this morning trying to figure out how to turn on some
> additional flags in superbuild such as my new filter plugin, Fortran
> coprocessing adaptor, and Phasta adaptor, but cannot succeed. Hope to get
> some hint on how to do so. Apparently, passing those flags in by adding
> corresponding –D to configuretools only resulted in CMake warning
> indicating those manually-specified variables were not used. Have tried to
> change some cmake files in superbuild, but does not seem to have any
> effect. Really appreciate any suggestions you can offer here to turn on
> those additional flags which are important for our coprocessing purposes.*
> ***
>
> ** **
>
> Thanks,****
>
> ** **
>
> Hong****
>
> ** **
>
> ** **
>
> *From:* David E DeMarle [mailto:dave.demarle at kitware.com]
> *Sent:* Sunday, September 01, 2013 1:56 PM
>
> *To:* Hong Yi
> *Cc:* Vanmoer, Mark W; paraview at paraview.org
> *Subject:* Re: [Paraview] Building on Titan using ParaViewSuperbuild****
>
> ** **
>
> Between 3.98 and 4.0 the syntax of the option to turn off MPI4PY changes
> like so:****
>
> I recommend staying with 4.0, but try reverting this change if you need
> 3.98 for some reason.****
>
> ** **
>
> -------------- CMake/crosscompile/xk7_gnu/ParaViewDefaults.cmake
> --------------****
>
> index 5835693..68c2bea 100644****
>
> @@ -3,7 +3,8 @@ list(APPEND PARAVIEW_OPTIONS
> "-DBUILD_SHARED_LIBS:BOOL=OFF")****
>
>  list(APPEND PARAVIEW_OPTIONS "-DBUILD_TESTING:BOOL=OFF")****
>
>  list(APPEND PARAVIEW_OPTIONS "-DPARAVIEW_BUILD_QT_GUI:BOOL=OFF")****
>
>  #****
>
> -list(APPEND PARAVIEW_OPTIONS "-DENABLE_MPI4PY:BOOL=OFF")****
>
> +list(APPEND PARAVIEW_OPTIONS "-DPARAVIEW_USE_SYSTEM_MPI4PY:BOOL=ON")****
>
> +#****
>
>  list(APPEND PARAVIEW_OPTIONS "-DPARAVIEW_USE_MPI:BOOL=ON")****
>
>  #/opt/cray/mpt/5.5.5/gni/mpich2-gnu/47/lib/****
>
>  #libmpich.a ****
>
> ** **
>
>
> ****
>
> David E DeMarle
> Kitware, Inc.
> R&D Engineer
> 21 Corporate Drive
> Clifton Park, NY 12065-8662
> Phone: 518-881-4909****
>
> ** **
>
> On Sat, Aug 31, 2013 at 4:10 PM, Hong Yi <hongyi at renci.org> wrote:****
>
> Just a quick update. Like Mark, I can build ParaView 4.0.1 with Superbuild
> successfully without issues using CMake 2.8.11.2 on Titan. However, I
> cannot build Paraview 3.98.1 with Superbuild using the same CMake version.
> I still got the same linking error from mpi4py.MPI.c in the final stage of
> linking after 100% built for the cross stage (see below for details) if I
> built version 3.98.1 using superbuild. Perhaps the latest version of
> Superbuild has some sort of incompatibility with the older version of
> ParaView for xk7_gnu target? Anyway, looks like we have to go with the
> latest ParaView version for in-situ visualization on Titan since superbuild
> does not work with older 3.98.1 version for us, which may very well prove a
> good thing for us in the long run due to a lot of improvement made on
> Catalyst in the latest ParaView version. ****
>
> ** **
>
> Thanks,****
>
> Hong****
>
>  ****
>  ------------------------------
>
> *From:* paraview-bounces at paraview.org [paraview-bounces at paraview.org] on
> behalf of Hong Yi [hongyi at renci.org]
> *Sent:* Friday, August 30, 2013 5:43 PM
> *To:* Vanmoer, Mark W; David E DeMarle
> *Cc:* paraview at paraview.org****
>
>
> *Subject:* Re: [Paraview] Building on Titan using ParaViewSuperbuild****
>
> ** **
>
> Thanks for the info, Mark. Looks like it is sensible for me to try newer
> version of CMake 2.8.11.2 and see how it goes.****
>
>  ****
>
> On a somewhat related question: I am trying to pass in some CMake flags to
> make it build my new filter plugin as well as FortranAdaptor for
> coprocessing (yes, I discovered also that coprocessing/catalyst is built by
> default by superbuild, but FortranAdaptor is turned off by default). I
> tried to pass it in by adding corresponding –D to configuretools, but got
> CMake warning as well indicating those manually-specified variables were
> not used. I am wondering whether I can do it by directly changing
> CMakeCache.txt under paraview/src/paraview-build to force the corresponding
> flags to be on…****
>
>  ****
>
> Thanks,****
>
> Hong****
>
>  ****
>
> *From:* Vanmoer, Mark W [mailto:mvanmoer at illinois.edu]
> *Sent:* Friday, August 30, 2013 5:34 PM
> *To:* Hong Yi; David E DeMarle
> *Cc:* paraview at paraview.org
> *Subject:* RE: [Paraview] Building on Titan using ParaViewSuperbuild****
>
>  ****
>
> Hi Hong,****
>
> I was able to get David’s instructions to work using CMake 2.8.11.2 and
> ParaView 4.0.1. The build process seems to be sensitive to versions.****
>
> Mark ****
>
>  ****
>
> *From:* Hong Yi [mailto:hongyi at renci.org <hongyi at renci.org>]
> *Sent:* Friday, August 30, 2013 3:52 PM
> *To:* David E DeMarle; Vanmoer, Mark W
> *Cc:* paraview at paraview.org
> *Subject:* RE: [Paraview] Building on Titan using ParaViewSuperbuild****
>
>  ****
>
> I tried to follow the instructions and the configuration scripts to build
> ParaView for compute nodes on Titan. It built successfully without issues
> for the TOOLS stage, but when doing final linking for paraview in the CROSS
> stage, I got numerous similar linking error from different lines such as
> the following:
>
> ----------------
> /....../ParaView/ParaViewSuperbuild/cross/paraview/src/paraview/Utilities/mpi4py/src/mpi4py.MPI.c:
> In function '__pyx_pf_6mpi4py_3MPI_4File_54Sync':
> /....../ParaView/ParaViewSuperbuild/cross/paraview/src/paraview/Utilities/mpi4py/src/mpi4py.MPI.c:89682:7:
> error: '_save' undeclared (first use in this function)
> ----------------
>
> The same error message were raised from the same file mpi4py.MPI.c from
> different lines.
>
> I am using CMake version 2.8.10.2 which is provided by Titan, and using
> the ParaView source tree version 3.98 with one additional filter I have
> developed.
>
> Any idea on what could cause the linking error?
>
> Thanks,
> Hong ****
>  ------------------------------
>
> *From:* David E DeMarle [dave.demarle at kitware.com]
> *Sent:* Thursday, August 29, 2013 4:08 PM
> *To:* Vanmoer, Mark W
> *Cc:* Hong Yi; paraview at paraview.org
> *Subject:* Re: [Paraview] Building on Titan using ParaViewSuperbuild****
>
> On Thu, Aug 29, 2013 at 3:51 PM, Vanmoer, Mark W <mvanmoer at illinois.edu>
> wrote:****
>
> So coprocessing will not be built using the below instructions? I would
> have mentioned that, but coprocessing appears to****
>
>  still be part of a regular, non-cross-compile build, so I figured it was
> part of ENABLE_paraview ****
>
>   ****
>
> The coprocessing plugin, which adds things to the GUI to make it easy to
> record coprocessing pipeline setups doesn't need to be turned on since that
> lives in the client only. (It is like python trace or state recording, but
> tailored to recording in-situ setups).****
>
>  ****
>
> Catalyst (the stripped down version of ParaView server that a simulation
> code can link to and use to run those recorded pipelines quickly) is not
> yet an option in ParaViewSuperbuild. To cross compile Catalyst a bit more
> work will be required. It will follow the same plan as how the ParaView
> server is compiled, but I just haven't tried it. When I did cross compile
> Catalyst last year at this time I did the same steps that
> ParaViewSuperbuild's TOOLS and CROSS build passes did, just by hand.****
>
>  Also, for the below configcross.sh, do we need to pass in a CMake
> variable telling it where the tools build dir is located?****
>
>   ****
>
> That should be an option that you can easily set, but it isn't sorry.****
>
>  ****
>
>  CMake/CrossCompilationMacros.cmake assumes it can find it one directory
> up and over like so:****
>
> macro(find_hosttools)****
>
>   set(PARAVIEW_HOSTTOOLS_DIR
> ${CMAKE_BINARY_DIR}/../tools/paraview/src/paraview-build/ CACHE PATH****
>
>     "Location of host built paraview compile tools directory")****
>
>   set(PYTHON_HOST_EXE ${CMAKE_BINARY_DIR}/../tools/install/bin/python
> CACHE PATH****
>
>     "Location of host built python executable")****
>
>   set(PYTHON_HOST_LIBDIR ${CMAKE_BINARY_DIR}/../tools/install/lib CACHE
> PATH****
>
>     "Location of host built python libraries")****
>
>   set(BOOST_HOST_INCLUDEDIR ${CMAKE_BINARY_DIR}/../tools/install/include
> CACHE PATH****
>
>     "Location of host built boost headers")****
>
> endmacro()****
>
>  ****
>
> You could predefine all four of those if you like.****
>
>  ****
>
>  Thanks,****
>
> Mark****
>
>  ****
>
> *From:* David E DeMarle [mailto:dave.demarle at kitware.com]
> *Sent:* Thursday, August 29, 2013 1:41 PM
> *To:* Hong Yi
> *Cc:* Vanmoer, Mark W; paraview at paraview.org****
>
>
> *Subject:* Re: [Paraview] Building on Titan using ParaViewSuperbuild****
>
>  ****
>
> On Thu, Aug 29, 2013 at 2:13 PM, Hong Yi <hongyi at renci.org> wrote:****
>
> Hi David,
>
> I just started to try superbuild on Titan also. I don't see you set
> ENABLE_MPI to be true in your configure script. Could you confirm whether
> ENABLE_MPI needs to be set to TRUE in order for ParaView to run on Titan in
> parallel? Since my purpose is to link our****
>
>  ****
>
> The ENABLE_MPI flag at the Superbuild level is unrelated. It has a purpose
> only when CROSS_BUILD_STAGE=HOST, that is when making ParaView binary
> installers for desktops from Superbuild. ****
>
>  ****
>
> You shouldn't turn it on in the TOOLS or CROSS stages. Instead let the
> CROSS stage use the system installed MPI. It does that by turning
> PARAVIEW_USE_MPI=ON when it configures the ParaView sub-build. See
> CMake/crosscompile/xk7_gnu to see where it does that, and to see the other
> flags it uses.****
>
>  ****
>
>  simulation code (already built statically with CMake on Titan) to
> ParaView CoProcessing libraries (I am using version 3.98.1) for in-situ
> visualization on Titan, so in this case, do I have to set ENABLE_paraview
> to true and do I need to enable OSMesa for ParaView to resort to off-screen
> rendering for in-situ visualization?  ****
>
>   ****
>
> The CROSS stage turns on Python, Mesa and ParaView. Titan's accelerators
> don't really run X11, so Mesa is the only option for rendering there.****
>
>  ****
>
>  Although I can build ParaView from source on Titan login nodes, I am not
> able to run it on compute nodes, so I am starting to try superbuild hoping
> to be able to cross build ParaView libraries to run in-situ visualization
> on Titan.****
>
>   ****
>
> I've cross compiled Catalyst itself before on a bluegene. I did it
> manually before SuperBuild existed. I will see if I can dig up my config
> scripts. Cross compiling Catalyst should be more or less that same thing as
> cross compiling ParaView, but a bit faster and easier because their is less
> code involved.****
>
>  ****
>
>  Thanks,
> Hong****
>  ------------------------------
>
> *From:* paraview-bounces at paraview.org [paraview-bounces at paraview.org] on
> behalf of David E DeMarle [dave.demarle at kitware.com]
> *Sent:* Thursday, August 29, 2013 1:21 PM****
>
>
> *To:* Vanmoer, Mark W
> *Cc:* paraview at paraview.org
> *Subject:* Re: [Paraview] Building on Titan using ParaViewSuperbuild****
>
>  ****
>
> Your tools build is pointing to the compiler wrapper that you normally
> would use to make code for the back end. ****
>
> The tools build should just use plain old local gcc since we only build
> things at that point that run on the login node.****
>
>  ****
>
> Try these setup scripts:****
>
> I source configtools.sh to set up my environment before I build the
> compile tools, and configcross.sh before before I cross compile ParaView.*
> ***
>
>  ****
>
> configtools.sh****
>
> #use my own cmake, system one is too old****
>
> setenv PATH
> /autofs/na4_proj/csc035/demarle/pvdev/titan/cmake-build/bin:${PATH}****
>
> #switch compiler to compile code for front end****
>
> module unload PrgEnv-pgi****
>
> module load gcc****
>
> #configure settings for to build compile tools only****
>
> cmake \****
>
>   -DCROSS_BUILD_STAGE:STRING=TOOLS -Dcross_target:STRING=xk7_gnu \****
>
>   -DCMAKE_BUILD_TYPE:STRING=Release \****
>
>   -DBUILD_TESTING:BOOL=FALSE \****
>
>   -DParaView_FROM_GIT:BOOL=OFF \****
>
>   -DENABLE_paraview:BOOL=TRUE \****
>
>   -DENABLE_boost:BOOL=TRUE \****
>
>   -DENABLE_python:BOOL=TRUE \****
>
>   -DENABLE_portfwd:BOOL=FALSE \****
>
>   ../../ParaViewSuperbuild****
>
>  ****
>
>  ****
>
> then make****
>
>  ****
>
> configcross.sh****
>
> #use my own cmake, system one is too old****
>
> setenv PATH
> /autofs/na4_proj/csc035/demarle/pvdev/titan/cmake-build/bin:${PATH}****
>
> #switch compiler to compile code for back end****
>
> module unload PrgEnv-pgi****
>
> module unload gcc****
>
> module load PrgEnv-gnu****
>
> #not sure why module load wasn't sufficient, but ended up needing to force
> ****
>
> #cmake to choose the right compiler****
>
> setenv CC /opt/cray/xt-asyncpe/5.17/bin/cc****
>
> setenv CXX /opt/cray/xt-asyncpe/5.17/bin/CC****
>
> #configure settings to cross compile python, (mesa - implied), and paraview
> ****
>
> cmake \****
>
>   -DCROSS_BUILD_STAGE:STRING=CROSS -Dcross_target:STRING=xk7_gnu \****
>
>   -DCMAKE_BUILD_TYPE:STRING=Release \****
>
>   -DBUILD_TESTING:BOOL=TRUE \****
>
>   -DParaView_FROM_GIT:BOOL=OFF \****
>
>   -DENABLE_paraview:BOOL=TRUE \****
>
>   -DENABLE_python:BOOL=TRUE \****
>
>   ../../ParaViewSuperbuild****
>
>  ****
>
> then make again****
>
>  ****
>
>
> ****
>
> David E DeMarle
> Kitware, Inc.
> R&D Engineer
> 21 Corporate Drive
> Clifton Park, NY 12065-8662
> Phone: 518-881-4909****
>
>  ****
>
> On Tue, Aug 27, 2013 at 4:26 PM, Vanmoer, Mark W <mvanmoer at illinois.edu>
> wrote:****
>
> Hi, I'm trying to follow the advice on building ParaView on Titan using
> the ParaViewSuperbuild method from an earlier discussion in June.****
>
>  ****
>
> When I run make in the "TOOLS" directory I get the following error:****
>
>  ****
>
> [ 66%] Building CXX object
> Utilities/ProcessXML/CMakeFiles/kwProcessXML.dir/ProcessXML.cxx.o****
>
> Linking CXX executable ../../bin/vtkkwProcessXML-pv4.0****
>
> /usr/bin/ld: attempted static link of dynamic object
> `../../lib/libvtkCommonCore-pv4.0.so.1'****
>
> collect2: error: ld returned 1 exit status****
>
> make[6]: *** [bin/vtkkwProcessXML-pv4.0] Error 1****
>
> make[5]: *** [Utilities/ProcessXML/CMakeFiles/kwProcessXML.dir/all] Error 2
> ****
>
> make[4]: *** [CMakeFiles/pvCompileTools.dir/rule] Error 2****
>
> make[3]: *** [pvCompileTools] Error 2****
>
> CMake Error at
> /ccs/home/vanmoer/builds/superbuild/tools-build/pv-paraview-build.cmake:26
> (message):****
>
>   Failed!!!****
>
>  ****
>
>  ****
>
> make[2]: *** [paraview/src/paraview-stamp/paraview-build] Error 1****
>
>  ****
>
> I don't see a BUILD_SHARED that I can toggle. All of the
> CMAKE_SHARED_LINKER_FLAGS* vars are empty, if those are related. ****
>
>  ****
>
> Any suggestions?****
>
>  ****
>
> Thanks,****
>
> Mark****
>
>  ****
>
>  ****
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at:
> http://paraview.org/Wiki/ParaView
>
> Follow this link to subscribe/unsubscribe:
> http://www.paraview.org/mailman/listinfo/paraview****
>
>  ****
>
>   ****
>
>   ****
>
> ** **
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.paraview.org/pipermail/paraview/attachments/20130905/95da8a4a/attachment-0001.htm>