[Paraview] Building on Titan using ParaViewSuperbuild

[David E DeMarle]

Between 3.98 and 4.0 the syntax of the option to turn off MPI4PY changes
like so:
I recommend staying with 4.0, but try reverting this change if you need
3.98 for some reason.

-------------- CMake/crosscompile/xk7_gnu/ParaViewDefaults.cmake
--------------
index 5835693..68c2bea 100644
@@ -3,7 +3,8 @@ list(APPEND PARAVIEW_OPTIONS "-DBUILD_SHARED_LIBS:BOOL=OFF")
 list(APPEND PARAVIEW_OPTIONS "-DBUILD_TESTING:BOOL=OFF")
 list(APPEND PARAVIEW_OPTIONS "-DPARAVIEW_BUILD_QT_GUI:BOOL=OFF")
 #
-list(APPEND PARAVIEW_OPTIONS "-DENABLE_MPI4PY:BOOL=OFF")
+list(APPEND PARAVIEW_OPTIONS "-DPARAVIEW_USE_SYSTEM_MPI4PY:BOOL=ON")
+#
 list(APPEND PARAVIEW_OPTIONS "-DPARAVIEW_USE_MPI:BOOL=ON")
 #/opt/cray/mpt/5.5.5/gni/mpich2-gnu/47/lib/
 #libmpich.a


David E DeMarle
Kitware, Inc.
R&D Engineer
21 Corporate Drive
Clifton Park, NY 12065-8662
Phone: 518-881-4909


On Sat, Aug 31, 2013 at 4:10 PM, Hong Yi <hongyi at renci.org> wrote:

>  Just a quick update. Like Mark, I can build ParaView 4.0.1 with
> Superbuild successfully without issues using CMake 2.8.11.2 on Titan.
> However, I cannot build Paraview 3.98.1 with Superbuild using the same
> CMake version. I still got the same linking error from mpi4py.MPI.c in
> the final stage of linking after 100% built for the cross stage (see below
> for details) if I built version 3.98.1 using superbuild. Perhaps the latest
> version of Superbuild has some sort of incompatibility with the older
> version of ParaView for xk7_gnu target? Anyway, looks like we have to go
> with the latest ParaView version for in-situ visualization on Titan since
> superbuild does not work with older 3.98.1 version for us, which may very
> well prove a good thing for us in the long run due to a lot of improvement
> made on Catalyst in the latest ParaView version.
>
>  Thanks,
> Hong
>
>  ------------------------------
> *From:* paraview-bounces at paraview.org [paraview-bounces at paraview.org] on
> behalf of Hong Yi [hongyi at renci.org]
> *Sent:* Friday, August 30, 2013 5:43 PM
> *To:* Vanmoer, Mark W; David E DeMarle
> *Cc:* paraview at paraview.org
>
> *Subject:* Re: [Paraview] Building on Titan using ParaViewSuperbuild
>
>   Thanks for the info, Mark. Looks like it is sensible for me to try
> newer version of CMake 2.8.11.2 and see how it goes.
>
>
>
> On a somewhat related question: I am trying to pass in some CMake flags to
> make it build my new filter plugin as well as FortranAdaptor for
> coprocessing (yes, I discovered also that coprocessing/catalyst is built by
> default by superbuild, but FortranAdaptor is turned off by default). I
> tried to pass it in by adding corresponding –D to configuretools, but got
> CMake warning as well indicating those manually-specified variables were
> not used. I am wondering whether I can do it by directly changing
> CMakeCache.txt under paraview/src/paraview-build to force the corresponding
> flags to be on…
>
>
>
> Thanks,
>
> Hong
>
>
>
> *From:* Vanmoer, Mark W [mailto:mvanmoer at illinois.edu]
> *Sent:* Friday, August 30, 2013 5:34 PM
> *To:* Hong Yi; David E DeMarle
> *Cc:* paraview at paraview.org
> *Subject:* RE: [Paraview] Building on Titan using ParaViewSuperbuild
>
>
>
> Hi Hong,
>
> I was able to get David’s instructions to work using CMake 2.8.11.2 and
> ParaView 4.0.1. The build process seems to be sensitive to versions.
>
> Mark
>
>
>
> *From:* Hong Yi [mailto:hongyi at renci.org <hongyi at renci.org>]
> *Sent:* Friday, August 30, 2013 3:52 PM
> *To:* David E DeMarle; Vanmoer, Mark W
> *Cc:* paraview at paraview.org
> *Subject:* RE: [Paraview] Building on Titan using ParaViewSuperbuild
>
>
>
> I tried to follow the instructions and the configuration scripts to build
> ParaView for compute nodes on Titan. It built successfully without issues
> for the TOOLS stage, but when doing final linking for paraview in the CROSS
> stage, I got numerous similar linking error from different lines such as
> the following:
>
> ----------------
> /....../ParaView/ParaViewSuperbuild/cross/paraview/src/paraview/Utilities/mpi4py/src/mpi4py.MPI.c:
> In function '__pyx_pf_6mpi4py_3MPI_4File_54Sync':
> /....../ParaView/ParaViewSuperbuild/cross/paraview/src/paraview/Utilities/mpi4py/src/mpi4py.MPI.c:89682:7:
> error: '_save' undeclared (first use in this function)
> ----------------
>
> The same error message were raised from the same file mpi4py.MPI.c from
> different lines.
>
> I am using CMake version 2.8.10.2 which is provided by Titan, and using
> the ParaView source tree version 3.98 with one additional filter I have
> developed.
>
> Any idea on what could cause the linking error?
>
> Thanks,
> Hong
>  ------------------------------
>
> *From:* David E DeMarle [dave.demarle at kitware.com]
> *Sent:* Thursday, August 29, 2013 4:08 PM
> *To:* Vanmoer, Mark W
> *Cc:* Hong Yi; paraview at paraview.org
> *Subject:* Re: [Paraview] Building on Titan using ParaViewSuperbuild
>
> On Thu, Aug 29, 2013 at 3:51 PM, Vanmoer, Mark W <mvanmoer at illinois.edu>
> wrote:
>
> So coprocessing will not be built using the below instructions? I would
> have mentioned that, but coprocessing appears to
>
>  still be part of a regular, non-cross-compile build, so I figured it was
> part of ENABLE_paraview
>
>
>
> The coprocessing plugin, which adds things to the GUI to make it easy to
> record coprocessing pipeline setups doesn't need to be turned on since that
> lives in the client only. (It is like python trace or state recording, but
> tailored to recording in-situ setups).
>
>
>
> Catalyst (the stripped down version of ParaView server that a simulation
> code can link to and use to run those recorded pipelines quickly) is not
> yet an option in ParaViewSuperbuild. To cross compile Catalyst a bit more
> work will be required. It will follow the same plan as how the ParaView
> server is compiled, but I just haven't tried it. When I did cross compile
> Catalyst last year at this time I did the same steps that
> ParaViewSuperbuild's TOOLS and CROSS build passes did, just by hand.
>
>  Also, for the below configcross.sh, do we need to pass in a CMake
> variable telling it where the tools build dir is located?
>
>
>
> That should be an option that you can easily set, but it isn't sorry.
>
>
>
>  CMake/CrossCompilationMacros.cmake assumes it can find it one directory
> up and over like so:
>
> macro(find_hosttools)
>
>   set(PARAVIEW_HOSTTOOLS_DIR
> ${CMAKE_BINARY_DIR}/../tools/paraview/src/paraview-build/ CACHE PATH
>
>     "Location of host built paraview compile tools directory")
>
>   set(PYTHON_HOST_EXE ${CMAKE_BINARY_DIR}/../tools/install/bin/python
> CACHE PATH
>
>     "Location of host built python executable")
>
>   set(PYTHON_HOST_LIBDIR ${CMAKE_BINARY_DIR}/../tools/install/lib CACHE
> PATH
>
>     "Location of host built python libraries")
>
>   set(BOOST_HOST_INCLUDEDIR ${CMAKE_BINARY_DIR}/../tools/install/include
> CACHE PATH
>
>     "Location of host built boost headers")
>
> endmacro()
>
>
>
> You could predefine all four of those if you like.
>
>
>
>  Thanks,
>
> Mark
>
>
>
> *From:* David E DeMarle [mailto:dave.demarle at kitware.com]
> *Sent:* Thursday, August 29, 2013 1:41 PM
> *To:* Hong Yi
> *Cc:* Vanmoer, Mark W; paraview at paraview.org
>
>
> *Subject:* Re: [Paraview] Building on Titan using ParaViewSuperbuild
>
>
>
> On Thu, Aug 29, 2013 at 2:13 PM, Hong Yi <hongyi at renci.org> wrote:
>
> Hi David,
>
> I just started to try superbuild on Titan also. I don't see you set
> ENABLE_MPI to be true in your configure script. Could you confirm whether
> ENABLE_MPI needs to be set to TRUE in order for ParaView to run on Titan in
> parallel? Since my purpose is to link our
>
>
>
> The ENABLE_MPI flag at the Superbuild level is unrelated. It has a purpose
> only when CROSS_BUILD_STAGE=HOST, that is when making ParaView binary
> installers for desktops from Superbuild.
>
>
>
> You shouldn't turn it on in the TOOLS or CROSS stages. Instead let the
> CROSS stage use the system installed MPI. It does that by turning
> PARAVIEW_USE_MPI=ON when it configures the ParaView sub-build. See
> CMake/crosscompile/xk7_gnu to see where it does that, and to see the other
> flags it uses.
>
>
>
>  simulation code (already built statically with CMake on Titan) to
> ParaView CoProcessing libraries (I am using version 3.98.1) for in-situ
> visualization on Titan, so in this case, do I have to set ENABLE_paraview
> to true and do I need to enable OSMesa for ParaView to resort to off-screen
> rendering for in-situ visualization?
>
>
>
> The CROSS stage turns on Python, Mesa and ParaView. Titan's accelerators
> don't really run X11, so Mesa is the only option for rendering there.
>
>
>
>  Although I can build ParaView from source on Titan login nodes, I am not
> able to run it on compute nodes, so I am starting to try superbuild hoping
> to be able to cross build ParaView libraries to run in-situ visualization
> on Titan.
>
>
>
> I've cross compiled Catalyst itself before on a bluegene. I did it
> manually before SuperBuild existed. I will see if I can dig up my config
> scripts. Cross compiling Catalyst should be more or less that same thing as
> cross compiling ParaView, but a bit faster and easier because their is less
> code involved.
>
>
>
>  Thanks,
> Hong
>  ------------------------------
>
> *From:* paraview-bounces at paraview.org [paraview-bounces at paraview.org] on
> behalf of David E DeMarle [dave.demarle at kitware.com]
> *Sent:* Thursday, August 29, 2013 1:21 PM
>
>
> *To:* Vanmoer, Mark W
> *Cc:* paraview at paraview.org
> *Subject:* Re: [Paraview] Building on Titan using ParaViewSuperbuild
>
>
>
> Your tools build is pointing to the compiler wrapper that you normally
> would use to make code for the back end.
>
> The tools build should just use plain old local gcc since we only build
> things at that point that run on the login node.
>
>
>
> Try these setup scripts:
>
> I source configtools.sh to set up my environment before I build the
> compile tools, and configcross.sh before before I cross compile ParaView.
>
>
>
> configtools.sh
>
> #use my own cmake, system one is too old
>
> setenv PATH
> /autofs/na4_proj/csc035/demarle/pvdev/titan/cmake-build/bin:${PATH}
>
> #switch compiler to compile code for front end
>
> module unload PrgEnv-pgi
>
> module load gcc
>
> #configure settings for to build compile tools only
>
> cmake \
>
>   -DCROSS_BUILD_STAGE:STRING=TOOLS -Dcross_target:STRING=xk7_gnu \
>
>   -DCMAKE_BUILD_TYPE:STRING=Release \
>
>   -DBUILD_TESTING:BOOL=FALSE \
>
>   -DParaView_FROM_GIT:BOOL=OFF \
>
>   -DENABLE_paraview:BOOL=TRUE \
>
>   -DENABLE_boost:BOOL=TRUE \
>
>   -DENABLE_python:BOOL=TRUE \
>
>   -DENABLE_portfwd:BOOL=FALSE \
>
>   ../../ParaViewSuperbuild
>
>
>
>
>
> then make
>
>
>
> configcross.sh
>
> #use my own cmake, system one is too old
>
> setenv PATH
> /autofs/na4_proj/csc035/demarle/pvdev/titan/cmake-build/bin:${PATH}
>
> #switch compiler to compile code for back end
>
> module unload PrgEnv-pgi
>
> module unload gcc
>
> module load PrgEnv-gnu
>
> #not sure why module load wasn't sufficient, but ended up needing to force
>
> #cmake to choose the right compiler
>
> setenv CC /opt/cray/xt-asyncpe/5.17/bin/cc
>
> setenv CXX /opt/cray/xt-asyncpe/5.17/bin/CC
>
> #configure settings to cross compile python, (mesa - implied), and paraview
>
> cmake \
>
>   -DCROSS_BUILD_STAGE:STRING=CROSS -Dcross_target:STRING=xk7_gnu \
>
>   -DCMAKE_BUILD_TYPE:STRING=Release \
>
>   -DBUILD_TESTING:BOOL=TRUE \
>
>   -DParaView_FROM_GIT:BOOL=OFF \
>
>   -DENABLE_paraview:BOOL=TRUE \
>
>   -DENABLE_python:BOOL=TRUE \
>
>   ../../ParaViewSuperbuild
>
>
>
> then make again
>
>
>
>
>  David E DeMarle
> Kitware, Inc.
> R&D Engineer
> 21 Corporate Drive
> Clifton Park, NY 12065-8662
> Phone: 518-881-4909
>
>
>
> On Tue, Aug 27, 2013 at 4:26 PM, Vanmoer, Mark W <mvanmoer at illinois.edu>
> wrote:
>
> Hi, I'm trying to follow the advice on building ParaView on Titan using
> the ParaViewSuperbuild method from an earlier discussion in June.
>
>
>
> When I run make in the "TOOLS" directory I get the following error:
>
>
>
> [ 66%] Building CXX object
> Utilities/ProcessXML/CMakeFiles/kwProcessXML.dir/ProcessXML.cxx.o
>
> Linking CXX executable ../../bin/vtkkwProcessXML-pv4.0
>
> /usr/bin/ld: attempted static link of dynamic object
> `../../lib/libvtkCommonCore-pv4.0.so.1'
>
> collect2: error: ld returned 1 exit status
>
> make[6]: *** [bin/vtkkwProcessXML-pv4.0] Error 1
>
> make[5]: *** [Utilities/ProcessXML/CMakeFiles/kwProcessXML.dir/all] Error 2
>
> make[4]: *** [CMakeFiles/pvCompileTools.dir/rule] Error 2
>
> make[3]: *** [pvCompileTools] Error 2
>
> CMake Error at
> /ccs/home/vanmoer/builds/superbuild/tools-build/pv-paraview-build.cmake:26
> (message):
>
>   Failed!!!
>
>
>
>
>
> make[2]: *** [paraview/src/paraview-stamp/paraview-build] Error 1
>
>
>
> I don't see a BUILD_SHARED that I can toggle. All of the
> CMAKE_SHARED_LINKER_FLAGS* vars are empty, if those are related.
>
>
>
> Any suggestions?
>
>
>
> Thanks,
>
> Mark
>
>
>
>
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at:
> http://paraview.org/Wiki/ParaView
>
> Follow this link to subscribe/unsubscribe:
> http://www.paraview.org/mailman/listinfo/paraview
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.paraview.org/pipermail/paraview/attachments/20130901/0dc00a52/attachment-0001.htm>