VTK/Cray XT3 Compilation

This document shows the steps to compile VTK for the service and compute nodes of the Cray XT3 (Red Storm) at Sandia National Laboratories. Check the main Red Storm web site <http://rsweb.sandia.gov> for a more complete discussion about how to compile for the Cray XT3.

Building VTK for Red Storm Service Nodes

Load appropriate modules

Cray uses modules [1] to set appropriate environment variables, paths, and compilers required for cross compiling for the compute or service nodes. I used the following script to load modules for service-node compilation.

# Compiling for service nodes:
. /opt/modules/default/init/bash

# Unload pgi and gnu catamount modules
module unload PrgEnv-pgi
module unload PrgEnv-gnu

# Load papi (maybe not necessary)
module load papi

You need to either execute each of these commands in your active shell, or source the script. I did this by putting the script in ~/.local/share/loadmods.snos and then creating an alias

alias loadmods.snos='source /home/raoldfi/.local/share/loadmods.snos'

.

Build/install CMAKE

Since VTK requires cmake, you need to download cmake from http://www.cmake.org and follow their instructions to install (make sure you have the snos modules loaded). I did the following without any problems.

[cmake-2.6.2] % ./build --prefix=${HOME}/.local
[cmake-2.6.2] % make install

Build/install VTK

Since the service nodes use Suse Linux, building and installing is trivial. From the VTK directory, this is what I did...

[VTK] % mkdir build/snos
[VTK] % cd build/snos
[snos] % ccmake ../..
[snos] % make install

During the ccmake step, I set prefix to ${HOME}/.local.

Building VTK for Red Storm Compute Nodes

Now for the hard part. Red Storm compute nodes use a the catamount operating system that has no support for

• fork(), exec(), system(), or dynamic process management,
• TCP/IP protocol stacks (MPI is used instead)
• I/O or network events other than MPI
• Dynamic linking and dynamically loaded libraries
• Unix-style interprocess communication (pipes, fifos, etc.)

Since VTK was designed primarily for Clusters with full OS support, we have to be a little careful.

Load appropriate modules

The standard supported modules for catamount are the PGI compilers. Gnu compilers are available and work, but they are not really supported. I source the following script to load the right modules cross compiling for catamount.

# Cross-compile modules for catamount

# Load the appropriate bash environment
. /opt/modules/default/init/bash

# Unload GNU modules for catamount
module unload PrgEnv-gnu

# load PGI modules for catamount
module load PrgEnv-pgi

# Load papi for accurate timings
module load papi

Use the same approach described in the previous section to load the modules into your shell environment.

TIFF Support

The Tiff image support can be tricky, here's an excerpt from Pat Marion:

If you look in the vtk build directory for your service node build, the Suse build, you should see a file named VTKCompileToolsConfig.cmake. This file lists the paths of some executables. These executables can be imported and used when crosscompile build. This is where the vtkmkg3states executable comes from that addresses the tiff error:

make[2]: *** No rule to make target `../../Utilities/vtktiff/vtkmkg3states', needed by `Utilities/vtktiff/tif_fax3sm.c'

To import the compile tools, add this to your cmake command line:

-DVTKCompileTools_DIR=$vtk_suse_build_dir

Configure VTK

To cross compile VTK for catamount, we have to specify some configuration options in a Toolchain file and remove some VTK features that are not supported in catamount.

Create a Toolchain file

The toolchain file tells cmake how to compile the code. I modified the Paraview toolchain file (thanks Alan) to create the following.

# -------- Toolchain-catamount.cmake ---------
# Specify the target system (this allows cross-compiling)
SET(CMAKE_SYSTEM_NAME Catamount)

# specify the cross compiler
SET(CMAKE_C_COMPILER   cc --target=catamount)
SET(CMAKE_CXX_COMPILER CC --target=catamount)

# set the search path for the environment coming with the compiler
# and a directory where you can install your own compiled software
set(CMAKE_FIND_ROOT_PATH
    /opt/xt-pe/default
    /opt/xt-mpt/default/mpich2-64/GP2
  )

# adjust the default behaviour of the FIND_XXX() commands:
# search headers and libraries in the target environment, search
# programs in the host environment
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

# Parallel configuration
SET(VTK_USE_PARALLEL ON)
SET(VTK_USE_MPI ON)

Using ccmake

The next step is to run ccmake from the build directory. This will generate all the necessary make files to build VTK.

[builddir] % ccmake -DCMAKE_TOOLCHAIN_FILE=Toolchain-catamount.cmake [PATH TO VTK SOURCE]

After calling ccmake, hit 'c' to configure and build the cache. Now, go get some coffee. This step took about 10 minutes. When it completes you will get a couple of errors about VTKCompileTools and LARGE_FILE_SUPPORT). I just ignored these... hit 'e'.

Once in the ccmake gui, hit 'c' again to read the Toolchain file and reconfigure. I ignored the error about not finding the OPENGL_INCLUDE files.

Now we need to manually remove features that are not supported by catamount, and set various other configuration. I did the following in ccmake's advanced mode:

• VTK_USE_METAIO = OFF (metaio uses unsupported network features)
• VTK_USE_RENDERING = OFF (fixes the OPENGL issue)
• BUILD_EXAMPLES = ON
• CMAKE_INSTALL_PREFIX = [my installation directory]

After configuring one more time, hit 'c', hit 'g' to generate the files and exit.

Build/Install VTK

At this point, we just need to make and install the libraries.

[builddir] make;
[builddir] make install;

Expect a number of warnings about typecasts, unreachable statements, and such; as well as warnings about unsupported features in catamount. These could turn out to be a problem if one of the vtk libraries uses these features.