Difference between revisions of "BuildingPythonWithCMake"

From KitwarePublic
Jump to: navigation, search
m (Building Python 2.7.1)
m (Building Python 2.7.1)
 
Line 39: Line 39:
 
The commands shown above will check out Python and the CMake files in the same directory, which will be both a CVS and a svn working directory at the same time. While this is a bit unusual it works, since svn and CVS use different meta data directories (.svn/ vs. CVS/).
 
The commands shown above will check out Python and the CMake files in the same directory, which will be both a CVS and a svn working directory at the same time. While this is a bit unusual it works, since svn and CVS use different meta data directories (.svn/ vs. CVS/).
  
= Building Python 2.7.1 =
+
= Building Python 2.7.1 to 2.7.3 =
  
 
The build system of "David Sansome" looks very promising.  
 
The build system of "David Sansome" looks very promising.  

Latest revision as of 22:22, 19 September 2014

Building Python 2.5.1

Get the Python sources for release 2.5.1, e.g. from http://www.python.org/download/ . If you are using Python 2.5.1 you'll need to apply the following patch, otherwise Python doesn't build with dynamic loading disabled and Unicode disabled: Python-2.5.1-build-without-shared-libs-and-unicode.diff

There are two ways to get the CMake files for building Python:

Unpack that file and copy them over to the Python source directory, so that the toplevel CMakeLists.txt ends up in the toplevel Python directory.

  • you can download the CMake files using cvs

The CMake files for building Python 2.5.1 are in the ParaView3 cvs, in the branch Python-2_5_1-BRANCH. You can get them like this:

$ cvs -d :pserver:anoncvs@www.paraview.org:/cvsroot/ParaView3 login
<empty password>
$ cvs -d :pserver:anoncvs@www.paraview.org:/cvsroot/ParaView3 co -d CMakeBuildForPython -r Python-2_5_1-BRANCH ParaView3/Utilities/CMakeBuildForPython

Then copy these files over to the Python source directory, so that the toplevel CMakeLists.txt ends up in the toplevel Python directory.

Building svn Python or Python >= 2.6.0

With Python from svn (not Python 3000) or Python 2.6.0 (not released yet, September 2007), the patch mentioned above for handling unicode and disabled dynamic loading isn't necessary anymore. Here's how you can get Python from Python svn and the CMake files for Python, which are in ParaView3 cvs, HEAD;

$ svn co http://svn.python.org/projects/python/trunk python-with-cmake
$ cvs -d :pserver:anoncvs@www.paraview.org:/cvsroot/ParaView3 login
<empty password>
$ cvs -d :pserver:anoncvs@www.paraview.org:/cvsroot/ParaView3 co -d python-with-cmake ParaView3/Utilities/CMakeBuildForPython 

The commands shown above will check out Python and the CMake files in the same directory, which will be both a CVS and a svn working directory at the same time. While this is a bit unusual it works, since svn and CVS use different meta data directories (.svn/ vs. CVS/).

Building Python 2.7.1 to 2.7.3

The build system of "David Sansome" looks very promising.

It allows a lot of flexibility in the way python can be built and embedded in existing application.

See https://github.com/davidsansome/python-cmake-buildsystem

Platform specific build notes

Building Python for Linux or Windows

Just run CMake on it as usual, then make, make install. Under Windows this is tested with VisualStudio 2003. Under Windows the tests in ConfigureChecks.cmake are not actually executed, but the pyconfig.h for Windows which is part of the Python sources is used as is. This is actually not necessary, the tests could be executed the same way as they are executed under Windows, but to get this working probably some flags, include files etc. have to be adjusted, and I didn't have the time to do this yet. As a difference to the original Python build for Windows not all modules are built into a huge python.dll, but instead they are built as separate plugins (as under Linux).

Building Python for other operating systems

Building Python using CMake for other operating systems like OS X, *BSD etc. shouldn't require a lot of work, maybe it already works out of the box. Probably some configure checks have to be tweaked.


Cross compiling Python for BlueGene/L

Cross compiling Python for BlueGene is easy, since in python-with-cmake/CMake/ there is a file TryRunResults-Python-bgl-gcc.cmake, which contains the results for the TRY_RUN() tests which are executed by CMake when configuring Python. Since we are cross compiling CMake can't actually run these tests, but the results have to be supplied manually. This is done in this case by preloading the CMake cache using the mentioned file:

~/src/python-with-cmake/ $ 
~/src/python-with-cmake/ $ mkdir build-bgl
~/src/python-with-cmake/~$ cd build-bgl
~/src/python-with-cmake/build-bgl/ $ cmake -DCMAKE_TOOLCHAIN_FILE=~/Toolchain-BlueGeneL-gcc.cmake -DCMAKE_INSTALL_PREFIX=~/bgl-install -C ../CMake/TryRunResults-Python-bgl-gcc.cmake ..
...
~/src/python-with-cmake/build-bgl/ $ make
...
~/src/python-with-cmake/build-bgl/ $ make install
...

Cross compiling Python for Cray Xt3/ Catamount

Cross compiling Python for Cray Xt3/Catamount is easy, since in python-with-cmake/CMake/ there is a file TryRunResults-Python-catamount-gcc.cmake, which contains the results for the TRY_RUN() tests which are executed by CMake when configuring Python. Since we are cross compiling CMake can't actually run these tests, but the results have to be supplied manually. This is done in this case by preloading the CMake cache using the mentioned file:

~/src/python-with-cmake/ $ 
~/src/python-with-cmake/ $ mkdir build-catamount-gcc
~/src/python-with-cmake/~$ cd build-catamount-gcc
~/src/python-with-cmake/build-catamount-gcc/ $ cmake -DCMAKE_TOOLCHAIN_FILE=~/Toolchain-Catamount-gcc.cmake -DCMAKE_INSTALL_PREFIX=~/catamount-install -C ../CMake/TryRunResults-Python-catamount-gcc.cmake ..
...

The python module pwd has to be disabled by setting MODULE_pwd_ENABLE to OFF, since the getpw*() functions are not available under Catamount.

~/src/python-with-cmake/build-catamount/ $ make
...
~/src/python-with-cmake/build-catamount/ $ make install
...

When linking it all together there are a lot of warnings about posix_* functions, where the linker warns that they will fail at runtime. They come from the posix module. But this has to be built, it's presence itself is used by python to detect that it's running on a POSIX-like platform.

Cross compiling Python for another platform

~/src/python-with-cmake/ $ 
~/src/python-with-cmake/ $ mkdir build-ecos
~/src/python-with-cmake/~$ cd build-ecos
~/src/python-with-cmake/build-ecos/ $ cmake -DCMAKE_TOOLCHAIN_FILE=~/Toolchain-eCos-synth.cmake -DCMAKE_INSTALL_PREFIX=~/ecos-install ..
...

This will finish with two errors, one for each TRY_RUN(), since this is cross compiling and CMake doesn't try to execute the executables. CMake will have created a file TryRunResults.cmake, This will finish with two errors:

-- Performing Test HAVE_BROKEN_NICE
CMake Error: TRY_RUN() invoked in cross-compiling mode, please set the following cache variables appropriatly:
   HAVE_BROKEN_NICE_EXITCODE (advanced)
For details see ~/src/python-with-cmake/build-ecos/TryRunResults.cmake
-- Performing Test HAVE_BROKEN_NICE - Failed
-- Performing Test HAVE_BROKEN_POLL
CMake Error: TRY_RUN() invoked in cross-compiling mode, please set the following cache variables appropriatly:
   HAVE_BROKEN_POLL_EXITCODE (advanced)
For details see ~/src/python-with-cmake/build-ecos/TryRunResults.cmake
-- Performing Test HAVE_BROKEN_POLL - Failed

These are two system tests, where cmake used TRY_RUN() to test an executable, but since they are cross compiled, it didn't try to run them but instead produced these error messages. As the messages say have a look at ~/src/python-with-cmake/build-ecos/TryRunResults.cmake . You will find the executables which were not run in the build directory, named cmTryCompileExec-HAVE_BROKEN_NICE_EXITCODE and cmTryCompileExec-HAVE_BROKEN_POLL_EXITCODE. You can try to run them on the target to see what there exit code is and put these results into the TryRunResults.cmake file. After that you should copy this file to a safe place, otherwise it gets overwritten on the next CMake run and your changes are lost. Then use this file to preload the CMake cache with the results from the TRY_RUN():

~/src/python-with-cmake/build-ecos/ $ cmake -C ../CMake/TryRunResults-Python-ecos-synth.cmake ..
...
~/src/python-with-cmake/build-ecos/ $ make
...
~/src/python-with-cmake/build-ecos/ $ make install
...

Current status

What works

  • Builds on Linux, Windows and cross compiles to eCos, IBM BlueGene/L and Cray Xt3/Catamount
  • the python library can be built as shared or static library, RPATH is supported
  • each module can be enabled and disabled separatly
  • on systems without shared libs everything is build static, modules are linked into the python library
  • packages can be created using cpack (or "make package"): rpm, deb, tgz, stgz, zip, nsis and more

TODO

  • Enable the configure checks on Windows, so the premade pyconfig.h file can go away.
  • Trying to build it on other platforms like *BSD, Apple. This shouldn't be much work.
  • Adding the remaining modules to the build. Should be just work, no major problems expected.
  • Add rules to bytecompile (or how it is called) the python modules to .pyc files
  • The python executable is currently installed with the version number in their name.

Potential problems

Module not found

If the Python executable and the python library have been built and installed successfully and you can execute simple things, it still can happen that some modules are not found. Currently not all Python modules are built and not everything is installed. In this case have a look at python-with-svn/CMakeLists.txt and search in the list of ADD_PYTHON_MODULE() commands if your module is included. Some modules are disabled by default, you can enable building them by setting the option BUILD_UNTESTED_MODULES to TRUE. Other modules are still completely missing. If this is the case, figure out what the source files for this module are and add it to this list. If it is built and installed, but Python still doesn't find it, it's time for debugging. A good start might be python-with-svn/Python/import.c .