[vtkusers] python (2.7) problems compiling with MSVC 2010 64bit

Wed Jun 8 17:50:39 EDT 2011

Hi David,

Thanks a lot for your comments.

You are right that my setup is far from the beaten track. I need to
compile python myself (to consistently compile other packages against
it with the same compiler/settings) but there is no installation
option available for Windows, or is hidden so deep that I could not
find it. I believe your argumentation does not cover two originally
mentioned issues 1) portable self contained precompiled python+vtk
environments, e.g. to be distributed to students or customers who will
not want to hear about PATH or PYTHONPATH 2) vtk DLLs depending on
externals like DLL, where vtk+python would fail out of the box (note
that Qt is normally built in place and not registered with Windows).

Regards,
Dominik

On Wed, Jun 8, 2011 at 11:24 PM, David Gobbi <david.gobbi at gmail.com> wrote:
> Hi Dominik,
>
> I disagree with the "assemble the build into a packaged folder" part
> of your discussion, because I don't think people should be doing this.
> On Linux you can specify a prefix with "./configure --prefix=/some/directory"
> and do a simple make install, the result will be a typical directory
> layout that is suitable for system-wide use.  The python install
> automatically segregates the binaries and libraries for different python
> versions, so multiple versions of python can easily be installed this way.
>
> Ditto for Windows, the default install puts each python in its own
> directory e.g. Python26, Python27, etc. so there are no collisions.
> Making python accessible system-wide is as simple as installing it
> where it is accessible to all users, this can be on a network drive
> since python does not need any registry entries in order to work
> properly.
>
> Even if you want to have one central network-wide installation that
> can be used by different people running on different platforms, this
> can be done in standard ways through ./configure.
>
> The bottom line is this: if you customize the directory structure of
> your python installation then you have to set up the PATHs yourself
> and/or copy the VTK dlls into the appropriate directories.
>
>  - David
>
>
> On Wed, Jun 8, 2011 at 2:35 PM, Dominik Szczerba <dominik at itis.ethz.ch> wrote:
>> On Wed, Jun 8, 2011 at 10:07 PM, David Gobbi <david.gobbi at gmail.com> wrote:
>>> Hi Dominik,
>>>
>>> I'm a bit confused about the paths you give in your examples:
>>>
>>> C:\python2.7.1\bin
>>> /usr/local/pack/python-2.7.1/bin
>>>
>>> On my Windows system (using the installer from python.org) I'd see
>>> something like this:
>>>
>>> C:\Python27\   (no bin directory!)
>>> C:\Python27\DLLs\
>>> C:\Python27\Lib\site-packages\
>>
>> What I called a system installed python was not immediately meant to
>> be the "official pre-compiled python". I actually have built one
>> myself and "installed" it by making it accessible by any user. It has
>> bin, include and lib subfolders, but yes, one can assemble it in any
>> other way one likes. I do not in fact know the actual structure of the
>> "official pre-compiled" installer.
>>
>>> On my Linux systems (ubuntu 8.04 and 10.04) after doing a typical
>>> install, I'd see something like this:
>>>
>>> /usr/local/bin/python2.7 (the executable)
>>> /usr/local/lib/python2.7/site-packages/
>>>
>>> Can you give some details about your system and how you have done
>>> your python installation so that I can understand why your paths are so
>>> different?
>>
>> What I do on linux is exactly the same as what I described above for
>> Windows: I compile it myself and package myself - I did not mean
>> calling "make install" on the custom compiled code.
>>
>> Bottom line: should another caption be added as 1) "Using official
>> installers", where you would correct the paths; and the current first
>> caption be renamed to "using custom system wide installation"? You may
>> also opt to remove te current first part and replace it by the one
>> related to the official installers. Hopefully the current second part
>> "system independent installation" part, should not raise any concerns,
>> and this one is most relevant for me.
>>
>> Regards,
>> Dominik
>>
>>>
>>>  - David
>>>
>>>
>>> On Wed, Jun 8, 2011 at 1:19 PM, Dominik Szczerba <dominik at itis.ethz.ch> wrote:
>>>> Hi David,
>>>>
>>>> I tried editing http://www.vtk.org/Wiki/VTK/Python_Wrapping_FAQ but
>>>> was denied the editing rights. Below my proposed addition (as bullet
>>>> 1.6) which I would have found useful when resolving the issue.
>>>>
>>>> Regards,
>>>> Dominik
>>>>
>>>> === Installation ===
>>>>
>>>> In order to use the python enabled VTK you need to resolve the
>>>> required DLL dependencies, i.e., you have to make sure all DLLs/shared
>>>> objects can find each other. For simple setups "vtkpython" might work,
>>>> but in case you compiled VTK with external dependencies (e.g. QT)
>>>> chances are it will not, as there is no easy way such dependencies be
>>>> resolved automatically. There are two scenarios covered here.
>>>>
>>>> ==== System installed python ====
>>>>
>>>> On Windows, you need to make sure that the path containing python.exe
>>>> and python.dll is in the PATH variable. You can set it system wide in
>>>> the system advanced settings, or from a terminal like this:
>>>>
>>>> PATH=%PATH%;C:\python2.7.1\bin
>>>>
>>>> On a recent linux distribution with a system installed python a
>>>> similar step should not be needed, but just in case you have installed
>>>> your own, you would need an equivalent like:
>>>>
>>>> export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/pack/python-2.7.1/bin
>>>>
>>>> ==== System independent python ====
>>>>
>>>> Suppose you want to have several independent python versions along
>>>> with their packages and you do not want to register any of them with
>>>> the system. In such a case, for the best compatibility and
>>>> consistency, you should compile python from source yourself, then
>>>> assemble the build into a packaged folder containing three
>>>> sub-folders: bin, include and lib. "bin" should contain python
>>>> executable and the dlls/shared objects, "include" the header files and
>>>> "lib" a subfolder site-packages. Now all there is to do is to naively
>>>> copy all dependent dll's/so's from the vtk build bin folder into the
>>>> custom python package's bin folder, and the vtk python package
>>>> subfolder from Wrapping/Python/vtk into  lib/site-packages. Now double
>>>> clicking python executable should bring up a python console, from
>>>> which you should be able to do things like:
>>>>
>>>> from vtk import *
>>>> image = vtkImageData()
>>>> print image
>>>>
>>>>
>>>>
>>>> On Wed, Jun 8, 2011 at 7:41 PM, David Gobbi <david.gobbi at gmail.com> wrote:
>>>>> Hi Dominik,
>>>>>
>>>>> It is a given that when using python.exe, either the DLLs have to be
>>>>> copied into the path, or the PATH and PYTHONPATH variables have
>>>>> to be set.  So I'll just leave that part of the question aside, hopefully
>>>>> someone from Kitware who deals with the VTK installation process can
>>>>> answer.
>>>>>
>>>>> The vtkpython executable is a different story, though, because it is
>>>>> supposed to find the VTK dlls automatically.  It does have some
>>>>> limitations: it cannot know all of the library dependencies, so it
>>>>> cannot be expected to know where to find the Qt DLLs, and these
>>>>> will have to be in the PATH.  When you used the dependency walker,
>>>>> did you find any dependencies that were unusual or unexpected?
>>>>>
>>>>> If you put together a wiki page, that would definitely be useful for
>>>>> other users, I'd be willing to review it for you.  You can also look
>>>>> at VTK/Wrapping/Python/README.txt to see if it has relevant
>>>>> information.
>>>>>
>>>>>  - David
>>>>>
>>>>>
>>>>> On Wed, Jun 8, 2011 at 10:59 AM, Dominik Szczerba <dominik at itis.ethz.ch> wrote:
>>>>>> I finally got 2.7 to work. Please review, if this is not dirty hacking
>>>>>> I can commit a small wiki page with a HOWTO.
>>>>>>
>>>>>> Both python26 and python27 will work if ALL required DLL's will be
>>>>>> copied over to python\bin. Which ones I required can be debugged with
>>>>>> the dependency walker. This includes, e.g., the Qt DLL's, because I
>>>>>> was compiling with Qt GUI support. Probably compiling without Qt, or
>>>>>> appending the missing DLL paths to PATH would work too, but I have not
>>>>>> tested it.
>>>>>>
>>>>>> I do not see how executing 'vtkpython' can work in such cases. I found
>>>>>> no documentation about possible setup scenarios and required system
>>>>>> variables.
>>>>>>
>>>>>> Best regards,
>>>>>> Dominik
>>>>>>
>>>>>> On Wed, Jun 8, 2011 at 5:52 PM, David Gobbi <david.gobbi at gmail.com> wrote:
>>>>>>> I can't speak to why python 2.7 would be different from python 2.6,
>>>>>>> because I have never tried it... hopefully someone else can answer
>>>>>>> that question.
>>>>>>>
>>>>>>>  - David
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jun 8, 2011 at 7:08 AM, Dominik Szczerba <dominik at itis.ethz.ch> wrote:
>>>>>>>> I have now manually copied everything from vtk's build
>>>>>>>> bin\RelWithDebInfo folder to python's bin folder and I have manually
>>>>>>>> copied Wrapping\Python\vtk to python's lib\site-packages folder.
>>>>>>>> Such treatment seems to work for python 2.6.7 (= "import vtk" works),
>>>>>>>> but does not work for python 2.7.1. Note, I have no python/vtk in the
>>>>>>>> system path and pythonhome and pythonpath are not set.
>>>>>>>>
>>>>>>>> Am I hacking and this will not work the next release, or is the right
>>>>>>>> way to go? Why would 2.7 not work on Windows? Works on linux all
>>>>>>>> right.
>>>>>>>>
>>>>>>>> Thanks for any more hints,
>>>>>>>> Dominik
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jun 8, 2011 at 12:59 PM, Dominik Szczerba <dominik at itis.ethz.ch> wrote:
>>>>>>>>> Hi David,
>>>>>>>>>
>>>>>>>>> I have tried python 2.6.7 with the same failure. Is this a known issue
>>>>>>>>> for Windows? My configure settings are pretty straight forward: using
>>>>>>>>> shared libs, paths to python/include/lib, enable wrapping, disable TK,
>>>>>>>>> did I forget anything?
>>>>>>>>>
>>>>>>>>> Is there somewhere a comprehensive information what should reside / be
>>>>>>>>> copied where such that I can use vtk with python? I never saw one,
>>>>>>>>> even under http://www.vtk.org/Wiki/VTK/Python_Wrapping_FAQ
>>>>>>>>> I do not (in fact, can not) use vtkpython (that does not work for now
>>>>>>>>> anyway). I need to create a custom python+vtk environment, totally
>>>>>>>>> unregistered with the system (so I can have multiple versions).
>>>>>>>>>
>>>>>>>>> Many thanks and best regards,
>>>>>>>>> Dominik
>>>>>>>>>
>>>>>>>>> On Tue, Jun 7, 2011 at 11:46 PM, David Gobbi <david.gobbi at gmail.com> wrote:
>>>>>>>>>> Hi Dominik,
>>>>>>>>>>
>>>>>>>>>> It might be as simple as cmake not properly detecting python, I know that
>>>>>>>>>> it was unable to detect python-2.7 until very recently, in fact even my
>>>>>>>>>> copy of cmake-2.8.2 is missing the bits that it needs to find python-2.7.
>>>>>>>>>>
>>>>>>>>>> Take a look through your CMakeCache.txt file to make sure that all of the
>>>>>>>>>> python-related variables are set.
>>>>>>>>>>
>>>>>>>>>> Also, now that you know for certain that it is a path-related issue, make
>>>>>>>>>> sure to clear the PYTHONHOME variable or it will get you into trouble
>>>>>>>>>> in the future.
>>>>>>>>>>
>>>>>>>>>>  - David
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Jun 7, 2011 at 3:36 PM, Dominik Szczerba <dominik at itis.ethz.ch> wrote:
>>>>>>>>>>> Hi David,
>>>>>>>>>>>
>>>>>>>>>>> Thanks, indeed, there was no PYTHONHOME set. But setting it brings me
>>>>>>>>>>> only one step further. The next is the error when calling "import
>>>>>>>>>>> vtk". See the error at the end. I saw a similar thread when you wrote
>>>>>>>>>>> it is about mixing release and debug for python. It is not the case
>>>>>>>>>>> here, both python and vtk is self compiled with the same compiler
>>>>>>>>>>> settings (Release). I blind tried various combinations of PATH and
>>>>>>>>>>> PYTHONPATH to no avail. I will be happy about some more ideas to try.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Dominik
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Jun 7, 2011 at 11:26 PM, David Gobbi <david.gobbi at gmail.com> wrote:
>>>>>>>>>>>> Hi Dominik,
>>>>>>>>>>>>
>>>>>>>>>>>> The "site" module is the first to be imported, so if it is missing,
>>>>>>>>>>>> then the "PYTHONHOME" (or equivalent) is not set or is incorrect and
>>>>>>>>>>>> python is unable to find its own modules.
>>>>>>>>>>>>
>>>>>>>>>>>> If you set PYTHONHOME manually (i.e. as an environment variable) to
>>>>>>>>>>>> the directory where python-2.7 resides, then I am guessing that these
>>>>>>>>>>>> tests will pass.
>>>>>>>>>>>>
>>>>>>>>>>>> So... the next question is, why is this happening?  Evidently the
>>>>>>>>>>>> vtkpython executable is missing some critical information that allows
>>>>>>>>>>>> it to find the python-2.7 directory on its own.  The relevant code in
>>>>>>>>>>>> VTK is Wrapping/Python/vtkPythonAppInit.cxx
>>>>>>>>>>>>
>>>>>>>>>>>> None of the dashboard machines seem to use python-2.7.
>>>>>>>>>>>>
>>>>>>>>>>>>  - David
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Jun 7, 2011 at 2:44 PM, Dominik Szczerba <dominik at itis.ethz.ch> wrote:
>>>>>>>>>>>>> Below some output from testing log. I get exactly the same error
>>>>>>>>>>>>> (ImportError: No module named site) by just invoking vtkpython from
>>>>>>>>>>>>> command line.
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>> 59/251 Testing: PythonSmokePython-image
>>>>>>>>>>>>> 59/251 Test: PythonSmokePython-image
>>>>>>>>>>>>> Command: "E:/build/vtk-5.6.1/bin/Release/vtkpython.exe"
>>>>>>>>>>>>> "E:/pack/vtk-5.6.1/Common/Testing/Python/PythonSmoke.py"
>>>>>>>>>>>>> Directory: E:/build/vtk-5.6.1/Common/Testing/Python
>>>>>>>>>>>>> "PythonSmokePython-image" start time: Jun 07 22:27 W. Europe Daylight Time
>>>>>>>>>>>>> Output:
>>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>> ImportError: No module named site
>>>>>>>>>>>>> <end of output>
>>>>>>>>>>>>> Test time =   2.22 sec
>>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>> Test Failed.
>>>>>>>>>>>>> "PythonSmokePython-image" end time: Jun 07 22:27 W. Europe Daylight Time
>>>>>>>>>>>>> "PythonSmokePython-image" time elapsed: 00:00:02
>>>>>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Jun 7, 2011 at 10:37 PM, Dominik Szczerba <dominik at itis.ethz.ch> wrote:
>>>>>>>>>>>>>> I have compiled python 2.7 in Release mode on MSVC 2010 64bit. I have
>>>>>>>>>>>>>> built vtk-5.6.1 against it (with wrapping enabled). However, I get the
>>>>>>>>>>>>>> below errors when testing the build.
>>>>>>>>>>>>>> My setup is rather standard, I was therefore wondering if there could
>>>>>>>>>>>>>> be something obviously wrong, or 2.7 is simply not yet supported.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Dominik
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> E:\build\vtk-5.6.1>ctest -R Python
>>>>>>>>>>>>>> Test project E:/build/vtk-5.6.1
>>>>>>>>>>>>>>    Start  59: PythonSmokePython-image
>>>>>>>>>>>>>> 1/6 Test  #59: PythonSmokePython-image ......................***Failed    2.22 s
>>>>>>>>>>>>>> ec
>>>>>>>>>>>>>>    Start  60: TestWeakrefPython-image
>>>>>>>>>>>>>> 2/6 Test  #60: TestWeakrefPython-image ......................***Failed    0.09 s
>>>>>>>>>>>>>> ec
>>>>>>>>>>>>>>    Start  61: TestNumpySupportPython-image
>>>>>>>>>>>>>> 3/6 Test  #61: TestNumpySupportPython-image .................***Failed    0.09 s
>>>>>>>>>>>>>> ec
>>>>>>>>>>>>>>    Start  62: TestTerminationCrashPython-image
>>>>>>>>>>>>>> 4/6 Test  #62: TestTerminationCrashPython-image .............***Failed    0.08 s
>>>>>>>>>>>>>> ec
>>>>>>>>>>>>>>    Start  63: TestExecuteMethodFinalizeCrashPython-image
>>>>>>>>>>>>>> 5/6 Test  #63: TestExecuteMethodFinalizeCrashPython-image ...***Failed    0.09 s
>>>>>>>>>>>>>> ec
>>>>>>>>>>>>>>    Start 237: kMeansDistanceCalculatorPython
>>>>>>>>>>>>>> 6/6 Test #237: kMeansDistanceCalculatorPython ...............***Failed    0.08 s
>>>>>>>>>>>>>> ec
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 0% tests passed, 6 tests failed out of 6
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Total Test time (real) =   3.50 sec
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The following tests FAILED:
>>>>>>>>>>>>>>         59 - PythonSmokePython-image (Failed)
>>>>>>>>>>>>>>         60 - TestWeakrefPython-image (Failed)
>>>>>>>>>>>>>>         61 - TestNumpySupportPython-image (Failed)
>>>>>>>>>>>>>>         62 - TestTerminationCrashPython-image (Failed)
>>>>>>>>>>>>>>         63 - TestExecuteMethodFinalizeCrashPython-image (Failed)
>>>>>>>>>>>>>>        237 - kMeansDistanceCalculatorPython (Failed)
>>>>>>>>>>>>>> Errors while running CTest
>>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Powered by www.kitware.com
>>>>>>>>>>>>>
>>>>>>>>>>>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please keep messages on-topic and check the VTK FAQ at: http://www.vtk.org/Wiki/VTK_FAQ
>>>>>>>>>>>>>
>>>>>>>>>>>>> Follow this link to subscribe/unsubscribe:
>>>>>>>>>>>>> http://www.vtk.org/mailman/listinfo/vtkusers
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>