[vtk-developers] Re: VTK Python Installation [was: VTK 5.0 branch in mid-August]

[David Gobbi]

I am not very familiar with the ITK python wrappers (I've only used them 
a couple times), but since I'm very familiar with the VTK wrappers I 
decided to enter the fray.

Brad King wrote:

> Prabhu Ramachandran wrote:
>
>>  1. AFAICT, this is a totally non-standard hack.  I've never seen a
>>     single Python package using os.chdir with a hard coded path to
>>     import extension modules.
>
> Charl P. Botha wrote:
> > Indeed.  When building standalone Python and ITK based binaries, I
> > have to hack around the hard-coded paths in the ITK Python
> > installation.  Not a biggie, but every extra piece of code is an extra
> > point of failure.
> Prabhu Ramachandran wrote:
> > Still a hack, still needs hard coded paths.
>
> Where are you guys seeing a hard-coded path in the itkbase design?  
> The whole point of it is to allow the binary to be distributed and 
> executed from anywhere.  When I first wrote it I put it on the ITK 1.6 
> CD and python scripts could be executed by drag-and-drop onto the 
> executable on the CD in whatever drive it was sitting or in whatever 
> path to which the user had copied it.
>
> The installed version of itkbase.py gets this package directory:
>
>   SET(ITK_CSWIG_PACKAGE_DIR "os.path.dirname(selfpath)")
>
> This allows it to compute the location of the rest of the python 
> wrappers wherever it may be.  Then you just have to build with 
> CMAKE_SKIP_RPATH set to ON and the installation will have no reference 
> to the original build tree.

If I understand where Prabhu is coming from, his contention is that the 
python wrappers should not "automatically" find the .so files.  By 
definition, anything that you do in the python code to automatically 
locate these files is a hack.  I tend to agree with him.

I think it is fine to have a special vtkpython or itkpython executable 
that features fancy code for specifying the location of .py or .so 
files, but such code doesn't belong in the wrappers themselves.  If I'm 
running python on UNIX and I do "import vtk" or "import itkbase" then I 
expect the modules to be found purely based on conventional means 
(PYTHONPATH, a .pth file, LD_LIBRARY_PATH, ld.so.conf).

>> OTOH, the Apple approach is simple and standard.  Build
>> libvtk*PythonD.so's and install them in the same place the other VTK
>> shared libraries (or dll's) are installed.  These libraries need to be
>> on the linkers path or on PATH (both are standard things that users,
>> packagers and admins know how to change).  Then build the
>> libvtk*Python.so extension modules that link to the
>> libvtk*PythonD.so's.  This is perfectly standard in the Python world.
>> These extension modules will be small so there is no significant space
>> overhead.
>
>
> The itkbase design allows a binary to be extracted from a tarball and 
> executed out-of-the box without setting any PATH or LD_LIBRARY_PATH.  
> I do not see how the Apple design addresses this.  When the .so is 
> loaded how does it find its ...D.dylib libraries?

Truth be told, I think that you and Prabhu have very different goals, 
and very different ideas about how the wrappers will be used.   My ideas 
are more in line with Prabhu, I think.

First, allowing a binary to be "extracted from a tarball an executed 
out-of-the-box" is fine, but only if it can be done without any tricks 
in the .py files or .so files.  Any tricks should be cointained within 
executables such as "vtkpython".

Having stuff execute "out-of-the-box" without installation should be a 
secondary goal.  The primary goal is that things should work if 
installed in the right places on the system (the python modules in 
site-packages, and any .so files that aren't python modules in /usr/lib, 
/usr/local/lib, etc).

Of course it's also my opinion that VTK's use of rpath is one of the 
worst crimes against UNIX convention that I've ever seen, which is why 
you might want to take my opinions with a grain of salt.

So getting to the heart of the matter with why the Apple solution might 
be good for other UNIX systems:  The apple solution is good because it 
allows the python modules to be installed in a site-packages or 
somewhere in the PYTHONPATH, without those modules having to also be 
somewere in the LD_LIBRARY_PATH or ld.so.conf.

To put it briefly, if our current batch of *Python.so files are split 
into *Python.so and *PythonD.so files, and if all these files are put 
into the right places (where "right" is defined by Python and UNIX 
conventions and not by any other ideals), then the system will work 
without any of these things that Prabhu and myself refer to as "hacks".

As it stands right now, the *Python.so files need to be in python's path 
as well as the system runtime-library path at the same time.  This is 
only possible by using LD_LIBRARY_PATH, by doing nasty hacks in the .py 
files, or by (shudder) using rpath.

 - David