[Paraview] Python interpreter not being created for coprocessing?

Tue Sep 25 15:15:19 EDT 2012

On Tue, Sep 25, 2012 at 12:21 PM, Vanmoer, Mark W <mvanmoer at illinois.edu>wrote:

>  I pulled it directly from stage/13460_CP_FortranAdaptor.
>

Great, then it was me after all that fixed it!

> ****
>
> ** **
>
> I’m now at the point where it runs great on a single node with GPUs, I’m
> trying to extend it to multiple nodes, which I think involves setting up
> vtkMultiBlockDataSets? I was going to look through the NPICAdaptor for
> hints on that.
>

By single node I'm assuming that you mean a single process, correct? What's
the type of grid that you're planning on using? The PhastaAdaptor is an
example of dealing with an unstructured grid. I also just updated the
coprocessing tutorials page (now at
http://paraview.org/Wiki/Coprocessing_Tutorials) to include the tutorial
I'll be doing at SC12. It should have some useful information for doing
this. You could just do a polydata input, just set the center of the sphere
source in your example based on the process rank (e.g.
Sphere->SetCenter(10*rank, 0, 0); ) so that it's easy to tell the
difference between each processes input part/domain partition.

Andy

> ****
>
> ** **
>
> Mark****
>
> ** **
>
> ** **
>
> *From:* Andy Bauer [mailto:andy.bauer at kitware.com]
> *Sent:* Tuesday, September 25, 2012 10:13 AM
>
> *To:* Vanmoer, Mark W
> *Cc:* paraview at paraview.org
> *Subject:* Re: [Paraview] Python interpreter not being created for
> coprocessing?****
>
> ** **
>
> Did you get my changes from ParaView's stage branch? My changes didn't
> make it into ParaView master yet so I'm not sure what changed that caused
> things to work now. Well, if it's working I'm glad -- I'm not sure I can
> personally take credit for it though :)
>
> Let me know how it goes with the GPU stuff.
>
> Andy****
>
> On Mon, Sep 24, 2012 at 5:48 PM, Vanmoer, Mark W <mvanmoer at illinois.edu>
> wrote:****
>
> Hi Andy,****
>
>  ****
>
> I’ve tested this and it works with both gcc/mpich2 and pgi/openmpi. ****
>
>  ****
>
> Thanks again!****
>
> Mark****
>
>  ****
>
> *From:* Vanmoer, Mark W
> *Sent:* Wednesday, September 19, 2012 1:30 PM
> *To:* Andy Bauer
> *Cc:* paraview at paraview.org
> *Subject:* RE: [Paraview] Python interpreter not being created for
> coprocessing?****
>
>  ****
>
> Hi Andy, that sounds like some deep debugging, thanks for your effort.
> I'll test this and let you know. It's on my to do list to learn CMake, I've
> had the book on my desk for a month now.
>
> Also, I made another Fortran example that's closer in spirit to the wiki
> example, which uses the DataGenerator class as the C++ does, if you're
> interested.
>
> Thanks,
> Mark****
>   ------------------------------
>
> *From: *Andy Bauer
> *Sent: *9/19/2012 1:12 PM
> *To: *Vanmoer, Mark W
> *Cc: *paraview at paraview.org
> *Subject: *Re: [Paraview] Python interpreter not being created for
> coprocessing?****
>
> Hi Mark,
>
> I finally got around to fixing this. It's in the stage branch
> (remotes/stage/13460_CP_FortranAdaptor) and hopefully will go into the
> master branch on the next gatekeeper review. There were 2 issues fixed in
> this:
> 1) for using newer GCC compilers and other compilers that hide symbols in
> the library (http://gcc.gnu.org/wiki/Visibility) the FortranAdaptor API
> methods weren't available to other libraries and executables trying to link
> to them.
> 2) Since 3.14.1 I changed around the way ParaView gets initialized through
> the coprocessing library (now through vtkCPProcessor::Initialize() as it
> should be instead of vtkCPPythonScriptPipeline::Initialize()). I forgot to
> make the change in the FortranAdaptorAPI.cxx which caused the python
> interpretor to not be initialized properly, as you noticed.
>
> Thanks for the help in finding this problem. Sorry for the slow response
> on it. I created a CMakeLists.txt file to help build as I've been using
> CMake too long and can't remember how to work with Makefiles anymore! I'm
> attaching it in case you'd like to use it. I didn't test it with an install
> of ParaView as I really only work directly with ParaView builds.
>
> Let me know if this solves your problems. I seem to remember there being
> more as far as using CUDA and PGI but hopefully this gets you close.
>
> By the way, I'll probably use what you sent me to create a Fortran test or
> example to help with making sure it doesn't get broken in the future.
>
> Andy****
>
> On Wed, Sep 12, 2012 at 4:42 PM, Vanmoer, Mark W <mvanmoer at illinois.edu>
> wrote:****
>
> Tarball attached, it’s not exactly the wiki example because I didn’t want
> to bother with passing a DataGenerator object back and forth from C++ to
> Fortran, so I just make a sphere. The pipeline script is a shrink filter
> generated by pvclient.****
>
>  ****
>
> I checked CMake and it looks like it’s finding the same version of python.
> ****
>
> Mark****
>
>  ****
>
> *From:* Andy Bauer [mailto:andy.bauer at kitware.com]
> *Sent:* Wednesday, September 12, 2012 1:02 PM****
>
>
> *To:* Vanmoer, Mark W
> *Cc:* paraview at paraview.org
> *Subject:* Re: [Paraview] Python interpreter not being created for
> coprocessing?****
>
>  ****
>
> Can you send me your Fortran version of the wiki example?
>
> From what I remember, CMake treats the Python interpretor/executable
> different than the header files and libraries. By this I mean that it will
> pick up the python header files and libraries from one version of python
> and the python interpretor will be from a different version. I don't think
> that's what's going on but you may want to check on that in CMake.
>
> Andy****
>
> On Tue, Sep 11, 2012 at 11:21 AM, Vanmoer, Mark W <mvanmoer at illinois.edu>
> wrote:****
>
> Hi Andy,****
>
>  ****
>
> Sorry, I lost track of where I was in debugging. What I had was an
> openmpi/pgi debug build that passed the ctest -R CoPro tests. But the app
> I'm working on still crashes when trying to create a python interpreter.**
> **
>
>  ****
>
> I started over from scratch working with the Wiki example, using gcc 4.4.6
> and mpich2 1.4.1p1.****
>
>  ****
>
> I have the C++ wiki example working with both the ParaView-3.14.1-Source
> tarball and with git commit 2537cabbf0cc9b39f17f347105ee18c559d0041a.****
>
>  ****
>
> I then made a Fortran version of the wiki example. It works with the
> ParaView-3.14.1-Source tarball build, but crashes when built against the
> git branch build. Seems to be at the same spot as the pgi build, when it
> tries to return the python interpreter instance. I stepped through with gdb
> and don’t see anything obvious. It seems to crash when it tries to create
> an interpreter to add the python script to the module search path.****
>
>  ****
>
> Mark****
>
> *From:* Andy Bauer [mailto:andy.bauer at kitware.com]
> *Sent:* Thursday, August 30, 2012 10:26 PM
> *To:* Vanmoer, Mark W
> *Cc:* paraview at paraview.org
> *Subject:* Re: [Paraview] Python interpreter not being created for
> coprocessing?****
>
>  ****
>
> Glad to hear that it's working. As for the parallel test failure, I think
> you can ignore that for now. I need to fix it but I just haven't gotten
> around to it yet. I think it's either a parallel rendering/compositing
> issue or a problem with the actors's visibility in the views getting
> screwed up. Both of those things are not my area of expertise so I've been
> reluctant to go and fix it. Now that you've publicly shamed me I'm going to
> have to bite the bullet and fix it :)  Hopefully by the end of next week it
> will be fixed. I created a mantis issue (
> http://paraview.org/Bug/view.php?id=13417) if you want to follow it.
>
> Andy****
>
> On Thu, Aug 30, 2012 at 6:19 PM, Vanmoer, Mark W <mvanmoer at illinois.edu>
> wrote:****
>
> Hi Andy, ****
>
> Not sure what changed, but I was able to build pgi/openmpi and got to the
> point you did below, where the parallel test fails with “CPGrid0 image
> compare failed.” ****
>
> Can I ignore this error? Anyway, the python interpreter seems to be
> created, so that issue is closed.****
>
> Mark****
>
>  ****
>
> *From:* Andy Bauer [mailto:andy.bauer at kitware.com]
> *Sent:* Wednesday, August 29, 2012 12:15 PM****
>
>
> *To:* Vanmoer, Mark W
> *Cc:* paraview at paraview.org
> *Subject:* Re: [Paraview] Python interpreter not being created for
> coprocessing?****
>
>  ****
>
> Ok, glad to hear that coprocessing is working in some fashion. Now to
> figure out how to get it running properly with your desired configuration.
> Unfortunately at this point I don't know how to move forward except for
> trying to piecemeal from having it working with GCC without everything you
> need to working with PGI, CUDA and everything else.
>
> Andy****
>
> On Tue, Aug 28, 2012 at 2:51 PM, Vanmoer, Mark W <mvanmoer at illinois.edu>
> wrote:****
>
> [mvanmoer at forge ParaView]$ git log origin/master | head -1****
>
> commit 2537cabbf0cc9b39f17f347105ee18c559d0041a****
>
>  ****
>
> I should mention I do have a gcc, mvapich2 compiled version of
> coprocessing compiled on Forge and it works great. However any program
> built with PGI/openmpi code (required because of the CUDA Fortran) crashes
> with MPI errors. That’s why I was trying to build ParaView with
> PGI/openmpi. I guess I could try gcc/openmpi and see if that linked to
> pgi/openmpi will run. ****
>
>  ****
>
> Mark****
>
>  ****
>
>  ****
>
> *From:* Andy Bauer [mailto:andy.bauer at kitware.com]
> *Sent:* Tuesday, August 28, 2012 12:19 PM****
>
>
> *To:* Vanmoer, Mark W
> *Cc:* paraview at paraview.org
> *Subject:* Re: [Paraview] Python interpreter not being created for
> coprocessing?****
>
>  ****
>
> Hmm, ParaView has been undergoing a lot of changes lately so it's been a
> bit unstable. I just checked out a clean version and built it without any
> problems. The SHA from that was 2537cabbf0cc9b39f17f347105ee18c559d0041a.
> What version are you on?
>
> Have you tried building with GCC instead of PGI? I'm thinking of ways to
> simplify the build since I haven't tried building with PGI+CUDA+MPI+Python
> myself.
>
> Andy****
>
> On Mon, Aug 27, 2012 at 3:31 PM, Vanmoer, Mark W <mvanmoer at illinois.edu>
> wrote:****
>
> Hi Andy, I just did a fresh git clone and ccmake gives me****
>
> Make Error at VTK/CMake/vtkModuleTop.cmake:29 (message):****
>
>    No such module "vtkWrappingTools" needed by
> "vtkUtilitiesWrapClientServer"****
>
> Call Stack (most recent call first):****
>
>    VTK/CMake/vtkModuleTop.cmake:45 (vtk_module_check)****
>
>    VTK/CMake/vtkModuleTop.cmake:52 (vtk_module_check)****
>
>    VTK/CMakeLists.txt:334 (include)****
>
>  ****
>
> Mark****
>
>  ****
>
> *From:* Andy Bauer [mailto:andy.bauer at kitware.com]
> *Sent:* Monday, August 27, 2012 11:59 AM****
>
>
> *To:* Vanmoer, Mark W
> *Cc:* paraview at paraview.org
> *Subject:* Re: [Paraview] Python interpreter not being created for
> coprocessing?****
>
>  ****
>
> Hi Mark,
>
> With the modularization of VTK and ParaView the coprocessing tests were
> temporarily in a bad state. The serial ones are fixed now (the parallel
> runs but gives incorrect results for the screenshot). Is it possible for
> you to update your version of ParaView and retry this?
>
> Thanks,
> Andy****
>
> On Fri, Aug 24, 2012 at 4:03 PM, Vanmoer, Mark W <mvanmoer at illinois.edu>
> wrote:****
>
> Hi, I rebuilt with ParaViewData directory set and it’s only finding 1 out
> 3 of the CoProcessing tests. Did I forget a step?****
>
>  ****
>
> $ ctest --verbose --output-on-failure --output-log ctest.out -R CoProcess*
> ***
>
>  ****
>
> UpdateCTestConfiguration  from
> :/usr/apps/vis/ParaView/repo/withTesting/DartConfiguration.tcl****
>
> Parse Config
> file:/usr/apps/vis/ParaView/repo/withTesting/DartConfiguration.tcl****
>
> UpdateCTestConfiguration  from
> :/usr/apps/vis/ParaView/repo/withTesting/DartConfiguration.tcl****
>
> Parse Config
> file:/usr/apps/vis/ParaView/repo/withTesting/DartConfiguration.tcl****
>
> Test project /usr/apps/vis/ParaView/repo/withTesting****
>
> Constructing a list of tests****
>
> Done constructing a list of tests****
>
> Checking test dependency graph...****
>
> Checking test dependency graph end****
>
> test 29****
>
>     Start  29: vtkCoProcessor-HeaderTest****
>
>  ****
>
> 29: Test command: /usr/bin/python
> "/usr/apps/vis/ParaView/repo/ParaView/VTK/Testing/Core/HeaderTesting.py"
> "/usr/apps/vis/ParaView/repo/ParaView/CoProcessing/Core"
> "VTKCOPROCESSOR_EXPORT"****
>
> 29: Test timeout computed to be: 1500****
>
> 29: Use export macro: VTKCOPROCESSOR_EXPORT****
>
> 1/3 Test  #29: vtkCoProcessor-HeaderTest ........   Passed    0.37 sec****
>
> test 100****
>
>     Start 100: CoProcessingTestPythonScript****
>
>  ****
>
> 100: Test command: /usr/apps/vis/CMake/2.8.9/bin/cmake "-Dcfg=Release"
> "-P"
> "/usr/apps/vis/ParaView/repo/withTesting/CoProcessing/CoProcessingTestPythonScript.cmake"
> ****
>
> 100: Test timeout computed to be: 1500****
>
> 100: CMake Error at
> /usr/apps/vis/ParaView/repo/withTesting/CoProcessing/CoProcessingTestPythonScript.cmake:6
> (message):****
>
> 100:   ****
>
> 100:
> '/usr/apps/vis/ParaView/repo/withTesting/ThirdParty/protobuf/vtkprotobuf/bin/Release/CoProcessingPythonScriptExample.exe'
> ****
>
> 100:   does not exist****
>
> 100: ****
>
> 100: ****
>
> 2/3 Test #100: CoProcessingTestPythonScript .....***Failed    0.79 sec****
>
> CMake Error at
> /usr/apps/vis/ParaView/repo/withTesting/CoProcessing/CoProcessingTestPythonScript.cmake:6
> (message):****
>
>   ****
>
>
>   '/usr/apps/vis/ParaView/repo/withTesting/ThirdParty/protobuf/vtkprotobuf/bin/Release/CoProcessingPythonScriptExample.exe'
> ****
>
>   does not exist****
>
>  ****
>
>  ****
>
>  ****
>
> test 101****
>
>     Start 101: PCoProcessingTestPythonScript****
>
>  ****
>
> 101: Test command: /usr/apps/vis/CMake/2.8.9/bin/cmake "-Dcfg=Release"
> "-P"
> "/usr/apps/vis/ParaView/repo/withTesting/CoProcessing/PCoProcessingTestPythonScript.cmake"
> ****
>
> 101: Test timeout computed to be: 1500****
>
> 101: CMake Error at
> /usr/apps/vis/ParaView/repo/withTesting/CoProcessing/PCoProcessingTestPythonScript.cmake:6
> (message):****
>
> 101:   ****
>
> 101:
> '/usr/apps/vis/ParaView/repo/withTesting/ThirdParty/protobuf/vtkprotobuf/bin/Release/CoProcessingPythonScriptExample.exe'
> ****
>
> 101:   does not exist****
>
> 101: ****
>
> 101: ****
>
> 3/3 Test #101: PCoProcessingTestPythonScript ....***Failed    0.01 sec****
>
> CMake Error at
> /usr/apps/vis/ParaView/repo/withTesting/CoProcessing/PCoProcessingTestPythonScript.cmake:6
> (message):****
>
>   ****
>
>
>   '/usr/apps/vis/ParaView/repo/withTesting/ThirdParty/protobuf/vtkprotobuf/bin/Release/CoProcessingPythonScriptExample.exe'
> ****
>
>   does not exist****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
> The following tests passed:****
>
>                 vtkCoProcessor-HeaderTest****
>
>  ****
>
> 33% tests passed, 2 tests failed out of 3****
>
>  ****
>
> Total Test time (real) =   8.14 sec****
>
>  ****
>
> The following tests FAILED:****
>
>                 100 - CoProcessingTestPythonScript (Failed)****
>
>                 101 - PCoProcessingTestPythonScript (Failed)****
>
> Errors while running CTest****
>
>  ****
>
> Mark****
>
> *From:* Andy Bauer [mailto:andy.bauer at kitware.com]
> *Sent:* Thursday, August 23, 2012 3:06 PM****
>
>
> *To:* Vanmoer, Mark W
> *Cc:* paraview at paraview.org
> *Subject:* Re: [Paraview] Python interpreter not being created for
> coprocessing?****
>
>  ****
>
>  ****
>
> On Thu, Aug 23, 2012 at 4:00 PM, Vanmoer, Mark W <mvanmoer at illinois.edu>
> wrote:****
>
> Two questions – is there a way to tell ctest where the ParaViewData is
> without having to go back into cmake and rebuilding?****
>
>
> Not possible for most, if not all, tests. The reason is that the test
> doesn't get added unless ParaViewData is available.
>  ****
>
>  And 2, is there a way to tell ctest to not use MPIEXEC_MAX_NUMPROCS when
> running tests?****
>
>
> Many parallel tests, including the parallel coprocessing test, usually
> just have it hard coded to use 2 or 4 processes for testing.
>  ****
>
>   ****
>
> Mark****
>
> *From:* Andy Bauer [mailto:andy.bauer at kitware.com]
> *Sent:* Thursday, August 23, 2012 1:32 PM****
>
>
> *To:* Vanmoer, Mark W
> *Cc:* paraview at paraview.org
> *Subject:* Re: [Paraview] Python interpreter not being created for
> coprocessing?****
>
>  ****
>
> It looks like you've built it properly. At least I don't see anything that
> looks wrong. I think the reason you didn't get any coprocessing tests is
> that they depend on ParaViewDAta. Maybe the next step is to download
> ParaViewData (git://paraview.org/ParaViewData.git) and then set the
> location ParaViewData in cmake. After that, run the coprocessing tests
> (ctest -R CoProcess) and show me the output from that.
>
> Andy****
>
> On Wed, Aug 22, 2012 at 12:26 PM, Vanmoer, Mark W <mvanmoer at illinois.edu>
> wrote:****
>
> Hi Andy,****
>
>  ****
>
> I set PARAVIEW_ENABLE_PYTHON to ON, is more needed? It did build pvpython,
> I don’t know if that’s separate thing entirely, though.****
>
> $ pvserver –version****
>
> paraview version 3.14.1-1549-g5f34f40****
>
>  ****
>
> $ python –version****
>
> Python 2.6.6****
>
>  ****
>
> $ cmake –version****
>
> cmake version 2.8.9****
>
>  ****
>
> CMakeCache.txt is attached.****
>
>  ****
>
> I also just noticed that ctest –R CoProcessing returns No tests were
> found!!!, yet it does have other tests.****
>
> Let me know what else you need.****
>
>  ****
>
> Mark****
>
>  ****
>
> *From:* Andy Bauer [mailto:andy.bauer at kitware.com]
> *Sent:* Wednesday, August 22, 2012 10:49 AM
> *To:* Vanmoer, Mark W
> *Cc:* paraview at paraview.org
> *Subject:* Re: [Paraview] Python interpreter not being created for
> coprocessing?****
>
>  ****
>
> Hi Mark,
>
> I think I can guess what the problem is but can't be certain without some
> more information. My guess is that you're using ParaView 3.14.1 for this
> and you're trying to use the fortran interface but haven't built ParaView
> with Python enabled. There were some dependency issues there that have been
> ironed out since then though. If that's not the case though I'll need more
> information on versions and maybe sending your CMakeCache.txt in your
> ParaView build directory.
>
> Andy****
>
> On Wed, Aug 22, 2012 at 11:12 AM, Vanmoer, Mark W <mvanmoer at illinois.edu>
> wrote:****
>
> Hello,****
>
>  ****
>
> I'm setting up someone's code to use coprocessing with OSMesa and
> openmpi/pgi/cuda fortran on NCSA's Forge using ParaView from the git repo.
> I'm getting segfaults and pgdbg gives me this error when
> coprocessorinitialize is called:****
>
>  ****
>
> [0] Signalled SIGSEGV at 0x2B62188CFD9A, function
> vtkCPPythonHelper::GetPythonInterpretor(), file
> /usr/apps/vis/ParaView/repo/ParaView/CoProcessing/CoProcessor/vtkCPPythonHelper.cxx,
> line 136****
>
> 0x2B62188CFD9A:  48 8B 41 38            movq   56(%rcx),%rax****
>
>  ****
>
> which is:****
>
> vtkPVPythonInterpretor* vtkCPPythonHelper::GetPythonInterpretor()****
>
> {****
>
>   return Instance->PythonInterpretor;****
>
> }****
>
>  ****
>
> The call stack is****
>
> vtkCPPythonHelper::GetPythonInterpretor() line 136 in vtkCPPythonHelper.cxx
> ****
>
> vtkCPPythonScriptPipeline::Initialize(const char *) line 69 in
> vtkCPPythonScriptPipeline.cxx****
>
> coprocessorinitialize_ line 88 in FortranAdaptorAPI.cxx****
>
>  ****
>
>  ****
>
> So is an Instance not getting created? I have the system python in my path.
> ****
>
>  ****
>
> Thanks,****
>
> Mark****
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at:
> http://paraview.org/Wiki/ParaView
>
> Follow this link to subscribe/unsubscribe:
> http://www.paraview.org/mailman/listinfo/paraview****
>
>  ****
>
>  ****
>
>   ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
> ** **
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.paraview.org/pipermail/paraview/attachments/20120925/e4539de5/attachment-0001.htm>