[Paraview] Connecting to Catalyst-Enabled Simulation Running on Remote Cluster

Wed May 20 03:54:43 EDT 2015

I haven't used the examples, so I'm afraid I can't help a lot there.

On 05/20/2015 09:47 AM, Ufuk Utku Turuncoglu (BE) wrote:
> Hi Timo,
> 
> Thanks for your help. Actually, to keep it simple, i am using my laptop
> for both server and client. So, i think there is no need to open ssh
> tunnel in this case and also hash will not be an issue.
> 
> Steps:
> 
> 1 - installed Paraview 4.3.1 source under Mac OS 10.10 by enabling
> Python + OpenMPI + Development files and ParaView works fine.
> 2 - i compiled ParaViewCatalystExampleCode without any problem
> 3 - i run server: mpirun -np 2 ./pvserver (on port 11111)
> 4 - i run ParaView and connect to server and Catalyst (on port 22222)
> using menus
> 5 - i go to CxxMultiPieceExample directory and i run the code: mpirun
> -np 2 ./CxxMultiPieceExample SampleScripts/feslicescript.py
> 
> in this case, i modified SampleScripts/feslicescript.py as
> coprocessor.EnableLiveVisualization(True) to enable live visualization.
> In this case, the code is running but i am getting
> 
> Invalid output port : pqPipelineSource::getOutputPort( 0 ). Available
> number of output ports: 0
> 
> error under ParaView and also server crashes with following error,
> 
> ERROR: In
> /Users/turuncu/Qsync/progs/paraview-4.3.1/src/VTK/Parallel/Core/vtkSocketCommunicator.cxx,
> line 812
> vtkSocketCommunicator (0x7f993cf20b80): Could not receive tag. 1
> 
> [epsilon:07510] *** Process received signal ***
> [epsilon:07510] Signal: Segmentation fault: 11 (11)
> [epsilon:07510] Signal code: Address not mapped (1)
> [epsilon:07510] Failing at address: 0x30
> [epsilon:07510] [ 0] 0   libsystem_platform.dylib 0x00007fff851f1f1a
> _sigtramp + 26
> [epsilon:07510] [ 1] 0   ??? 0x00007f993b8578a2 0x0 + 140296105326754
> [epsilon:07510] [ 2] 0   libvtkCommonSystem-pv4.3.1.dylib
> 0x000000010fd5c9d9 _ZN9vtkSocket4SendEPKvi + 41
> [epsilon:07510] [ 3] 0   libvtkParallelCore-pv4.3.1.dylib
> 0x000000010f313803 _ZN21vtkSocketCommunicator10SendTaggedEPKviiiPKc + 67
> [epsilon:07510] [ 4] 0   libvtkParallelCore-pv4.3.1.dylib
> 0x000000010f3131a1 _ZN21vtkSocketCommunicator13SendVoidArrayEPKvxiii + 4625
> [epsilon:07510] [ 5] 0 libvtkPVClientServerCoreCore-pv4.3.1.dylib
> 0x000000010d5738e2 _ZN15vtkCommunicator4SendEPKcxii + 98
> [epsilon:07510] [ 6] 0 libvtkPVClientServerCoreCore-pv4.3.1.dylib
> 0x000000010d56fa4d _ZN25vtkMultiProcessController4SendEPKcxii + 77
> [epsilon:07510] [ 7] 0 libvtkPVClientServerCoreCore-pv4.3.1.dylib
> 0x000000010d56eca7 _ZN20vtkPVProgressHandler14RefreshMessageEPKc + 151
> [epsilon:07510] [ 8] 0 libvtkPVClientServerCoreCore-pv4.3.1.dylib
> 0x000000010d56d99b
> _ZN20vtkPVProgressHandler14OnMessageEventEP9vtkObjectmPv + 91
> [epsilon:07510] [ 9] 0 libvtkPVClientServerCoreCore-pv4.3.1.dylib
> 0x000000010d5719b8
> _ZN9vtkObject22vtkClassMemberCallbackI20vtkPVProgressHandlerEclEPS_mPv +
> 344
> [epsilon:07510] [10] 0   libvtkCommonCore-pv4.3.1.dylib
> 0x000000011056c7d2 _ZN24vtkObjectCommandInternal7ExecuteEP9vtkObjectmPv
> + 98
> [epsilon:07510] [11] 0   libvtkCommonCore-pv4.3.1.dylib
> 0x0000000110569a7b _ZN16vtkSubjectHelper11InvokeEventEmPvP9vtkObject + 4731
> [epsilon:07510] [12] 0   libvtkCommonCore-pv4.3.1.dylib
> 0x000000011056a052 _ZN9vtkObject11InvokeEventEmPv + 66
> [epsilon:07510] [13] 0   libvtkCommonCore-pv4.3.1.dylib
> 0x0000000110571a7b _ZN15vtkOutputWindow11DisplayTextEPKc + 219
> [epsilon:07510] [14] 0   libvtkCommonCore-pv4.3.1.dylib
> 0x0000000110571ac6 _ZN15vtkOutputWindow16DisplayErrorTextEPKc + 54
> [epsilon:07510] [15] 0   libvtkCommonCore-pv4.3.1.dylib
> 0x000000011057165c _Z31vtkOutputWindowDisplayErrorTextPKc + 44
> [epsilon:07510] [16] 0   libvtkParallelCore-pv4.3.1.dylib
> 0x000000010f315619 _ZN21vtkSocketCommunicator13ReceiveTaggedEPviiiPKc + 761
> [epsilon:07510] [17] 0   libvtkParallelCore-pv4.3.1.dylib
> 0x000000010f315196 _ZN21vtkSocketCommunicator16ReceiveVoidArrayEPvxiii +
> 4486
> [epsilon:07510] [18] 0   libvtkParallelCore-pv4.3.1.dylib
> 0x000000010f2d88a2 _ZN15vtkCommunicator7ReceiveEPhxii + 98
> [epsilon:07510] [19] 0   libvtkParallelCore-pv4.3.1.dylib
> 0x000000010f2e91a1 _ZN25vtkMultiProcessController11ProcessRMIsEii + 193
> [epsilon:07510] [20] 0 libvtkPVClientServerCoreCore-pv4.3.1.dylib
> 0x000000010d5bb89d
> _ZN26vtkTCPNetworkAccessManager21ProcessEventsInternalEmb + 2717
> [epsilon:07510] [21] 0 libvtkPVClientServerCoreCore-pv4.3.1.dylib
> 0x000000010d5bba72 _ZN26vtkTCPNetworkAccessManager13ProcessEventsEm + 34
> [epsilon:07510] [22] 0   pvserver 0x000000010ba83caa
> _ZL8RealMainiPPcN16vtkProcessModule12ProcessTypesE + 730
> [epsilon:07510] [23] 0   pvserver 0x000000010ba839b7 main + 39
> [epsilon:07510] [24] 0   libdyld.dylib 0x00007fff8cbe75c9 start + 1
> [epsilon:07510] *** End of error message ***
> --------------------------------------------------------------------------
> mpirun noticed that process rank 0 with PID 7510 on node epsilon exited
> on signal 11 (Segmentation fault: 11).
> --------------------------------------------------------------------------
> 
> So, i am unable to run Catalyst tests for now. They produces output
> *.vtm files (using writers in the pipeline) if i disable live
> visualization. Please let me know, if you have any suggestion.
> 
> Regards,
> 
> --ufuk
> 
> On 20/05/15 10:31, Timo Oster wrote:
>> Hi,
>>
>> I have indeed run into the problem with "Could not receive tag. 1". In
>> my case I had to make sure that the client and server versions were
>> exactly the same. When connecting, the vtkSocketCommunicators on client
>> and server exchange a version hash that must match. This hash is
>> generated while building ParaView. The file that is generated can be
>> found in your build directory under
>> VTK/Parallel/Core/vtkSocketCommunicatorHash.h. Check that your client
>> and server have the same hash value in this file.
>>
>> I am assuming that you have correctly set up an ssh tunnel between
>> client and server to enable the connection (ssh -L 11111:localhost:11111
>> user at server). The steps I take to run a in-situ case on a single remote
>> workstation are listed in my previous mail.
>>
>> BTW I am using the "trunk" version of ParaView from the git repository,
>> which is close to version 4.3.1, and I'm running Linux on both client
>> and server (different distributions).
>>
>> Regards,
>>
>> Timo
>>
>> On 05/19/2015 06:57 PM, u.utku.turuncoglu at be.itu.edu.tr wrote:
>>> Hi Timo,
>>>
>>> Have you ever tried to test Catalyst examples in your case.
>>>
>>> https://github.com/Kitware/ParaViewCatalystExampleCode
>>>
>>> I am also trying to run Catalyst examples but i could not. In my case, i
>>> am able to install Paraview (4.3.1) from source under Mac OS 10.10
>>> and the
>>> Catalys examples but when i try to run the examples i am getting error
>>> like,
>>>
>>> ERROR: In
>>> /Users/turuncu/Qsync/progs/paraview-4.3.1/src/VTK/Parallel/Core/vtkSocketCommunicator.cxx,
>>>
>>> line 812
>>> vtkSocketCommunicator (0x7f86a52658e0): Could not receive tag. 1
>>>
>>> or
>>>
>>> ERROR: In
>>> /Users/turuncu/Qsync/progs/paraview-4.3.1/src/VTK/Common/System/vtkSocket.cxx,
>>>
>>> line 572
>>> vtkClientSocket (0x7fcefc939fe0): Socket error in call to send.
>>> Broken pipe.
>>>
>>> So, if you don't mind could you share the details of your work
>>> environment
>>> (OS, Paraview version, etc.) and the steps to run the in-situ
>>> example. So,
>>> that could also help me to find the source of the problem.
>>>
>>> Regards,
>>>
>>> --ufuk
>>>
>>>> Hi all,
>>>>
>>>> in an effort to enable live visualization of our simulation code, I
>>>> have
>>>> written a Catalyst adaptor for it. The live visualization is working
>>>> great when the ParaView client runs on the same machine as the
>>>> simulation, even when the simulation runs in parallel using mpi.
>>>>
>>>> Now I want to do live visualization of a simulation running on a remote
>>>> cluster. I am able to get this to work for the simulation running on
>>>> the
>>>> login node of the cluster:
>>>>
>>>> 1. Tunnel an ssh connection of port 11111 to the cluster:
>>>>      ssh -L 11111:localhost:11111 server
>>>> 2. In the shell that opens, start a paraview server (in the background
>>>> with '&')
>>>> 3. Run the simulation with mpirun (simulation runs only on login node)
>>>> 4. Start my local ParaView client and connect to localhost:11111 to
>>>> connect to the remote pvserver through the ssh tunnel
>>>> 5. In the client, connect to Catalyst (port 22222)
>>>> 6. A provider for the simulation is created and the live visualization
>>>> works
>>>>
>>>> Now I want to do the same for simulations started on the remote cluster
>>>> via the batch job system. In this scenario, the parallel processes of
>>>> the simulation will run on different (randomly chosen) nodes on the
>>>> cluster. How do I go about getting a connection from my local client to
>>>> the Catalyst instances running on those nodes?
>>>>
>>>> I imagine I will need to set up ssh tunnels from the nodes to the login
>>>> node where the pvserver is running. I've tried adding a ssh tunnel line
>>>> to the job script that is executed when the batch job starts. I've
>>>> tried
>>>> forwarding and reverse-forwarding port 22222 (ssh -L and ssh -R) to no
>>>> avail. The best I get is "ERROR: In
>>>> /.../ParaView_git/VTK/Common/System/vtkSocket.cxx, line 206
>>>> vtkServerSocket (0x18e0930): Socket error in call to bind. Address
>>>> already in use."
>>>>
>>>> My knowledge in networking and ssh is limited, so any pointers to how I
>>>> would go about this are greatly appreciated.
>>>>
>>>> Regards,
>>>>
>>>> Timo
>>>> _______________________________________________
>>>> Powered by www.kitware.com
>>>>
>>>> Visit other Kitware open-source projects at
>>>> http://www.kitware.com/opensource/opensource.html
>>>>
>>>> Please keep messages on-topic and check the ParaView Wiki at:
>>>> http://paraview.org/Wiki/ParaView
>>>>
>>>> Search the list archives at: http://markmail.org/search/?q=ParaView
>>>>
>>>> Follow this link to subscribe/unsubscribe:
>>>> http://public.kitware.com/mailman/listinfo/paraview
>>>>
> 