[Paraview] Connecting to Catalyst-Enabled Simulation Running on Remote Cluster

Wed May 20 03:47:01 EDT 2015

Hi Timo,

Thanks for your help. Actually, to keep it simple, i am using my laptop 
for both server and client. So, i think there is no need to open ssh 
tunnel in this case and also hash will not be an issue.

Steps:

1 - installed Paraview 4.3.1 source under Mac OS 10.10 by enabling 
Python + OpenMPI + Development files and ParaView works fine.
2 - i compiled ParaViewCatalystExampleCode without any problem
3 - i run server: mpirun -np 2 ./pvserver (on port 11111)
4 - i run ParaView and connect to server and Catalyst (on port 22222) 
using menus
5 - i go to CxxMultiPieceExample directory and i run the code: mpirun 
-np 2 ./CxxMultiPieceExample SampleScripts/feslicescript.py

in this case, i modified SampleScripts/feslicescript.py as 
coprocessor.EnableLiveVisualization(True) to enable live visualization. 
In this case, the code is running but i am getting

Invalid output port : pqPipelineSource::getOutputPort( 0 ). Available 
number of output ports: 0

error under ParaView and also server crashes with following error,

ERROR: In 
/Users/turuncu/Qsync/progs/paraview-4.3.1/src/VTK/Parallel/Core/vtkSocketCommunicator.cxx, 
line 812
vtkSocketCommunicator (0x7f993cf20b80): Could not receive tag. 1

[epsilon:07510] *** Process received signal ***
[epsilon:07510] Signal: Segmentation fault: 11 (11)
[epsilon:07510] Signal code: Address not mapped (1)
[epsilon:07510] Failing at address: 0x30
[epsilon:07510] [ 0] 0   libsystem_platform.dylib 0x00007fff851f1f1a 
_sigtramp + 26
[epsilon:07510] [ 1] 0   ??? 0x00007f993b8578a2 0x0 + 140296105326754
[epsilon:07510] [ 2] 0   libvtkCommonSystem-pv4.3.1.dylib 
0x000000010fd5c9d9 _ZN9vtkSocket4SendEPKvi + 41
[epsilon:07510] [ 3] 0   libvtkParallelCore-pv4.3.1.dylib 
0x000000010f313803 _ZN21vtkSocketCommunicator10SendTaggedEPKviiiPKc + 67
[epsilon:07510] [ 4] 0   libvtkParallelCore-pv4.3.1.dylib 
0x000000010f3131a1 _ZN21vtkSocketCommunicator13SendVoidArrayEPKvxiii + 4625
[epsilon:07510] [ 5] 0 libvtkPVClientServerCoreCore-pv4.3.1.dylib 
0x000000010d5738e2 _ZN15vtkCommunicator4SendEPKcxii + 98
[epsilon:07510] [ 6] 0 libvtkPVClientServerCoreCore-pv4.3.1.dylib 
0x000000010d56fa4d _ZN25vtkMultiProcessController4SendEPKcxii + 77
[epsilon:07510] [ 7] 0 libvtkPVClientServerCoreCore-pv4.3.1.dylib 
0x000000010d56eca7 _ZN20vtkPVProgressHandler14RefreshMessageEPKc + 151
[epsilon:07510] [ 8] 0 libvtkPVClientServerCoreCore-pv4.3.1.dylib 
0x000000010d56d99b 
_ZN20vtkPVProgressHandler14OnMessageEventEP9vtkObjectmPv + 91
[epsilon:07510] [ 9] 0 libvtkPVClientServerCoreCore-pv4.3.1.dylib 
0x000000010d5719b8 
_ZN9vtkObject22vtkClassMemberCallbackI20vtkPVProgressHandlerEclEPS_mPv + 344
[epsilon:07510] [10] 0   libvtkCommonCore-pv4.3.1.dylib 
0x000000011056c7d2 _ZN24vtkObjectCommandInternal7ExecuteEP9vtkObjectmPv + 98
[epsilon:07510] [11] 0   libvtkCommonCore-pv4.3.1.dylib 
0x0000000110569a7b _ZN16vtkSubjectHelper11InvokeEventEmPvP9vtkObject + 4731
[epsilon:07510] [12] 0   libvtkCommonCore-pv4.3.1.dylib 
0x000000011056a052 _ZN9vtkObject11InvokeEventEmPv + 66
[epsilon:07510] [13] 0   libvtkCommonCore-pv4.3.1.dylib 
0x0000000110571a7b _ZN15vtkOutputWindow11DisplayTextEPKc + 219
[epsilon:07510] [14] 0   libvtkCommonCore-pv4.3.1.dylib 
0x0000000110571ac6 _ZN15vtkOutputWindow16DisplayErrorTextEPKc + 54
[epsilon:07510] [15] 0   libvtkCommonCore-pv4.3.1.dylib 
0x000000011057165c _Z31vtkOutputWindowDisplayErrorTextPKc + 44
[epsilon:07510] [16] 0   libvtkParallelCore-pv4.3.1.dylib 
0x000000010f315619 _ZN21vtkSocketCommunicator13ReceiveTaggedEPviiiPKc + 761
[epsilon:07510] [17] 0   libvtkParallelCore-pv4.3.1.dylib 
0x000000010f315196 _ZN21vtkSocketCommunicator16ReceiveVoidArrayEPvxiii + 
4486
[epsilon:07510] [18] 0   libvtkParallelCore-pv4.3.1.dylib 
0x000000010f2d88a2 _ZN15vtkCommunicator7ReceiveEPhxii + 98
[epsilon:07510] [19] 0   libvtkParallelCore-pv4.3.1.dylib 
0x000000010f2e91a1 _ZN25vtkMultiProcessController11ProcessRMIsEii + 193
[epsilon:07510] [20] 0 libvtkPVClientServerCoreCore-pv4.3.1.dylib 
0x000000010d5bb89d 
_ZN26vtkTCPNetworkAccessManager21ProcessEventsInternalEmb + 2717
[epsilon:07510] [21] 0 libvtkPVClientServerCoreCore-pv4.3.1.dylib 
0x000000010d5bba72 _ZN26vtkTCPNetworkAccessManager13ProcessEventsEm + 34
[epsilon:07510] [22] 0   pvserver 0x000000010ba83caa 
_ZL8RealMainiPPcN16vtkProcessModule12ProcessTypesE + 730
[epsilon:07510] [23] 0   pvserver 0x000000010ba839b7 main + 39
[epsilon:07510] [24] 0   libdyld.dylib 0x00007fff8cbe75c9 start + 1
[epsilon:07510] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 7510 on node epsilon exited 
on signal 11 (Segmentation fault: 11).
--------------------------------------------------------------------------

So, i am unable to run Catalyst tests for now. They produces output 
*.vtm files (using writers in the pipeline) if i disable live 
visualization. Please let me know, if you have any suggestion.

Regards,

--ufuk

On 20/05/15 10:31, Timo Oster wrote:
> Hi,
>
> I have indeed run into the problem with "Could not receive tag. 1". In
> my case I had to make sure that the client and server versions were
> exactly the same. When connecting, the vtkSocketCommunicators on client
> and server exchange a version hash that must match. This hash is
> generated while building ParaView. The file that is generated can be
> found in your build directory under
> VTK/Parallel/Core/vtkSocketCommunicatorHash.h. Check that your client
> and server have the same hash value in this file.
>
> I am assuming that you have correctly set up an ssh tunnel between
> client and server to enable the connection (ssh -L 11111:localhost:11111
> user at server). The steps I take to run a in-situ case on a single remote
> workstation are listed in my previous mail.
>
> BTW I am using the "trunk" version of ParaView from the git repository,
> which is close to version 4.3.1, and I'm running Linux on both client
> and server (different distributions).
>
> Regards,
>
> Timo
>
> On 05/19/2015 06:57 PM, u.utku.turuncoglu at be.itu.edu.tr wrote:
>> Hi Timo,
>>
>> Have you ever tried to test Catalyst examples in your case.
>>
>> https://github.com/Kitware/ParaViewCatalystExampleCode
>>
>> I am also trying to run Catalyst examples but i could not. In my case, i
>> am able to install Paraview (4.3.1) from source under Mac OS 10.10 and the
>> Catalys examples but when i try to run the examples i am getting error
>> like,
>>
>> ERROR: In
>> /Users/turuncu/Qsync/progs/paraview-4.3.1/src/VTK/Parallel/Core/vtkSocketCommunicator.cxx,
>> line 812
>> vtkSocketCommunicator (0x7f86a52658e0): Could not receive tag. 1
>>
>> or
>>
>> ERROR: In
>> /Users/turuncu/Qsync/progs/paraview-4.3.1/src/VTK/Common/System/vtkSocket.cxx,
>> line 572
>> vtkClientSocket (0x7fcefc939fe0): Socket error in call to send. Broken pipe.
>>
>> So, if you don't mind could you share the details of your work environment
>> (OS, Paraview version, etc.) and the steps to run the in-situ example. So,
>> that could also help me to find the source of the problem.
>>
>> Regards,
>>
>> --ufuk
>>
>>> Hi all,
>>>
>>> in an effort to enable live visualization of our simulation code, I have
>>> written a Catalyst adaptor for it. The live visualization is working
>>> great when the ParaView client runs on the same machine as the
>>> simulation, even when the simulation runs in parallel using mpi.
>>>
>>> Now I want to do live visualization of a simulation running on a remote
>>> cluster. I am able to get this to work for the simulation running on the
>>> login node of the cluster:
>>>
>>> 1. Tunnel an ssh connection of port 11111 to the cluster:
>>>      ssh -L 11111:localhost:11111 server
>>> 2. In the shell that opens, start a paraview server (in the background
>>> with '&')
>>> 3. Run the simulation with mpirun (simulation runs only on login node)
>>> 4. Start my local ParaView client and connect to localhost:11111 to
>>> connect to the remote pvserver through the ssh tunnel
>>> 5. In the client, connect to Catalyst (port 22222)
>>> 6. A provider for the simulation is created and the live visualization
>>> works
>>>
>>> Now I want to do the same for simulations started on the remote cluster
>>> via the batch job system. In this scenario, the parallel processes of
>>> the simulation will run on different (randomly chosen) nodes on the
>>> cluster. How do I go about getting a connection from my local client to
>>> the Catalyst instances running on those nodes?
>>>
>>> I imagine I will need to set up ssh tunnels from the nodes to the login
>>> node where the pvserver is running. I've tried adding a ssh tunnel line
>>> to the job script that is executed when the batch job starts. I've tried
>>> forwarding and reverse-forwarding port 22222 (ssh -L and ssh -R) to no
>>> avail. The best I get is "ERROR: In
>>> /.../ParaView_git/VTK/Common/System/vtkSocket.cxx, line 206
>>> vtkServerSocket (0x18e0930): Socket error in call to bind. Address
>>> already in use."
>>>
>>> My knowledge in networking and ssh is limited, so any pointers to how I
>>> would go about this are greatly appreciated.
>>>
>>> Regards,
>>>
>>> Timo
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Please keep messages on-topic and check the ParaView Wiki at:
>>> http://paraview.org/Wiki/ParaView
>>>
>>> Search the list archives at: http://markmail.org/search/?q=ParaView
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://public.kitware.com/mailman/listinfo/paraview
>>>