[Paraview] crash when picking with 'p' for stream tracer : Tag mismatch: got 1, expecting 22222

[Burlen Loring]

another data point on this, a similar crash occurs seemingly randomly 
(see output below). I'm seeing this crash or a variant of it on pretty 
much all the systems I'm using. Sometimes I can run for 5min and other a 
few hours, but I'm crashing alot always with:  Tag mismatch: got 1, 
expecting 22222.


ERROR: In 
/home/bloring/builds/ParaView/VTK/Parallel/Core/vtkSocketCommunicator.cxx, 
line 878
vtkSocketCommunicator (0xf2d270): Tag mismatch: got 1, expecting 22222.

pvserver: 
/home/bloring/builds/ParaView/VTK/Parallel/Core/vtkMultiProcessStream.cxx:637: 
vtkMultiProcessStream& vtkMultiProcessStream::operator>>(unsigned int&): 
Assertion `this->Internals->Data.front() == vtkInternals::uint32_value' 
failed.
Generic Warning: In 
/home/bloring/builds/ParaView/VTK/Parallel/MPI/vtkMPICommunicator.cxx, 
line 99

MPI had an error
------------------------------------------------

A process has failed, error stack:
PMPI_Barrier(426)...........: MPI_Barrier(MPI_COMM_WORLD) failed
MPIR_Barrier_impl(294)......:
MPIR_Barrier_or_coll_fn(121):
MPIR_Barrier_intra(83)......:
dequeue_and_set_error(888)..: Communication error with rank 0
MPIR_Barrier_impl(308)......:
MPIR_Bcast_impl(1369).......:
MPIR_Bcast_intra(1199)......:
MPIR_Bcast_binomial(220)....: Failure during collective

Program Stack:
------------------------------------------------
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkParallelMPI-pv4.0.so.1 
[0x2b773f94aea7]
/home/bloring/installs/mpich/3.0.4/lib/libmpich.so.10(MPIR_Err_return_comm+0x1bd) 
[0x2b77418465bd]
/home/bloring/installs/mpich/3.0.4/lib/libmpich.so.10(MPI_Barrier+0xda) 
[0x2b77418bd7ca]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkParallelMPI-pv4.0.so.1 
[0x2b773f94ac95]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkParallelMPI-pv4.0.so.1(_ZN18vtkMPICommunicator18BroadcastVoidArrayEPvxii+0x2f) 
[0x2b773f94e3ab]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkParallelCore-pv4.0.so.1 
[0x2b773fb9b4bc]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkParallelCore-pv4.0.so.1(_ZN15vtkCommunicator9BroadcastER21vtkMultiProcessStreami+0x110) 
[0x2b773fb9a732]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVClientServerCoreRendering-pv4.0.so.1 
[0x2b774d5c7901]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVClientServerCoreRendering-pv4.0.so.1(_ZN30vtkPVSynchronizedRenderWindows20SatelliteStartRenderEP15vtkRenderWindow+0x46) 
[0x2b774d60938a]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVClientServerCoreRendering-pv4.0.so.1(_ZN30vtkPVSynchronizedRenderWindows17HandleStartRenderEP15vtkRenderWindow+0xc0) 
[0x2b774d608f36]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVClientServerCoreRendering-pv4.0.so.1 
[0x2b774d60db58]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkCommonCore-pv4.0.so.1 
[0x2b77431ba183]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkCommonCore-pv4.0.so.1(_ZN9vtkObject11InvokeEventEmPv+0x3d) 
[0x2b77431ba693]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkRenderingCore-pv4.0.so.1(_ZN15vtkRenderWindow6RenderEv+0x1e7) 
[0x2b773b696c2d]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkRenderingCore-pv4.0.so.1(_ZN19vtkHardwareSelector14CaptureBuffersEv+0x473) 
[0x2b773b5fb60b]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVVTKExtensionsRendering-pv4.0.so.1(_ZN21vtkPVHardwareSelector13PrepareSelectEv+0xef) 
[0x2b774e61ce93]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVVTKExtensionsRendering-pv4.0.so.1(_ZN21vtkPVHardwareSelector6SelectEPi+0x1c) 
[0x2b774e61cee8]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVClientServerCoreRendering-pv4.0.so.1(_ZN15vtkPVRenderView6SelectEiPi+0x9f) 
[0x2b774d5f714b]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVClientServerCoreRendering-pv4.0.so.1(_ZN15vtkPVRenderView11SelectCellsEPi+0x25) 
[0x2b774d5f6e99]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVServerManagerApplication-pv4.0.so.1(_ZN15vtkPVRenderView11SelectCellsEiiii+0x44) 
[0x2b7739ab6736]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVServerManagerApplication-pv4.0.so.1(_Z22vtkPVRenderViewCommandP26vtkClientServerInterpreterP13vtkObjectBasePKcRK21vtkClientServerStreamRS5_+0x22dc) 
[0x2b7739ab214f]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkClientServer-pv4.0.so.1(_ZN26vtkClientServerInterpreter20ProcessCommandInvokeERK21vtkClientServerStreami+0x18e) 
[0x2b773ec67564]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkClientServer-pv4.0.so.1(_ZN26vtkClientServerInterpreter17ProcessOneMessageERK21vtkClientServerStreami+0x135) 
[0x2b773ec66c9d]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkClientServer-pv4.0.so.1(_ZN26vtkClientServerInterpreter13ProcessStreamERK21vtkClientServerStream+0x2f) 
[0x2b773ec66b37]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVServerImplementationCore-pv4.0.so.1(_ZN16vtkPVSessionCore21ExecuteStreamInternalERK21vtkClientServerStreamb+0xdb) 
[0x2b773a91b7bf]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVServerImplementationCore-pv4.0.so.1(_ZN16vtkPVSessionCore30ExecuteStreamSatelliteCallbackEv+0xd5) 
[0x2b773a91b6a3]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkPVServerImplementationCore-pv4.0.so.1 
[0x2b773a91938f]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkParallelCore-pv4.0.so.1(_ZN25vtkMultiProcessController10ProcessRMIEiPvii+0x3a6) 
[0x2b773fba72f2]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkParallelCore-pv4.0.so.1(_ZN25vtkMultiProcessController11ProcessRMIsEii+0x63c) 
[0x2b773fba6e88]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/libvtkParallelCore-pv4.0.so.1(_ZN25vtkMultiProcessController11ProcessRMIsEv+0x22) 
[0x2b773fba684a]
   /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/pvserver 
[0x401d30]
/home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/pvserver(main+0x25) 
[0x401dca]
   /lib64/libc.so.6(__libc_start_main+0xf4) [0x3fab01d994]
   /home/bloring/installs/ParaView/4.0.0/lib/paraview-4.0/pvserver 
[0x401a09]




On 05/31/2013 03:15 PM, Burlen Loring wrote:
> can also reproduce with the point source (open data, change to surface 
> rep, add point source, pick)
> vtk dataset that can be used to reproduce: 
> http://hpcvis.com/downloads/pr1-vti.tar.gz
>
> On 05/31/2013 01:54 PM, Burlen Loring wrote:
>> Hi All,
>>
>> With git master (first noticed a few days ago, rebased this am) I'm 
>> hitting crash when attempting to use the stream tracer interactive 
>> picking a seed.
>>
>> 1) load the data,
>> 2) change to surface rep,
>> 3) add stream tracer filter
>> 4) hit the 'p' key to select a seed -> crash.
>>
>> I've only been able to reproduce with a real dataset (1024^3 image 
>> mesh). the small data I've tried from ParaViewData doesn't lead to 
>> the crash,  However, there's plenty of ram (first encountered using 
>> 32 ranks 4 ranks per node 48g per node on longhorn reduced to 4 ranks 
>> 24g  on my workstation for debugging). I can share this dataset if it 
>> helps to debug, or you may have to use a large dataset to trigger.
>>
>> filed a bug report: http://paraview.org/Bug/view.php?id=14107
>>
>> here's cerr output and a stack
>>
>> 3437 ERROR: In 
>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Parallel/Core/vtkSocketCommunicator.cxx, 
>> line 877
>> 3438 vtkSocketCommunicator (0x2efd2d0): Tag mismatch: got 1, 
>> expecting 22222.
>> 3439
>> 3440 pvserver: 
>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Parallel/Core/vtkMultiProcessStream.cxx:637: 
>> vtkMultiProcessStream &vtkMultiProcessStream::operator>>(unsigned int 
>> &): Assertion `this->Internals->Data.front() == vtkInter 
>> nals::uint32_value' failed.
>>
>> #0  0x00000033aea30215 in raise () from /lib64/libc.so.6
>> #1  0x00000033aea31cc0 in abort () from /lib64/libc.so.6
>> #2  0x00000033aea29696 in __assert_fail () from /lib64/libc.so.6
>> #3  0x00002b6ae159c3ed in vtkMultiProcessStream::operator>> 
>> (this=0x7fffea94ed68, value=@0x7fffea94ed1c)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Parallel/Core/vtkMultiProcessStream.cxx:637
>> #4  0x00002b6aef371cdd in 
>> vtkPVSynchronizedRenderWindows::LoadWindowAndLayout (this=0x2835d50, 
>> window=0x26e9650, stream=@0x7fffea94ed68)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerCore/Rendering/vtkPVSynchronizedRenderWindows.cxx:998
>> #5  0x00002b6aef37155b in 
>> vtkPVSynchronizedRenderWindows::RootStartRender (this=0x2835d50, 
>> renWin=0x26e9650)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerCore/Rendering/vtkPVSynchronizedRenderWindows.cxx:901
>> #6  0x00002b6aef3711aa in 
>> vtkPVSynchronizedRenderWindows::HandleStartRender (this=0x2835d50, 
>> renWin=0x26e9650)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerCore/Rendering/vtkPVSynchronizedRenderWindows.cxx:814
>> #7  0x00002b6aef36dc5a in 
>> vtkPVSynchronizedRenderWindows::vtkObserver::Execute (this=0x282f660, 
>> ocaller=0x26e9650, eventId=3)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerCore/Rendering/vtkPVSynchronizedRenderWindows.cxx:219
>> #8  0x00002b6ae374c9ff in vtkSubjectHelper::InvokeEvent 
>> (this=0x282e970, event=3, callData=0x0, self=0x26e9650) at 
>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Common/Core/vtkObject.cxx:620
>> #9  0x00002b6ae374d028 in vtkObject::InvokeEvent (this=0x26e9650, 
>> event=3, callData=0x0) at 
>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Common/Core/vtkObject.cxx:789
>> #10 0x00002b6adc9ee77a in vtkRenderWindow::Render (this=0x26e9650) at 
>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Rendering/Core/vtkRenderWindow.cxx:313
>> #11 0x00002b6af42443f7 in vtkXOpenGLRenderWindow::Render 
>> (this=0x26e9650) at 
>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Rendering/OpenGL/vtkXOpenGLRenderWindow.cxx:1826
>> #12 0x00002b6adc9183d6 in vtkHardwareSelector::CaptureBuffers 
>> (this=0x26e9460) at 
>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Rendering/Core/vtkHardwareSelector.cxx:288
>> #13 0x00002b6af05119ef in vtkPVHardwareSelector::PrepareSelect 
>> (this=0x26e9460) at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/VTKExtensions/Rendering/vtkPVHardwareSelector.cxx:62
>> #14 0x00002b6af0511a54 in vtkPVHardwareSelector::Select 
>> (this=0x26e9460, region=0x7fffea94f750)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/VTKExtensions/Rendering/vtkPVHardwareSelector.cxx:75
>> #15 0x00002b6aef353e31 in vtkPVRenderView::Select (this=0x2833ce0, 
>> fieldAssociation=1, region=0x7fffea94f750)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerCore/Rendering/vtkPVRenderView.cxx:597
>> #16 0x00002b6aef353a19 in vtkPVRenderView::SelectCells 
>> (this=0x2833ce0, region=0x7fffea94f750)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerCore/Rendering/vtkPVRenderView.cxx:556
>> #17 0x00002b6ada8b1064 in vtkPVRenderView::SelectCells 
>> (this=0x2833ce0, region0=218, region1=142, region2=218, region3=142)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerCore/Rendering/vtkPVRenderView.h:246
>> #18 0x00002b6ada8ab300 in vtkPVRenderViewCommand (arlu=0x227b4f0, 
>> ob=0x2833ce0, method=0x3dfbd19 "SelectCells", msg=@0x7fffea951538, 
>> resultStream=@0x227bdf0)
>>     at 
>> /scratch/01237/bloring/ParaView/next/PV/Wrapping/ClientServer/vtkPVRenderViewClientServer.cxx:545
>> #19 0x00002b6ae04b896e in 
>> vtkClientServerInterpreter::ProcessCommandInvoke (this=0x227b4f0, 
>> css=@0x7fffea951b90, midx=0)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerStream/vtkClientServerInterpreter.cxx:379
>> #20 0x00002b6ae04b7d5b in 
>> vtkClientServerInterpreter::ProcessOneMessage (this=0x227b4f0, 
>> css=@0x7fffea951b90, message=0)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerStream/vtkClientServerInterpreter.cxx:214
>> #21 0x00002b6ae04b7c0e in vtkClientServerInterpreter::ProcessStream 
>> (this=0x227b4f0, css=@0x7fffea951b90)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerStream/vtkClientServerInterpreter.cxx:183
>> #22 0x00002b6adba173cd in vtkPVSessionCore::ExecuteStreamInternal 
>> (this=0x227b990, stream=@0x7fffea951b90, ignore_errors=false)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ServerImplementation/Core/vtkPVSessionCore.cxx:665
>> #23 0x00002b6adba1711e in vtkPVSessionCore::ExecuteStream 
>> (this=0x227b990, location=21, stream=@0x7fffea951b90, 
>> ignore_errors=false)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ServerImplementation/Core/vtkPVSessionCore.cxx:635
>> ---Type <return> to continue, or q <return> to quit---
>> #24 0x00002b6adba11dde in vtkPVSessionBase::ExecuteStream 
>> (this=0x1a6ae50, location=21, stream=@0x7fffea951b90, 
>> ignore_errors=false)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ServerImplementation/Core/vtkPVSessionBase.cxx:173
>> #25 0x00002b6adba284b4 in 
>> vtkPVSessionServer::OnClientServerMessageRMI (this=0x1a6ae50, 
>> message=0x3cd3d00, message_length=16)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ServerImplementation/Core/vtkPVSessionServer.cxx:510
>> #26 0x00002b6adba25d2e in (anonymous namespace)::RMICallback 
>> (localArg=0x1a6ae50, remoteArg=0x3cd3d00, remoteArgLength=16)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ServerImplementation/Core/vtkPVSessionServer.cxx:56
>> #27 0x00002b6ae158abcc in vtkMultiProcessController::ProcessRMI 
>> (this=0x2efd1a0, remoteProcessId=1, arg=0x3cd3d00, argLength=16, 
>> rmiTag=55625)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Parallel/Core/vtkMultiProcessController.cxx:650
>> #28 0x00002b6ae158a48f in vtkMultiProcessController::ProcessRMIs 
>> (this=0x2efd1a0, reportErrors=0, dont_loop=1)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/VTK/Parallel/Core/vtkMultiProcessController.cxx:596
>> #29 0x00002b6adbec2d07 in 
>> vtkTCPNetworkAccessManager::ProcessEventsInternal (this=0x174f090, 
>> timeout_msecs=0, do_processing=true)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerCore/Core/vtkTCPNetworkAccessManager.cxx:268
>> #30 0x00002b6adbec25d0 in vtkTCPNetworkAccessManager::ProcessEvents 
>> (this=0x174f090, timeout_msecs=0)
>>     at 
>> /scratch/01237/bloring/ParaView/next/ParaView/ParaViewCore/ClientServerCore/Core/vtkTCPNetworkAccessManager.cxx:165
>> #31 0x00000000004035e0 in RealMain (argc=5, argv=0x7fffea953418, 
>> type=vtkProcessModule::PROCESS_SERVER) at 
>> /scratch/01237/bloring/ParaView/next/ParaView/CommandLineExecutables/pvserver_common.h:90
>> #32 0x000000000040305d in main (argc=5, argv=0x7fffea953418) at 
>> /scratch/01237/bloring/ParaView/next/ParaView/CommandLineExecutables/pvserver.cxx:21 
>>
>>
>