[Paraview] reducing startup time on cray

Burlen Loring bloring at lbl.gov
Fri Feb 6 13:55:25 EST 2015 

Hi All,

I want to share a performance issue related to PV start up time, which 
in general on our cray's is significant usability issue, especially on 
Hopper.

Earlier this week NERSC did some upgrades on one of our Crays which 
introduced some issues that prevent ParaView from starting. In the end 
we found that the issue is a system issue (file system or python 
problem) and not ParaView, but this exposed a performance issue in 
ParaView. ParaView would connect then hang when creating the render 
view. The debugger shows ParaView stuck in a python import call(stack is 
below) from vtkMatplotlibMathTextUtilities::CheckMPLAvailability. This 
function initializes the python interpreter and tries to "import 
matplotlib".

Importing a python module (that's not built into the core of python) is 
an extremely expensive and unscalable operation in HPC setting. Software 
like python and ParaView, at least at NERSC, lives on slow remote file 
systems and in parallel the import hammers these file systems from each 
process. Python is essential for batch processing, but when using the 
interactive ParaView we should delay/avoid python imports as much as 
possible , especially during start up. This is a scalability issue in 
that most user aren't going to use the tex feature, but all users 
startup time is impacted, especially as job size grows.

I think in the case of ParaView's matplotlib tex features the run time 
check for matplotlib could probably be deferred until someone actually 
tries to use tex. What do you think? This would help keep startup time fast.

Burlen

proc_set{0}: #0 0x00002aaaae6b8fd0 in __open_nocancel
proc_set{0}: #1 0x00002aaaae65e483 in __GI__IO_file_open
proc_set{0}: #2 0x00002aaaae65e5fb in _IO_new_file_fopen
proc_set{0}: #3 0x00002aaaae6522f4 in __fopen_internal
proc_set{0}: #4 0x00002aaaaddb833c in find_module at Python/import.c:1556
proc_set{0}: #5 0x00002aaaaddba470 in import_submodule at 
Python/import.c:2689
proc_set{0}: #6 0x00002aaaaddbb0e3 in load_next at Python/import.c:2515
proc_set{0}: #7 0x00002aaaaddbb0e3 in import_module_level at 
Python/import.c:2224
proc_set{0}: #8 0x00002aaaaddbb0e3 in PyImport_ImportModuleLevel at 
Python/import.c:2288
proc_set{0}: #9 0x00002aaaadd998cf in builtin___import__ at 
Python/bltinmodule.c:49
proc_set{0}: #10 0x00002aaaadceae93 in PyObject_Call at 
Objects/abstract.c:2529
proc_set{0}: #11 0x00002aaaadd9b327 in PyEval_CallObjectWithKeywords at 
Python/ceval.c:3890
proc_set{0}: #12 0x00002aaaadd9e296 in PyEval_EvalFrameEx at 
Python/ceval.c:2333
proc_set{0}: #13 0x00002aaaadda198d in PyEval_EvalCodeEx at 
Python/ceval.c:3253
proc_set{0}: #14 0x00002aaaadda1ac2 in PyEval_EvalCode at Python/ceval.c:667
proc_set{0}: #15 0x00002aaaaddb924c in PyImport_ExecCodeModuleEx at 
Python/import.c:709
proc_set{0}: #16 0x00002aaaaddb95ae in load_source_module at 
Python/import.c:1099
proc_set{0}: #17 0x00002aaaaddba4a9 in import_submodule at 
Python/import.c:2700
proc_set{0}: #18 0x00002aaaaddbb0e3 in load_next at Python/import.c:2515
proc_set{0}: #19 0x00002aaaaddbb0e3 in import_module_level at 
Python/import.c:2224
proc_set{0}: #20 0x00002aaaaddbb0e3 in PyImport_ImportModuleLevel at 
Python/import.c:2288
proc_set{0}: #21 0x00002aaaadd998cf in builtin___import__ at 
Python/bltinmodule.c:49
proc_set{0}: #22 0x00002aaaadceae93 in PyObject_Call at 
Objects/abstract.c:2529
proc_set{0}: #23 0x00002aaaadd9b327 in PyEval_CallObjectWithKeywords at 
Python/ceval.c:3890
proc_set{0}: #24 0x00002aaaadd9e296 in PyEval_EvalFrameEx at 
Python/ceval.c:2333
proc_set{0}: #25 0x00002aaaadda198d in PyEval_EvalCodeEx at 
Python/ceval.c:3253
proc_set{0}: #26 0x00002aaaadda1ac2 in PyEval_EvalCode at Python/ceval.c:667
proc_set{0}: #27 0x00002aaaaddb924c in PyImport_ExecCodeModuleEx at 
Python/import.c:709
proc_set{0}: #28 0x00002aaaaddb95ae in load_source_module at 
Python/import.c:1099
proc_set{0}: #29 0x00002aaaaddba4a9 in import_submodule at 
Python/import.c:2700
proc_set{0}: #30 0x00002aaaaddbb567 in load_next at Python/import.c:2519
proc_set{0}: #31 0x00002aaaaddbb567 in import_module_level at 
Python/import.c:2224
proc_set{0}: #32 0x00002aaaaddbb567 in PyImport_ImportModuleLevel at 
Python/import.c:2288
proc_set{0}: #33 0x00002aaaadd998cf in builtin___import__ at 
Python/bltinmodule.c:49
proc_set{0}: #34 0x00002aaaadceae93 in PyObject_Call at 
Objects/abstract.c:2529
proc_set{0}: #35 0x00002aaaadd9b327 in PyEval_CallObjectWithKeywords at 
Python/ceval.c:3890
proc_set{0}: #36 0x00002aaaadd9e296 in PyEval_EvalFrameEx at 
Python/ceval.c:2333
proc_set{0}: #37 0x00002aaaadda198d in PyEval_EvalCodeEx at 
Python/ceval.c:3253
proc_set{0}: #38 0x00002aaaadda1ac2 in PyEval_EvalCode at Python/ceval.c:667
proc_set{0}: #39 0x00002aaaaddb924c in PyImport_ExecCodeModuleEx at 
Python/import.c:709
proc_set{0}: #40 0x00002aaaaddb95ae in load_source_module at 
Python/import.c:1099
proc_set{0}: #41 0x00002aaaaddba9e9 in load_package at Python/import.c:1166
proc_set{0}: #42 0x00002aaaaddba4a9 in import_submodule at 
Python/import.c:2700
proc_set{0}: #43 0x00002aaaaddbb0e3 in load_next at Python/import.c:2515
proc_set{0}: #44 0x00002aaaaddbb0e3 in import_module_level at 
Python/import.c:2224
proc_set{0}: #45 0x00002aaaaddbb0e3 in PyImport_ImportModuleLevel at 
Python/import.c:2288
proc_set{0}: #46 0x00002aaaadd998cf in builtin___import__ at 
Python/bltinmodule.c:49
proc_set{0}: #47 0x00002aaaadceae93 in PyObject_Call at 
Objects/abstract.c:2529
proc_set{0}: #48 0x00002aaaadceafeb in call_function_tail at 
Objects/abstract.c:2561
proc_set{0}: #49 0x00002aaaadceafeb in PyObject_CallFunction at 
Objects/abstract.c:2585
proc_set{0}: #50 0x00002aaaaddbbeb2 in PyImport_Import at 
Python/import.c:2886
proc_set{0}: #51 0x00002aaaaddbc02a in PyImport_ImportModule at 
Python/import.c:2129
proc_set{0}: #52 0x0000000004766407 in 
vtkMatplotlibMathTextUtilities::CheckMPLAvailability at 
/global/common/edison/graphics/ParaView/builds/ParaView/VTK/Rendering/Matplotlib/vtkMatplotlibMathTextUtilities.cxx:75
proc_set{0}: #53 0x000000000476681a in 
vtkMatplotlibMathTextUtilities::New at 
/global/common/edison/graphics/ParaView/builds/ParaView/VTK/Rendering/Matplotlib/vtkMatplotlibMathTextUtilities.cxx:119
proc_set{0}: #54 0x000000000476abed in 
vtkObjectFactoryCreatevtkMatplotlibMathTextUtilities at 
/global/common/edison/graphics/ParaView/builds/PV-4.2.0/VTK/Rendering/Matplotlib/vtkRenderingMatplotlibObjectFactory.cxx:27
proc_set{0}: #55 0x0000000007336149 in vtkObjectFactory::CreateObject at 
/global/common/edison/graphics/ParaView/builds/ParaView/VTK/Common/Core/vtkObjectFactory.cxx:488
proc_set{0}: #56 0x0000000007334a8a in vtkObjectFactory::CreateInstance 
at 
/global/common/edison/graphics/ParaView/builds/ParaView/VTK/Common/Core/vtkObjectFactory.cxx:62
proc_set{0}: #57 0x0000000005a85a61 in vtkMathTextUtilities::GetInstance 
at 
/global/common/edison/graphics/ParaView/builds/ParaView/VTK/Rendering/FreeType/vtkMathTextUtilities.cxx:58
proc_set{0}: #58 0x0000000005a85566 in 
vtkMathTextFreeTypeTextRenderer::vtkMathTextFreeTypeTextRenderer at 
/global/common/edison/graphics/ParaView/builds/ParaView/VTK/Rendering/FreeType/vtkMathTextFreeTypeTextRenderer.cxx:489
proc_set{0}: #59 0x0000000005a81b35 in 
vtkMathTextFreeTypeTextRenderer::New at 
/global/common/edison/graphics/ParaView/builds/ParaView/VTK/Rendering/FreeType/vtkMathTextFreeTypeTextRenderer.cxx:26
proc_set{0}: #60 0x0000000005a8dc29 in 
vtkObjectFactoryCreatevtkMathTextFreeTypeTextRenderer at 
/global/common/edison/graphics/ParaView/builds/PV-4.2.0/VTK/Rendering/FreeType/vtkRenderingFreeTypeObjectFactory.cxx:27
proc_set{0}: #61 0x0000000007336149 in vtkObjectFactory::CreateObject at 
/global/common/edison/graphics/ParaView/builds/ParaView/VTK/Common/Core/vtkObjectFactory.cxx:488
proc_set{0}: #62 0x0000000007334a8a in vtkObjectFactory::CreateInstance 
at 
/global/common/edison/graphics/ParaView/builds/ParaView/VTK/Common/Core/vtkObjectFactory.cxx:62
proc_set{0}: #63 0x000000000601cec9 in vtkTextRenderer::GetInstance at 
/global/common/edison/graphics/ParaView/builds/ParaView/VTK/Rendering/Core/vtkTextRenderer.cxx:79
proc_set{0}: #64 0x0000000005a87f11 in vtkTextActor::vtkTextActor at 
/global/common/edison/graphics/ParaView/builds/ParaView/VTK/Rendering/FreeType/vtkTextActor.cxx:105
proc_set{0}: #65 0x0000000005a87a34 in vtkTextActor::New at 
/global/common/edison/graphics/ParaView/builds/ParaView/VTK/Rendering/FreeType/vtkTextActor.cxx:35
proc_set{0}: #66 0x000000000501535d in 
vtkTextRepresentation::vtkTextRepresentation at 
/global/common/edison/graphics/ParaView/builds/ParaView/VTK/Interaction/Widgets/vtkTextRepresentation.cxx:62
proc_set{0}: #67 0x00000000050152e0 in vtkTextRepresentation::New at 
/global/common/edison/graphics/ParaView/builds/ParaView/VTK/Interaction/Widgets/vtkTextRepresentation.cxx:54
proc_set{0}: #68 0x000000000386c735 in 
vtkNew<vtkTextRepresentation>::vtkNew at 
/global/common/edison/graphics/ParaView/builds/ParaView/VTK/Common/Core/vtkNew.h:66
proc_set{0}: #69 0x0000000003865225 in vtkPVRenderView::vtkPVRenderView 
at 
/global/common/edison/graphics/ParaView/builds/ParaView/ParaViewCore/ClientServerCore/Rendering/vtkPVRenderView.cxx:154
proc_set{0}: #70 0x0000000003864eac in vtkPVRenderView::New at 
/global/common/edison/graphics/ParaView/builds/ParaView/ParaViewCore/ClientServerCore/Rendering/vtkPVRenderView.cxx:140
proc_set{0}: #71 0x00000000032f9975 in 
vtkPVRenderViewClientServerNewCommand at 
/global/common/edison/graphics/ParaView/builds/PV-4.2.0/Wrapping/ClientServer/vtkPVRenderViewClientServer.cxx:13
proc_set{0}: #72 0x000000000474760f in 
vtkClientServerInterpreter::NewInstance at 
/global/common/edison/graphics/ParaView/builds/ParaView/ParaViewCore/ClientServerStream/vtkClientServerInterpreter.cxx:1068
proc_set{0}: #73 0x00000000036a1b80 in vtkSIProxy::CreateVTKObjects at 
/global/common/edison/graphics/ParaView/builds/ParaView/ParaViewCore/ServerImplementation/Core/vtkSIProxy.cxx:303
proc_set{0}: #74 0x00000000036a04cd in vtkSIProxy::Push at 
/global/common/edison/graphics/ParaView/builds/ParaView/ParaViewCore/ServerImplementation/Core/vtkSIProxy.cxx:112
proc_set{0}: #75 0x00000000036848bf in 
vtkPVSessionCore::PushStateInternal at 
/global/common/edison/graphics/ParaView/builds/ParaView/ParaViewCore/ServerImplementation/Core/vtkPVSessionCore.cxx:486
proc_set{0}: #76 0x0000000003684b43 in vtkPVSessionCore::PushState at 
/global/common/edison/graphics/ParaView/builds/ParaView/ParaViewCore/ServerImplementation/Core/vtkPVSessionCore.cxx:529
proc_set{0}: #77 0x0000000003681f69 in vtkPVSessionBase::PushState at 
/global/common/edison/graphics/ParaView/builds/ParaView/ParaViewCore/ServerImplementation/Core/vtkPVSessionBase.cxx:142
proc_set{0}: #78 0x000000000368edf2 in 
vtkPVSessionServer::OnClientServerMessageRMI at 
/global/common/edison/graphics/ParaView/builds/ParaView/ParaViewCore/ServerImplementation/Core/vtkPVSessionServer.cxx:495
proc_set{0}: #79 0x000000000368d29c in at 
/global/common/edison/graphics/ParaView/builds/ParaView/ParaViewCore/ServerImplementation/Core/vtkPVSessionServer.cxx:56
proc_set{0}: #80 0x0000000004b0a589 in 
vtkMultiProcessController::ProcessRMI at 
/global/common/edison/graphics/ParaView/builds/ParaView/VTK/Parallel/Core/vtkMultiProcessController.cxx:774
proc_set{0}: #81 0x0000000004b0a145 in 
vtkMultiProcessController::ProcessRMIs at 
/global/common/edison/graphics/ParaView/builds/ParaView/VTK/Parallel/Core/vtkMultiProcessController.cxx:720
proc_set{0}: #82 0x00000000039c1f95 in 
vtkTCPNetworkAccessManager::ProcessEventsInternal at 
/global/common/edison/graphics/ParaView/builds/ParaView/ParaViewCore/ClientServerCore/Core/vtkTCPNetworkAccessManager.cxx:268
proc_set{0}: #83 0x00000000039c1b32 in 
vtkTCPNetworkAccessManager::ProcessEvents at 
/global/common/edison/graphics/ParaView/builds/ParaView/ParaViewCore/ClientServerCore/Core/vtkTCPNetworkAccessManager.cxx:165
proc_set{0}: #84 0x0000000001e8c2e9 in RealMain at 
/global/common/edison/graphics/ParaView/builds/ParaView/CommandLineExecutables/pvserver_common.h:91
proc_set{0}: #85 0x0000000001e8c3b3 in main at 
/global/common/edison/graphics/ParaView/builds/ParaView/CommandLineExecutables/pvserver.cxx:27

More information about the ParaView mailing list