<div dir="ltr">I am trying to run a script on a cluster with Paraview in parallel. The cluster architecture is 16cores and 64GB per node. The script reads in two XMF files, one contains a grid of 2002x100x1000 points along with one variable. There is a loop to perform two iterations:<div><br></div><div><div>try: paraview.simple</div><div>except: from paraview.simple import *</div><div>paraview.simple._DisableFirstRenderCameraReset()</div><div><br></div><div>RenderView1 = GetRenderView()</div><div>RenderView1.Background = [1.0, 1.0, 1.0]</div><div>view = GetActiveView()</div><div>view.ViewSize = [ 800, 500 ]</div><div><br></div><div>RenderView1.CenterAxesVisibility = 0</div><div>#RenderView1.OrientationAxesVisibility = 0</div><div><br></div><div>#Define variables</div><div>wdir = '/lustre/groups/balarasgrp/nikosb/Research/Simulations/golfball/stationary/S120/Re200000/gridC1/run_wm'</div><div>#wdir = '/Users/nikosb/colone/Research/Simulations/golfball/stationary/S120/Re200000/gridC1/run_wm'</div><div>#wdir = '/Users/nikosb/Research/Simulations/SPHERE/Re100/gridC1/run'</div><div>Qval = 0.1</div><div>cfmin = -0.001</div><div>cfmax = 0.005</div><div><br></div><div>index = [None]*9999</div><div><br></div><div>m = 0</div><div>for i in range(0, 9):</div><div>  for j in range(0, 9):</div><div>    for k in range(0, 9):</div><div>      for l in range(0, 9):</div><div>        index[m] = chr(i+48)+chr(j+48)+chr(k+48)+chr(l+48)</div><div>        m = m+1</div><div><br></div><div><br></div><div>for i in range(1,3):</div><div><br></div><div>  cf_VP_xmf = XDMFReader( FileName=wdir+'/VPfield/XDMF/cf_VP'+index[i]+'.xmf' )</div><div><br></div><div>  cf_VP_xmf.Sets = []</div><div>  cf_VP_xmf.Grids = ['STL Grid']</div><div>  cf_VP_xmf.PointArrays = ['Cf']</div><div><br></div><div>  RenderView1 = GetRenderView()</div><div>  RenderView1.CenterOfRotation = [6.2286853790283203e-06, 2.5033950805664062e-06, 7.1227550506591797e-06]</div><div><br></div><div>  DataRepresentation1 = Show()</div><div>  DataRepresentation1.EdgeColor = [0.0, 0.0, 0.50000762951094835]</div><div>  DataRepresentation1.SelectionPointFieldDataArrayName = 'Cf'</div><div>  DataRepresentation1.ColorArrayName = ('POINT_DATA', 'Cf')</div><div>  DataRepresentation1.ScalarOpacityUnitDistance = 0.033928799990483226</div><div>  DataRepresentation1.ScaleFactor = 0.099998557567596444</div><div><br></div><div>  a1_Cf_PVLookupTable = GetLookupTableForArray( "Cf", 1, RGBPoints=[-0.0099065126851201057, 0.23000000000000001, 0.29899999999999999, 0.754, 0.0097341262735426426, 0.86499999999999999, 0.86499999999999999, 0.86499999999999999, 0.029374765232205391, 0.70599999999999996, 0.016, 0.14999999999999999], VectorMode='Magnitude', NanColor=[0.25, 0.0, 0.0], ColorSpace='Diverging', ScalarRangeInitialized=1.0 )</div></div><div><div>  a1_Cf_PiecewiseFunction = CreatePiecewiseFunction( Points=[-0.0099065126851201057, 0.0, 0.5, 0.0, 0.029374765232205391, 1.0, 0.5, 0.0] )</div><div><br></div><div>  DataRepresentation1.ScalarOpacityFunction = a1_Cf_PiecewiseFunction</div><div>  DataRepresentation1.LookupTable = a1_Cf_PVLookupTable</div><div><br></div><div>  a1_Cf_PVLookupTable.ScalarOpacityFunction = a1_Cf_PiecewiseFunction</div><div><br></div><div>  a1_Cf_PVLookupTable.RGBPoints = [cfmin, 0.23000000000000001, 0.29899999999999999, 0.754, -0.002, 0.86499999999999999, 0.86499999999999999, 0.86499999999999999, cfmax, 0.70599999999999996, 0.016, 0.14999999999999999]</div><div>  a1_Cf_PVLookupTable.ScalarOpacityFunction = a1_Cf_PiecewiseFunction</div><div>  a1_Cf_PVLookupTable.LockScalarRange = 1</div><div><br></div><div>  RenderView1.CameraPosition = [6.2286853790283203e-06, 2.5033950805664062e-06, 3.3460035306026636]</div><div>  RenderView1.CameraFocalPoint = [6.2286853790283203e-06, 2.5033950805664062e-06, 7.1227550506591797e-06]</div><div>  RenderView1.CameraClippingRange = [2.3175641399614015, 4.6486514901655163]</div><div>  RenderView1.CameraParallelScale = 0.86600759519558368</div><div><br></div><div>  Contour1 = Contour( PointMergeMethod="Uniform Binning" )</div><div><br></div><div>  Contour1.PointMergeMethod = "Uniform Binning"</div><div>  Contour1.ContourBy = ['POINTS', 'Cf']</div><div>  Contour1.Isosurfaces = [0.0097341262735426426]</div><div><br></div><div>  DataRepresentation2 = Show()</div><div>  DataRepresentation2.ColorArrayName = ('POINT_DATA', '')</div><div>  DataRepresentation2.ScaleFactor = 0.099971860647201538</div><div>  DataRepresentation2.SelectionPointFieldDataArrayName = 'Cf'</div><div>  DataRepresentation2.EdgeColor = [0.0, 0.0, 0.50000762951094835]</div><div><br></div><div>  DataRepresentation2.Representation = 'Wireframe'</div><div>  DataRepresentation2.AmbientColor = [0.0, 0.0, 0.0]</div><div><br></div><div>  DataRepresentation1.Visibility = 0</div><div><br></div><div>  RenderView1.CameraClippingRange = [2.7470738944403328, 4.2397684828477349]</div><div><br></div><div>  DataRepresentation1.Visibility = 1</div><div><br></div><div>  RenderView1.CameraClippingRange = [2.3175641399614015, 4.6486514901655163]</div><div><br></div><div>  Q_VP_xmf = XDMFReader( FileName=wdir+'/VPfield/XDMF/Q_VP'+index[i]+'.xmf' )</div><div><br></div><div>  Contour1.Isosurfaces = [0.0]</div><div><br></div><div>  Q_VP_xmf.Sets = []</div><div>  Q_VP_xmf.Grids = ['Eulerian Grid']</div><div>  Q_VP_xmf.PointArrays = ['Q']</div><div><br></div><div>  DataRepresentation3 = Show()</div><div>  DataRepresentation3.EdgeColor = [0.0, 0.0, 0.50000762951094835]</div><div>  DataRepresentation3.SelectionPointFieldDataArrayName = 'Q'</div><div>  DataRepresentation3.ScalarOpacityUnitDistance = 0.0045778956118952439</div><div>  DataRepresentation3.Representation = 'Outline'</div></div><div><div>  DataRepresentation3.ScaleFactor = 0.25585924386978148</div><div><br></div><div>  RenderView1.CameraClippingRange = [0.30866115510384229, 6.1311800002592065]</div><div><br></div><div>  Contour2 = Contour( PointMergeMethod="Uniform Binning" )</div><div><br></div><div>  Contour2.PointMergeMethod = "Uniform Binning"</div><div>  Contour2.ContourBy = ['POINTS', 'Q']</div><div>  Contour2.Isosurfaces = [-348374.28125]</div><div><br></div><div>  DataRepresentation4 = Show()</div><div>  DataRepresentation4.ScaleFactor = 0.1348903000354767</div><div>  DataRepresentation4.SelectionPointFieldDataArrayName = 'Normals'</div><div>  DataRepresentation4.EdgeColor = [0.0, 0.0, 0.50000762951094835]</div><div><br></div><div>  Contour2.Isosurfaces = [0.5]</div><div><br></div><div>  DataRepresentation3.Visibility = 0</div><div><br></div><div>  RenderView1.CameraClippingRange = [0.46635874330917981, 5.5760844897764175]</div><div><br></div><div>  RenderView1.CameraViewUp = [-1.0, 0.0, 0.0]</div><div>  RenderView1.CameraPosition = [-5.0, 0.0, 5.0]</div><div>  RenderView1.CameraClippingRange = [3.0180305778980254, 7.5069525367021557]</div><div>  RenderView1.CameraFocalPoint = [0.0, 0.0, 5.0]</div><div>  RenderView1.CameraParallelScale = 1.565457167140391</div><div>  RenderView1.CenterOfRotation = [0.0, 0.0, 0.0]</div><div><br></div><div>  RenderView1.CameraFocalPoint = [0.0, 0.0, 0.5]</div><div>  RenderView1.CameraClippingRange = [0.86197012554183883, 4.2263653610655147]</div><div>  RenderView1.CameraPosition = [0.0, -2.5, 0.0]</div><div><br></div><div>  WriteImage(wdir+'/POSTPRO/IMAGES/Q_cf_VP'+index[i]+'.jpg')</div><div><br></div><div>  Delete(DataRepresentation4)</div><div>  Delete(Contour2)</div><div>  Delete(DataRepresentation3)</div><div>  Delete(Q_VP_xmf)</div><div><br></div><div>  Delete(DataRepresentation2)</div><div>  Delete(Contour1)</div><div>  Delete(DataRepresentation1)</div><div>  Delete(cf_VP_xmf)</div><div><br></div></div><div>The script was created by running an interactive Paraview session and recording the commands. I then modified it slightly to add the loop and some variables.</div><div><br></div><div>I have been able to run the script successfully on 1,2,16 and 32 cores successfully and it loops twice. However when I try to run on 64 cores I get the following error message:</div><div><br></div><div><div>[node116:58372] 63 more processes have sent help message help-mpi-common-cuda.txt / dlopen failed</div><div>[node116:58372] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages</div><div>terminate called after throwing an instance of 'std::bad_alloc'</div><div>  what():  std::bad_alloc</div><div>[node125:37180] *** Process received signal ***</div><div>[node125:37180] Signal: Aborted (6)</div><div>[node125:37180] Signal code:  (-6)</div><div>[node125:37180] [ 0] /lib64/libpthread.so.0(+0xf710) [0x2aaab28f3710]</div><div>[node125:37180] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x2aaab479f625]</div><div>[node125:37180] [ 2] /lib64/libc.so.6(abort+0x175) [0x2aaab47a0e05]</div><div>[node125:37180] [ 3] /usr/lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x12d) [0x2aaab408aa7d]</div><div>[node125:37180] [ 4] /usr/lib64/libstdc++.so.6(+0xbcbd6) [0x2aaab4088bd6]</div><div>[node125:37180] [ 5] /usr/lib64/libstdc++.so.6(+0xbcc03) [0x2aaab4088c03]</div><div>[node125:37180] [ 6] /usr/lib64/libstdc++.so.6(+0xbcd22) [0x2aaab4088d22]</div><div>[node125:37180] [ 7] /usr/lib64/libstdc++.so.6(_Znwm+0x7d) [0x2aaab408911d]</div><div>[node125:37180] [ 8] /usr/lib64/libstdc++.so.6(_Znam+0x9) [0x2aaab40891d9]</div><div>[node125:37180] [ 9] /c1/apps/paraview/4.1.0/lib/libvtkIOXdmf2-pv4.1.so.1(_ZN16vtkXdmfHeavyData10ReadPointsEP12XdmfGeometryPiS2_+0x3a6) [0x2aaab716310c]</div><div>[node125:37180] [10] /c1/apps/paraview/4.1.0/lib/libvtkIOXdmf2-pv4.1.so.1(_ZN16vtkXdmfHeavyData21RequestStructuredGridEP8XdmfGrid+0xfb) [0x2aaab716292f]</div><div>[node125:37180] [11] /c1/apps/paraview/4.1.0/lib/libvtkIOXdmf2-pv4.1.so.1(_ZN16vtkXdmfHeavyData15ReadUniformDataEP8XdmfGrid+0x14c) [0x2aaab71614ce]</div><div>[node125:37180] [12] /c1/apps/paraview/4.1.0/lib/libvtkIOXdmf2-pv4.1.so.1(_ZN16vtkXdmfHeavyData8ReadDataEP8XdmfGrid+0xd1) [0x2aaab7160c91]</div><div>[node125:37180] [13] /c1/apps/paraview/4.1.0/lib/libvtkIOXdmf2-pv4.1.so.1(_ZN16vtkXdmfHeavyData8ReadDataEv+0x4f) [0x2aaab7160a0f]</div><div>[node125:37180] [14] /c1/apps/paraview/4.1.0/lib/libvtkIOXdmf2-pv4.1.so.1(_ZN13vtkXdmfReader11RequestDataEP14vtkInformationPP20vtkInformationVectorS3_+0x31c) [0x2aaab716daf0]</div><div>[node125:37180] [15] /c1/apps/paraview/4.1.0/lib/libvtkIOLegacy-pv4.1.so.1(_ZN13vtkDataReader14ProcessRequestEP14vtkInformationPP20vtkInformationVectorS3_+0x62) [0x2aaab0b3b9be]</div><div>[node125:37180] [16] /c1/apps/paraview/4.1.0/lib/libvtkIOXdmf2-pv4.1.so.1(_ZN13vtkXdmfReader14ProcessRequestEP14vtkInformationPP20vtkInformationVectorS3_+0x76) [0x2aaab716c702]</div><div>[node125:37180] [17] /c1/apps/paraview/4.1.0/lib/libvtkCommonExecutionModel-pv4.1.so.1(_ZN12vtkExecutive13CallAlgorithmEP14vtkInformationiPP20vtkInformationVectorS3_+0x8d) [0x2aaab10c63d1]</div><div>[node125:37180] [18] /c1/apps/paraview/4.1.0/lib/libvtkCommonExecutionModel-pv4.1.so.1(_ZN23vtkDemandDrivenPipeline11ExecuteDataEP14vtkInformationPP20vtkInformationVectorS3_+0x76) [0x2aaab10c0b1c]</div><div>[node125:37180] [19] /c1/apps/paraview/4.1.0/lib/libvtkCommonExecutionModel-pv4.1.so.1(_ZN24vtkCompositeDataPipeline11ExecuteDataEP14vtkInformationPP20vtkInformationVectorS3_+0x468) [0x2aaab10b9950]</div><div>[node125:37180] [20] /c1/apps/paraview/4.1.0/lib/libvtkCommonExecutionModel-pv4.1.so.1(_ZN23vtkDemandDrivenPipeline14ProcessRequestEP14vtkInformationPP20vtkInformationVectorS3_+0x522) [0x2aaab10c0248]</div><div>[node125:37180] [21] /c1/apps/paraview/4.1.0/lib/libvtkCommonExecutionModel-pv4.1.so.1(_ZN32vtkStreamingDemandDrivenPipeline14ProcessRequestEP14vtkInformationPP20vtkInformationVectorS3_+0xa6c) [0x2aaab10e5ace]</div><div>[node125:37180] [22] /c1/apps/paraview/4.1.0/lib/libvtkCommonExecutionModel-pv4.1.so.1(_ZN24vtkCompositeDataPipeline15ForwardUpstreamEP14vtkInformation+0x2d6) [0x2aaab10bbb6e]</div><div>[node125:37180] [23] /c1/apps/paraview/4.1.0/lib/libvtkCommonExecutionModel-pv4.1.so.1(_ZN23vtkDemandDrivenPipeline14ProcessRequestEP14vtkInformationPP20vtkInformationVectorS3_+0x48a) [0x2aaab10c01b0]</div><div>[node125:37180] [24] /c1/apps/paraview/4.1.0/lib/libvtkCommonExecutionModel-pv4.1.so.1(_ZN32vtkStreamingDemandDrivenPipeline14ProcessRequestEP14vtkInformationPP20vtkInformationVectorS3_+0xa6c) [0x2aaab10e5ace]</div><div>[node125:37180] [25] /c1/apps/paraview/4.1.0/lib/libvtkCommonExecutionModel-pv4.1.so.1(_ZN23vtkDemandDrivenPipeline10UpdateDataEi+0x2fb) [0x2aaab10c0965]</div><div>[node125:37180] [26] /c1/apps/paraview/4.1.0/lib/libvtkCommonExecutionModel-pv4.1.so.1(_ZN32vtkStreamingDemandDrivenPipeline6UpdateEi+0xf8) [0x2aaab10e5d3e]</div><div>[node125:37180] [27] /c1/apps/paraview/4.1.0/lib/libvtkPVServerImplementationCore-pv4.1.so.1(_ZN16vtkSISourceProxy14UpdatePipelineEidb+0x196) [0x2aaaac77ef72]</div><div>[node125:37180] [28] /c1/apps/paraview/4.1.0/lib/libvtkPVServerManagerApplication-pv4.1.so.1(_Z23vtkSISourceProxyCommandP26vtkClientServerInterpreterP13vtkObjectBasePKcRK21vtkClientServerStreamRS5_Pv+0x6ee) [0x2aaaab7f27bd]</div><div>[node125:37180] [29] /c1/apps/paraview/4.1.0/lib/libvtkClientServer-pv4.1.so.1(_ZN26vtkClientServerInterpreter19CallCommandFunctionEPKcP13vtkObjectBaseS1_RK21vtkClientServerStreamRS4_+0x343) [0x2aaaafb57aa5]</div><div>[node125:37180] *** End of error message ***</div></div><div><br></div><div>I am afraid there is an issue with memory leakage or improper memory allocation and deallocation. At this point I do not know how to proceed to debug it. Is there an obvious problem in in the script and the way datasets are deallocated at the end of each iteration?</div><div><br></div><div>Regards,</div><div><br></div><div>Nikos</div></div>