[From nobody Mon Jul 9 10:19:44 2007 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: [Paraview] Newbie question about paraview's parallel visualization procedure Date: Thu, 5 Jul 2007 14:03:21 -0600 In-Reply-To: <20070705182437.9152.qmail@f5mail15.rediffmail.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [Paraview] Newbie question about paraview's parallel visualization procedure thread-index: Ace/MgX7zjASG+q0Q3mr8WbThmJggQABhiTA References: <20070705182437.9152.qmail@f5mail15.rediffmail.com> From: "Moreland, Kenneth" <kmorel@sandia.gov> To: "Raashid Baig" <raashid.b@rediffmail.com>, paraview@paraview.org Content-Type: multipart/alternative; boundary="----_=_NextPart_003_01C7BF3F.7CBDB200" Content-Transfer-Encoding: 7bit This is a multi-part message in MIME format. ------_=_NextPart_003_01C7BF3F.7CBDB200 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Raashid, =20 First, let me assure you that ParaView's parallel visualization and rendering are efficient and scalable. We have been able to use ParaView to visualize 6 billion cell grids and have clocked rendering speeds of over 8 billion polygons per second (albeit with much larger clusters than you have set up). =20 As far as your problems are concerned, I'm going to have to do a lot of guessing because I cannot replicate your problem and there are missing pieces of information. Looking at your mpirun command, it appears that the DISPLAY variable is set to the same value on every process. That is, all processes in the ParaView server are creating a remote (for them) X window on 10.101.11.71. If that is the case, it is very, very bad. Basically, that would mean that all of the rendering is being funneled into one computer. Thus, instead of actually getting parallel rendering, you are getting serial rendering plus the added overhead of moving OpenGL primitives and images across the network plus the standard overhead of parallel rendering. Note that because you are only using 3 computers, I would not expect the parallel gains to be much higher than the natural overhead of the parallel algorithms. If your computers are using a standard Ethernet connection, the relatively high latency and low bandwidth will further erode the parallel gains. =20 The fact that your parallel job is crashing when you have a parallel job with a sizeable data set suggests that you might not be using parallel rendering at all. You can shut off parallel rendering with the 3D view properties or ParaView will shut it off for you if the server cannot create the appropriate rendering windows. When parallel rendering is shut off, ParaView does the only thing left: funnel all of the geometry to the client. =20 Verify that you are actually doing parallel rendering. Start by running in client/server mode (don't bother with parallel standalone: its usefulness is basically limited to debugging). Go to 3D View Properties, verify that the checkbox next to "Composite" is checked and move the slider all the way to the left (0 MBytes). If everything is working right, parallel rendering should now always be used. The easiest way to verify that is to observe the subsampling, which only happens on interactive rendering when rendering in parallel. Make sure that the checkbox next to "Subsample Rate" is checked and move the slider to the right to 4 Pixels or more. Now when you rotate your 3D rendering, it should look pixilated. =20 Offhand, I cannot think of another reason for your memory failures. I just used the Wavelet source to create a 125,000,000 cell structured mesh on three nodes of our visualization cluster (each with 1 MB each), then created 5 isosurfaces (~7.4 million cells and ~3.7 points) and rendered it just fine. =20 One last piece of advice: don't run D3 on your structured data. ParaView can balance your structured data without any help. D3 is designed for unstructured data. The first thing it will do is convert the structured data to unstructured, which is a much less efficient representation. It makes more sense to run D3 on the output of the contour filter, but that is probably not worthwhile unless the contour is very big and very unbalanced. =20 -Ken =20 _____ =20 From: paraview-bounces+kmorel=3Dsandia.gov@paraview.org [mailto:paraview-bounces+kmorel=3Dsandia.gov@paraview.org] On Behalf Of Raashid Baig Sent: Thursday, July 05, 2007 12:25 PM To: paraview@paraview.org Subject: [Paraview] Newbie question about paraview's parallel visualization procedure =20 I am a newbie to parallel visualization and ParaView in general, I have some doubts which may be very fundamental. =20 I am using ParaView2.6.X and trying to do some performance analysis and dataset limit of ParaView in general as a part of an academic project. I want to see what are the gains in CPU & memory usage and time required to render a particular dataset when ParaView is run in parallel. All machines are Intel Core Duo 2, 2GB DDR2 RAM, and 2GB swap, all machines have NVidia 7series graphics cards and have Debian/Ubuntu Linux I have structured data-sets with point attributes in form of VTK legacy files. When I run paraview in stand-alone mode on a single machine, I am able to draw 5 iso-surface for structured grid datasets upto 5 million points. But when I put the same VTK legacy files on 3 different computers (all have ParaView 2.6.X installed) and run ParaView through LAM/MPI in distributed stand alone mode (as described in the ParaView book) on these 3 machines, the performance decreases for the isosurface case. Not only does the time taken to render final image, cpu/memory usage increase, I am only able to visualize till 1 million grid points. After 1 million point the get an error (this particular error is for a dataset of 2 million) : raashid@ged:~/src/cvs/paraview-unix/bin$ mpirun -np 3 -x DISPLAY=3D10.101.11.71:0.0 ./paraview ErrorMessage # Error or warning: There was a VTK Error in file: /home/raashid/src/cvs/ParaView/VTK/Common/vtkDataArrayTemplate.txx (266) vtkIdTypeArray (0xe041768): Unable to allocate 59135376 elements of size 4 bytes. ErrorMessage end ------------------------------------------------------------------------ ----- One of the processes started by mpirun has exited with a nonzero exit code. This typically indicates that the process finished in error. If your process did not finish in error, be sure to include a "return 0" or "exit(0)" in your C code before exiting the application. PID 16283 failed on node n0 (10.101.11.71) due to signal 11. ------------------------------------------------------------------------ ----- I get the same error as above if I use a D3 filter before applying the iso-contour. Following the ideas of a technical report I tried to open the VTK legacy file when ParaView server was running on 3 different computers and the client was running on a different computer, apply the D3 filter and then save the file as PVTK unstructured grid format (*.pvtu, with 1 ghost levels), close the paraview, start a fresh paraview session with the same configuration (ParaView server was running on 3 different computers and the client was running on a different computer) load the master *.pvtu file and do the same iso-contour example as above, the server again crashes and displays the following error.(this particular error is for 2.5 million grid points) On the server side I get the message : ------------------------------------------------------------------------ ------ raashid@ender:~/src/cvs/paraview-unix/bin$ mpirun -np 3 -x DISPLAY=3D10.101.11.71:0.0 ./pvserver Listen on port: 11111 Waiting for client... Client connected. Process id: 0 >> ERROR: In /home/raashid/src/cvs/ParaView/VTK/Common/vtkDataArrayTemplate.txx, line 95 vtkIdTypeArray (0xd243258): Unable to allocate 75216436 elements of size 4 bytes. Process id: 0 >> ERROR: In /home/raashid/src/cvs/ParaView/Servers/Common/vtkProcessModule.cxx, line 1187 vtkProcessModule (0xb395560): Exception: Insufficient memory exception. ------------------------------------------------------------------------ ------ On the client side I get the message ------------------------------------------------------------------------ ------- raashid@ged:~/src/cvs/paraview-unix/bin$ ./pvclient --server-host=3D10.101.11.72 Connected to server ErrorMessage # Error or warning: There was a VTK Error in file: /home/raashid/src/cvs/ParaView/Servers/Common/vtkProcessModule.cxx (1171) vtkProcessModule (0xc0c7530): Received exception from server: Insufficient memory exception. ErrorMessage end ErrorMessage # Error or warning: There was a VTK Error in file: /home/raashid/src/cvs/ParaView/Servers/Common/vtkServerConnection.cxx (67) vtkServerConnection (0xc0ee1b8): Server Connection Closed! ErrorMessage end ------------------------------------------------------------------------ -------- Am I using the wrong file type, if yes what should I use ? =20 Is there some threshold limit I am violating ? Please help me with this problem, because I am in serious trouble.=20 =20 ------_=_NextPart_003_01C7BF3F.7CBDB200 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = xmlns:o=3D"urn:schemas-microsoft-com:office:office" = xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns=3D"http://www.w3.org/TR/REC-html40"> <head> <META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; = charset=3Dus-ascii"> <meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)"> <!--[if !mso]> <style> v\:* {behavior:url(#default#VML);} o\:* {behavior:url(#default#VML);} w\:* {behavior:url(#default#VML);} .shape {behavior:url(#default#VML);} </style> <![endif]--> <style> <!-- /* Font Definitions */ @font-face {font-family:Tahoma; panose-1:2 11 6 4 3 5 4 4 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:12.0pt; font-family:"Times New Roman";} a:link, span.MsoHyperlink {color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {color:purple; text-decoration:underline;} p {mso-margin-top-alt:auto; margin-right:0in; mso-margin-bottom-alt:auto; margin-left:0in; font-size:12.0pt; font-family:"Times New Roman";} span.EmailStyle18 {mso-style-type:personal-reply; font-family:"Courier New"; color:maroon; font-weight:normal; font-style:normal; text-decoration:none none;} @page Section1 {size:8.5in 11.0in; margin:1.0in 1.25in 1.0in 1.25in;} div.Section1 {page:Section1;} --> </style> <!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--> </head> <body lang=3DEN-US link=3Dblue vlink=3Dpurple> <div class=3DSection1> <p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3D"Courier = New"><span style=3D'font-size:10.0pt;font-family:"Courier = New";color:maroon'>Raashid,<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3D"Courier = New"><span style=3D'font-size:10.0pt;font-family:"Courier = New";color:maroon'><o:p>&nbsp;</o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3D"Courier = New"><span style=3D'font-size:10.0pt;font-family:"Courier New";color:maroon'>First, = let me assure you that ParaView&#8217;s parallel visualization and rendering = are efficient and scalable. &nbsp;We have been able to use ParaView to = visualize 6 billion cell grids and have clocked rendering speeds of over 8 billion = polygons per second (albeit with much larger clusters than you have set = up).<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3D"Courier = New"><span style=3D'font-size:10.0pt;font-family:"Courier = New";color:maroon'><o:p>&nbsp;</o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3D"Courier = New"><span style=3D'font-size:10.0pt;font-family:"Courier New";color:maroon'>As far = as your problems are concerned, I&#8217;m going to have to do a lot of guessing = because I cannot replicate your problem and there are missing pieces of = information.&nbsp; Looking at your mpirun command, it appears that the DISPLAY variable is = set to the same value on every process. &nbsp;That is, all processes in the = ParaView server are creating a remote (for them) X window on 10.101.11.71. = &nbsp;If that is the case, it is very, very bad. &nbsp;Basically, that would mean that = all of the rendering is being funneled into one computer. &nbsp;Thus, instead = of actually getting parallel rendering, you are getting serial rendering = plus the added overhead of moving OpenGL primitives and images across the network = plus the standard overhead of parallel rendering.&nbsp; Note that because you = are only using 3 computers, I would not expect the parallel gains to be much = higher than the natural overhead of the parallel algorithms. &nbsp;If your = computers are using a standard Ethernet connection, the relatively high latency = and low bandwidth will further erode the parallel = gains.<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3D"Courier = New"><span style=3D'font-size:10.0pt;font-family:"Courier = New";color:maroon'><o:p>&nbsp;</o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3D"Courier = New"><span style=3D'font-size:10.0pt;font-family:"Courier New";color:maroon'>The = fact that your parallel job is crashing when you have a parallel job with a = sizeable data set suggests that you might not be using parallel rendering at all. = &nbsp;You can shut off parallel rendering with the 3D view properties or ParaView = will shut it off for you if the server cannot create the appropriate = rendering windows.&nbsp; When parallel rendering is shut off, ParaView does the = only thing left: funnel all of the geometry to the = client.<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3D"Courier = New"><span style=3D'font-size:10.0pt;font-family:"Courier = New";color:maroon'><o:p>&nbsp;</o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3D"Courier = New"><span style=3D'font-size:10.0pt;font-family:"Courier New";color:maroon'>Verify = that you are actually doing parallel rendering. &nbsp;Start by running in = client/server mode (don&#8217;t bother with parallel standalone: its usefulness is = basically limited to debugging).&nbsp; Go to 3D View Properties, verify that the = checkbox next to &#8220;Composite&#8221; is checked and move the slider all the = way to the left (0 MBytes). &nbsp;If everything is working right, parallel = rendering should now always be used. &nbsp;The easiest way to verify that is to = observe the subsampling, which only happens on interactive rendering when = rendering in parallel. &nbsp;Make sure that the checkbox next to &#8220;Subsample = Rate&#8221; is checked and move the slider to the right to 4 Pixels or more. = &nbsp;Now when you rotate your 3D rendering, it should look = pixilated.<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3D"Courier = New"><span style=3D'font-size:10.0pt;font-family:"Courier = New";color:maroon'><o:p>&nbsp;</o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3D"Courier = New"><span style=3D'font-size:10.0pt;font-family:"Courier = New";color:maroon'>Offhand, I cannot think of another reason for your memory failures. &nbsp;I just = used the Wavelet source to create a 125,000,000 cell structured mesh on three = nodes of our visualization cluster (each with 1 MB each), then created 5 = isosurfaces (~7.4 million cells and ~3.7 points) and rendered it just = fine.<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3D"Courier = New"><span style=3D'font-size:10.0pt;font-family:"Courier = New";color:maroon'><o:p>&nbsp;</o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3D"Courier = New"><span style=3D'font-size:10.0pt;font-family:"Courier New";color:maroon'>One = last piece of advice: don&#8217;t run D3 on your structured data. &nbsp;ParaView = can balance your structured data without any help.&nbsp; D3 is designed for unstructured data.&nbsp; The first thing it will do is convert the = structured data to unstructured, which is a much less efficient representation. = &nbsp;It makes more sense to run D3 on the output of the contour filter, but that = is probably not worthwhile unless the contour is very big and very = unbalanced.<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3D"Courier = New"><span style=3D'font-size:10.0pt;font-family:"Courier = New";color:maroon'><o:p>&nbsp;</o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3D"Courier = New"><span style=3D'font-size:10.0pt;font-family:"Courier = New";color:maroon'>-Ken<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dmaroon face=3D"Courier = New"><span style=3D'font-size:10.0pt;font-family:"Courier = New";color:maroon'><o:p>&nbsp;</o:p></span></font></p> <div style=3D'border:none;border-left:solid blue 1.5pt;padding:0in 0in = 0in 4.0pt'> <div> <div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font = size=3D3 face=3D"Times New Roman"><span style=3D'font-size:12.0pt'> <hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1> </span></font></div> <p class=3DMsoNormal><b><font size=3D2 face=3DTahoma><span = style=3D'font-size:10.0pt; font-family:Tahoma;font-weight:bold'>From:</span></font></b><font = size=3D2 face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma'> = paraview-bounces+kmorel=3Dsandia.gov@paraview.org [mailto:paraview-bounces+kmorel=3Dsandia.gov@paraview.org] <b><span style=3D'font-weight:bold'>On Behalf Of </span></b>Raashid Baig<br> <b><span style=3D'font-weight:bold'>Sent:</span></b> Thursday, July 05, = 2007 12:25 PM<br> <b><span style=3D'font-weight:bold'>To:</span></b> = paraview@paraview.org<br> <b><span style=3D'font-weight:bold'>Subject:</span></b> [Paraview] = Newbie question about paraview's parallel visualization = procedure</span></font><o:p></o:p></p> </div> <p class=3DMsoNormal><font size=3D3 face=3D"Times New Roman"><span = style=3D'font-size: 12.0pt'><o:p>&nbsp;</o:p></span></font></p> <p><font size=3D3 face=3D"Times New Roman"><span = style=3D'font-size:12.0pt'>I am a newbie to parallel visualization and ParaView in general, I<br> have some doubts which may be very fundamental.&nbsp; <br> <br> I am using ParaView2.6.X and trying to do some performance analysis<br> and dataset limit of ParaView in general as a part of an academic<br> project. I want to see what are the gains in CPU &amp; memory usage = and<br> time required to render a particular dataset when ParaView is run in<br> parallel. All machines are Intel Core Duo 2, 2GB DDR2 RAM, and 2GB<br> swap, all machines have NVidia 7series graphics cards and have<br> Debian/Ubuntu Linux<br> <br> I have structured data-sets with point attributes in form of VTK<br> legacy files.&nbsp; When I run paraview in stand-alone mode on a = single<br> machine, I am able to draw 5 iso-surface for structured grid = datasets<br> upto 5 million points. But when I put the same VTK legacy files on 3<br> different computers (all have ParaView 2.6.X installed) and run<br> ParaView through LAM/MPI in distributed stand alone mode (as = described<br> in the ParaView book) on these 3 machines, the performance decreases<br> for the isosurface case. Not only does the time taken to render = final<br> image, cpu/memory usage increase, I am only able to visualize till 1<br> million grid points. After 1 million point the get an error (this<br> particular error is for a dataset of 2 million) :<br> <br> raashid@ged:~/src/cvs/paraview-unix/bin$ mpirun -np 3 -x DISPLAY=3D10.101.11.71:0.0 ./paraview<br> ErrorMessage<br> # Error or warning: There was a VTK Error in file: = /home/raashid/src/cvs/ParaView/VTK/Common/vtkDataArrayTemplate.txx (266)<br> vtkIdTypeArray (0xe041768): Unable to allocate 59135376 elements of size = 4 bytes.<br> ErrorMessage end<br> -------------------------------------------------------------------------= ----<br> One of the processes started by mpirun has exited with a nonzero = exit<br> code.&nbsp; This typically indicates that the process finished in = error.<br> If your process did not finish in error, be sure to include a = &quot;return<br> 0&quot; or &quot;exit(0)&quot; in your C code before exiting the = application.<br> <br> PID 16283 failed on node n0 (10.101.11.71) due to signal 11.<br> -------------------------------------------------------------------------= ----<br> <br> I get the same error as above if I use a D3 filter before applying = the<br> iso-contour.<br> <br> <br> Following the ideas of a technical report I tried to open the VTK<br> legacy file when ParaView server was running on 3 different = computers<br> and the client was running on a different computer, apply the D3<br> filter and then save the file as PVTK unstructured grid format<br> (*.pvtu, with 1 ghost levels), close the paraview, start a fresh<br> paraview session with the same configuration (ParaView server was<br> running on 3 different computers and the client was running on a<br> different computer) load the master *.pvtu file and do the same<br> iso-contour example as above, the server again crashes and displays<br> the following error.(this particular error is for 2.5 million grid<br> points)<br> <br> On the server side I get the message :<br> -------------------------------------------------------------------------= -----<br> raashid@ender:~/src/cvs/paraview-unix/bin$ mpirun -np 3 -x DISPLAY=3D10.101.11.71:0.0 ./pvserver<br> Listen on port: 11111<br> Waiting for client...<br> Client connected.<br> Process id: 0 &gt;&gt; ERROR: In /home/raashid/src/cvs/ParaView/VTK/Common/vtkDataArrayTemplate.txx, line = 95<br> vtkIdTypeArray (0xd243258): Unable to allocate 75216436 elements of size = 4 bytes.<br> <br> Process id: 0 &gt;&gt; ERROR: In /home/raashid/src/cvs/ParaView/Servers/Common/vtkProcessModule.cxx, line = 1187<br> vtkProcessModule (0xb395560): Exception: Insufficient memory = exception.<br> -------------------------------------------------------------------------= -----<br> <br> On the client side I get the message<br> -------------------------------------------------------------------------= ------<br> raashid@ged:~/src/cvs/paraview-unix/bin$ ./pvclient = --server-host=3D10.101.11.72<br> Connected to server<br> ErrorMessage<br> # Error or warning: There was a VTK Error in file: /home/raashid/src/cvs/ParaView/Servers/Common/vtkProcessModule.cxx = (1171)<br> vtkProcessModule (0xc0c7530): Received exception from server: = Insufficient memory exception.<br> ErrorMessage end<br> ErrorMessage<br> # Error or warning: There was a VTK Error in file: /home/raashid/src/cvs/ParaView/Servers/Common/vtkServerConnection.cxx = (67)<br> vtkServerConnection (0xc0ee1b8): Server Connection Closed!<br> ErrorMessage end<br> -------------------------------------------------------------------------= -------<br> <br> Am I using the wrong file type, if yes what should I use ?&nbsp; <br> Is there some threshold limit I am violating ?<br> <br> Please help me with this problem, because I am in serious trouble. = <o:p></o:p></span></font></p> <p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><font size=3D3 face=3D"Times New Roman"><span = style=3D'font-size:12.0pt'><o:p>&nbsp;</o:p></span></font></p> </div> </div> </body> </html> ------_=_NextPart_003_01C7BF3F.7CBDB200-- ]