[Paraview] [EXTERNAL] horrible slow performance -- I must be doing something very wrong

Thu Oct 31 11:57:30 EDT 2013

Thanks!  Very good explanation.

On Oct 31, 2013, at 7:48 AM, "Moreland, Kenneth" <kmorel at sandia.gov<mailto:kmorel at sandia.gov>>
 wrote:

It is not correct that ParaView is rendering every face in the mesh (at least, it most certainly should not be and has been my experience that it is not). ParaView only renders the external faces of the grid. As Berk mentioned in a separate email, that is about 1.6 M quads. That should be reasonable for a single modern GPU, but could be an issue for Mesa.

However, it is also not quite right that adding more servers simply divides the data among the processes (even if surface elements were divided evenly). This is because when the ParaView rendering system extracts external faces, it does so without requesting ghost cells. Thus, you are also getting the faces in between the partitions. So on 4 servers, instead of getting the 400K quads per server you would expect, you are actually getting around 655K quads. Add the extra overhead of parallel rendering, and it might not actually be faster at all than a single server.

Running on 16 processes, you should be getting about 260K quads per server, which I guess is about right for the rendering performance on your service. Interestingly, 16 processes falls right into the rule of thumb provided in the ParaView tutorial (http://www.paraview.org/Wiki/The_ParaView_Tutorial).

-Ken

From: <Cook>, Rich <cook47 at llnl.gov<mailto:cook47 at llnl.gov>>
Date: Wednesday, October 30, 2013 6:18 PM
To: Walter Scott <wascott at sandia.gov<mailto:wascott at sandia.gov>>
Cc: "paraview at paraview.org<mailto:paraview at paraview.org>" <paraview at paraview.org<mailto:paraview at paraview.org>>
Subject: Re: [Paraview] [EXTERNAL] horrible slow performance -- I must be doing something very wrong

I guess paraview doesn't have the benefit of knowing I'm rendering a giant cube in space, and has to do as you say.  To me it seems obvious because I do have a cube, but in general I guess what you are saying is reasonable.  I'm certainly not coming up with any way to do it another way off the top of my head.  :-)
-- Rich

On Oct 30, 2013, at 5:10 PM, "Scott, W Alan" <wascott at sandia.gov<mailto:wascott at sandia.gov>>
 wrote:

I’m pretty sure that what happens is everything gets rendered – all bazillion cells, and then z buffer sorting puts the right part of the object closest to the eye.  Thus, lots and lots and lots of triangles for those few 16 servers...

Alan

From: Cook, Rich [mailto:cook47 at llnl.gov<http://llnl.gov/>]
Sent: Wednesday, October 30, 2013 6:09 PM
To: Scott, W Alan
Cc: paraview at paraview.org<mailto:paraview at paraview.org>
Subject: Re: [EXTERNAL] [Paraview] horrible slow performance -- I must be doing something very wrong

4 nodes was plenty, but I needed 16 servers to do the job.
I was just surprised because I figured once the data was read, all that was needed was the exterior faces and that shouldn't require much bandwidth or computation.  The nodes were not gasping for memory so I'm not sure still why it's so slow.  VisIt acts the same, however.  Guess I was just plain ol' wrong!  bleh

On Oct 30, 2013, at 5:06 PM, "Scott, W Alan" <wascott at sandia.gov<mailto:wascott at sandia.gov>>
 wrote:

Yep, I should have thought of that.  You basically have a billion cells/8, or 125 million cells.  You are basically throwing a large desktop computer at it with 4 servers...  I would probably be running something like 16 nodes (128 cores/processes) and see how it goes...

Alan

From: Cook, Rich [mailto:cook47 at llnl.gov<http://llnl.gov/>]
Sent: Wednesday, October 30, 2013 6:03 PM
To: Scott, W Alan
Cc: paraview at paraview.org<mailto:paraview at paraview.org>
Subject: Re: [EXTERNAL] [Paraview] horrible slow performance -- I must be doing something very wrong

I think I just wasn't throwing enough horsepower at it.  I underestimated the rendering requirements of a 512^3 mesh.  My bad.  Adding more servers sped things up dramatically.  Duh.

On Oct 30, 2013, at 3:33 PM, "Cook, Rich" <cook47 at llnl.gov<mailto:cook47 at llnl.gov>>
 wrote:

On Oct 29, 2013, at 5:45 PM, "Scott, W Alan" <wascott at sandia.gov<mailto:wascott at sandia.gov>> wrote:

Rich,
You are probably doing something wrong.  :-)  I assume you are using PV 4.0.1.

After setting up the local client/ remote server, using more than one server, try the following:
·         Help/ About.  Click on Connection Information.  You should see the correct number of Processes (i.e., pvservers).  Now you know you are running in parallel with MPI.
Yep.

·         Edit/ Settings/ Server.  Change Remote Render Threshold to 0.  You are now rendering remotely on the cluster.   Make the ParaView window quite small.  Tools/ Timer log.  Clear.  Close.  Sources/ Sphere.  Apply.  Spin it.   Tools/ Timer log.  Is it fast enough?  On redsky, going through a disgusting number of layers from my ParaView client to my desktop (think X forwarding), I am getting 15 to 30 frames/second.  Clear the timer log.
0.054 sec/render == 18.5 FPS

·         Make ParaView full screen.  Spin, spin.  Timer log.  I get about 10 frames/second interactive, 2 frames /second still.  (This represents time with my finger on the mouse, time with finger off the mouse).  This tells you how fast your network is pushing 2d images from the server to the client.  (As I said, for me, it is a SSLLOOWW pipe between me and my client.)
About 4.5-5 FPS interactive, 3.5-5.0 still
I guess this means my network from our cluster to my desktop is crappy, I guess because of ssh tunnel?  Tunnel is necessary due to topology.

·         Make ParaView small screened once again.  Change the sphere’s Theta resolution to 1000 and Phi Resolution to 1000.  You now have around 2 million cells.  Spin, spin.  Timer log.  This represents the speed of the cluster, since we are having to render all of those triangles (cells) (actually vertexes) in the Mesa3d software renderer.
Interactive Render, 0.229963 seconds

    OpenGL Dev Render,  7e-05 seconds

    OpenGL Dev Render,  0.000525 seconds

    OpenGL Dev Render,  1.6e-05 seconds

Interactive Render,  0.212467 seconds

    OpenGL Dev Render,  6.7e-05 seconds

    OpenGL Dev Render,  0.000532 seconds

    OpenGL Dev Render,  1.7e-05 seconds

Still Render,  0.734462 seconds

    OpenGL Dev Render,  0.000118 seconds

    OpenGL Dev Render,  0.000521 seconds

    OpenGL Dev Render,  1.6e-05 seconds

Still Render,  0.723525 seconds

·         Clear the timer log.  Delete the Sphere.  View/ Memory Inspector.  (This lets you know if you are overflowing memory.  Further, you should have as many pvservers as you think you have.  If you see one, you have an MPI issue.)  Sources/ Wavelet.

No memory problems.  Expected number of servers.
Should I do something with the wavelet?

By the way, if I were running the client locally (which I often do), I will get frame rates over 60/second connecting to the clusters.  Connecting to clusters across state lines will slow things down to 4 to 10 FPS or less.

Huge, hero size data will also slow things down a lot.

Another thing to try is use the Process Id Scalars filter.  That will show you how your data is being spread around.  It is very possible that your data is being read through a serial reader, and only going into one process (server).

I thought this would show the problem, but it looks like data is decomposed correctly.

Don’t turn volume rendering on or opacity on (at first, anyway).

If that doesn’t help, I will be in tomorrow (and tonight for another hour) – if we need to chat.

Would like that -- must still be missing something obvious…

Alan

From: paraview-bounces at paraview.org<mailto:paraview-bounces at paraview.org> [mailto:paraview-bounces at paraview.org<mailto:bounces at paraview.org>] On Behalf Of Cook, Rich
Sent: Tuesday, October 29, 2013 6:16 PM
To: paraview at paraview.org<mailto:paraview at paraview.org>
Subject: [EXTERNAL] [Paraview] horrible slow performance -- I must be doing something very wrong

Hello
I am running paraview in client-server mode.  I start the client on my desktop, run pvserver in parallel with four instances and connect my client to the "ringleader" pvserver through an ssh tunnel.  The data is a 512x512x512 rectilinear grid in Miranda format.
I open my data and attempt to render a simple picture of my data and it is so slow I'm going to kill myself.  Maybe one frame every 5 or 10 seconds.  I'm imagining that the geometry is being shipped to my client, and the client is doing the rendering, but I don't know that.
Am I doing some basic thing wrong?  How can I get reasonable performance?
Thanks

--
✐Richard Cook
✇ Lawrence Livermore National Laboratory
Bldg-453 Rm-4024, Mail Stop L-557
7000 East Avenue,  Livermore, CA, 94550, USA
☎ (office) (925) 423-9605
☎ (fax) (925) 423-6961
---
Information Management & Graphics Grp., Services & Development Div., Integrated Computing & Communications Dept.
(opinions expressed herein are mine and not those of LLNL)

--
✐Richard Cook
✇ Lawrence Livermore National Laboratory
Bldg-453 Rm-4024, Mail Stop L-557
7000 East Avenue,  Livermore, CA, 94550, USA
☎ (office) (925) 423-9605
☎ (fax) (925) 423-6961
---
Information Management & Graphics Grp., Services & Development Div., Integrated Computing & Communications Dept.
(opinions expressed herein are mine and not those of LLNL)

--
✐Richard Cook
✇ Lawrence Livermore National Laboratory
Bldg-453 Rm-4024, Mail Stop L-557
7000 East Avenue,  Livermore, CA, 94550, USA
☎ (office) (925) 423-9605
☎ (fax) (925) 423-6961
---
Information Management & Graphics Grp., Services & Development Div., Integrated Computing & Communications Dept.
(opinions expressed herein are mine and not those of LLNL)

--
✐Richard Cook
✇ Lawrence Livermore National Laboratory
Bldg-453 Rm-4024, Mail Stop L-557
7000 East Avenue,  Livermore, CA, 94550, USA
☎ (office) (925) 423-9605
☎ (fax) (925) 423-6961
---
Information Management & Graphics Grp., Services & Development Div., Integrated Computing & Communications Dept.
(opinions expressed herein are mine and not those of LLNL)

--
✐Richard Cook
✇ Lawrence Livermore National Laboratory
Bldg-453 Rm-4024, Mail Stop L-557
7000 East Avenue,  Livermore, CA, 94550, USA
☎ (office) (925) 423-9605
☎ (fax) (925) 423-6961
---
Information Management & Graphics Grp., Services & Development Div., Integrated Computing & Communications Dept.
(opinions expressed herein are mine and not those of LLNL)

--
✐Richard Cook
✇ Lawrence Livermore National Laboratory
Bldg-453 Rm-4024, Mail Stop L-557
7000 East Avenue,  Livermore, CA, 94550, USA
☎ (office) (925) 423-9605
☎ (fax) (925) 423-6961
---
Information Management & Graphics Grp., Services & Development Div., Integrated Computing & Communications Dept.
(opinions expressed herein are mine and not those of LLNL)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.paraview.org/pipermail/paraview/attachments/20131031/e9488427/attachment-0001.htm>