No subject

Mon Aug 10 18:29:21 EDT 2009

> i tried running the same data using np1 until np10, and i get the followi=
ng
> reading for still render, the rest of the readings are quite consistent,
> which makes me wonder why the time (still render) decrease initially but
> increase eventually? shouldn't it become much faster when more nodes are
> used? all my server nodes are identical to each other (intel core 2 duo,
> 2.66 GHz)
>
>
ParaView doesn't use parallelism to scale in terms of processing speed, it
uses it to scale in terms of achievable data size. That means, if you have
sufficient memory on one machine, processing the data on two machines isn't
likely to give you any speed up and will usually slow it down because of th=
e
extra code and especially communication that has to happen in the parallel
program (Ahmdahls law). If the problem is too big for one machine, then the
only alternative is to split the data and run it on many machines
(Gustafsen's law). That said, if the data is nearly to big for one machine,
for instance when swapping, then splitting it and running it on several
machines will make it faster for a while until the overhead starts to
dominate the performance.

>
> ----------------------------------------------
>   number of process used still render (seconds)  np1 7.16043  np2 3.93390
> np3 3.05784  np4 3.02900  np5 3.02851  np6 3.04962  np7 3.43479  np8
> 3.47883  np9 3.71554  np10 3.80835
> ----------------------------------------------
>
>
> appreciate your reply!
>
> regards,
> chewping
>
>
> ------------------------------
> From: kmorel at sandia.gov
> To: lcp81um at msn.com; paraview at paraview.org
> CC: kitware at kitware.com
> Date: Mon, 28 Sep 2009 08:37:08 -0600
> Subject: Re: [Paraview] How to interpret timer log
>
>
> Both scenarios are wrong.  ParaView will not push out data from process 0
> to processes 1-3 unless you explicitly run a filter that does that (or th=
e
> reader does that internally, but I know of no such reader).  What is
> actually happening is more along the lines of:
>
>
>    1. Processes 0-3 each read in a partition of data from the file.
>    2. Each process extracts polygonal geometry from their local data.
>    3. Per your settings, ParaView decides to send the geometry to the
>    client.  The data is collected to process 0 and sent to the client.
>
>
> The reason you are not seeing vtkFileSeriesReader on all of the servers i=
s
> that there is a threshold in the timer log to not show anything that
> executes under a certain amount of time (by default 0.01 seconds).  If yo=
u
> change the Time Threshold to Show All, you should be able to see everythi=
ng
> that executes, even if it completes immediately.
>
> You should note that how readers read partitions is determined by the
> reader itself.  Many of the readers do not really handle partitioned
> reading.  Thus, the reader will do something na=EFve like read everything=
 on
> process 0.  Based on your timings, that is probably what is happening to
> you.  That is, processes 1-3 probably have empty data.  You never specifi=
ed
> what format of data you are reader, so I cannot answer the data completel=
y.
>  However, if you want to know how your data is partitioned on the server =
(at
> least, before rendering), you can run the Process Ids filter.
>
> -Ken
>
>
> On 9/24/09 9:09 PM, "chew ping" <lcp81um at msn.com <http://msn.com>> wrote:
>
> Hi all,
>
> I'm doing parallel rendering using 1 client (dual core laptop) and 2
> cluster servers (dual core desktop)
> below is the timer log result i collected when i run: mpirun -np 4 pvserv=
er
>
>
> -------------------------------------------------------------------------=
----------------------
> Local Process
> Still Render, 3.029 seconds
> Execute vtkMPIMoveData id: 457, 1.98248 seconds
>
>
> Server, Process 0
> Execute vtkFileSeriesReader id: 176, 0.637821 seconds
> Execute vtkMPIMoveData id: 457, 1.49186 seconds
> Dataserver gathering to 0, 0.829525 seconds
> Dataserver sending to client, 0.661658 seconds
>
> Server, Process 1
> Execute vtkMPIMoveData id: 457, 0.141821 seconds
> Dataserver gathering to 0, 0.141544 seconds
>
> Server, Process 2
> Execute vtkMPIMoveData id: 457, 0.243584 seconds
> Dataserver gathering to 0, 0.243318 seconds
>
> Server, Process 3
> Execute vtkMPIMoveData id: 457, 0.191589 seconds
> Dataserver gathering to 0, 0.191303 seconds
>
>
> -------------------------------------------------------------------------=
----------------------------
>
> i have difficulty interpreting the timer log, my guess is:
>
> Scenario 1:
> Process 0 reads the whole data, disseminate the dats into 4 pieces, then
> distribute to itself and Process 1&2&3,
> each node will process the data and send it back Process 0,
> Process 0 gather all data and send it back to client,
> client renders the data
>
> Scenario 2:
> Process 0 reads the whole data, distribute the whole data to Process
> 0&1&2&3,
> each node will 'take' their own piece of data to process, then send it ba=
ck
> Process 0,
> Process 0 gather all data and send it back to client,
> client renders the data
>
> Which scenario is the correct one? or both are wrong?
>
> is there any resources i could refer to find what does it mean by: Execut=
e
> vtkFileSeriesReader, Execute vtkMPIMoveData?
>
> any help / feedback is highly appreciated!
> thanks!
>
> regards,
> chewping
>
>
>
>
>
>
>
>
>    ****      Kenneth Moreland
>     ***      Sandia National Laboratories
> ***********
> *** *** ***  email: kmorel at sandia.gov <http://sandia.gov>
> **  ***  **  phone: (505) 844-8919
>     ***      web:   http://www.cs.unm.edu/~kmorel<http://www.cs.unm.edu/%=
7Ekmorel>
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at:
> http://paraview.org/Wiki/ParaView
>
> Follow this link to subscribe/unsubscribe:
> http://www.paraview.org/mailman/listinfo/paraview
>
>

--0016368e1d5873f76e0474cb7fdb
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I counted more than one question (six in fact) in that last email. :) <d=
iv class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"border-=
left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left=
: 1ex;">

<div><font style=3D"font-size: 10pt;" size=3D"2"><span style=3D"font-size: =
10pt;"><font face=3D"Tahoma, Verdana, Helvetica, Arial">does these mean tha=
t, at the client&#39;s end, it took 3.029 seconds to renders the geometry s=
end back from process 0?<br>

</div></blockquote><div> I believe the 3.029 second=
s includes the transfer time, but I may be incorrect on that. Someone pleas=
e correct me if I am wrong. =A0 </div><blockquote class=3D"gmail_quot=
e" style=3D"border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt =
0.8ex; padding-left: 1ex;">

<div>what does still ren=
der means? </div></blockquote><div> While you drag =
the mouse and move the camera paraview does interactive renders. Iteractive=
 renders will (depending on data and the preference settings) use subsample=
d geometry and subsampled images. Once you release the mouse, ParaView does=
 a still render at full resolution.

<br>=A0</div><blockquote class=3D"gmail_quote" style=3D"border-left: 1px so=
lid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div=
><font style=3D"font-size: 10pt;" size=3D"2"><span style=3D"font-size: 10pt=
;"><font face=3D"Tahoma, Verdana, Helvetica, Arial">how about the third lin=
e? after still render, why does the MPI need to move data? </font></span></=
font></div>

</blockquote><div> See answer to question 1, the MPI time (and processin=
g) is being counted as part of the rendering. Since it is &quot;part of&quo=
t; the rendering, it shows up after in the log (even if it happened first).=

</div><div>=A0</div><blockquote class=3D"gmail_quote" style=3D"border-left:=
 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex=
;"><div><font style=3D"font-size: 10pt;" size=3D"2"><span style=3D"font-siz=
e: 10pt;"><font face=3D"Tahoma, Verdana, Helvetica, Arial">which data? </fo=
nt></span></font></div>

</blockquote><div><br>Geometry (the surface of whatever your data is).<br>=
=A0<br></div><blockquote class=3D"gmail_quote" style=3D"border-left: 1px so=
lid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div=
><font style=3D"font-size: 10pt;" size=3D"2"><span style=3D"font-size: 10pt=
;"><font face=3D"Tahoma, Verdana, Helvetica, Arial">to where?<br>

</font></span></font></div></blockquote><div><br>From the root node of the =
data server to the client.<br><br></div><blockquote class=3D"gmail_quote" s=
tyle=3D"border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8e=
x; padding-left: 1ex;">

<div> i tried running=
 the same data using np1 until np10, and i get the following reading for st=
ill render=
, the
rest of the readings are quite consistent, which makes me wonder why
the time (still render) decrease initially but increase eventually? shouldn=
&#39;t it
become much faster when more nodes are used? all my server nodes are identi=
cal to each other (intel core 2 duo, 2.66 GHz) 
 </div></blockquote><div> ParaView doesn&#39;t u=
se parallelism to scale in terms of processing speed, it uses it to scale i=
n terms of achievable data size. That means, if you have sufficient memory =
on one machine, processing the data on two machines isn&#39;t likely to giv=
e you any speed up and will usually slow it down because of the extra code =
and especially communication that has to happen in the parallel program (Ah=
mdahls law). If the problem is too big for one machine, then the only alter=
native is to split the data and run it on many machines (Gustafsen&#39;s la=
w). That said, if the data is nearly to big for one machine, for instance w=
hen swapping, then splitting it and running it on several machines will mak=
e it faster for a while until the overhead starts to dominate the performan=
ce.

<br>=A0</div><blockquote class=3D"gmail_quote" style=3D"border-left: 1px so=
lid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div=
><font style=3D"font-size: 10pt;" size=3D"2"><span style=3D"font-size: 10pt=
;"><font face=3D"Tahoma, Verdana, Helvetica, Arial"><br>

----------------------------------------------<br></font></span></font>

=09
=09
=09
=09
=09
=09

<table border=3D"0" cellspacing=3D"0" cols=3D"2" frame=3D"void" height=3D"1=
87" rules=3D"none" width=3D"268">
	<colgroup><col width=3D"176"><col width=3D"145"></colgroup>
	<tbody>
		<tr>
			<td align=3D"left" height=3D"17" width=3D"176">number of process used </=
td>
			<td align=3D"left" width=3D"145">still render (seconds)</td>
		</tr>
		<tr>
			<td align=3D"left" height=3D"17">np1</td>
			<td align=3D"right">7.16043</td>
		</tr>
		<tr>
			<td align=3D"left" height=3D"17">np2</td>
			<td align=3D"right">3.93390</td>
		</tr>
		<tr>
			<td align=3D"left" height=3D"17">np3</td>
			<td align=3D"right">3.05784</td>
		</tr>
		<tr>
			<td align=3D"left" height=3D"17">np4</td>
			<td align=3D"right">3.02900</td>
		</tr>
		<tr>
			<td align=3D"left" height=3D"17">np5</td>
			<td align=3D"right">3.02851</td>
		</tr>
		<tr>
			<td align=3D"left" height=3D"17">np6</td>
			<td align=3D"right">3.04962</td>
		</tr>
		<tr>
			<td align=3D"left" height=3D"17">np7</td>
			<td align=3D"right">3.43479</td>
		</tr>
		<tr>
			<td align=3D"left" height=3D"17">np8</td>
			<td align=3D"right">3.47883</td>
		</tr>
		<tr>
			<td align=3D"left" height=3D"17">np9</td>
			<td align=3D"right">3.71554</td>
		</tr>
		<tr>
			<td align=3D"left" height=3D"17">np10</td>
			<td align=3D"right">3.80835</td>
		</tr>
	</tbody>
</table>
<br><font style=3D"font-size: 10pt;" size=3D"2"><span style=3D"font-size: 1=
0pt;"><font face=3D"Tahoma, Verdana, Helvetica, Arial">--------------------=
--------------------------<br><br>=A0<br>appreciate your reply!<br><br>rega=
rds,<br>

chewping<br><br><br></font></span></font><hr>From: <a href=3D"mailto:kmorel=
@sandia.gov" target=3D"_blank">kmorel at sandia.gov</a><br>To: <a href=3D"mail=
to:lcp81um at msn.com" target=3D"_blank">lcp81um at msn.com</a>; <a href=3D"mailt=
o:paraview at paraview.org" target=3D"_blank">paraview at paraview.org</a><br>

CC: <a href=3D"mailto:kitware at kitware.com" target=3D"_blank">kitware at kitwar=
e.com</a><br>Date: Mon, 28 Sep 2009 08:37:08 -0600<br>Subject: Re: [Paravie=
w] How to interpret timer log<div><div></div><div class=3D"h5"><br><br>

Both scenarios are wrong. =A0ParaView will not push out data from p=
rocess 0 to processes 1-3 unless you explicitly run a filter that does that=
 (or the reader does that internally, but I know of no such reader). =A0Wha=
t is actually happening is more along the lines of:

<ol><li><sp=
an style=3D"font-size: 11pt;">Processes 0-3 each read in a partition of dat=
a from the file.
</li><li><s=
pan style=3D"font-size: 11pt;">Each process extracts polygonal geometry fro=
m their local data.
</li><li><s=
pan style=3D"font-size: 11pt;">Per your settings, ParaView decides to send =
the geometry to the client. =A0The data is collected to process 0 and sent =
to the client.

</li></ol><=
span style=3D"font-size: 11pt;"> 
The reason you are not seeing vtkFileSeriesReader on all of the servers is =
that there is a threshold in the timer log to not show anything that execut=
es under a certain amount of time (by default 0.01 seconds). =A0If you chan=
ge the Time Threshold to Show All, you should be able to see everything tha=
t executes, even if it completes immediately.

You should note that how readers read partitions is determined by the reade=
r itself. =A0Many of the readers do not really handle partitioned reading. =
=A0Thus, the reader will do something na=EFve like read everything on proce=
ss 0. =A0Based on your timings, that is probably what is happening to you. =
=A0That is, processes 1-3 probably have empty data. =A0You never specified =
what format of data you are reader, so I cannot answer the data completely.=
 =A0However, if you want to know how your data is partitioned on the server=
 (at least, before rendering), you can run the Process Ids filter.

<br>
-Ken<br>
<br>
<br>
On 9/24/09 9:09 PM, &quot;chew ping&quot; &lt;<a href=3D"http://msn.com" ta=
rget=3D"_blank">lcp81um at msn.com</a>&gt; wrote:<br>
<br>
</span></font><blockquote><font size=3D"2"><font face=3D"Verdana, Helvetica=
, Arial"><span style=3D"font-size: 10pt;">Hi all,<br>
=A0<br>
I&#39;m doing parallel rendering using 1 client (dual core laptop) and 2 cl=
uster servers (dual core desktop)<br>
below is the timer log result i collected when i run: mpirun -np 4 pvserver=
<br>
=A0<br>
---------------------------------------------------------------------------=
--------------------<br>
</span></font><span style=3D"font-size: 10pt;"><font face=3D"Tahoma, Verdan=
a, Helvetica, Arial">Local Process<br>
Still Render, 3.029 seconds<br>
Execute vtkMPIMoveData id: 457, 1.98248 seconds<br>
=A0<br>
=A0<br>
Server, Process 0<br>
Execute vtkFileSeriesReader id: 176, 0.637821 seconds<br>
Execute vtkMPIMoveData id: 457, 1.49186 seconds<br>
Dataserver gathering to 0, 0.829525 seconds<br>
Dataserver sending to client, 0.661658 seconds<br>
</font><font face=3D"Verdana, Helvetica, Arial"> <br>
</font><font face=3D"Tahoma, Verdana, Helvetica, Arial">Server, Process 1<b=
r>
Execute vtkMPIMoveData id: 457, 0.141821 seconds<br>
Dataserver gathering to 0, 0.141544 seconds<br>
</font><font face=3D"Verdana, Helvetica, Arial"> <br>
</font><font face=3D"Tahoma, Verdana, Helvetica, Arial">Server, Process 2<b=
r>
Execute vtkMPIMoveData id: 457, 0.243584 seconds<br>
Dataserver gathering to 0, 0.243318 seconds<br>
</font><font face=3D"Verdana, Helvetica, Arial"> <br>
</font><font face=3D"Tahoma, Verdana, Helvetica, Arial">Server, Process 3<b=
r>
Execute vtkMPIMoveData id: 457, 0.191589 seconds<br>
Dataserver gathering to 0, 0.191303 seconds<br>
</font><font face=3D"Verdana, Helvetica, Arial"> <br>
---------------------------------------------------------------------------=
--------------------------<br>
=A0<br>
i have difficulty interpreting the timer log, my guess is:<br>
=A0<br>
Scenario 1: <br>
Process 0 reads the whole data, disseminate the dats into 4 pieces, then di=
stribute to itself and Process 1&amp;2&amp;3,<br>
each node will process the data and send it back Process 0,<br>
Process 0 gather all data and send it back to client,<br>
client renders the data<br>
=A0<br>
Scenario 2:<br>
Process 0 reads the whole data, distribute the whole data to Process 0&amp;=
1&amp;2&amp;3,<br>
each node will &#39;take&#39; their own piece of data to process, then send=
 it back Process 0,<br>
Process 0 gather all data and send it back to client,<br>
client renders the data<br>
=A0=A0=A0<br>
Which scenario is the correct one? or both are wrong?<br>
=A0<br>
is there any resources i could refer to find what does it mean by: Execute =
vtkFileSeriesReader, Execute vtkMPIMoveData?<br>
=A0<br>
any help / feedback is highly appreciated!<br>
thanks!<br>
=A0<br>
regards,<br>
chewping<br>
=A0<br>
=A0<br>
=A0<br>
=A0<br>
=A0=A0=A0=A0=A0=A0=A0<br>
</font></span></font></blockquote><font size=3D"2"><span style=3D"font-size=
: 10pt;"><font face=3D"Verdana, Helvetica, Arial"><br>
</font><font face=3D"Consolas, Courier New, Courier"><br>
=A0=A0=A0**** =A0=A0=A0=A0=A0Kenneth Moreland<br>
=A0=A0=A0=A0*** =A0=A0=A0=A0=A0Sandia National Laboratories<br>
*********** =A0<br>
*** *** *** =A0email: <a href=3D"http://sandia.gov" target=3D"_blank">kmore=
l at sandia.gov</a><br>
** =A0*** =A0** =A0phone: (505) 844-8919<br>
=A0=A0=A0=A0*** =A0=A0=A0=A0=A0web: =A0=A0<a href=3D"http://www.cs.unm.edu/=
%7Ekmorel" target=3D"_blank">http://www.cs.unm.edu/~kmorel</a><br>
</font></span></font><font face=3D"Calibri, Verdana, Helvetica, Arial"><spa=
n style=3D"font-size: 11pt;"><br>
</span></font> 		 	   		  </div></div></div>
<br>_______________________________________________<br>
Powered by <a href=3D"http://www.kitware.com" target=3D"_blank">www.kitware=
.com</a><br>
<br>
Visit other Kitware open-source projects at <a href=3D"http://www.kitware.c=
om/opensource/opensource.html" target=3D"_blank">http://www.kitware.com/ope=
nsource/opensource.html</a><br>
<br>
Please keep messages on-topic and check the ParaView Wiki at: <a href=3D"ht=
tp://paraview.org/Wiki/ParaView" target=3D"_blank">http://paraview.org/Wiki=
/ParaView</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href=3D"http://www.paraview.org/mailman/listinfo/paraview" target=3D"_bl=
ank">http://www.paraview.org/mailman/listinfo/paraview</a><br>
<br></blockquote></div><br>

--0016368e1d5873f76e0474cb7fdb--