<div class="gmail_quote">On Wed, Feb 27, 2013 at 1:48 PM, Ben Burnett <span dir="ltr"><<a href="mailto:benjam.arlyn@gmail.com" target="_blank">benjam.arlyn@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Sort of, I have a high level plan of splitting data, parrallel computing, and then sending data peice by piece to the client via websockets.</blockquote><div><br></div><div>Can you add text to the Wiki in this regards? So that we can find our key issues and corresponding solution and then may be each of us can pick a task for this investigation? </div>
<div>I really liked the wiki page you created and I think it would be good to have discussion on Wiki then on the google document but I will leave that to you. </div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I need to know what the data structure will be, and what metadata will need to be attached to each piece of data that is streamed.</blockquote><div> </div><div>I am thinking that we need to think about this but before that we need to find out what library can read part of data and perform analysis on it which could be aggregated later by the frontend. We will need: </div>
<div><br></div><div>1. Ability to read data from a region</div><div>2. Ability to read complete meta data</div><div>3. Ability to compute statistics on that data for analysis purposes. </div><div><br></div><div>I think CDAT can do this or VTK as well. What others? </div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Should this be GeoJSON? I haven't looked into it much yet.</blockquote><div><br></div><div>I think whatever we choose, we can mostly convert it into GeoJSON (except image data). How do we send image data across could be a question as well. </div>
<div><br></div><div>Thanks,</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><br>
<br><div class="gmail_quote">On Wed, Feb 27, 2013 at 1:43 PM, Aashish Chaudhary <span dir="ltr"><<a href="mailto:aashish.chaudhary@kitware.com" target="_blank">aashish.chaudhary@kitware.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
It works now. I had bad installation of mpi and then on top of that I got the wrong text pasted on the pi.py file. <div><br></div><div>Thanks. Have you plan the next steps? </div><div><div><div>
<br>
<br><div class="gmail_quote">On Wed, Feb 27, 2013 at 1:39 PM, Aashish Chaudhary <span dir="ltr"><<a href="mailto:aashish.chaudhary@kitware.com" target="_blank">aashish.chaudhary@kitware.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Found the problem (and pasted one file wrong)<div><div><div><br><br><div class="gmail_quote">
On Wed, Feb 27, 2013 at 1:38 PM, Aashish Chaudhary <span dir="ltr"><<a href="mailto:aashish.chaudhary@kitware.com" target="_blank">aashish.chaudhary@kitware.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">yes, <div><br></div><div><b>pi.py</b></div><div><div>from mpi4py import MPI</div><div>import numpy</div><div><br></div>
<div>comm = MPI.Comm.Get_parent()</div><div>size = comm.Get_size()</div><div>rank = comm.Get_rank()</div>
<div><br></div><div>N = numpy.array(0, dtype='i')</div><div>comm.Bcast([N, <a href="http://MPI.INT" target="_blank">MPI.INT</a>], root=0)</div><div>h = 1.0 / N; s = 0.0</div><div>for i in range(rank, N, size):</div>
<div> x = h * (i + 0.5)</div>
<div> s += 4.0 / (1.0 + x**2)</div><div>PI = numpy.array(s * h, dtype='d')</div><div>comm.Reduce([PI, MPI.DOUBLE], None,</div><div> op=MPI.SUM, root=0)</div><div><br></div><div>comm.Disconnect()</div>
</div><div><br></div><div><b>cpi.py</b></div><div><div>#!/usr/bin/env python</div><div>from mpi4py import MPI</div><div>import numpy</div><div><br></div><div>comm = MPI.Comm.Get_parent()</div><div>size = comm.Get_size()</div>
<div>rank = comm.Get_rank()</div><div><br></div><div>N = numpy.array(0, dtype='i')</div><div>comm.Bcast([N, <a href="http://MPI.INT" target="_blank">MPI.INT</a>], root=0)</div><div>h = 1.0 / N; s = 0.0</div><div>
for i in range(rank, N, size):</div>
<div> x = h * (i + 0.5)</div><div> s += 4.0 / (1.0 + x**2)</div><div>PI = numpy.array(s * h, dtype='d')</div><div>comm.Reduce([PI, MPI.DOUBLE], None,</div><div> op=MPI.SUM, root=0)</div><div><br>
</div>
<div>comm.Disconnect()</div><div style="font-weight:bold"><br></div></div><div><div><div><br></div><div><br><br><div class="gmail_quote">On Wed, Feb 27, 2013 at 1:29 PM, Ben Burnett <span dir="ltr"><<a href="mailto:benjam.arlyn@gmail.com" target="_blank">benjam.arlyn@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">When you ran the example, did you create two python files, one for the master (e.g. pi.py), one for the worker (cpi.py), and then run the master file?<div>
<div><br><br><div class="gmail_quote">On Wed, Feb 27, 2013 at 1:23 PM, Aashish Chaudhary <span dir="ltr"><<a href="mailto:aashish.chaudhary@kitware.com" target="_blank">aashish.chaudhary@kitware.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">That's what I did to verify..here is the error I got: <div><br></div><div><div>Traceback (most recent call last):</div>
<div> File "test.py", line 5, in <module></div><div> size = comm.Get_size()</div>
<div> File "Comm.pyx", line 68, in mpi4py.MPI.Comm.Get_size (src/mpi4py.MPI.c:57940)</div><div>mpi4py.MPI.Exception: Invalid communicator, error stack:</div><div>MPI_Comm_size(111): MPI_Comm_size(MPI_COMM_NULL, size=0x7fffeda7bd3c) failed</div>
<div>MPI_Comm_size(69).: Null communicator</div><div><div><div><br></div><br><div class="gmail_quote">On Wed, Feb 27, 2013 at 1:14 PM, Ben Burnett <span dir="ltr"><<a href="mailto:benjam.arlyn@gmail.com" target="_blank">benjam.arlyn@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">The Pi example came from <a href="http://mpi4py.scipy.org/docs/usrman/tutorial.html" target="_blank">http://mpi4py.scipy.org/docs/usrman/tutorial.html</a>, I just modified it to run in a websocket handler on cherrypy. <div>
<br></div><div>
It looks like it's failing to spawn the processes using MPI. Do you get any exceptions in the log?</div><div><br></div><div>You could try first running the Pi example from the above website in python to make sure MPI is working on your machine.<div>
<div><br>
<br><div class="gmail_quote">On Wed, Feb 27, 2013 at 12:46 PM, Aashish Chaudhary <span dir="ltr"><<a href="mailto:aashish.chaudhary@kitware.com" target="_blank">aashish.chaudhary@kitware.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>@Ben, </div><div><br></div><div>The char exampled worked like charm, but the PI example didn't. I don't see the value of PI. Did you write that code or its from the ws4py test / example?</div><div><br></div>
<div>
Thanks</div><div><br></div><div><br></div><div> 1 [27/Feb/2013:12:44:38] ENGINE Listening for SIGHUP.</div><div> 2 [27/Feb/2013:12:44:38] ENGINE Listening for SIGTERM.</div><div> 3 [27/Feb/2013:12:44:38] ENGINE Listening for SIGUSR1.</div>
<div> 4 [27/Feb/2013:12:44:38] ENGINE Bus STARTING</div><div> 5 [27/Feb/2013:12:44:38] ENGINE Forking once.</div><div> 6 [27/Feb/2013:12:44:38] ENGINE Forking twice.</div><div> 7 [27/Feb/2013:12:44:38] ENGINE Daemonized to PID: 24810</div>
<div> 8 [27/Feb/2013:12:44:38] Starting WebSocket processing</div><div> 9 [27/Feb/2013:12:44:38] ENGINE PID 24810 written to '/home/aashish/workspace/build/deploy/ogs.pid'.</div><div> 10 [27/Feb/2013:12:44:38] ENGINE Started monitor thread '_TimeoutMonitor'.</div>
<div> 11 [27/Feb/2013:12:44:38] ENGINE Started monitor thread 'Autoreloader'.</div><div> 12 [27/Feb/2013:12:44:38] ENGINE Serving on <a href="http://127.0.0.1:8080" target="_blank">127.0.0.1:8080</a></div><div> 13 [27/Feb/2013:12:44:38] ENGINE Bus STARTED</div>
<div> 14 [27/Feb/2013:12:44:41] Handler created: <websocket_pi.PiWebSocketHandler object at 0x275f410></div><div> 15 [27/Feb/2013:12:44:41] Managing WebSocket connection from <a href="http://127.0.0.1:56898" target="_blank">127.0.0.1:56898</a></div>
<div> 16 [27/Feb/2013:12:44:46] something</div><div> 17 [27/Feb/2013:12:44:46] Recieved message 101</div><div> 18 [27/Feb/2013:12:44:46] Dir /home/aashish/workspace/build/deploy</div><div> 19 [27/Feb/2013:12:44:46] Path /home/aashish/workspace/build/deploy/cpi.py</div>
<div><div>
<br><div class="gmail_quote">On Wed, Feb 27, 2013 at 12:15 PM, Aashish Chaudhary <span dir="ltr"><<a href="mailto:aashish.chaudhary@kitware.com" target="_blank">aashish.chaudhary@kitware.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Nice..I will have a look at it today. <br><br><div class="gmail_quote"><div>On Wed, Feb 27, 2013 at 12:10 PM, Ben Burnett <span dir="ltr"><<a href="mailto:benjam.arlyn@gmail.com" target="_blank">benjam.arlyn@gmail.com</a>></span> wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>I just pushed the branch websockets-mpi, and added a wiki page with some notes on it <a href="https://github.com/OpenGeoscience/opengeoscience/wiki/Testing-Websockets-and-MPI" target="_blank">https://github.com/OpenGeoscience/opengeoscience/wiki/Testing-Websockets-and-MPI</a>
<br></div>_______________________________________________<br>
Opengeoscience-developers mailing list<br>
<a href="mailto:Opengeoscience-developers@public.kitware.com" target="_blank">Opengeoscience-developers@public.kitware.com</a><br>
<a href="http://public.kitware.com/cgi-bin/mailman/listinfo/opengeoscience-developers" target="_blank">http://public.kitware.com/cgi-bin/mailman/listinfo/opengeoscience-developers</a><br>
<br></blockquote></div><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br>| Aashish Chaudhary <br>| R&D Engineer <br>| Kitware Inc. <br>| <a href="http://www.kitware.com" target="_blank">www.kitware.com</a>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br>| Aashish Chaudhary <br>| R&D Engineer <br>| Kitware Inc. <br>| <a href="http://www.kitware.com" target="_blank">www.kitware.com</a>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>| Aashish Chaudhary <br>| R&D Engineer <br>| Kitware Inc. <br>| <a href="http://www.kitware.com" target="_blank">www.kitware.com</a>
</div></div></div>
</blockquote></div><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>| Aashish Chaudhary <br>| R&D Engineer <br>| Kitware Inc. <br>| <a href="http://www.kitware.com" target="_blank">www.kitware.com</a>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>| Aashish Chaudhary <br>| R&D Engineer <br>| Kitware Inc. <br>| <a href="http://www.kitware.com" target="_blank">www.kitware.com</a>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>| Aashish Chaudhary <br>| R&D Engineer <br>| Kitware Inc. <br>| <a href="http://www.kitware.com" target="_blank">www.kitware.com</a>
</div>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>| Aashish Chaudhary <br>| R&D Engineer <br>| Kitware Inc. <br>| <a href="http://www.kitware.com">www.kitware.com</a>