[Opengeoscience-developers] Websockets and MPI

Wed Feb 27 13:58:32 EST 2013

On Wed, Feb 27, 2013 at 1:48 PM, Ben Burnett <benjam.arlyn at gmail.com> wrote:

> Sort of, I have a high level plan of splitting data, parrallel computing,
> and then sending data peice by piece to the client via websockets.

Can you add text to the Wiki in this regards? So that we can find our key
issues and corresponding solution and then may be each of us can pick a
task for this investigation?
I really liked the wiki page you created and I think it would be good to
have discussion on Wiki then on the google document but I will leave that
to you.

> I need to know what the data structure will be, and what metadata will
> need to be attached to each piece of data that is streamed.

I am thinking that we need to think about this but before that we need to
find out what library can read part of data and perform analysis on it
which could be aggregated later by the frontend. We will need:

1. Ability to read data from a region
2. Ability to read complete meta data
3. Ability to compute statistics on that data for analysis purposes.

I think CDAT can do this or VTK as well. What others?

> Should this be GeoJSON? I haven't looked into it much yet.

I think whatever we choose, we can mostly convert it into GeoJSON (except
image data). How do we send image data across could be a question as well.

Thanks,

>
>
> On Wed, Feb 27, 2013 at 1:43 PM, Aashish Chaudhary <
> aashish.chaudhary at kitware.com> wrote:
>
>> It works now. I had bad installation of mpi and then on top of that I got
>> the wrong text pasted on the pi.py file.
>>
>> Thanks. Have you plan the next steps?
>>
>>
>> On Wed, Feb 27, 2013 at 1:39 PM, Aashish Chaudhary <
>> aashish.chaudhary at kitware.com> wrote:
>>
>>> Found the problem (and pasted one file wrong)
>>>
>>>
>>> On Wed, Feb 27, 2013 at 1:38 PM, Aashish Chaudhary <
>>> aashish.chaudhary at kitware.com> wrote:
>>>
>>>> yes,
>>>>
>>>> *pi.py*
>>>> from mpi4py import MPI
>>>> import numpy
>>>>
>>>> comm = MPI.Comm.Get_parent()
>>>> size = comm.Get_size()
>>>> rank = comm.Get_rank()
>>>>
>>>> N = numpy.array(0, dtype='i')
>>>> comm.Bcast([N, MPI.INT], root=0)
>>>> h = 1.0 / N; s = 0.0
>>>> for i in range(rank, N, size):
>>>>     x = h * (i + 0.5)
>>>>     s += 4.0 / (1.0 + x**2)
>>>> PI = numpy.array(s * h, dtype='d')
>>>> comm.Reduce([PI, MPI.DOUBLE], None,
>>>>             op=MPI.SUM, root=0)
>>>>
>>>> comm.Disconnect()
>>>>
>>>> *cpi.py*
>>>> #!/usr/bin/env python
>>>> from  mpi4py import MPI
>>>> import numpy
>>>>
>>>> comm = MPI.Comm.Get_parent()
>>>> size = comm.Get_size()
>>>> rank = comm.Get_rank()
>>>>
>>>> N = numpy.array(0, dtype='i')
>>>> comm.Bcast([N, MPI.INT], root=0)
>>>> h = 1.0 / N; s = 0.0
>>>> for i in range(rank, N, size):
>>>>     x = h * (i + 0.5)
>>>>     s += 4.0 / (1.0 + x**2)
>>>> PI = numpy.array(s * h, dtype='d')
>>>> comm.Reduce([PI, MPI.DOUBLE], None,
>>>>             op=MPI.SUM, root=0)
>>>>
>>>>  comm.Disconnect()
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Feb 27, 2013 at 1:29 PM, Ben Burnett <benjam.arlyn at gmail.com>wrote:
>>>>
>>>>> When you ran the example, did you create two python files, one for the
>>>>> master (e.g. pi.py), one for the worker (cpi.py), and then run the master
>>>>> file?
>>>>>
>>>>>
>>>>> On Wed, Feb 27, 2013 at 1:23 PM, Aashish Chaudhary <
>>>>> aashish.chaudhary at kitware.com> wrote:
>>>>>
>>>>>> That's what I did to verify..here is the error I got:
>>>>>>
>>>>>> Traceback (most recent call last):
>>>>>>   File "test.py", line 5, in <module>
>>>>>>     size = comm.Get_size()
>>>>>>   File "Comm.pyx", line 68, in mpi4py.MPI.Comm.Get_size
>>>>>> (src/mpi4py.MPI.c:57940)
>>>>>> mpi4py.MPI.Exception: Invalid communicator, error stack:
>>>>>> MPI_Comm_size(111): MPI_Comm_size(MPI_COMM_NULL, size=0x7fffeda7bd3c)
>>>>>> failed
>>>>>> MPI_Comm_size(69).: Null communicator
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 27, 2013 at 1:14 PM, Ben Burnett <benjam.arlyn at gmail.com>wrote:
>>>>>>
>>>>>>> The Pi example came from
>>>>>>> http://mpi4py.scipy.org/docs/usrman/tutorial.html, I just modified
>>>>>>> it to run in a websocket handler on cherrypy.
>>>>>>>
>>>>>>> It looks like it's failing to spawn the processes using MPI. Do you
>>>>>>> get any exceptions in the log?
>>>>>>>
>>>>>>> You could try first running the Pi example from the above website in
>>>>>>> python to make sure MPI is working on your machine.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Feb 27, 2013 at 12:46 PM, Aashish Chaudhary <
>>>>>>> aashish.chaudhary at kitware.com> wrote:
>>>>>>>
>>>>>>>> @Ben,
>>>>>>>>
>>>>>>>> The char exampled worked like charm, but the PI example didn't.  I
>>>>>>>> don't see the value of PI. Did you write that code or its from the ws4py
>>>>>>>> test / example?
>>>>>>>>
>>>>>>>>  Thanks
>>>>>>>>
>>>>>>>>
>>>>>>>>   1 [27/Feb/2013:12:44:38] ENGINE Listening for SIGHUP.
>>>>>>>>   2 [27/Feb/2013:12:44:38] ENGINE Listening for SIGTERM.
>>>>>>>>   3 [27/Feb/2013:12:44:38] ENGINE Listening for SIGUSR1.
>>>>>>>>   4 [27/Feb/2013:12:44:38] ENGINE Bus STARTING
>>>>>>>>   5 [27/Feb/2013:12:44:38] ENGINE Forking once.
>>>>>>>>   6 [27/Feb/2013:12:44:38] ENGINE Forking twice.
>>>>>>>>   7 [27/Feb/2013:12:44:38] ENGINE Daemonized to PID: 24810
>>>>>>>>   8 [27/Feb/2013:12:44:38]  Starting WebSocket processing
>>>>>>>>   9 [27/Feb/2013:12:44:38] ENGINE PID 24810 written to
>>>>>>>> '/home/aashish/workspace/build/deploy/ogs.pid'.
>>>>>>>>  10 [27/Feb/2013:12:44:38] ENGINE Started monitor thread
>>>>>>>> '_TimeoutMonitor'.
>>>>>>>>  11 [27/Feb/2013:12:44:38] ENGINE Started monitor thread
>>>>>>>> 'Autoreloader'.
>>>>>>>>  12 [27/Feb/2013:12:44:38] ENGINE Serving on 127.0.0.1:8080
>>>>>>>>  13 [27/Feb/2013:12:44:38] ENGINE Bus STARTED
>>>>>>>>  14 [27/Feb/2013:12:44:41]  Handler created:
>>>>>>>> <websocket_pi.PiWebSocketHandler object at 0x275f410>
>>>>>>>>  15 [27/Feb/2013:12:44:41]  Managing WebSocket connection from
>>>>>>>> 127.0.0.1:56898
>>>>>>>>  16 [27/Feb/2013:12:44:46]  something
>>>>>>>>  17 [27/Feb/2013:12:44:46]  Recieved message 101
>>>>>>>>  18 [27/Feb/2013:12:44:46]  Dir /home/aashish/workspace/build/deploy
>>>>>>>>  19 [27/Feb/2013:12:44:46]  Path
>>>>>>>> /home/aashish/workspace/build/deploy/cpi.py
>>>>>>>>
>>>>>>>> On Wed, Feb 27, 2013 at 12:15 PM, Aashish Chaudhary <
>>>>>>>> aashish.chaudhary at kitware.com> wrote:
>>>>>>>>
>>>>>>>>> Nice..I will have a look at it today.
>>>>>>>>>
>>>>>>>>> On Wed, Feb 27, 2013 at 12:10 PM, Ben Burnett <
>>>>>>>>> benjam.arlyn at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I just pushed the branch websockets-mpi, and added a wiki page
>>>>>>>>>> with some notes on it
>>>>>>>>>> https://github.com/OpenGeoscience/opengeoscience/wiki/Testing-Websockets-and-MPI
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Opengeoscience-developers mailing list
>>>>>>>>>> Opengeoscience-developers at public.kitware.com
>>>>>>>>>>
>>>>>>>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/opengeoscience-developers
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> | Aashish Chaudhary
>>>>>>>>> | R&D Engineer
>>>>>>>>> | Kitware Inc.
>>>>>>>>> | www.kitware.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> | Aashish Chaudhary
>>>>>>>> | R&D Engineer
>>>>>>>> | Kitware Inc.
>>>>>>>> | www.kitware.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> | Aashish Chaudhary
>>>>>> | R&D Engineer
>>>>>> | Kitware Inc.
>>>>>> | www.kitware.com
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> | Aashish Chaudhary
>>>> | R&D Engineer
>>>> | Kitware Inc.
>>>> | www.kitware.com
>>>>
>>>
>>>
>>>
>>> --
>>> | Aashish Chaudhary
>>> | R&D Engineer
>>> | Kitware Inc.
>>> | www.kitware.com
>>>
>>
>>
>>
>> --
>> | Aashish Chaudhary
>> | R&D Engineer
>> | Kitware Inc.
>> | www.kitware.com
>>
>
>

-- 
| Aashish Chaudhary
| R&D Engineer
| Kitware Inc.
| www.kitware.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/opengeoscience-developers/attachments/20130227/b556cf03/attachment.html>