From tristan.wright at kitware.com  Tue Apr 11 19:38:31 2017
From: tristan.wright at kitware.com (Tristan Wright)
Date: Tue, 11 Apr 2017 17:38:31 -0600
Subject: [Girder-users] Missing requirement ctk_cli?
Message-ID: <CACJDoNTD7oyAWHHJ_XM6LaDiMLhHcUyAx_dv5bxVvoV78rwuug@mail.gmail.com>

On master the test `server_item_tasks.tasks` failing. The plugin
`item_tasks` isn't loading because of an import error:

153:     from . import cli_parser, constants
>>
> 153:   File "/Users/tristan/Envs/girder/girder/plugins/item_tasks/server/cli_parser.py",
>> line 1, in <module>
>
> 153:     import ctk_cli
>
> 153: ImportError: No module named ctk_cli
>
>
Pip-installing the latest ctk_cli package fixes this. ctk_cli isn't listed
in requirements.txt or requirements_dev.txt, should it be?

-- 
- Tristan Wright

*R&D Engineer*
*Kitware Inc.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170411/52b2a106/attachment.html>

From david.manthey at kitware.com  Wed Apr 12 08:18:53 2017
From: david.manthey at kitware.com (David Manthey)
Date: Wed, 12 Apr 2017 08:18:53 -0400
Subject: [Girder-users] Missing requirement ctk_cli?
In-Reply-To: <CACJDoNTD7oyAWHHJ_XM6LaDiMLhHcUyAx_dv5bxVvoV78rwuug@mail.gmail.com>
References: <CACJDoNTD7oyAWHHJ_XM6LaDiMLhHcUyAx_dv5bxVvoV78rwuug@mail.gmail.com>
Message-ID: <CANnzgbCdEAFJ-hXVZ9MFe120JuY=07cGx58GLKZV1Kjtm3qZXQ@mail.gmail.com>

I might suggest we use this version of ctk-cli:
pip install --upgrade 'git+https://github.com/cdeepakroy/ctk-cli'
See https://github.com/commontk/ctk-cli/pull/8 for details why.

On Tue, Apr 11, 2017 at 7:38 PM, Tristan Wright <tristan.wright at kitware.com>
wrote:

> On master the test `server_item_tasks.tasks` failing. The plugin
> `item_tasks` isn't loading because of an import error:
>
> 153:     from . import cli_parser, constants
>>>
>> 153:   File "/Users/tristan/Envs/girder/girder/plugins/item_tasks/server/cli_parser.py",
>>> line 1, in <module>
>>
>> 153:     import ctk_cli
>>
>> 153: ImportError: No module named ctk_cli
>>
>>
> Pip-installing the latest ctk_cli package fixes this. ctk_cli isn't listed
> in requirements.txt or requirements_dev.txt, should it be?
>
> --
> - Tristan Wright
>
> *R&D Engineer*
> *Kitware Inc.*
>
> _______________________________________________
> Girder-users mailing list
> Girder-users at public.kitware.com
> http://public.kitware.com/mailman/listinfo/girder-users
>
>


-- 
David Manthey
R&D Engineer
Kitware Inc.
(518) 881-4439
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170412/0715c4d9/attachment.html>

From jchris.fillionr at kitware.com  Wed Apr 12 08:21:43 2017
From: jchris.fillionr at kitware.com (Jean-Christophe Fillion-Robin)
Date: Wed, 12 Apr 2017 08:21:43 -0400
Subject: [Girder-users] Missing requirement ctk_cli?
In-Reply-To: <CANnzgbCdEAFJ-hXVZ9MFe120JuY=07cGx58GLKZV1Kjtm3qZXQ@mail.gmail.com>
References: <CACJDoNTD7oyAWHHJ_XM6LaDiMLhHcUyAx_dv5bxVvoV78rwuug@mail.gmail.com>
 <CANnzgbCdEAFJ-hXVZ9MFe120JuY=07cGx58GLKZV1Kjtm3qZXQ@mail.gmail.com>
Message-ID: <CANLz8R7bmcnDbtfj8Wc7FEfJPFdm04SqSzhRRn9QmD+JbTWiOw@mail.gmail.com>

As discussed with Deepak, I plan on integrating outstanding PR to ctk_cli.
Jc

On Wed, Apr 12, 2017 at 8:18 AM, David Manthey <david.manthey at kitware.com>
wrote:

> I might suggest we use this version of ctk-cli:
> pip install --upgrade 'git+https://github.com/cdeepakroy/ctk-cli'
> See https://github.com/commontk/ctk-cli/pull/8 for details why.
>
> On Tue, Apr 11, 2017 at 7:38 PM, Tristan Wright <
> tristan.wright at kitware.com> wrote:
>
>> On master the test `server_item_tasks.tasks` failing. The plugin
>> `item_tasks` isn't loading because of an import error:
>>
>> 153:     from . import cli_parser, constants
>>>>
>>> 153:   File "/Users/tristan/Envs/girder/girder/plugins/item_tasks/server/cli_parser.py",
>>>> line 1, in <module>
>>>
>>> 153:     import ctk_cli
>>>
>>> 153: ImportError: No module named ctk_cli
>>>
>>>
>> Pip-installing the latest ctk_cli package fixes this. ctk_cli isn't
>> listed in requirements.txt or requirements_dev.txt, should it be?
>>
>> --
>> - Tristan Wright
>>
>> *R&D Engineer*
>> *Kitware Inc.*
>>
>> _______________________________________________
>> Girder-users mailing list
>> Girder-users at public.kitware.com
>> http://public.kitware.com/mailman/listinfo/girder-users
>>
>>
>
>
> --
> David Manthey
> R&D Engineer
> Kitware Inc.
> (518) 881-4439
>
> _______________________________________________
> Girder-users mailing list
> Girder-users at public.kitware.com
> http://public.kitware.com/mailman/listinfo/girder-users
>
>


-- 
+1 919 869 8849
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170412/5ee1992d/attachment.html>

From Brian.Chapman at utah.edu  Thu Apr 13 13:45:29 2017
From: Brian.Chapman at utah.edu (Brian E Chapman)
Date: Thu, 13 Apr 2017 17:45:29 +0000
Subject: [Girder-users] Find a file with a given name in a Collection
Message-ID: <97DE2675-613A-407E-9A39-1EB74D704A88@umail.utah.edu>

Hi Everyone,

I?m interested using the Python API to find all the items in a collection that match a name pattern. I?ve written a recursive function that goes through the items and folders (using the client listItem and listFolder methods) in the collection but it seems incredibly slow and I?m assuming there must be a better way of doing it. I?m using girder_client vs 2.0.0


Brian E Chapman, Ph.D
Associate Professor
Department of Radiology and Imaging Sciences
University of Utah

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170413/5ebac03e/attachment.html>

From zach.mullen at kitware.com  Thu Apr 13 14:29:04 2017
From: zach.mullen at kitware.com (Zach Mullen)
Date: Thu, 13 Apr 2017 14:29:04 -0400
Subject: [Girder-users] Find a file with a given name in a Collection
In-Reply-To: <97DE2675-613A-407E-9A39-1EB74D704A88@umail.utah.edu>
References: <97DE2675-613A-407E-9A39-1EB74D704A88@umail.utah.edu>
Message-ID: <CAPzddEHar+0-Q92t0aLHOscFCddR6zzx0GkV==OUKmgTHUDvMQ@mail.gmail.com>

Hi Brian,

Items actually keep track of what top-level resource they live under in
their model data.  This isn't exposed via the core web API, so you'd want
to expose it via a plugin that would create a new endpoint.  You could do
this (mostly) in the database, by using find() with a query like the
following:

results = self.model('item').find({
    'name': {'$regex': searchRegex},
    'baseParentType': 'collection',
    'baseParentId': collectionId
})
return list(self.model('item').filterResultsByPermission(results,
self.getCurrentUser(), level=AccessType.READ, limit=limit, offset=offset))


It would probably also be a good idea to add the relevant indexes to the
item collection to make such searching much faster.


HTH,



Zach Mullen
Kitware, Inc.
919-869-8858

On Thu, Apr 13, 2017 at 1:45 PM, Brian E Chapman <Brian.Chapman at utah.edu>
wrote:

> Hi Everyone,
>
> I?m interested using the Python API to find all the items in a collection
> that match a name pattern. I?ve written a recursive function that goes
> through the items and folders (using the client listItem and listFolder
> methods) in the collection but it seems incredibly slow and I?m assuming
> there must be a better way of doing it. I?m using girder_client vs 2.0.0
>
>
> Brian E Chapman, Ph.D
> Associate Professor
> Department of Radiology and Imaging Sciences
> University of Utah
>
>
> _______________________________________________
> Girder-users mailing list
> Girder-users at public.kitware.com
> http://public.kitware.com/mailman/listinfo/girder-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170413/60b52fc5/attachment.html>

From caguero at osrfoundation.org  Thu Apr 20 19:41:06 2017
From: caguero at osrfoundation.org (=?UTF-8?Q?Carlos_Ag=C3=BCero?=)
Date: Thu, 20 Apr 2017 16:41:06 -0700
Subject: [Girder-users] Deploy Girder on AWS with Elastic Beanstalk
Message-ID: <CAMGA_6LF7mBWpWKLuNLNA3xDQ_wSGg9x9t9ZQ3e4tpN+Wm3kdg@mail.gmail.com>

Hello,

I'm new to Girder and deployed a local instance on my own development
machine without problems following your documentation (nice documentation
by the way!).

I'd like to deploy another Girder instance on AWS and I've read the
"Deploy" section of the Administrator documentation. It seems to cover a
few options but none of them is AWS.

I normally use Elastic Beanstalk (EB) that allows you to configure a load
balancer that will spin up machines depending on the server demand. This
model seems compatible with Girder as long as the database is deployed in a
separate machine separated from the servers, to make sure that there's only
one machine running the database.

Does anyone have experience, documentation or suggestions deploying Girder
on AWS with EB?

Thanks!
Carlos
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170420/a2dfc836/attachment.html>

From michael.grauer at kitware.com  Thu Apr 20 20:01:50 2017
From: michael.grauer at kitware.com (Michael Grauer)
Date: Thu, 20 Apr 2017 20:01:50 -0400
Subject: [Girder-users] Deploy Girder on AWS with Elastic Beanstalk
In-Reply-To: <CAMGA_6LF7mBWpWKLuNLNA3xDQ_wSGg9x9t9ZQ3e4tpN+Wm3kdg@mail.gmail.com>
References: <CAMGA_6LF7mBWpWKLuNLNA3xDQ_wSGg9x9t9ZQ3e4tpN+Wm3kdg@mail.gmail.com>
Message-ID: <CAKx26+HJAuc0tEgJ7Ys+dQOPFfcrtvo8fisfKOT=T3O4iAuYZA@mail.gmail.com>

Hi Carlos,

Thanks for trying out Girder, I'm glad you appreciate our docs!  We've done
a number of deployments to AWS, but they've all been EC2 instances.  EB is
on our roadmap, we'd like to start playing with it over the next couple
months, but don't have much concrete to share yet.  We'd love for you to
keep us informed about your progress, or let us know any stumbling blocks
you run into.

The plan you describe sounds like what we were going to attempt, put Mongo
in a separate EC2 instance, and then have EB bring up Dockerized (seems
easiest, and we already have a Docker image in Girder's repo) Girder
containers.  We normally would put Girder behind Nginx or Apache.  Were you
going to use ELB for your load balancer, and is that what you normally use
for your EB deployments?  How to best use ELB + possibly other proxy
servers if necessary (e.g., can you set all of the proxy rules in ELB to
allow stream notifications? where do you terminate TLS?) + Girder in EB are
open questions for us.

Let us know if you have more specific questions as well.

Thanks,
Mike



On Thu, Apr 20, 2017 at 7:41 PM, Carlos Ag?ero <caguero at osrfoundation.org>
wrote:

> Hello,
>
> I'm new to Girder and deployed a local instance on my own development
> machine without problems following your documentation (nice documentation
> by the way!).
>
> I'd like to deploy another Girder instance on AWS and I've read the
> "Deploy" section of the Administrator documentation. It seems to cover a
> few options but none of them is AWS.
>
> I normally use Elastic Beanstalk (EB) that allows you to configure a load
> balancer that will spin up machines depending on the server demand. This
> model seems compatible with Girder as long as the database is deployed in a
> separate machine separated from the servers, to make sure that there's only
> one machine running the database.
>
> Does anyone have experience, documentation or suggestions deploying Girder
> on AWS with EB?
>
> Thanks!
> Carlos
>
>
>
> _______________________________________________
> Girder-users mailing list
> Girder-users at public.kitware.com
> http://public.kitware.com/mailman/listinfo/girder-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170420/a264bc2d/attachment.html>

From caguero at osrfoundation.org  Wed Apr 26 17:02:04 2017
From: caguero at osrfoundation.org (=?UTF-8?Q?Carlos_Ag=C3=BCero?=)
Date: Wed, 26 Apr 2017 14:02:04 -0700
Subject: [Girder-users] Deploy Girder on AWS with Elastic Beanstalk
In-Reply-To: <CAKx26+HJAuc0tEgJ7Ys+dQOPFfcrtvo8fisfKOT=T3O4iAuYZA@mail.gmail.com>
References: <CAMGA_6LF7mBWpWKLuNLNA3xDQ_wSGg9x9t9ZQ3e4tpN+Wm3kdg@mail.gmail.com>
 <CAKx26+HJAuc0tEgJ7Ys+dQOPFfcrtvo8fisfKOT=T3O4iAuYZA@mail.gmail.com>
Message-ID: <CAMGA_6L141_oKdAeD19cAX31RaZjTyJateMN2nE5j4uAJnyQ_Q@mail.gmail.com>

hi Michael,

Thanks for your reply! Sorry that I didn't follow-up before but I've been
exploring and partially documenting the process.

I managed to set-up an EC2 machine hosting the Mongo database. Also, I
configured Elastic Beanstalk with a single Docker container (girder/girder)
and a load balancer. The instances are running Girder and using the
external database. I associated an SSL certificate to the load balancer.
All the connections between the users and load balancer are secured and
between the load balancer and the instances go though regular HTTP. I think
this is OK because the load balancer and the instances are within the
internal VPN that cannot be sniffed. The TSL termination happens on the
load balancer.

A similar thing occurs with the database instance. The Mongo port (27017)
is configured with a security rule that allows access only from a machine
within the same VPN (the Girder instances in this case).

The default EB configuration uses NGINX, that redirects requests to the
Docker container. I still have an open issue for redirecting the non-https
requests to https without breaking the health checker monitor that EB uses.
I've done this in the past but this is the first time that I do it with EB
+ docker.

I'm in the process of documenting the process here:

https://bitbucket.org/osrf/propshop-girder/overview

On Thu, Apr 20, 2017 at 5:01 PM, Michael Grauer <michael.grauer at kitware.com>
wrote:

> Hi Carlos,
>
> Thanks for trying out Girder, I'm glad you appreciate our docs!  We've
> done a number of deployments to AWS, but they've all been EC2 instances.
> EB is on our roadmap, we'd like to start playing with it over the next
> couple months, but don't have much concrete to share yet.  We'd love for
> you to keep us informed about your progress, or let us know any stumbling
> blocks you run into.
>
> The plan you describe sounds like what we were going to attempt, put Mongo
> in a separate EC2 instance, and then have EB bring up Dockerized (seems
> easiest, and we already have a Docker image in Girder's repo) Girder
> containers.  We normally would put Girder behind Nginx or Apache.  Were you
> going to use ELB for your load balancer, and is that what you normally use
> for your EB deployments?  How to best use ELB + possibly other proxy
> servers if necessary (e.g., can you set all of the proxy rules in ELB to
> allow stream notifications? where do you terminate TLS?) + Girder in EB are
> open questions for us.
>
> Let us know if you have more specific questions as well.
>
> Thanks,
> Mike
>
>
>
> On Thu, Apr 20, 2017 at 7:41 PM, Carlos Ag?ero <caguero at osrfoundation.org>
> wrote:
>
>> Hello,
>>
>> I'm new to Girder and deployed a local instance on my own development
>> machine without problems following your documentation (nice documentation
>> by the way!).
>>
>> I'd like to deploy another Girder instance on AWS and I've read the
>> "Deploy" section of the Administrator documentation. It seems to cover a
>> few options but none of them is AWS.
>>
>> I normally use Elastic Beanstalk (EB) that allows you to configure a load
>> balancer that will spin up machines depending on the server demand. This
>> model seems compatible with Girder as long as the database is deployed in a
>> separate machine separated from the servers, to make sure that there's only
>> one machine running the database.
>>
>> Does anyone have experience, documentation or suggestions deploying
>> Girder on AWS with EB?
>>
>> Thanks!
>> Carlos
>>
>>
>>
>> _______________________________________________
>> Girder-users mailing list
>> Girder-users at public.kitware.com
>> http://public.kitware.com/mailman/listinfo/girder-users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170426/8354b2e9/attachment.html>

From caguero at osrfoundation.org  Wed Apr 26 18:25:51 2017
From: caguero at osrfoundation.org (=?UTF-8?Q?Carlos_Ag=C3=BCero?=)
Date: Wed, 26 Apr 2017 15:25:51 -0700
Subject: [Girder-users] C++ client library
Message-ID: <CAMGA_6Km+=Lhz-Dm9TUpAmc1bpVi+BrMmB1eL8S1SDpoK7xnKQ@mail.gmail.com>

hi to everyone!

I noticed that Swagger has been used inside Girder. I don't know anything
about Swagger but the Girder online documentation for visualizing the API
and even requesting API calls is pretty awesome.

We're considering to write a C++ client library for Girder and I found that
Swagger supports the generation of client libraries in multiple languages
(including C++). Does anyone know if this is possible and/or a good
approach? My first instinct is to write a custom library using libcurl and
a json parser as dependencies (jsoncpp maybe) but I'd love to hear if
someone has suggestions about Swagger.

Thanks!
Carlos
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170426/240bf765/attachment.html>

From jchris.fillionr at kitware.com  Wed Apr 26 18:34:05 2017
From: jchris.fillionr at kitware.com (Jean-Christophe Fillion-Robin)
Date: Wed, 26 Apr 2017 18:34:05 -0400
Subject: [Girder-users] C++ client library
In-Reply-To: <CAMGA_6Km+=Lhz-Dm9TUpAmc1bpVi+BrMmB1eL8S1SDpoK7xnKQ@mail.gmail.com>
References: <CAMGA_6Km+=Lhz-Dm9TUpAmc1bpVi+BrMmB1eL8S1SDpoK7xnKQ@mail.gmail.com>
Message-ID: <CANLz8R5u6Gh2gXEXKAOrXEDeXh7RY7WuNeAJT9RCTQuV6VTMXg@mail.gmail.com>

Hi Carlos,

While I don't have experience with Swagger + cpp generation, if Qt is a
dependency of your stack you could look into using qRestAPI,
<https://github.com/commontk/qRestAPI>it is a simple library to perform
synchronous or asynchronous REST API calls.

Instead of JsonCpp, I would also suggest to use RapidJson
<http://rapidjson.org/>, it performs order of magnitude faster.

Hth
Jc

On Wed, Apr 26, 2017 at 6:25 PM, Carlos Ag?ero <caguero at osrfoundation.org>
wrote:

> hi to everyone!
>
> I noticed that Swagger has been used inside Girder. I don't know anything
> about Swagger but the Girder online documentation for visualizing the API
> and even requesting API calls is pretty awesome.
>
> We're considering to write a C++ client library for Girder and I found
> that Swagger supports the generation of client libraries in multiple
> languages (including C++). Does anyone know if this is possible and/or a
> good approach? My first instinct is to write a custom library using libcurl
> and a json parser as dependencies (jsoncpp maybe) but I'd love to hear if
> someone has suggestions about Swagger.
>
> Thanks!
> Carlos
>
>
>
> _______________________________________________
> Girder-users mailing list
> Girder-users at public.kitware.com
> http://public.kitware.com/mailman/listinfo/girder-users
>
>


-- 
+1 919 869 8849
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170426/8eac441b/attachment.html>

From caguero at osrfoundation.org  Wed Apr 26 18:40:07 2017
From: caguero at osrfoundation.org (=?UTF-8?Q?Carlos_Ag=C3=BCero?=)
Date: Wed, 26 Apr 2017 15:40:07 -0700
Subject: [Girder-users] C++ client library
In-Reply-To: <CANLz8R5u6Gh2gXEXKAOrXEDeXh7RY7WuNeAJT9RCTQuV6VTMXg@mail.gmail.com>
References: <CAMGA_6Km+=Lhz-Dm9TUpAmc1bpVi+BrMmB1eL8S1SDpoK7xnKQ@mail.gmail.com>
 <CANLz8R5u6Gh2gXEXKAOrXEDeXh7RY7WuNeAJT9RCTQuV6VTMXg@mail.gmail.com>
Message-ID: <CAMGA_6JY-ckz1tkZGu4RVih_U=BcxiDJ+jqVoO49DWwr7Sgv6A@mail.gmail.com>

Jc, thanks for the pointers!



On Wed, Apr 26, 2017 at 3:34 PM, Jean-Christophe Fillion-Robin <
jchris.fillionr at kitware.com> wrote:

> Hi Carlos,
>
> While I don't have experience with Swagger + cpp generation, if Qt is a
> dependency of your stack you could look into using qRestAPI,
> <https://github.com/commontk/qRestAPI>it is a simple library to perform
> synchronous or asynchronous REST API calls.
>
> Instead of JsonCpp, I would also suggest to use RapidJson
> <http://rapidjson.org/>, it performs order of magnitude faster.
>
> Hth
> Jc
>
> On Wed, Apr 26, 2017 at 6:25 PM, Carlos Ag?ero <caguero at osrfoundation.org>
> wrote:
>
>> hi to everyone!
>>
>> I noticed that Swagger has been used inside Girder. I don't know anything
>> about Swagger but the Girder online documentation for visualizing the API
>> and even requesting API calls is pretty awesome.
>>
>> We're considering to write a C++ client library for Girder and I found
>> that Swagger supports the generation of client libraries in multiple
>> languages (including C++). Does anyone know if this is possible and/or a
>> good approach? My first instinct is to write a custom library using libcurl
>> and a json parser as dependencies (jsoncpp maybe) but I'd love to hear if
>> someone has suggestions about Swagger.
>>
>> Thanks!
>> Carlos
>>
>>
>>
>> _______________________________________________
>> Girder-users mailing list
>> Girder-users at public.kitware.com
>> http://public.kitware.com/mailman/listinfo/girder-users
>>
>>
>
>
> --
> +1 919 869 8849 <(919)%20869-8849>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170426/86e1f187/attachment-0001.html>

From zach.mullen at kitware.com  Thu Apr 27 09:41:22 2017
From: zach.mullen at kitware.com (Zach Mullen)
Date: Thu, 27 Apr 2017 09:41:22 -0400
Subject: [Girder-users] Deploy Girder on AWS with Elastic Beanstalk
In-Reply-To: <CAMGA_6L141_oKdAeD19cAX31RaZjTyJateMN2nE5j4uAJnyQ_Q@mail.gmail.com>
References: <CAMGA_6LF7mBWpWKLuNLNA3xDQ_wSGg9x9t9ZQ3e4tpN+Wm3kdg@mail.gmail.com>
 <CAKx26+HJAuc0tEgJ7Ys+dQOPFfcrtvo8fisfKOT=T3O4iAuYZA@mail.gmail.com>
 <CAMGA_6L141_oKdAeD19cAX31RaZjTyJateMN2nE5j4uAJnyQ_Q@mail.gmail.com>
Message-ID: <CAPzddEFXQYuFA6Cxk2BmEenPeo0qZoW=tm=nh1m=CT8GdDqL7A@mail.gmail.com>

This is great news, thanks for reaching out, Carlos!

Zach Mullen
Kitware, Inc.
919-869-8858

On Wed, Apr 26, 2017 at 5:02 PM, Carlos Ag?ero <caguero at osrfoundation.org>
wrote:

> hi Michael,
>
> Thanks for your reply! Sorry that I didn't follow-up before but I've been
> exploring and partially documenting the process.
>
> I managed to set-up an EC2 machine hosting the Mongo database. Also, I
> configured Elastic Beanstalk with a single Docker container (girder/girder)
> and a load balancer. The instances are running Girder and using the
> external database. I associated an SSL certificate to the load balancer.
> All the connections between the users and load balancer are secured and
> between the load balancer and the instances go though regular HTTP. I think
> this is OK because the load balancer and the instances are within the
> internal VPN that cannot be sniffed. The TSL termination happens on the
> load balancer.
>
> A similar thing occurs with the database instance. The Mongo port (27017)
> is configured with a security rule that allows access only from a machine
> within the same VPN (the Girder instances in this case).
>
> The default EB configuration uses NGINX, that redirects requests to the
> Docker container. I still have an open issue for redirecting the non-https
> requests to https without breaking the health checker monitor that EB uses.
> I've done this in the past but this is the first time that I do it with EB
> + docker.
>
> I'm in the process of documenting the process here:
>
> https://bitbucket.org/osrf/propshop-girder/overview
>
> On Thu, Apr 20, 2017 at 5:01 PM, Michael Grauer <
> michael.grauer at kitware.com> wrote:
>
>> Hi Carlos,
>>
>> Thanks for trying out Girder, I'm glad you appreciate our docs!  We've
>> done a number of deployments to AWS, but they've all been EC2 instances.
>> EB is on our roadmap, we'd like to start playing with it over the next
>> couple months, but don't have much concrete to share yet.  We'd love for
>> you to keep us informed about your progress, or let us know any stumbling
>> blocks you run into.
>>
>> The plan you describe sounds like what we were going to attempt, put
>> Mongo in a separate EC2 instance, and then have EB bring up Dockerized
>> (seems easiest, and we already have a Docker image in Girder's repo) Girder
>> containers.  We normally would put Girder behind Nginx or Apache.  Were you
>> going to use ELB for your load balancer, and is that what you normally use
>> for your EB deployments?  How to best use ELB + possibly other proxy
>> servers if necessary (e.g., can you set all of the proxy rules in ELB to
>> allow stream notifications? where do you terminate TLS?) + Girder in EB are
>> open questions for us.
>>
>> Let us know if you have more specific questions as well.
>>
>> Thanks,
>> Mike
>>
>>
>>
>> On Thu, Apr 20, 2017 at 7:41 PM, Carlos Ag?ero <caguero at osrfoundation.org
>> > wrote:
>>
>>> Hello,
>>>
>>> I'm new to Girder and deployed a local instance on my own development
>>> machine without problems following your documentation (nice documentation
>>> by the way!).
>>>
>>> I'd like to deploy another Girder instance on AWS and I've read the
>>> "Deploy" section of the Administrator documentation. It seems to cover a
>>> few options but none of them is AWS.
>>>
>>> I normally use Elastic Beanstalk (EB) that allows you to configure a
>>> load balancer that will spin up machines depending on the server demand.
>>> This model seems compatible with Girder as long as the database is deployed
>>> in a separate machine separated from the servers, to make sure that there's
>>> only one machine running the database.
>>>
>>> Does anyone have experience, documentation or suggestions deploying
>>> Girder on AWS with EB?
>>>
>>> Thanks!
>>> Carlos
>>>
>>>
>>>
>>> _______________________________________________
>>> Girder-users mailing list
>>> Girder-users at public.kitware.com
>>> http://public.kitware.com/mailman/listinfo/girder-users
>>>
>>>
>>
>
> _______________________________________________
> Girder-users mailing list
> Girder-users at public.kitware.com
> http://public.kitware.com/mailman/listinfo/girder-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170427/e1ca33fb/attachment.html>

From michael.grauer at kitware.com  Thu Apr 27 10:04:30 2017
From: michael.grauer at kitware.com (Michael Grauer)
Date: Thu, 27 Apr 2017 10:04:30 -0400
Subject: [Girder-users] Deploy Girder on AWS with Elastic Beanstalk
In-Reply-To: <CAMGA_6L141_oKdAeD19cAX31RaZjTyJateMN2nE5j4uAJnyQ_Q@mail.gmail.com>
References: <CAMGA_6LF7mBWpWKLuNLNA3xDQ_wSGg9x9t9ZQ3e4tpN+Wm3kdg@mail.gmail.com>
 <CAKx26+HJAuc0tEgJ7Ys+dQOPFfcrtvo8fisfKOT=T3O4iAuYZA@mail.gmail.com>
 <CAMGA_6L141_oKdAeD19cAX31RaZjTyJateMN2nE5j4uAJnyQ_Q@mail.gmail.com>
Message-ID: <CAKx26+GKd1zUKnxpBT+-KGc_FkrUFLNP=mUQ8CidPDzJS47QsQ@mail.gmail.com>

Hi Carlos,

Thanks for sharing this!

I hope you don't mind if I ask you some follow-up questions :)  Just trying
to understand your setup and choices in more detail.

I agree that your security setup sounds reasonable, HTTPS to load balancer,
HTTP from load balancer to Girder (though I have more questions on this
below), assuming the instances are not visible to the outside world and
only to the LB via the VPN, and Mongo/Instances talking to each other
inside the same VPN.  Out of curiosity (rather than suggesting a policy),
how do you handle ssh, are each of the machines accessible to ssh or do you
have a VPN ssh gateway machine?

When you say load balancer, does that mean Elastic Load Balancer or
something else?  I'm confused about how you use Nginx, are you using ELB +
Nginx, and if so how does ELB hand off to Nginx?  Where does Nginx live, is
it in a separate Docker container that redirects to the Girder instances?

Thanks,
Mike


On Wed, Apr 26, 2017 at 5:02 PM, Carlos Ag?ero <caguero at osrfoundation.org>
wrote:

> hi Michael,
>
> Thanks for your reply! Sorry that I didn't follow-up before but I've been
> exploring and partially documenting the process.
>
> I managed to set-up an EC2 machine hosting the Mongo database. Also, I
> configured Elastic Beanstalk with a single Docker container (girder/girder)
> and a load balancer. The instances are running Girder and using the
> external database. I associated an SSL certificate to the load balancer.
> All the connections between the users and load balancer are secured and
> between the load balancer and the instances go though regular HTTP. I think
> this is OK because the load balancer and the instances are within the
> internal VPN that cannot be sniffed. The TSL termination happens on the
> load balancer.
>
> A similar thing occurs with the database instance. The Mongo port (27017)
> is configured with a security rule that allows access only from a machine
> within the same VPN (the Girder instances in this case).
>
> The default EB configuration uses NGINX, that redirects requests to the
> Docker container. I still have an open issue for redirecting the non-https
> requests to https without breaking the health checker monitor that EB uses.
> I've done this in the past but this is the first time that I do it with EB
> + docker.
>
> I'm in the process of documenting the process here:
>
> https://bitbucket.org/osrf/propshop-girder/overview
>
> On Thu, Apr 20, 2017 at 5:01 PM, Michael Grauer <
> michael.grauer at kitware.com> wrote:
>
>> Hi Carlos,
>>
>> Thanks for trying out Girder, I'm glad you appreciate our docs!  We've
>> done a number of deployments to AWS, but they've all been EC2 instances.
>> EB is on our roadmap, we'd like to start playing with it over the next
>> couple months, but don't have much concrete to share yet.  We'd love for
>> you to keep us informed about your progress, or let us know any stumbling
>> blocks you run into.
>>
>> The plan you describe sounds like what we were going to attempt, put
>> Mongo in a separate EC2 instance, and then have EB bring up Dockerized
>> (seems easiest, and we already have a Docker image in Girder's repo) Girder
>> containers.  We normally would put Girder behind Nginx or Apache.  Were you
>> going to use ELB for your load balancer, and is that what you normally use
>> for your EB deployments?  How to best use ELB + possibly other proxy
>> servers if necessary (e.g., can you set all of the proxy rules in ELB to
>> allow stream notifications? where do you terminate TLS?) + Girder in EB are
>> open questions for us.
>>
>> Let us know if you have more specific questions as well.
>>
>> Thanks,
>> Mike
>>
>>
>>
>> On Thu, Apr 20, 2017 at 7:41 PM, Carlos Ag?ero <caguero at osrfoundation.org
>> > wrote:
>>
>>> Hello,
>>>
>>> I'm new to Girder and deployed a local instance on my own development
>>> machine without problems following your documentation (nice documentation
>>> by the way!).
>>>
>>> I'd like to deploy another Girder instance on AWS and I've read the
>>> "Deploy" section of the Administrator documentation. It seems to cover a
>>> few options but none of them is AWS.
>>>
>>> I normally use Elastic Beanstalk (EB) that allows you to configure a
>>> load balancer that will spin up machines depending on the server demand.
>>> This model seems compatible with Girder as long as the database is deployed
>>> in a separate machine separated from the servers, to make sure that there's
>>> only one machine running the database.
>>>
>>> Does anyone have experience, documentation or suggestions deploying
>>> Girder on AWS with EB?
>>>
>>> Thanks!
>>> Carlos
>>>
>>>
>>>
>>> _______________________________________________
>>> Girder-users mailing list
>>> Girder-users at public.kitware.com
>>> http://public.kitware.com/mailman/listinfo/girder-users
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170427/ba5f4031/attachment.html>

From caguero at osrfoundation.org  Thu Apr 27 13:33:25 2017
From: caguero at osrfoundation.org (=?UTF-8?Q?Carlos_Ag=C3=BCero?=)
Date: Thu, 27 Apr 2017 10:33:25 -0700
Subject: [Girder-users] Deploy Girder on AWS with Elastic Beanstalk
In-Reply-To: <CAKx26+GKd1zUKnxpBT+-KGc_FkrUFLNP=mUQ8CidPDzJS47QsQ@mail.gmail.com>
References: <CAMGA_6LF7mBWpWKLuNLNA3xDQ_wSGg9x9t9ZQ3e4tpN+Wm3kdg@mail.gmail.com>
 <CAKx26+HJAuc0tEgJ7Ys+dQOPFfcrtvo8fisfKOT=T3O4iAuYZA@mail.gmail.com>
 <CAMGA_6L141_oKdAeD19cAX31RaZjTyJateMN2nE5j4uAJnyQ_Q@mail.gmail.com>
 <CAKx26+GKd1zUKnxpBT+-KGc_FkrUFLNP=mUQ8CidPDzJS47QsQ@mail.gmail.com>
Message-ID: <CAMGA_6J0jiP9qCz0q1Br5k1jFKcc56Ud-2o=2LzN7fjJgcJLGQ@mail.gmail.com>

On Thu, Apr 27, 2017 at 7:04 AM, Michael Grauer <michael.grauer at kitware.com>
wrote:

> [...]
>


> I hope you don't mind if I ask you some follow-up questions :)  Just
> trying to understand your setup and choices in more detail.
>

Of course!

I agree that your security setup sounds reasonable, HTTPS to load balancer,
> HTTP from load balancer to Girder (though I have more questions on this
> below), assuming the instances are not visible to the outside world and
> only to the LB via the VPN, and Mongo/Instances talking to each other
> inside the same VPN.  Out of curiosity (rather than suggesting a policy),
> how do you handle ssh, are each of the machines accessible to ssh or do you
> have a VPN ssh gateway machine?
>

Both the Girder instances and the EC2 machine with the Mongo database have
public IP addresses, share the same VPC (private network) and all the
machines support ssh access from the outside. On the instances under the
Elastic Beanstalk (Girder instances), this is the default behavior (you can
create a key pair .pem file for sshing). On the EC2 instance with the Mongo
DB, you have to attach a security group that configures the access. The
inbound rule for port 22 is set to 0.0.0.0 allowing access from machines
outside the private network. On the other hand, the rule for port 27017
restricts the access only from machines belonging to the same VPC.



> When you say load balancer, does that mean Elastic Load Balancer or
> something else?  I'm confused about how you use Nginx, are you using ELB +
> Nginx, and if so how does ELB hand off to Nginx?  Where does Nginx live, is
> it in a separate Docker container that redirects to the Girder instances?
>

I meant the load balancer provided by Elastic Beanstalk. If I'm not wrong,
the request hits the Nginx running on the load balancer. Then, the request
is forwarded to one of the Girder instances running another Nginx. Then,
the request is forwarded to the HTTP server running within the Docker
container. You can SSH into all the machines (including the load balancer)
if you want to poke around and see configurations, logs, etc. I was playing
with /etc/nginx/conf.d/ configurations, restarting the service, like in any
regular machine.

EB offers a way (although a bit convoluted in my opinion) to overload Nginx
configuration files when you deploy new versions of your code. I'm doing
this for tweaking the Nginx configuration in the Girder instances for
redirecting non-https requests to https. The solution involves creating
files under the .ebextension directory with some specific syntax. I managed
to solve the http-->https redirection in a non-dockerized instance but I
still have some issues here using Docker for Girder. In particular, EB
includes a health checker that monitors the instances. When I enable the
redirection, the load balancer receives a 301 response with the https URL
redirection. The expected response was a 200 OK, and that makes the load
balancer think that the instances are not behaving correctly. Maybe you
have more experience than me dealing with Nginx configurations.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170427/7530de15/attachment.html>

From michael.grauer at kitware.com  Thu Apr 27 16:03:35 2017
From: michael.grauer at kitware.com (Michael Grauer)
Date: Thu, 27 Apr 2017 16:03:35 -0400
Subject: [Girder-users] Deploy Girder on AWS with Elastic Beanstalk
In-Reply-To: <CAMGA_6J0jiP9qCz0q1Br5k1jFKcc56Ud-2o=2LzN7fjJgcJLGQ@mail.gmail.com>
References: <CAMGA_6LF7mBWpWKLuNLNA3xDQ_wSGg9x9t9ZQ3e4tpN+Wm3kdg@mail.gmail.com>
 <CAKx26+HJAuc0tEgJ7Ys+dQOPFfcrtvo8fisfKOT=T3O4iAuYZA@mail.gmail.com>
 <CAMGA_6L141_oKdAeD19cAX31RaZjTyJateMN2nE5j4uAJnyQ_Q@mail.gmail.com>
 <CAKx26+GKd1zUKnxpBT+-KGc_FkrUFLNP=mUQ8CidPDzJS47QsQ@mail.gmail.com>
 <CAMGA_6J0jiP9qCz0q1Br5k1jFKcc56Ud-2o=2LzN7fjJgcJLGQ@mail.gmail.com>
Message-ID: <CAKx26+GXYCus7Kxy4=vXigi4FNWStu39ShAWAtCmpiTm4JCyKw@mail.gmail.com>

Thanks for sharing this Carlos, it is really helpful and will give us quite
a leg up when we get back to Girder on EBS.

We'll let you know if we run into those same 301 issues with the health
checker when we get there.


A quick question for you, what are your Girder instances, in terms of EBS
containers?  Are you using an EC2 instance with Nginx and a Dockerized
Girder?  At first I was thinking you were using the Docker runtime of EBS
with Girder, and I couldn't figure out where the secondary Nginx was
living, but it sounds more like EC2 (Nginx, Dockerized Girder) to me.




On Thu, Apr 27, 2017 at 1:33 PM, Carlos Ag?ero <caguero at osrfoundation.org>
wrote:

>
>
> On Thu, Apr 27, 2017 at 7:04 AM, Michael Grauer <michael.grauer at kitware
> .com> wrote:
>
>> [...]
>>
>
>
>> I hope you don't mind if I ask you some follow-up questions :)  Just
>> trying to understand your setup and choices in more detail.
>>
>
> Of course!
>
> I agree that your security setup sounds reasonable, HTTPS to load
>> balancer, HTTP from load balancer to Girder (though I have more questions
>> on this below), assuming the instances are not visible to the outside world
>> and only to the LB via the VPN, and Mongo/Instances talking to each other
>> inside the same VPN.  Out of curiosity (rather than suggesting a policy),
>> how do you handle ssh, are each of the machines accessible to ssh or do you
>> have a VPN ssh gateway machine?
>>
>
> Both the Girder instances and the EC2 machine with the Mongo database
> have public IP addresses, share the same VPC (private network) and all
> the machines support ssh access from the outside. On the instances under
> the Elastic Beanstalk (Girder instances), this is the default behavior (you
> can create a key pair .pem file for sshing). On the EC2 instance with the
> Mongo DB, you have to attach a security group that configures the access.
> The inbound rule for port 22 is set to 0.0.0.0 allowing access from
> machines outside the private network. On the other hand, the rule for port
> 27017 restricts the access only from machines belonging to the same VPC.
>
>
>
>> When you say load balancer, does that mean Elastic Load Balancer or
>> something else?  I'm confused about how you use Nginx, are you using ELB +
>> Nginx, and if so how does ELB hand off to Nginx?  Where does Nginx live, is
>> it in a separate Docker container that redirects to the Girder instances?
>>
>
> I meant the load balancer provided by Elastic Beanstalk. If I'm not
> wrong, the request hits the Nginx running on the load balancer. Then, the
> request is forwarded to one of the Girder instances running another Nginx.
> Then, the request is forwarded to the HTTP server running within the Docker
> container. You can SSH into all the machines (including the load balancer)
> if you want to poke around and see configurations, logs, etc. I was playing
> with /etc/nginx/conf.d/ configurations, restarting the service, like in
> any regular machine.
>
> EB offers a way (although a bit convoluted in my opinion) to overload
> Nginx configuration files when you deploy new versions of your code. I'm
> doing this for tweaking the Nginx configuration in the Girder instances
> for redirecting non-https requests to https. The solution involves creating
> files under the .ebextension directory with some specific syntax. I
> managed to solve the http-->https redirection in a non-dockerized
> instance but I still have some issues here using Docker for Girder. In
> particular, EB includes a health checker that monitors the instances. When
> I enable the redirection, the load balancer receives a 301 response with
> the https URL redirection. The expected response was a 200 OK, and that
> makes the load balancer think that the instances are not behaving
> correctly. Maybe you have more experience than me dealing with Nginx
> configurations.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170427/258a8d73/attachment.html>

From caguero at osrfoundation.org  Fri Apr 28 16:17:47 2017
From: caguero at osrfoundation.org (=?UTF-8?Q?Carlos_Ag=C3=BCero?=)
Date: Fri, 28 Apr 2017 13:17:47 -0700
Subject: [Girder-users] Deploy Girder on AWS with Elastic Beanstalk
In-Reply-To: <CAKx26+GXYCus7Kxy4=vXigi4FNWStu39ShAWAtCmpiTm4JCyKw@mail.gmail.com>
References: <CAMGA_6LF7mBWpWKLuNLNA3xDQ_wSGg9x9t9ZQ3e4tpN+Wm3kdg@mail.gmail.com>
 <CAKx26+HJAuc0tEgJ7Ys+dQOPFfcrtvo8fisfKOT=T3O4iAuYZA@mail.gmail.com>
 <CAMGA_6L141_oKdAeD19cAX31RaZjTyJateMN2nE5j4uAJnyQ_Q@mail.gmail.com>
 <CAKx26+GKd1zUKnxpBT+-KGc_FkrUFLNP=mUQ8CidPDzJS47QsQ@mail.gmail.com>
 <CAMGA_6J0jiP9qCz0q1Br5k1jFKcc56Ud-2o=2LzN7fjJgcJLGQ@mail.gmail.com>
 <CAKx26+GXYCus7Kxy4=vXigi4FNWStu39ShAWAtCmpiTm4JCyKw@mail.gmail.com>
Message-ID: <CAMGA_6KqnfytKYLWpUKK406L9CAR9XF+6+D+BV6nJo=mELOjSA@mail.gmail.com>

As far as I understand I'm using an EC2 instance with Nginx and a
Dockerized Girder.

Having said that, the load balancer machine also has its own Nginx server,
that's why there are two Nginx servers involved.

Nginx (load balancer)->Nginx (instance)->Girder Docker

On Thu, Apr 27, 2017 at 1:03 PM, Michael Grauer <michael.grauer at kitware.com>
wrote:

> Thanks for sharing this Carlos, it is really helpful and will give us
> quite a leg up when we get back to Girder on EBS.
>
> We'll let you know if we run into those same 301 issues with the health
> checker when we get there.
>
>
> A quick question for you, what are your Girder instances, in terms of EBS
> containers?  Are you using an EC2 instance with Nginx and a Dockerized
> Girder?  At first I was thinking you were using the Docker runtime of EBS
> with Girder, and I couldn't figure out where the secondary Nginx was
> living, but it sounds more like EC2 (Nginx, Dockerized Girder) to me.
>
>
>
>
> On Thu, Apr 27, 2017 at 1:33 PM, Carlos Ag?ero <caguero at osrfoundation.org>
> wrote:
>
>>
>>
>> On Thu, Apr 27, 2017 at 7:04 AM, Michael Grauer <michael.grauer at kitware
>> .com> wrote:
>>
>>> [...]
>>>
>>
>>
>>> I hope you don't mind if I ask you some follow-up questions :)  Just
>>> trying to understand your setup and choices in more detail.
>>>
>>
>> Of course!
>>
>> I agree that your security setup sounds reasonable, HTTPS to load
>>> balancer, HTTP from load balancer to Girder (though I have more questions
>>> on this below), assuming the instances are not visible to the outside world
>>> and only to the LB via the VPN, and Mongo/Instances talking to each other
>>> inside the same VPN.  Out of curiosity (rather than suggesting a policy),
>>> how do you handle ssh, are each of the machines accessible to ssh or do you
>>> have a VPN ssh gateway machine?
>>>
>>
>> Both the Girder instances and the EC2 machine with the Mongo database
>> have public IP addresses, share the same VPC (private network) and all
>> the machines support ssh access from the outside. On the instances under
>> the Elastic Beanstalk (Girder instances), this is the default behavior (you
>> can create a key pair .pem file for sshing). On the EC2 instance with
>> the Mongo DB, you have to attach a security group that configures the
>> access. The inbound rule for port 22 is set to 0.0.0.0 allowing access from
>> machines outside the private network. On the other hand, the rule for port
>> 27017 restricts the access only from machines belonging to the same VPC.
>>
>>
>>
>>> When you say load balancer, does that mean Elastic Load Balancer or
>>> something else?  I'm confused about how you use Nginx, are you using ELB +
>>> Nginx, and if so how does ELB hand off to Nginx?  Where does Nginx live, is
>>> it in a separate Docker container that redirects to the Girder instances?
>>>
>>
>> I meant the load balancer provided by Elastic Beanstalk. If I'm not
>> wrong, the request hits the Nginx running on the load balancer. Then,
>> the request is forwarded to one of the Girder instances running another
>> Nginx. Then, the request is forwarded to the HTTP server running within
>> the Docker container. You can SSH into all the machines (including the load
>> balancer) if you want to poke around and see configurations, logs, etc.
>> I was playing with /etc/nginx/conf.d/ configurations, restarting the
>> service, like in any regular machine.
>>
>> EB offers a way (although a bit convoluted in my opinion) to overload
>> Nginx configuration files when you deploy new versions of your code. I'm
>> doing this for tweaking the Nginx configuration in the Girder instances
>> for redirecting non-https requests to https. The solution involves creating
>> files under the .ebextension directory with some specific syntax. I
>> managed to solve the http-->https redirection in a non-dockerized
>> instance but I still have some issues here using Docker for Girder. In
>> particular, EB includes a health checker that monitors the instances. When
>> I enable the redirection, the load balancer receives a 301 response with
>> the https URL redirection. The expected response was a 200 OK, and that
>> makes the load balancer think that the instances are not behaving
>> correctly. Maybe you have more experience than me dealing with Nginx
>> configurations.
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170428/4c3a4cb3/attachment.html>

From michael.grauer at kitware.com  Fri Apr 28 16:45:04 2017
From: michael.grauer at kitware.com (Michael Grauer)
Date: Fri, 28 Apr 2017 16:45:04 -0400
Subject: [Girder-users] Deploy Girder on AWS with Elastic Beanstalk
In-Reply-To: <CAMGA_6KqnfytKYLWpUKK406L9CAR9XF+6+D+BV6nJo=mELOjSA@mail.gmail.com>
References: <CAMGA_6LF7mBWpWKLuNLNA3xDQ_wSGg9x9t9ZQ3e4tpN+Wm3kdg@mail.gmail.com>
 <CAKx26+HJAuc0tEgJ7Ys+dQOPFfcrtvo8fisfKOT=T3O4iAuYZA@mail.gmail.com>
 <CAMGA_6L141_oKdAeD19cAX31RaZjTyJateMN2nE5j4uAJnyQ_Q@mail.gmail.com>
 <CAKx26+GKd1zUKnxpBT+-KGc_FkrUFLNP=mUQ8CidPDzJS47QsQ@mail.gmail.com>
 <CAMGA_6J0jiP9qCz0q1Br5k1jFKcc56Ud-2o=2LzN7fjJgcJLGQ@mail.gmail.com>
 <CAKx26+GXYCus7Kxy4=vXigi4FNWStu39ShAWAtCmpiTm4JCyKw@mail.gmail.com>
 <CAMGA_6KqnfytKYLWpUKK406L9CAR9XF+6+D+BV6nJo=mELOjSA@mail.gmail.com>
Message-ID: <CAKx26+F3FKmRp=PvY9=yyDN214XKPt9Zvvd3jG4bsCqQRPqDdA@mail.gmail.com>

Great, thanks, that helps clear things up for me.

On Fri, Apr 28, 2017 at 4:17 PM, Carlos Ag?ero <caguero at osrfoundation.org>
wrote:

> As far as I understand I'm using an EC2 instance with Nginx and a
> Dockerized Girder.
>
> Having said that, the load balancer machine also has its own Nginx server,
> that's why there are two Nginx servers involved.
>
> Nginx (load balancer)->Nginx (instance)->Girder Docker
>
> On Thu, Apr 27, 2017 at 1:03 PM, Michael Grauer <
> michael.grauer at kitware.com> wrote:
>
>> Thanks for sharing this Carlos, it is really helpful and will give us
>> quite a leg up when we get back to Girder on EBS.
>>
>> We'll let you know if we run into those same 301 issues with the health
>> checker when we get there.
>>
>>
>> A quick question for you, what are your Girder instances, in terms of EBS
>> containers?  Are you using an EC2 instance with Nginx and a Dockerized
>> Girder?  At first I was thinking you were using the Docker runtime of EBS
>> with Girder, and I couldn't figure out where the secondary Nginx was
>> living, but it sounds more like EC2 (Nginx, Dockerized Girder) to me.
>>
>>
>>
>>
>> On Thu, Apr 27, 2017 at 1:33 PM, Carlos Ag?ero <caguero at osrfoundation.org
>> > wrote:
>>
>>>
>>>
>>> On Thu, Apr 27, 2017 at 7:04 AM, Michael Grauer <michael.grauer at kitware
>>> .com> wrote:
>>>
>>>> [...]
>>>>
>>>
>>>
>>>> I hope you don't mind if I ask you some follow-up questions :)  Just
>>>> trying to understand your setup and choices in more detail.
>>>>
>>>
>>> Of course!
>>>
>>> I agree that your security setup sounds reasonable, HTTPS to load
>>>> balancer, HTTP from load balancer to Girder (though I have more questions
>>>> on this below), assuming the instances are not visible to the outside world
>>>> and only to the LB via the VPN, and Mongo/Instances talking to each other
>>>> inside the same VPN.  Out of curiosity (rather than suggesting a policy),
>>>> how do you handle ssh, are each of the machines accessible to ssh or do you
>>>> have a VPN ssh gateway machine?
>>>>
>>>
>>> Both the Girder instances and the EC2 machine with the Mongo database
>>> have public IP addresses, share the same VPC (private network) and all
>>> the machines support ssh access from the outside. On the instances under
>>> the Elastic Beanstalk (Girder instances), this is the default behavior (you
>>> can create a key pair .pem file for sshing). On the EC2 instance with
>>> the Mongo DB, you have to attach a security group that configures the
>>> access. The inbound rule for port 22 is set to 0.0.0.0 allowing access from
>>> machines outside the private network. On the other hand, the rule for port
>>> 27017 restricts the access only from machines belonging to the same VPC
>>> .
>>>
>>>
>>>
>>>> When you say load balancer, does that mean Elastic Load Balancer or
>>>> something else?  I'm confused about how you use Nginx, are you using ELB +
>>>> Nginx, and if so how does ELB hand off to Nginx?  Where does Nginx live, is
>>>> it in a separate Docker container that redirects to the Girder instances?
>>>>
>>>
>>> I meant the load balancer provided by Elastic Beanstalk. If I'm not
>>> wrong, the request hits the Nginx running on the load balancer. Then,
>>> the request is forwarded to one of the Girder instances running another
>>> Nginx. Then, the request is forwarded to the HTTP server running within
>>> the Docker container. You can SSH into all the machines (including the load
>>> balancer) if you want to poke around and see configurations, logs, etc.
>>> I was playing with /etc/nginx/conf.d/ configurations, restarting the
>>> service, like in any regular machine.
>>>
>>> EB offers a way (although a bit convoluted in my opinion) to overload
>>> Nginx configuration files when you deploy new versions of your code.
>>> I'm doing this for tweaking the Nginx configuration in the Girder
>>> instances for redirecting non-https requests to https. The solution
>>> involves creating files under the .ebextension directory with some
>>> specific syntax. I managed to solve the http-->https redirection in a non-
>>> dockerized instance but I still have some issues here using Docker for
>>> Girder. In particular, EB includes a health checker that monitors the
>>> instances. When I enable the redirection, the load balancer receives a
>>> 301 response with the https URL redirection. The expected response was a
>>> 200 OK, and that makes the load balancer think that the instances are
>>> not behaving correctly. Maybe you have more experience than me dealing with
>>> Nginx configurations.
>>>
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170428/53faf9b2/attachment-0001.html>