From andres at ekumenlabs.com  Thu Jun 29 09:18:07 2017
From: andres at ekumenlabs.com (=?UTF-8?Q?Andr=C3=A9s_Fortier?=)
Date: Thu, 29 Jun 2017 10:18:07 -0300
Subject: [Girder-users] Querying by metadata
Message-ID: <CALiGwQvDSmrXc950wUcG=2outTyqW-Ls1RGRO5OunEU4XYuGNA@mail.gmail.com>

Hi all,
first of all, sorry of this is a trivial question, just getting started
with Girder. We are currently evaluating using Girder as a backend to store
resources (either a folder or a file). One of the requirements we have is
that a resource may have 0, 1 or more "types" (although it will most likely
be 1 for a starter), which can be plain strings. Also, we want to be able
to search resources by type, even if they are in different collections.

Initially we thought on attaching a *type* property to a resource metadata,
which could be an array of strings. However the search bar in the web
front-end doesn't seem to support search by metadata, so I was wondering:
is metadata search supported on a collection? if yes, how about
cross-collection search? If no, should I write a plugin to do that?

Any hints / pointers are much appreciated.

Thanks!
Andy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170629/645f10bb/attachment.html>

From zach.mullen at kitware.com  Thu Jun 29 10:48:11 2017
From: zach.mullen at kitware.com (Zach Mullen)
Date: Thu, 29 Jun 2017 10:48:11 -0400
Subject: [Girder-users] Querying by metadata
In-Reply-To: <CALiGwQvDSmrXc950wUcG=2outTyqW-Ls1RGRO5OunEU4XYuGNA@mail.gmail.com>
References: <CALiGwQvDSmrXc950wUcG=2outTyqW-Ls1RGRO5OunEU4XYuGNA@mail.gmail.com>
Message-ID: <CAPzddEH0pAto=ca_8KupZmeGSd+kcqqDwwZJ76HWThVN1rvehw@mail.gmail.com>

Hi Andy,

Thanks for reaching out! This is actually a common sort of use case, but
Girder out-of-the-box does not support it (yet). The main reason for that
is because these sorts of queries should typically be performed against a
database index so that they can scale up to large numbers of folders/items.
So, the recommended route for this case is to create a small plugin that
makes sure your desired search field is indexed. To achieve that, you'd add
a line like:

    ModelImporter.model('item').ensureIndex(['meta.type', {'sparse': True}])

Then, you'd want to probably add a small API endpoint to search by this
field, and perhaps even some UI augmentation to expose it somewhere. Let me
know if you need help with those other steps.

Thanks,

Zach Mullen
Kitware, Inc.
919-869-8858

On Thu, Jun 29, 2017 at 9:18 AM, Andr?s Fortier <andres at ekumenlabs.com>
wrote:
>
> Hi all,
> first of all, sorry of this is a trivial question, just getting started
with Girder. We are currently evaluating using Girder as a backend to store
resources (either a folder or a file). One of the requirements we have is
that a resource may have 0, 1 or more "types" (although it will most likely
be 1 for a starter), which can be plain strings. Also, we want to be able
to search resources by type, even if they are in different collections.
>
> Initially we thought on attaching a type property to a resource metadata,
which could be an array of strings. However the search bar in the web
front-end doesn't seem to support search by metadata, so I was wondering:
is metadata search supported on a collection? if yes, how about
cross-collection search? If no, should I write a plugin to do that?
>
> Any hints / pointers are much appreciated.
>
> Thanks!
> Andy
>
> _______________________________________________
> Girder-users mailing list
> Girder-users at public.kitware.com
> http://public.kitware.com/mailman/listinfo/girder-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170629/32013b10/attachment.html>

From andres at ekumenlabs.com  Thu Jun 29 11:22:26 2017
From: andres at ekumenlabs.com (=?UTF-8?Q?Andr=C3=A9s_Fortier?=)
Date: Thu, 29 Jun 2017 12:22:26 -0300
Subject: [Girder-users] Querying by metadata
In-Reply-To: <CAPzddEH0pAto=ca_8KupZmeGSd+kcqqDwwZJ76HWThVN1rvehw@mail.gmail.com>
References: <CALiGwQvDSmrXc950wUcG=2outTyqW-Ls1RGRO5OunEU4XYuGNA@mail.gmail.com>
 <CAPzddEH0pAto=ca_8KupZmeGSd+kcqqDwwZJ76HWThVN1rvehw@mail.gmail.com>
Message-ID: <CALiGwQs=hFQ0Jz1qjc+XRD3A=0LxQHa9-kc4WZX=Zujz7JtaUw@mail.gmail.com>

Hi Zach,
thanks for the quick reply. What you say makes as lot of sense, my next
question on this was going to be about performance and indexes for this
kind of search :).

Just to clarify, if we go this road, is there a way to query all
collections at once or we need to run the query for each collection? I'm
asking because we may need to create collections on the fly as the system
runs, so it would be great to be able to query all elements regardless of
the collection they belong to.

Thanks again,
Andy

On Thu, Jun 29, 2017 at 11:48 AM, Zach Mullen <zach.mullen at kitware.com>
wrote:

> Hi Andy,
>
> Thanks for reaching out! This is actually a common sort of use case, but
> Girder out-of-the-box does not support it (yet). The main reason for that
> is because these sorts of queries should typically be performed against a
> database index so that they can scale up to large numbers of folders/items.
> So, the recommended route for this case is to create a small plugin that
> makes sure your desired search field is indexed. To achieve that, you'd add
> a line like:
>
>     ModelImporter.model('item').ensureIndex(['meta.type', {'sparse':
> True}])
>
> Then, you'd want to probably add a small API endpoint to search by this
> field, and perhaps even some UI augmentation to expose it somewhere. Let me
> know if you need help with those other steps.
>
> Thanks,
>
> Zach Mullen
> Kitware, Inc.
> 919-869-8858
>
>
> On Thu, Jun 29, 2017 at 9:18 AM, Andr?s Fortier <andres at ekumenlabs.com>
> wrote:
> >
> > Hi all,
> > first of all, sorry of this is a trivial question, just getting started
> with Girder. We are currently evaluating using Girder as a backend to store
> resources (either a folder or a file). One of the requirements we have is
> that a resource may have 0, 1 or more "types" (although it will most likely
> be 1 for a starter), which can be plain strings. Also, we want to be able
> to search resources by type, even if they are in different collections.
> >
> > Initially we thought on attaching a type property to a resource
> metadata, which could be an array of strings. However the search bar in the
> web front-end doesn't seem to support search by metadata, so I was
> wondering: is metadata search supported on a collection? if yes, how about
> cross-collection search? If no, should I write a plugin to do that?
> >
> > Any hints / pointers are much appreciated.
> >
> > Thanks!
> > Andy
> >
> > _______________________________________________
> > Girder-users mailing list
> > Girder-users at public.kitware.com
> > http://public.kitware.com/mailman/listinfo/girder-users
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170629/86d31790/attachment.html>

From zach.mullen at kitware.com  Thu Jun 29 11:27:12 2017
From: zach.mullen at kitware.com (Zach Mullen)
Date: Thu, 29 Jun 2017 11:27:12 -0400
Subject: [Girder-users] Querying by metadata
In-Reply-To: <CALiGwQs=hFQ0Jz1qjc+XRD3A=0LxQHa9-kc4WZX=Zujz7JtaUw@mail.gmail.com>
References: <CALiGwQvDSmrXc950wUcG=2outTyqW-Ls1RGRO5OunEU4XYuGNA@mail.gmail.com>
 <CAPzddEH0pAto=ca_8KupZmeGSd+kcqqDwwZJ76HWThVN1rvehw@mail.gmail.com>
 <CALiGwQs=hFQ0Jz1qjc+XRD3A=0LxQHa9-kc4WZX=Zujz7JtaUw@mail.gmail.com>
Message-ID: <CAPzddEHw85VnHk9e0fXJdhg11MFY_1bM1v_nMKoFmZqXx21RCw@mail.gmail.com>

One of the main limitations (or perhaps features?) of mongodb is that a
query is applied to only one collection at a time, so your search function
would make one query per collection that you wish to search. So, in the
case you describe, you'd need to loop over some dynamic set of collections
and query each one. Or, create some secondary collection that aggregates
all of the data with this metadata field, and contains references to other
collections in its documents.

Thanks,

-Zach

On Thu, Jun 29, 2017 at 11:22 AM, Andr?s Fortier <andres at ekumenlabs.com>
wrote:

> Hi Zach,
> thanks for the quick reply. What you say makes as lot of sense, my next
> question on this was going to be about performance and indexes for this
> kind of search :).
>
> Just to clarify, if we go this road, is there a way to query all
> collections at once or we need to run the query for each collection? I'm
> asking because we may need to create collections on the fly as the system
> runs, so it would be great to be able to query all elements regardless of
> the collection they belong to.
>
> Thanks again,
> Andy
>
> On Thu, Jun 29, 2017 at 11:48 AM, Zach Mullen <zach.mullen at kitware.com>
> wrote:
>
>> Hi Andy,
>>
>> Thanks for reaching out! This is actually a common sort of use case, but
>> Girder out-of-the-box does not support it (yet). The main reason for that
>> is because these sorts of queries should typically be performed against a
>> database index so that they can scale up to large numbers of folders/items.
>> So, the recommended route for this case is to create a small plugin that
>> makes sure your desired search field is indexed. To achieve that, you'd add
>> a line like:
>>
>>     ModelImporter.model('item').ensureIndex(['meta.type', {'sparse':
>> True}])
>>
>> Then, you'd want to probably add a small API endpoint to search by this
>> field, and perhaps even some UI augmentation to expose it somewhere. Let me
>> know if you need help with those other steps.
>>
>> Thanks,
>>
>> Zach Mullen
>> Kitware, Inc.
>> 919-869-8858 <(919)%20869-8858>
>>
>>
>> On Thu, Jun 29, 2017 at 9:18 AM, Andr?s Fortier <andres at ekumenlabs.com>
>> wrote:
>> >
>> > Hi all,
>> > first of all, sorry of this is a trivial question, just getting started
>> with Girder. We are currently evaluating using Girder as a backend to store
>> resources (either a folder or a file). One of the requirements we have is
>> that a resource may have 0, 1 or more "types" (although it will most likely
>> be 1 for a starter), which can be plain strings. Also, we want to be able
>> to search resources by type, even if they are in different collections.
>> >
>> > Initially we thought on attaching a type property to a resource
>> metadata, which could be an array of strings. However the search bar in the
>> web front-end doesn't seem to support search by metadata, so I was
>> wondering: is metadata search supported on a collection? if yes, how about
>> cross-collection search? If no, should I write a plugin to do that?
>> >
>> > Any hints / pointers are much appreciated.
>> >
>> > Thanks!
>> > Andy
>> >
>> > _______________________________________________
>> > Girder-users mailing list
>> > Girder-users at public.kitware.com
>> > http://public.kitware.com/mailman/listinfo/girder-users
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170629/2549e71e/attachment-0001.html>

From jonathan.beezley at kitware.com  Thu Jun 29 12:40:53 2017
From: jonathan.beezley at kitware.com (Jonathan Beezley)
Date: Thu, 29 Jun 2017 12:40:53 -0400
Subject: [Girder-users] Querying by metadata
In-Reply-To: <CAPzddEHw85VnHk9e0fXJdhg11MFY_1bM1v_nMKoFmZqXx21RCw@mail.gmail.com>
References: <CALiGwQvDSmrXc950wUcG=2outTyqW-Ls1RGRO5OunEU4XYuGNA@mail.gmail.com>
 <CAPzddEH0pAto=ca_8KupZmeGSd+kcqqDwwZJ76HWThVN1rvehw@mail.gmail.com>
 <CALiGwQs=hFQ0Jz1qjc+XRD3A=0LxQHa9-kc4WZX=Zujz7JtaUw@mail.gmail.com>
 <CAPzddEHw85VnHk9e0fXJdhg11MFY_1bM1v_nMKoFmZqXx21RCw@mail.gmail.com>
Message-ID: <CAATBr3ayMYCboDCg=MqQujyYEr-z05hdiruzgKHFO0dvJHVA8w@mail.gmail.com>

For clarification, do you mean collections in the sense of mongodb
collections, or girder collections?  It shouldn't be any problem, for
example, searching all items regardless of the parent collection they
belong in.

On Thu, Jun 29, 2017 at 11:27 AM, Zach Mullen <zach.mullen at kitware.com>
wrote:

> One of the main limitations (or perhaps features?) of mongodb is that a
> query is applied to only one collection at a time, so your search function
> would make one query per collection that you wish to search. So, in the
> case you describe, you'd need to loop over some dynamic set of collections
> and query each one. Or, create some secondary collection that aggregates
> all of the data with this metadata field, and contains references to other
> collections in its documents.
>
> Thanks,
>
> -Zach
>
> On Thu, Jun 29, 2017 at 11:22 AM, Andr?s Fortier <andres at ekumenlabs.com>
> wrote:
>
>> Hi Zach,
>> thanks for the quick reply. What you say makes as lot of sense, my next
>> question on this was going to be about performance and indexes for this
>> kind of search :).
>>
>> Just to clarify, if we go this road, is there a way to query all
>> collections at once or we need to run the query for each collection? I'm
>> asking because we may need to create collections on the fly as the system
>> runs, so it would be great to be able to query all elements regardless of
>> the collection they belong to.
>>
>> Thanks again,
>> Andy
>>
>> On Thu, Jun 29, 2017 at 11:48 AM, Zach Mullen <zach.mullen at kitware.com>
>> wrote:
>>
>>> Hi Andy,
>>>
>>> Thanks for reaching out! This is actually a common sort of use case, but
>>> Girder out-of-the-box does not support it (yet). The main reason for that
>>> is because these sorts of queries should typically be performed against a
>>> database index so that they can scale up to large numbers of folders/items.
>>> So, the recommended route for this case is to create a small plugin that
>>> makes sure your desired search field is indexed. To achieve that, you'd add
>>> a line like:
>>>
>>>     ModelImporter.model('item').ensureIndex(['meta.type', {'sparse':
>>> True}])
>>>
>>> Then, you'd want to probably add a small API endpoint to search by this
>>> field, and perhaps even some UI augmentation to expose it somewhere. Let me
>>> know if you need help with those other steps.
>>>
>>> Thanks,
>>>
>>> Zach Mullen
>>> Kitware, Inc.
>>> 919-869-8858 <(919)%20869-8858>
>>>
>>>
>>> On Thu, Jun 29, 2017 at 9:18 AM, Andr?s Fortier <andres at ekumenlabs.com>
>>> wrote:
>>> >
>>> > Hi all,
>>> > first of all, sorry of this is a trivial question, just getting
>>> started with Girder. We are currently evaluating using Girder as a backend
>>> to store resources (either a folder or a file). One of the requirements we
>>> have is that a resource may have 0, 1 or more "types" (although it will
>>> most likely be 1 for a starter), which can be plain strings. Also, we want
>>> to be able to search resources by type, even if they are in different
>>> collections.
>>> >
>>> > Initially we thought on attaching a type property to a resource
>>> metadata, which could be an array of strings. However the search bar in the
>>> web front-end doesn't seem to support search by metadata, so I was
>>> wondering: is metadata search supported on a collection? if yes, how about
>>> cross-collection search? If no, should I write a plugin to do that?
>>> >
>>> > Any hints / pointers are much appreciated.
>>> >
>>> > Thanks!
>>> > Andy
>>> >
>>> > _______________________________________________
>>> > Girder-users mailing list
>>> > Girder-users at public.kitware.com
>>> > http://public.kitware.com/mailman/listinfo/girder-users
>>> >
>>>
>>
>>
>
> _______________________________________________
> Girder-users mailing list
> Girder-users at public.kitware.com
> http://public.kitware.com/mailman/listinfo/girder-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170629/56c335b8/attachment.html>

From andres at ekumenlabs.com  Thu Jun 29 14:17:45 2017
From: andres at ekumenlabs.com (=?UTF-8?Q?Andr=C3=A9s_Fortier?=)
Date: Thu, 29 Jun 2017 15:17:45 -0300
Subject: [Girder-users] Querying by metadata
In-Reply-To: <CAATBr3ayMYCboDCg=MqQujyYEr-z05hdiruzgKHFO0dvJHVA8w@mail.gmail.com>
References: <CALiGwQvDSmrXc950wUcG=2outTyqW-Ls1RGRO5OunEU4XYuGNA@mail.gmail.com>
 <CAPzddEH0pAto=ca_8KupZmeGSd+kcqqDwwZJ76HWThVN1rvehw@mail.gmail.com>
 <CALiGwQs=hFQ0Jz1qjc+XRD3A=0LxQHa9-kc4WZX=Zujz7JtaUw@mail.gmail.com>
 <CAPzddEHw85VnHk9e0fXJdhg11MFY_1bM1v_nMKoFmZqXx21RCw@mail.gmail.com>
 <CAATBr3ayMYCboDCg=MqQujyYEr-z05hdiruzgKHFO0dvJHVA8w@mail.gmail.com>
Message-ID: <CALiGwQuY3+WHqzieEUZsHGaHL7fDJbiQYOBvR7uOigGXuZkGoA@mail.gmail.com>

Thanks Zach and Jonathan for your replies! To answer Jonathan's question, I
was referring to Girder collections and not mongo (sorry for the confusion
here, it is unfortunate that the term "collection" is overloaded; I will
disambiguate from now on).

So, maybe stepping back a little sharing the high-level requirements may be
a better approach :). We are trying to build an application to manage
assets (xml, png, maybe source code, dlls, etc. An asset can also be
composed of many files, so basically an asset can be either a girder file
or folder). The application must provide both an administrative back end
(we think girder web client would be a fit) and REST API (again, girder
REST API seems a fit). We will later on roll out a front end (most likely a
Javascript SPA), backed by girder's the REST API.

Now, the typical workflow we envision is:

- Someone creates an account and adds assets to the app. Assets have an
associated type, which is independent of the file extension or mime-type,
so this is configured by the user. The user can then decide to keep the
asset private, public or shared with some users / group.
- It should be easy / performant to query all the assets of a given type
(plus some variations when it applies, like "all assets of a given type
that belong to user X", "all private assets of a given type that belong to
user X", etc).
- It should be easy for a user to download all its contents (e.g. for
personal backup).
- It should support asset versioning (I was going to discuss this in
another thread, just mentioning here to give a broad view).

The approach that we currently have in mind is to have a girder collection
per user (so that all the user assets are stored there) and tag the assets
with metadata to specify its type (hence the need to query by metadata
across different girder collections). Another approach could be to have a
folder per user in a single "assets" collection, pretty much like a linux
/home directory (which at least would remove the cross-girder-collection
query). But again, I'm *very* new to Girder, so it is most likely that I
don't know what the best use for girder collections are, so any hints are
much appreciated.

Thanks!
Andy




On Thu, Jun 29, 2017 at 1:40 PM, Jonathan Beezley <
jonathan.beezley at kitware.com> wrote:

> For clarification, do you mean collections in the sense of mongodb
> collections, or girder collections?  It shouldn't be any problem, for
> example, searching all items regardless of the parent collection they
> belong in.
>
> On Thu, Jun 29, 2017 at 11:27 AM, Zach Mullen <zach.mullen at kitware.com>
> wrote:
>
>> One of the main limitations (or perhaps features?) of mongodb is that a
>> query is applied to only one collection at a time, so your search function
>> would make one query per collection that you wish to search. So, in the
>> case you describe, you'd need to loop over some dynamic set of collections
>> and query each one. Or, create some secondary collection that aggregates
>> all of the data with this metadata field, and contains references to other
>> collections in its documents.
>>
>> Thanks,
>>
>> -Zach
>>
>> On Thu, Jun 29, 2017 at 11:22 AM, Andr?s Fortier <andres at ekumenlabs.com>
>> wrote:
>>
>>> Hi Zach,
>>> thanks for the quick reply. What you say makes as lot of sense, my next
>>> question on this was going to be about performance and indexes for this
>>> kind of search :).
>>>
>>> Just to clarify, if we go this road, is there a way to query all
>>> collections at once or we need to run the query for each collection? I'm
>>> asking because we may need to create collections on the fly as the system
>>> runs, so it would be great to be able to query all elements regardless of
>>> the collection they belong to.
>>>
>>> Thanks again,
>>> Andy
>>>
>>> On Thu, Jun 29, 2017 at 11:48 AM, Zach Mullen <zach.mullen at kitware.com>
>>> wrote:
>>>
>>>> Hi Andy,
>>>>
>>>> Thanks for reaching out! This is actually a common sort of use case,
>>>> but Girder out-of-the-box does not support it (yet). The main reason for
>>>> that is because these sorts of queries should typically be performed
>>>> against a database index so that they can scale up to large numbers of
>>>> folders/items. So, the recommended route for this case is to create a small
>>>> plugin that makes sure your desired search field is indexed. To achieve
>>>> that, you'd add a line like:
>>>>
>>>>     ModelImporter.model('item').ensureIndex(['meta.type', {'sparse':
>>>> True}])
>>>>
>>>> Then, you'd want to probably add a small API endpoint to search by this
>>>> field, and perhaps even some UI augmentation to expose it somewhere. Let me
>>>> know if you need help with those other steps.
>>>>
>>>> Thanks,
>>>>
>>>> Zach Mullen
>>>> Kitware, Inc.
>>>> 919-869-8858 <(919)%20869-8858>
>>>>
>>>>
>>>> On Thu, Jun 29, 2017 at 9:18 AM, Andr?s Fortier <andres at ekumenlabs.com>
>>>> wrote:
>>>> >
>>>> > Hi all,
>>>> > first of all, sorry of this is a trivial question, just getting
>>>> started with Girder. We are currently evaluating using Girder as a backend
>>>> to store resources (either a folder or a file). One of the requirements we
>>>> have is that a resource may have 0, 1 or more "types" (although it will
>>>> most likely be 1 for a starter), which can be plain strings. Also, we want
>>>> to be able to search resources by type, even if they are in different
>>>> collections.
>>>> >
>>>> > Initially we thought on attaching a type property to a resource
>>>> metadata, which could be an array of strings. However the search bar in the
>>>> web front-end doesn't seem to support search by metadata, so I was
>>>> wondering: is metadata search supported on a collection? if yes, how about
>>>> cross-collection search? If no, should I write a plugin to do that?
>>>> >
>>>> > Any hints / pointers are much appreciated.
>>>> >
>>>> > Thanks!
>>>> > Andy
>>>> >
>>>> > _______________________________________________
>>>> > Girder-users mailing list
>>>> > Girder-users at public.kitware.com
>>>> > http://public.kitware.com/mailman/listinfo/girder-users
>>>> >
>>>>
>>>
>>>
>>
>> _______________________________________________
>> Girder-users mailing list
>> Girder-users at public.kitware.com
>> http://public.kitware.com/mailman/listinfo/girder-users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170629/efd363bb/attachment-0001.html>

From zach.mullen at kitware.com  Thu Jun 29 14:26:12 2017
From: zach.mullen at kitware.com (Zach Mullen)
Date: Thu, 29 Jun 2017 14:26:12 -0400
Subject: [Girder-users] Querying by metadata
In-Reply-To: <CALiGwQuY3+WHqzieEUZsHGaHL7fDJbiQYOBvR7uOigGXuZkGoA@mail.gmail.com>
References: <CALiGwQvDSmrXc950wUcG=2outTyqW-Ls1RGRO5OunEU4XYuGNA@mail.gmail.com>
 <CAPzddEH0pAto=ca_8KupZmeGSd+kcqqDwwZJ76HWThVN1rvehw@mail.gmail.com>
 <CALiGwQs=hFQ0Jz1qjc+XRD3A=0LxQHa9-kc4WZX=Zujz7JtaUw@mail.gmail.com>
 <CAPzddEHw85VnHk9e0fXJdhg11MFY_1bM1v_nMKoFmZqXx21RCw@mail.gmail.com>
 <CAATBr3ayMYCboDCg=MqQujyYEr-z05hdiruzgKHFO0dvJHVA8w@mail.gmail.com>
 <CALiGwQuY3+WHqzieEUZsHGaHL7fDJbiQYOBvR7uOigGXuZkGoA@mail.gmail.com>
Message-ID: <CAPzddEFyuKzieek2eoeH2hr1LjUnXOWWzLO_uMk5DreVp5u87w@mail.gmail.com>

Ah, that is quite simple then, you actually have to only query a fixed set
of mongo collections, and data can be pulled from every girder collection.
If your assets are represented by items, you just need to do one query
against the item (mongo) collection. If they can also be represented by
folders, you'd look up folders as well. Exposing this via a REST endpoint
that respects access control policies is also trivial, so this definitely
sounds like a good fit for Girder.

Thanks,

-Zach


On Thu, Jun 29, 2017 at 2:17 PM, Andr?s Fortier <andres at ekumenlabs.com>
wrote:

> Thanks Zach and Jonathan for your replies! To answer Jonathan's question,
> I was referring to Girder collections and not mongo (sorry for the
> confusion here, it is unfortunate that the term "collection" is overloaded;
> I will disambiguate from now on).
>
> So, maybe stepping back a little sharing the high-level requirements may
> be a better approach :). We are trying to build an application to manage
> assets (xml, png, maybe source code, dlls, etc. An asset can also be
> composed of many files, so basically an asset can be either a girder file
> or folder). The application must provide both an administrative back end
> (we think girder web client would be a fit) and REST API (again, girder
> REST API seems a fit). We will later on roll out a front end (most likely a
> Javascript SPA), backed by girder's the REST API.
>
> Now, the typical workflow we envision is:
>
> - Someone creates an account and adds assets to the app. Assets have an
> associated type, which is independent of the file extension or mime-type,
> so this is configured by the user. The user can then decide to keep the
> asset private, public or shared with some users / group.
> - It should be easy / performant to query all the assets of a given type
> (plus some variations when it applies, like "all assets of a given type
> that belong to user X", "all private assets of a given type that belong to
> user X", etc).
> - It should be easy for a user to download all its contents (e.g. for
> personal backup).
> - It should support asset versioning (I was going to discuss this in
> another thread, just mentioning here to give a broad view).
>
> The approach that we currently have in mind is to have a girder collection
> per user (so that all the user assets are stored there) and tag the assets
> with metadata to specify its type (hence the need to query by metadata
> across different girder collections). Another approach could be to have a
> folder per user in a single "assets" collection, pretty much like a linux
> /home directory (which at least would remove the cross-girder-collection
> query). But again, I'm *very* new to Girder, so it is most likely that I
> don't know what the best use for girder collections are, so any hints are
> much appreciated.
>
> Thanks!
> Andy
>
>
>
>
> On Thu, Jun 29, 2017 at 1:40 PM, Jonathan Beezley <
> jonathan.beezley at kitware.com> wrote:
>
>> For clarification, do you mean collections in the sense of mongodb
>> collections, or girder collections?  It shouldn't be any problem, for
>> example, searching all items regardless of the parent collection they
>> belong in.
>>
>> On Thu, Jun 29, 2017 at 11:27 AM, Zach Mullen <zach.mullen at kitware.com>
>> wrote:
>>
>>> One of the main limitations (or perhaps features?) of mongodb is that a
>>> query is applied to only one collection at a time, so your search function
>>> would make one query per collection that you wish to search. So, in the
>>> case you describe, you'd need to loop over some dynamic set of collections
>>> and query each one. Or, create some secondary collection that aggregates
>>> all of the data with this metadata field, and contains references to other
>>> collections in its documents.
>>>
>>> Thanks,
>>>
>>> -Zach
>>>
>>> On Thu, Jun 29, 2017 at 11:22 AM, Andr?s Fortier <andres at ekumenlabs.com>
>>> wrote:
>>>
>>>> Hi Zach,
>>>> thanks for the quick reply. What you say makes as lot of sense, my next
>>>> question on this was going to be about performance and indexes for this
>>>> kind of search :).
>>>>
>>>> Just to clarify, if we go this road, is there a way to query all
>>>> collections at once or we need to run the query for each collection? I'm
>>>> asking because we may need to create collections on the fly as the system
>>>> runs, so it would be great to be able to query all elements regardless of
>>>> the collection they belong to.
>>>>
>>>> Thanks again,
>>>> Andy
>>>>
>>>> On Thu, Jun 29, 2017 at 11:48 AM, Zach Mullen <zach.mullen at kitware.com>
>>>> wrote:
>>>>
>>>>> Hi Andy,
>>>>>
>>>>> Thanks for reaching out! This is actually a common sort of use case,
>>>>> but Girder out-of-the-box does not support it (yet). The main reason for
>>>>> that is because these sorts of queries should typically be performed
>>>>> against a database index so that they can scale up to large numbers of
>>>>> folders/items. So, the recommended route for this case is to create a small
>>>>> plugin that makes sure your desired search field is indexed. To achieve
>>>>> that, you'd add a line like:
>>>>>
>>>>>     ModelImporter.model('item').ensureIndex(['meta.type', {'sparse':
>>>>> True}])
>>>>>
>>>>> Then, you'd want to probably add a small API endpoint to search by
>>>>> this field, and perhaps even some UI augmentation to expose it somewhere.
>>>>> Let me know if you need help with those other steps.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Zach Mullen
>>>>> Kitware, Inc.
>>>>> 919-869-8858 <(919)%20869-8858>
>>>>>
>>>>>
>>>>> On Thu, Jun 29, 2017 at 9:18 AM, Andr?s Fortier <andres at ekumenlabs.com>
>>>>> wrote:
>>>>> >
>>>>> > Hi all,
>>>>> > first of all, sorry of this is a trivial question, just getting
>>>>> started with Girder. We are currently evaluating using Girder as a backend
>>>>> to store resources (either a folder or a file). One of the requirements we
>>>>> have is that a resource may have 0, 1 or more "types" (although it will
>>>>> most likely be 1 for a starter), which can be plain strings. Also, we want
>>>>> to be able to search resources by type, even if they are in different
>>>>> collections.
>>>>> >
>>>>> > Initially we thought on attaching a type property to a resource
>>>>> metadata, which could be an array of strings. However the search bar in the
>>>>> web front-end doesn't seem to support search by metadata, so I was
>>>>> wondering: is metadata search supported on a collection? if yes, how about
>>>>> cross-collection search? If no, should I write a plugin to do that?
>>>>> >
>>>>> > Any hints / pointers are much appreciated.
>>>>> >
>>>>> > Thanks!
>>>>> > Andy
>>>>> >
>>>>> > _______________________________________________
>>>>> > Girder-users mailing list
>>>>> > Girder-users at public.kitware.com
>>>>> > http://public.kitware.com/mailman/listinfo/girder-users
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Girder-users mailing list
>>> Girder-users at public.kitware.com
>>> http://public.kitware.com/mailman/listinfo/girder-users
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170629/8a57920f/attachment.html>

From jonathan.beezley at kitware.com  Thu Jun 29 14:30:30 2017
From: jonathan.beezley at kitware.com (Jonathan Beezley)
Date: Thu, 29 Jun 2017 14:30:30 -0400
Subject: [Girder-users] Querying by metadata
In-Reply-To: <CAPzddEFyuKzieek2eoeH2hr1LjUnXOWWzLO_uMk5DreVp5u87w@mail.gmail.com>
References: <CALiGwQvDSmrXc950wUcG=2outTyqW-Ls1RGRO5OunEU4XYuGNA@mail.gmail.com>
 <CAPzddEH0pAto=ca_8KupZmeGSd+kcqqDwwZJ76HWThVN1rvehw@mail.gmail.com>
 <CALiGwQs=hFQ0Jz1qjc+XRD3A=0LxQHa9-kc4WZX=Zujz7JtaUw@mail.gmail.com>
 <CAPzddEHw85VnHk9e0fXJdhg11MFY_1bM1v_nMKoFmZqXx21RCw@mail.gmail.com>
 <CAATBr3ayMYCboDCg=MqQujyYEr-z05hdiruzgKHFO0dvJHVA8w@mail.gmail.com>
 <CALiGwQuY3+WHqzieEUZsHGaHL7fDJbiQYOBvR7uOigGXuZkGoA@mail.gmail.com>
 <CAPzddEFyuKzieek2eoeH2hr1LjUnXOWWzLO_uMk5DreVp5u87w@mail.gmail.com>
Message-ID: <CAATBr3YaR3ns3yP6ZkydHJBJYCHSCGdHQ+xQ-kbariasR985bg@mail.gmail.com>

I would also mention that user's can be treated like collections in that
they can contain folders.  (See my user on data.kitware.com for example[1])
 You may be able to get away with just using this rather than creating
collections for each user.

[1] https://data.kitware.com/#user/55087c098d777f5ff55d5e3a

On Thu, Jun 29, 2017 at 2:26 PM, Zach Mullen <zach.mullen at kitware.com>
wrote:

> Ah, that is quite simple then, you actually have to only query a fixed set
> of mongo collections, and data can be pulled from every girder collection.
> If your assets are represented by items, you just need to do one query
> against the item (mongo) collection. If they can also be represented by
> folders, you'd look up folders as well. Exposing this via a REST endpoint
> that respects access control policies is also trivial, so this definitely
> sounds like a good fit for Girder.
>
> Thanks,
>
> -Zach
>
>
> On Thu, Jun 29, 2017 at 2:17 PM, Andr?s Fortier <andres at ekumenlabs.com>
> wrote:
>
>> Thanks Zach and Jonathan for your replies! To answer Jonathan's question,
>> I was referring to Girder collections and not mongo (sorry for the
>> confusion here, it is unfortunate that the term "collection" is overloaded;
>> I will disambiguate from now on).
>>
>> So, maybe stepping back a little sharing the high-level requirements may
>> be a better approach :). We are trying to build an application to manage
>> assets (xml, png, maybe source code, dlls, etc. An asset can also be
>> composed of many files, so basically an asset can be either a girder file
>> or folder). The application must provide both an administrative back end
>> (we think girder web client would be a fit) and REST API (again, girder
>> REST API seems a fit). We will later on roll out a front end (most likely a
>> Javascript SPA), backed by girder's the REST API.
>>
>> Now, the typical workflow we envision is:
>>
>> - Someone creates an account and adds assets to the app. Assets have an
>> associated type, which is independent of the file extension or mime-type,
>> so this is configured by the user. The user can then decide to keep the
>> asset private, public or shared with some users / group.
>> - It should be easy / performant to query all the assets of a given type
>> (plus some variations when it applies, like "all assets of a given type
>> that belong to user X", "all private assets of a given type that belong to
>> user X", etc).
>> - It should be easy for a user to download all its contents (e.g. for
>> personal backup).
>> - It should support asset versioning (I was going to discuss this in
>> another thread, just mentioning here to give a broad view).
>>
>> The approach that we currently have in mind is to have a girder
>> collection per user (so that all the user assets are stored there) and tag
>> the assets with metadata to specify its type (hence the need to query by
>> metadata across different girder collections). Another approach could be to
>> have a folder per user in a single "assets" collection, pretty much like a
>> linux /home directory (which at least would remove the
>> cross-girder-collection query). But again, I'm *very* new to Girder, so it
>> is most likely that I don't know what the best use for girder collections
>> are, so any hints are much appreciated.
>>
>> Thanks!
>> Andy
>>
>>
>>
>>
>> On Thu, Jun 29, 2017 at 1:40 PM, Jonathan Beezley <
>> jonathan.beezley at kitware.com> wrote:
>>
>>> For clarification, do you mean collections in the sense of mongodb
>>> collections, or girder collections?  It shouldn't be any problem, for
>>> example, searching all items regardless of the parent collection they
>>> belong in.
>>>
>>> On Thu, Jun 29, 2017 at 11:27 AM, Zach Mullen <zach.mullen at kitware.com>
>>> wrote:
>>>
>>>> One of the main limitations (or perhaps features?) of mongodb is that a
>>>> query is applied to only one collection at a time, so your search function
>>>> would make one query per collection that you wish to search. So, in the
>>>> case you describe, you'd need to loop over some dynamic set of collections
>>>> and query each one. Or, create some secondary collection that aggregates
>>>> all of the data with this metadata field, and contains references to other
>>>> collections in its documents.
>>>>
>>>> Thanks,
>>>>
>>>> -Zach
>>>>
>>>> On Thu, Jun 29, 2017 at 11:22 AM, Andr?s Fortier <andres at ekumenlabs.com
>>>> > wrote:
>>>>
>>>>> Hi Zach,
>>>>> thanks for the quick reply. What you say makes as lot of sense, my
>>>>> next question on this was going to be about performance and indexes for
>>>>> this kind of search :).
>>>>>
>>>>> Just to clarify, if we go this road, is there a way to query all
>>>>> collections at once or we need to run the query for each collection? I'm
>>>>> asking because we may need to create collections on the fly as the system
>>>>> runs, so it would be great to be able to query all elements regardless of
>>>>> the collection they belong to.
>>>>>
>>>>> Thanks again,
>>>>> Andy
>>>>>
>>>>> On Thu, Jun 29, 2017 at 11:48 AM, Zach Mullen <zach.mullen at kitware.com
>>>>> > wrote:
>>>>>
>>>>>> Hi Andy,
>>>>>>
>>>>>> Thanks for reaching out! This is actually a common sort of use case,
>>>>>> but Girder out-of-the-box does not support it (yet). The main reason for
>>>>>> that is because these sorts of queries should typically be performed
>>>>>> against a database index so that they can scale up to large numbers of
>>>>>> folders/items. So, the recommended route for this case is to create a small
>>>>>> plugin that makes sure your desired search field is indexed. To achieve
>>>>>> that, you'd add a line like:
>>>>>>
>>>>>>     ModelImporter.model('item').ensureIndex(['meta.type', {'sparse':
>>>>>> True}])
>>>>>>
>>>>>> Then, you'd want to probably add a small API endpoint to search by
>>>>>> this field, and perhaps even some UI augmentation to expose it somewhere.
>>>>>> Let me know if you need help with those other steps.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Zach Mullen
>>>>>> Kitware, Inc.
>>>>>> 919-869-8858 <(919)%20869-8858>
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 29, 2017 at 9:18 AM, Andr?s Fortier <
>>>>>> andres at ekumenlabs.com> wrote:
>>>>>> >
>>>>>> > Hi all,
>>>>>> > first of all, sorry of this is a trivial question, just getting
>>>>>> started with Girder. We are currently evaluating using Girder as a backend
>>>>>> to store resources (either a folder or a file). One of the requirements we
>>>>>> have is that a resource may have 0, 1 or more "types" (although it will
>>>>>> most likely be 1 for a starter), which can be plain strings. Also, we want
>>>>>> to be able to search resources by type, even if they are in different
>>>>>> collections.
>>>>>> >
>>>>>> > Initially we thought on attaching a type property to a resource
>>>>>> metadata, which could be an array of strings. However the search bar in the
>>>>>> web front-end doesn't seem to support search by metadata, so I was
>>>>>> wondering: is metadata search supported on a collection? if yes, how about
>>>>>> cross-collection search? If no, should I write a plugin to do that?
>>>>>> >
>>>>>> > Any hints / pointers are much appreciated.
>>>>>> >
>>>>>> > Thanks!
>>>>>> > Andy
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > Girder-users mailing list
>>>>>> > Girder-users at public.kitware.com
>>>>>> > http://public.kitware.com/mailman/listinfo/girder-users
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Girder-users mailing list
>>>> Girder-users at public.kitware.com
>>>> http://public.kitware.com/mailman/listinfo/girder-users
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170629/7b0a7218/attachment-0001.html>

From andres at ekumenlabs.com  Thu Jun 29 14:48:23 2017
From: andres at ekumenlabs.com (=?UTF-8?Q?Andr=C3=A9s_Fortier?=)
Date: Thu, 29 Jun 2017 15:48:23 -0300
Subject: [Girder-users] Versioning files / folders
Message-ID: <CALiGwQucZx6bV2_g02qb9b4yJB+fRrWsKV=fNMVFSEwB6P7ndQ@mail.gmail.com>

Hi all,
just to close the loop on the previous thread requirements, we would also
need to version the assets a user uploads. We don't need any fancy support
(i.e. semver or similar), just an increasing integer would do it. What we
would need is:

- Updating an asset version each time it is re-uploaded (I assume the new
upload overwrites the existing data).
- Ability to get all the versions of a given asset.
- Ability to recover an asset by its version.
- (Ideally) ability to diff two versions of an asset (but there are tools
that already do this that we could integrate, so no big deal).

Again, assets can be both files or folders (where the latter may be a
tricky one to handle). Is there support for such a use case in Girder?

Thanks!
Andy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170629/b0df3cea/attachment.html>

From zach.mullen at kitware.com  Thu Jun 29 14:54:48 2017
From: zach.mullen at kitware.com (Zach Mullen)
Date: Thu, 29 Jun 2017 14:54:48 -0400
Subject: [Girder-users] Versioning files / folders
In-Reply-To: <CALiGwQucZx6bV2_g02qb9b4yJB+fRrWsKV=fNMVFSEwB6P7ndQ@mail.gmail.com>
References: <CALiGwQucZx6bV2_g02qb9b4yJB+fRrWsKV=fNMVFSEwB6P7ndQ@mail.gmail.com>
Message-ID: <CAPzddEEFBhtvjOtbfZaPrN-sPvTziyfxy=pk_fgAwfRNUs=Kog@mail.gmail.com>

Hi Andy,

I actually wrote a prototype plugin[1] that does pretty much exactly what
you described. Basically, when you upload the file, you just need to pass
an extra parameter that identifies its canonical name/path, and then it
auto-increments a version. The plugin contains a rudimentary view for
listing the revisions of an asset in reverse-chronological order.

[1] https://github.com/girder/data_revisions

Thanks,

-Zach



On Thu, Jun 29, 2017 at 2:48 PM, Andr?s Fortier <andres at ekumenlabs.com>
wrote:

> Hi all,
> just to close the loop on the previous thread requirements, we would also
> need to version the assets a user uploads. We don't need any fancy support
> (i.e. semver or similar), just an increasing integer would do it. What we
> would need is:
>
> - Updating an asset version each time it is re-uploaded (I assume the new
> upload overwrites the existing data).
> - Ability to get all the versions of a given asset.
> - Ability to recover an asset by its version.
> - (Ideally) ability to diff two versions of an asset (but there are tools
> that already do this that we could integrate, so no big deal).
>
> Again, assets can be both files or folders (where the latter may be a
> tricky one to handle). Is there support for such a use case in Girder?
>
> Thanks!
> Andy
>
> _______________________________________________
> Girder-users mailing list
> Girder-users at public.kitware.com
> http://public.kitware.com/mailman/listinfo/girder-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170629/5b61d223/attachment.html>

From zach.mullen at kitware.com  Thu Jun 29 14:55:50 2017
From: zach.mullen at kitware.com (Zach Mullen)
Date: Thu, 29 Jun 2017 14:55:50 -0400
Subject: [Girder-users] Versioning files / folders
In-Reply-To: <CAPzddEEFBhtvjOtbfZaPrN-sPvTziyfxy=pk_fgAwfRNUs=Kog@mail.gmail.com>
References: <CALiGwQucZx6bV2_g02qb9b4yJB+fRrWsKV=fNMVFSEwB6P7ndQ@mail.gmail.com>
 <CAPzddEEFBhtvjOtbfZaPrN-sPvTziyfxy=pk_fgAwfRNUs=Kog@mail.gmail.com>
Message-ID: <CAPzddEGby9rV-=g+AYwEZCxruCLxw16d8EqLcH=te_n5_eGxSg@mail.gmail.com>

BTW, the plugin I linked only supports revisions of a file, not a folder
(doing that is less trivial, and depends on your specific requirements).



On Thu, Jun 29, 2017 at 2:54 PM, Zach Mullen <zach.mullen at kitware.com>
wrote:

> Hi Andy,
>
> I actually wrote a prototype plugin[1] that does pretty much exactly what
> you described. Basically, when you upload the file, you just need to pass
> an extra parameter that identifies its canonical name/path, and then it
> auto-increments a version. The plugin contains a rudimentary view for
> listing the revisions of an asset in reverse-chronological order.
>
> [1] https://github.com/girder/data_revisions
>
> Thanks,
>
> -Zach
>
>
>
> On Thu, Jun 29, 2017 at 2:48 PM, Andr?s Fortier <andres at ekumenlabs.com>
> wrote:
>
>> Hi all,
>> just to close the loop on the previous thread requirements, we would also
>> need to version the assets a user uploads. We don't need any fancy support
>> (i.e. semver or similar), just an increasing integer would do it. What we
>> would need is:
>>
>> - Updating an asset version each time it is re-uploaded (I assume the new
>> upload overwrites the existing data).
>> - Ability to get all the versions of a given asset.
>> - Ability to recover an asset by its version.
>> - (Ideally) ability to diff two versions of an asset (but there are tools
>> that already do this that we could integrate, so no big deal).
>>
>> Again, assets can be both files or folders (where the latter may be a
>> tricky one to handle). Is there support for such a use case in Girder?
>>
>> Thanks!
>> Andy
>>
>> _______________________________________________
>> Girder-users mailing list
>> Girder-users at public.kitware.com
>> http://public.kitware.com/mailman/listinfo/girder-users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170629/9bc7083f/attachment.html>

From andres at ekumenlabs.com  Thu Jun 29 17:24:47 2017
From: andres at ekumenlabs.com (=?UTF-8?Q?Andr=C3=A9s_Fortier?=)
Date: Thu, 29 Jun 2017 18:24:47 -0300
Subject: [Girder-users] Querying by metadata
In-Reply-To: <CAATBr3YaR3ns3yP6ZkydHJBJYCHSCGdHQ+xQ-kbariasR985bg@mail.gmail.com>
References: <CALiGwQvDSmrXc950wUcG=2outTyqW-Ls1RGRO5OunEU4XYuGNA@mail.gmail.com>
 <CAPzddEH0pAto=ca_8KupZmeGSd+kcqqDwwZJ76HWThVN1rvehw@mail.gmail.com>
 <CALiGwQs=hFQ0Jz1qjc+XRD3A=0LxQHa9-kc4WZX=Zujz7JtaUw@mail.gmail.com>
 <CAPzddEHw85VnHk9e0fXJdhg11MFY_1bM1v_nMKoFmZqXx21RCw@mail.gmail.com>
 <CAATBr3ayMYCboDCg=MqQujyYEr-z05hdiruzgKHFO0dvJHVA8w@mail.gmail.com>
 <CALiGwQuY3+WHqzieEUZsHGaHL7fDJbiQYOBvR7uOigGXuZkGoA@mail.gmail.com>
 <CAPzddEFyuKzieek2eoeH2hr1LjUnXOWWzLO_uMk5DreVp5u87w@mail.gmail.com>
 <CAATBr3YaR3ns3yP6ZkydHJBJYCHSCGdHQ+xQ-kbariasR985bg@mail.gmail.com>
Message-ID: <CALiGwQsV9WNG4F1BozYGWFi_Av1kovFeKORRur9kui9hFcQ5ng@mail.gmail.com>

Great, thanks again Zach and Jonathan for your input. So, to summarize:

- While not supported out of the box, it shouldn't be hard to implement a
plugin to do metadata-based queries, either if using one or multiple Grider
collections to store the assets.
- As an alternative, we may consider treating the user like a collection
and implement metadata-based search there as well.

Thanks!
Andy

On Thu, Jun 29, 2017 at 3:30 PM, Jonathan Beezley <
jonathan.beezley at kitware.com> wrote:

> I would also mention that user's can be treated like collections in that
> they can contain folders.  (See my user on data.kitware.com for
> example[1])  You may be able to get away with just using this rather than
> creating collections for each user.
>
> [1] https://data.kitware.com/#user/55087c098d777f5ff55d5e3a
>
> On Thu, Jun 29, 2017 at 2:26 PM, Zach Mullen <zach.mullen at kitware.com>
> wrote:
>
>> Ah, that is quite simple then, you actually have to only query a fixed
>> set of mongo collections, and data can be pulled from every girder
>> collection. If your assets are represented by items, you just need to do
>> one query against the item (mongo) collection. If they can also be
>> represented by folders, you'd look up folders as well. Exposing this via a
>> REST endpoint that respects access control policies is also trivial, so
>> this definitely sounds like a good fit for Girder.
>>
>> Thanks,
>>
>> -Zach
>>
>>
>> On Thu, Jun 29, 2017 at 2:17 PM, Andr?s Fortier <andres at ekumenlabs.com>
>> wrote:
>>
>>> Thanks Zach and Jonathan for your replies! To answer Jonathan's
>>> question, I was referring to Girder collections and not mongo (sorry for
>>> the confusion here, it is unfortunate that the term "collection" is
>>> overloaded; I will disambiguate from now on).
>>>
>>> So, maybe stepping back a little sharing the high-level requirements may
>>> be a better approach :). We are trying to build an application to manage
>>> assets (xml, png, maybe source code, dlls, etc. An asset can also be
>>> composed of many files, so basically an asset can be either a girder file
>>> or folder). The application must provide both an administrative back end
>>> (we think girder web client would be a fit) and REST API (again, girder
>>> REST API seems a fit). We will later on roll out a front end (most likely a
>>> Javascript SPA), backed by girder's the REST API.
>>>
>>> Now, the typical workflow we envision is:
>>>
>>> - Someone creates an account and adds assets to the app. Assets have an
>>> associated type, which is independent of the file extension or mime-type,
>>> so this is configured by the user. The user can then decide to keep the
>>> asset private, public or shared with some users / group.
>>> - It should be easy / performant to query all the assets of a given type
>>> (plus some variations when it applies, like "all assets of a given type
>>> that belong to user X", "all private assets of a given type that belong to
>>> user X", etc).
>>> - It should be easy for a user to download all its contents (e.g. for
>>> personal backup).
>>> - It should support asset versioning (I was going to discuss this in
>>> another thread, just mentioning here to give a broad view).
>>>
>>> The approach that we currently have in mind is to have a girder
>>> collection per user (so that all the user assets are stored there) and tag
>>> the assets with metadata to specify its type (hence the need to query by
>>> metadata across different girder collections). Another approach could be to
>>> have a folder per user in a single "assets" collection, pretty much like a
>>> linux /home directory (which at least would remove the
>>> cross-girder-collection query). But again, I'm *very* new to Girder, so it
>>> is most likely that I don't know what the best use for girder collections
>>> are, so any hints are much appreciated.
>>>
>>> Thanks!
>>> Andy
>>>
>>>
>>>
>>>
>>> On Thu, Jun 29, 2017 at 1:40 PM, Jonathan Beezley <
>>> jonathan.beezley at kitware.com> wrote:
>>>
>>>> For clarification, do you mean collections in the sense of mongodb
>>>> collections, or girder collections?  It shouldn't be any problem, for
>>>> example, searching all items regardless of the parent collection they
>>>> belong in.
>>>>
>>>> On Thu, Jun 29, 2017 at 11:27 AM, Zach Mullen <zach.mullen at kitware.com>
>>>> wrote:
>>>>
>>>>> One of the main limitations (or perhaps features?) of mongodb is that
>>>>> a query is applied to only one collection at a time, so your search
>>>>> function would make one query per collection that you wish to search. So,
>>>>> in the case you describe, you'd need to loop over some dynamic set of
>>>>> collections and query each one. Or, create some secondary collection that
>>>>> aggregates all of the data with this metadata field, and contains
>>>>> references to other collections in its documents.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -Zach
>>>>>
>>>>> On Thu, Jun 29, 2017 at 11:22 AM, Andr?s Fortier <
>>>>> andres at ekumenlabs.com> wrote:
>>>>>
>>>>>> Hi Zach,
>>>>>> thanks for the quick reply. What you say makes as lot of sense, my
>>>>>> next question on this was going to be about performance and indexes for
>>>>>> this kind of search :).
>>>>>>
>>>>>> Just to clarify, if we go this road, is there a way to query all
>>>>>> collections at once or we need to run the query for each collection? I'm
>>>>>> asking because we may need to create collections on the fly as the system
>>>>>> runs, so it would be great to be able to query all elements regardless of
>>>>>> the collection they belong to.
>>>>>>
>>>>>> Thanks again,
>>>>>> Andy
>>>>>>
>>>>>> On Thu, Jun 29, 2017 at 11:48 AM, Zach Mullen <
>>>>>> zach.mullen at kitware.com> wrote:
>>>>>>
>>>>>>> Hi Andy,
>>>>>>>
>>>>>>> Thanks for reaching out! This is actually a common sort of use case,
>>>>>>> but Girder out-of-the-box does not support it (yet). The main reason for
>>>>>>> that is because these sorts of queries should typically be performed
>>>>>>> against a database index so that they can scale up to large numbers of
>>>>>>> folders/items. So, the recommended route for this case is to create a small
>>>>>>> plugin that makes sure your desired search field is indexed. To achieve
>>>>>>> that, you'd add a line like:
>>>>>>>
>>>>>>>     ModelImporter.model('item').ensureIndex(['meta.type',
>>>>>>> {'sparse': True}])
>>>>>>>
>>>>>>> Then, you'd want to probably add a small API endpoint to search by
>>>>>>> this field, and perhaps even some UI augmentation to expose it somewhere.
>>>>>>> Let me know if you need help with those other steps.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Zach Mullen
>>>>>>> Kitware, Inc.
>>>>>>> 919-869-8858 <(919)%20869-8858>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jun 29, 2017 at 9:18 AM, Andr?s Fortier <
>>>>>>> andres at ekumenlabs.com> wrote:
>>>>>>> >
>>>>>>> > Hi all,
>>>>>>> > first of all, sorry of this is a trivial question, just getting
>>>>>>> started with Girder. We are currently evaluating using Girder as a backend
>>>>>>> to store resources (either a folder or a file). One of the requirements we
>>>>>>> have is that a resource may have 0, 1 or more "types" (although it will
>>>>>>> most likely be 1 for a starter), which can be plain strings. Also, we want
>>>>>>> to be able to search resources by type, even if they are in different
>>>>>>> collections.
>>>>>>> >
>>>>>>> > Initially we thought on attaching a type property to a resource
>>>>>>> metadata, which could be an array of strings. However the search bar in the
>>>>>>> web front-end doesn't seem to support search by metadata, so I was
>>>>>>> wondering: is metadata search supported on a collection? if yes, how about
>>>>>>> cross-collection search? If no, should I write a plugin to do that?
>>>>>>> >
>>>>>>> > Any hints / pointers are much appreciated.
>>>>>>> >
>>>>>>> > Thanks!
>>>>>>> > Andy
>>>>>>> >
>>>>>>> > _______________________________________________
>>>>>>> > Girder-users mailing list
>>>>>>> > Girder-users at public.kitware.com
>>>>>>> > http://public.kitware.com/mailman/listinfo/girder-users
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Girder-users mailing list
>>>>> Girder-users at public.kitware.com
>>>>> http://public.kitware.com/mailman/listinfo/girder-users
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170629/b61e3011/attachment-0001.html>

From andres at ekumenlabs.com  Thu Jun 29 17:28:27 2017
From: andres at ekumenlabs.com (=?UTF-8?Q?Andr=C3=A9s_Fortier?=)
Date: Thu, 29 Jun 2017 18:28:27 -0300
Subject: [Girder-users] Versioning files / folders
In-Reply-To: <CAPzddEGby9rV-=g+AYwEZCxruCLxw16d8EqLcH=te_n5_eGxSg@mail.gmail.com>
References: <CALiGwQucZx6bV2_g02qb9b4yJB+fRrWsKV=fNMVFSEwB6P7ndQ@mail.gmail.com>
 <CAPzddEEFBhtvjOtbfZaPrN-sPvTziyfxy=pk_fgAwfRNUs=Kog@mail.gmail.com>
 <CAPzddEGby9rV-=g+AYwEZCxruCLxw16d8EqLcH=te_n5_eGxSg@mail.gmail.com>
Message-ID: <CALiGwQu2qaR9hAFUpxW-96xxszqCf5DJvavBrKpO0YMOuPQvOQ@mail.gmail.com>

Great, thanks Zach! Agreed that supporting folders can be tricky, however I
don't think we will need partial updates (i.e. add/remove/update an element
inside the folder) but just treat the folder as s whole, so I expect to be
able to handle them pretty similar to how you handle file versioning. I'll
ping you in the future once I try to extend your plugin.

Best,
Andy

On Thu, Jun 29, 2017 at 3:55 PM, Zach Mullen <zach.mullen at kitware.com>
wrote:

> BTW, the plugin I linked only supports revisions of a file, not a folder
> (doing that is less trivial, and depends on your specific requirements).
>
>
>
> On Thu, Jun 29, 2017 at 2:54 PM, Zach Mullen <zach.mullen at kitware.com>
> wrote:
>
>> Hi Andy,
>>
>> I actually wrote a prototype plugin[1] that does pretty much exactly what
>> you described. Basically, when you upload the file, you just need to pass
>> an extra parameter that identifies its canonical name/path, and then it
>> auto-increments a version. The plugin contains a rudimentary view for
>> listing the revisions of an asset in reverse-chronological order.
>>
>> [1] https://github.com/girder/data_revisions
>>
>> Thanks,
>>
>> -Zach
>>
>>
>>
>> On Thu, Jun 29, 2017 at 2:48 PM, Andr?s Fortier <andres at ekumenlabs.com>
>> wrote:
>>
>>> Hi all,
>>> just to close the loop on the previous thread requirements, we would
>>> also need to version the assets a user uploads. We don't need any fancy
>>> support (i.e. semver or similar), just an increasing integer would do it.
>>> What we would need is:
>>>
>>> - Updating an asset version each time it is re-uploaded (I assume the
>>> new upload overwrites the existing data).
>>> - Ability to get all the versions of a given asset.
>>> - Ability to recover an asset by its version.
>>> - (Ideally) ability to diff two versions of an asset (but there are
>>> tools that already do this that we could integrate, so no big deal).
>>>
>>> Again, assets can be both files or folders (where the latter may be a
>>> tricky one to handle). Is there support for such a use case in Girder?
>>>
>>> Thanks!
>>> Andy
>>>
>>> _______________________________________________
>>> Girder-users mailing list
>>> Girder-users at public.kitware.com
>>> http://public.kitware.com/mailman/listinfo/girder-users
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170629/39190e18/attachment.html>