[Girder-users] Querying by metadata

[Andrés Fortier]

Thanks Zach and Jonathan for your replies! To answer Jonathan's question, I
was referring to Girder collections and not mongo (sorry for the confusion
here, it is unfortunate that the term "collection" is overloaded; I will
disambiguate from now on).

So, maybe stepping back a little sharing the high-level requirements may be
a better approach :). We are trying to build an application to manage
assets (xml, png, maybe source code, dlls, etc. An asset can also be
composed of many files, so basically an asset can be either a girder file
or folder). The application must provide both an administrative back end
(we think girder web client would be a fit) and REST API (again, girder
REST API seems a fit). We will later on roll out a front end (most likely a
Javascript SPA), backed by girder's the REST API.

Now, the typical workflow we envision is:

- Someone creates an account and adds assets to the app. Assets have an
associated type, which is independent of the file extension or mime-type,
so this is configured by the user. The user can then decide to keep the
asset private, public or shared with some users / group.
- It should be easy / performant to query all the assets of a given type
(plus some variations when it applies, like "all assets of a given type
that belong to user X", "all private assets of a given type that belong to
user X", etc).
- It should be easy for a user to download all its contents (e.g. for
personal backup).
- It should support asset versioning (I was going to discuss this in
another thread, just mentioning here to give a broad view).

The approach that we currently have in mind is to have a girder collection
per user (so that all the user assets are stored there) and tag the assets
with metadata to specify its type (hence the need to query by metadata
across different girder collections). Another approach could be to have a
folder per user in a single "assets" collection, pretty much like a linux
/home directory (which at least would remove the cross-girder-collection
query). But again, I'm *very* new to Girder, so it is most likely that I
don't know what the best use for girder collections are, so any hints are
much appreciated.

Thanks!
Andy




On Thu, Jun 29, 2017 at 1:40 PM, Jonathan Beezley <
jonathan.beezley at kitware.com> wrote:

> For clarification, do you mean collections in the sense of mongodb
> collections, or girder collections?  It shouldn't be any problem, for
> example, searching all items regardless of the parent collection they
> belong in.
>
> On Thu, Jun 29, 2017 at 11:27 AM, Zach Mullen <zach.mullen at kitware.com>
> wrote:
>
>> One of the main limitations (or perhaps features?) of mongodb is that a
>> query is applied to only one collection at a time, so your search function
>> would make one query per collection that you wish to search. So, in the
>> case you describe, you'd need to loop over some dynamic set of collections
>> and query each one. Or, create some secondary collection that aggregates
>> all of the data with this metadata field, and contains references to other
>> collections in its documents.
>>
>> Thanks,
>>
>> -Zach
>>
>> On Thu, Jun 29, 2017 at 11:22 AM, Andrés Fortier <andres at ekumenlabs.com>
>> wrote:
>>
>>> Hi Zach,
>>> thanks for the quick reply. What you say makes as lot of sense, my next
>>> question on this was going to be about performance and indexes for this
>>> kind of search :).
>>>
>>> Just to clarify, if we go this road, is there a way to query all
>>> collections at once or we need to run the query for each collection? I'm
>>> asking because we may need to create collections on the fly as the system
>>> runs, so it would be great to be able to query all elements regardless of
>>> the collection they belong to.
>>>
>>> Thanks again,
>>> Andy
>>>
>>> On Thu, Jun 29, 2017 at 11:48 AM, Zach Mullen <zach.mullen at kitware.com>
>>> wrote:
>>>
>>>> Hi Andy,
>>>>
>>>> Thanks for reaching out! This is actually a common sort of use case,
>>>> but Girder out-of-the-box does not support it (yet). The main reason for
>>>> that is because these sorts of queries should typically be performed
>>>> against a database index so that they can scale up to large numbers of
>>>> folders/items. So, the recommended route for this case is to create a small
>>>> plugin that makes sure your desired search field is indexed. To achieve
>>>> that, you'd add a line like:
>>>>
>>>>     ModelImporter.model('item').ensureIndex(['meta.type', {'sparse':
>>>> True}])
>>>>
>>>> Then, you'd want to probably add a small API endpoint to search by this
>>>> field, and perhaps even some UI augmentation to expose it somewhere. Let me
>>>> know if you need help with those other steps.
>>>>
>>>> Thanks,
>>>>
>>>> Zach Mullen
>>>> Kitware, Inc.
>>>> 919-869-8858 <(919)%20869-8858>
>>>>
>>>>
>>>> On Thu, Jun 29, 2017 at 9:18 AM, Andrés Fortier <andres at ekumenlabs.com>
>>>> wrote:
>>>> >
>>>> > Hi all,
>>>> > first of all, sorry of this is a trivial question, just getting
>>>> started with Girder. We are currently evaluating using Girder as a backend
>>>> to store resources (either a folder or a file). One of the requirements we
>>>> have is that a resource may have 0, 1 or more "types" (although it will
>>>> most likely be 1 for a starter), which can be plain strings. Also, we want
>>>> to be able to search resources by type, even if they are in different
>>>> collections.
>>>> >
>>>> > Initially we thought on attaching a type property to a resource
>>>> metadata, which could be an array of strings. However the search bar in the
>>>> web front-end doesn't seem to support search by metadata, so I was
>>>> wondering: is metadata search supported on a collection? if yes, how about
>>>> cross-collection search? If no, should I write a plugin to do that?
>>>> >
>>>> > Any hints / pointers are much appreciated.
>>>> >
>>>> > Thanks!
>>>> > Andy
>>>> >
>>>> > _______________________________________________
>>>> > Girder-users mailing list
>>>> > Girder-users at public.kitware.com
>>>> > http://public.kitware.com/mailman/listinfo/girder-users
>>>> >
>>>>
>>>
>>>
>>
>> _______________________________________________
>> Girder-users mailing list
>> Girder-users at public.kitware.com
>> http://public.kitware.com/mailman/listinfo/girder-users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/girder-users/attachments/20170629/efd363bb/attachment-0001.html>