[Smtk-developers] Resource Proxies (not the ParaView kind)

[TJ Corona]

Hi David,

Thanks for responding!

>> I have been working on resource links today. ...
> 
> Does this fit in with making smtk::resource::{Resource,Component} share a common base class (e.g., PersistentObject) so that components can link to resources and vice versa? I think that's part of some of our use cases (notably the export operator).

It could, but I haven’t gone down that path yet.

>> In the process, I decided to try and tackle the case where a resource has links to another resource, and we don’t want to load that resource until we need to. The problem I ran into is the open-ended nature of this task: “until we need to” load a resource. 
> 
> I don't think it's open ended, so much as often the definition of "need" must be provided by a user (or a workflow), not a developer.

Cool. I came to a similar conclusion.

>> 1. When do we “need” a resource to load? When an operator uses it as a parameter? When a resource with links (think a model with an associated mesh) is operated on (if you remove a model face, should we remove the associated mesh part)?
> 
> Yes, links must be bidirectional so that a resource/component (let's just say "persistent object" from here on) can discover what links to it. Otherwise, it could be hard to avoid cyclic dependencies which we may want to prevent.

Good to know, that wasn’t part of my original design.

>> 2. If two resources link to the same resource proxy ...
> 
> what's a resource _proxy_? (And where is that in the documentation? :-)

Do you write documentation for a class you haven’t written? :-/

A resource proxy is identical to your OfflineResource. It is a proxy for a resource that can be retrieved when needed. Here and throughout, I am using the term “proxy” as "a figure that can be used to represent the value of something in a calculation” (Google’s definition, not mine).

>> and one of them resolves that resource, do we expect the resource link in the other resource to automagically update itself? Can two resources link to the same resource?
> 
> Yes, I think this goes back to the bidirectional nature of relationships.

If this is the case, we are going to be adding some potential minefields in the resource logic. The infinite recursion kind. There are 1,000,000 ways that can go wrong, especially with the open-ended API we expose for resources.

>> 3. Does this need to work in the absence of a resource manager (I hope not!)?
> 
> Everything I'm doing ends up having a resource manager, so I'm OK with it requiring a resource manager.

I actually decided to go the other way with this, avoiding requiring a resource manager to implement linking.

>> The problem becomes much more complex when you take into account thread safety, where many things can try to resolve a proxy at once.
> 
> Again, I'm not sure what a proxy is.

See above.

>> I have some ideas for how this all should work, but I would really like to hear from you all what concrete examples need to be supported.
> 
> 1. A mesh and model should be linked so that if a model gets modified, SMTK can reconcile (or unlink) linked meshes. We have talked about storing the "recipe" (sizing function) on the model so automatic remeshing _could_ happen (not that it should automatically). Another thing we could do to simplify the situation is to store the sizing function on the link itself.
> 
> 2. Attribute associations are really links. You might load a simulation attribute resource without the model, but if you start to export the model might need to be loaded for the exporter.
> 
> 3. Attribute RefItems are really links.
>   a. An attribute holding solver parameters might refer to an inflow/outflow/wall boundary condition attribute or an initial condition attribute. In a complex case, it is conceivable that those attributes might even live in different resources -- although we don't do that today. However, ...
>   b. I believe the "export" and "simulation" attributes are held in different resources today and "export" attributes reference "simulation" attributes. Certainly the export operator should be allowed to link to model, mesh, and attribute resources as well as components within them.
> 
> 4. We have talked about storing attributes directly on meshes, but I don't have a specific use case in mind.
> 
> 	David
> 
> PS. Here's my 2¢ for how to handle links:
> 
> 1. Create smtk::resource::PersistentObject which both Component and Resource inherit.

I agree, it may be the right time for this.

> 2. Add an ivar to PersistentObject named m_role of type std::string.
> 3. Change the "clean" ivar in Resource to an enum { CLEAN, DIRTY, OFFLINE } or add another bool that indicates the resource is not loaded so that access to its components is unavailable without further action.
> 4. Create smtk::resoource::OfflineComponent which inherits Component. It must point to a resource in the OFFLINE state. It should have a resolve() method that returns the proper value (obtained using the OfflineComponent's id()) if its parent resource is not OFFLINE.

This is pretty similar to what I first tried (except using resources, not components). The method resolve() pulls in the operation system which pulls in the attribute system and IO system, thus making SMTK Core a giant circular dependency again. I really, really don’t want to do this for the sake of syntactic sugar.

> 5. Add an ivar to PersistentObject named m_links of type: std::map<std::string, std::set<PersistentObjectPtr>>. The string in the map is a "role". I think I can hear TJ's teeth gnashing, but perhaps he's just about to suggest some template-fu instead of a string. Whatever creates links must ensure they are bidirectional.

You can hear that from NC? :-)

> 6. Add a LinkManager class which Resource instances may hold... that allows applications to set a policy for how OfflineComponent::resolve() should behave.

This is a lot of machinery for what we are trying to accomplish. I like the idea of a policy for resolving resources, though.

> Now we can represent offline persistent objects. The application has a chance to decide how/whether to resolve unloaded resources at the time anyone attempts to resolve a link: if the resource is offline, the link-manager will load it or not based on either the workflow or user input. If a component is offline, again, the link-manager is invoked. Then functions in the SMTK library just need to examine whether offline persistent objects are resolved into online values before continuing. In fact, the entire process could be made invisible to users of links: you ask for a persistent object's links with a particular role and are only given those which resolved. If links are unresolved, the link-manager has an opportunity to remove the link on a case-by-case basis.
> 
> Here's another 2¢: add some more metadata (acceptable values in the form of multimap<uniqueName,queryString>) to links and remove the attribute system's RefItem, ComponentItem, and ResourceItem.


> On Mar 15, 2018, at 9:36 AM, David Thompson <david.thompson at kitware.com> wrote:
> 
> Hi TJ,
> 
>> I have been working on resource links today. ...
> 
> Does this fit in with making smtk::resource::{Resource,Component} share a common base class (e.g., PersistentObject) so that components can link to resources and vice versa? I think that's part of some of our use cases (notably the export operator).
> 
>> In the process, I decided to try and tackle the case where a resource has links to another resource, and we don’t want to load that resource until we need to. The problem I ran into is the open-ended nature of this task: “until we need to” load a resource. 
> 
> I don't think it's open ended, so much as often the definition of "need" must be provided by a user (or a workflow), not a developer.
> 
>> 1. When do we “need” a resource to load? When an operator uses it as a parameter? When a resource with links (think a model with an associated mesh) is operated on (if you remove a model face, should we remove the associated mesh part)?
> 
> Yes, links must be bidirectional so that a resource/component (let's just say "persistent object" from here on) can discover what links to it. Otherwise, it could be hard to avoid cyclic dependencies which we may want to prevent.
> 
>> 2. If two resources link to the same resource proxy ...
> 
> what's a resource _proxy_? (And where is that in the documentation? :-)
> 
>> and one of them resolves that resource, do we expect the resource link in the other resource to automagically update itself? Can two resources link to the same resource?
> 
> Yes, I think this goes back to the bidirectional nature of relationships.
> 
>> 3. Does this need to work in the absence of a resource manager (I hope not!)?
> 
> Everything I'm doing ends up having a resource manager, so I'm OK with it requiring a resource manager.
> 
>> The problem becomes much more complex when you take into account thread safety, where many things can try to resolve a proxy at once.
> 
> Again, I'm not sure what a proxy is.
> 
>> I have some ideas for how this all should work, but I would really like to hear from you all what concrete examples need to be supported.
> 
> 1. A mesh and model should be linked so that if a model gets modified, SMTK can reconcile (or unlink) linked meshes. We have talked about storing the "recipe" (sizing function) on the model so automatic remeshing _could_ happen (not that it should automatically). Another thing we could do to simplify the situation is to store the sizing function on the link itself.
> 
> 2. Attribute associations are really links. You might load a simulation attribute resource without the model, but if you start to export the model might need to be loaded for the exporter.
> 
> 3. Attribute RefItems are really links.
>   a. An attribute holding solver parameters might refer to an inflow/outflow/wall boundary condition attribute or an initial condition attribute. In a complex case, it is conceivable that those attributes might even live in different resources -- although we don't do that today. However, ...
>   b. I believe the "export" and "simulation" attributes are held in different resources today and "export" attributes reference "simulation" attributes. Certainly the export operator should be allowed to link to model, mesh, and attribute resources as well as components within them.
> 
> 4. We have talked about storing attributes directly on meshes, but I don't have a specific use case in mind.
> 
> 	David
> 
> PS. Here's my 2¢ for how to handle links:
> 
> 1. Create smtk::resource::PersistentObject which both Component and Resource inherit.
> 2. Add an ivar to PersistentObject named m_role of type std::string.
> 3. Change the "clean" ivar in Resource to an enum { CLEAN, DIRTY, OFFLINE } or add another bool that indicates the resource is not loaded so that access to its components is unavailable without further action.
> 4. Create smtk::resoource::OfflineComponent which inherits Component. It must point to a resource in the OFFLINE state. It should have a resolve() method that returns the proper value (obtained using the OfflineComponent's id()) if its parent resource is not OFFLINE.
> 5. Add an ivar to PersistentObject named m_links of type: std::map<std::string, std::set<PersistentObjectPtr>>. The string in the map is a "role". I think I can hear TJ's teeth gnashing, but perhaps he's just about to suggest some template-fu instead of a string. Whatever creates links must ensure they are bidirectional.
> 6. Add a LinkManager class which Resource instances may hold... that allows applications to set a policy for how OfflineComponent::resolve() should behave.
> 
> Now we can represent offline persistent objects. The application has a chance to decide how/whether to resolve unloaded resources at the time anyone attempts to resolve a link: if the resource is offline, the link-manager will load it or not based on either the workflow or user input. If a component is offline, again, the link-manager is invoked. Then functions in the SMTK library just need to examine whether offline persistent objects are resolved into online values before continuing. In fact, the entire process could be made invisible to users of links: you ask for a persistent object's links with a particular role and are only given those which resolved. If links are unresolved, the link-manager has an opportunity to remove the link on a case-by-case basis.
> 
> Here's another 2¢: add some more metadata (acceptable values in the form of multimap<uniqueName,queryString>) to links and remove the attribute system's RefItem, ComponentItem, and ResourceItem.