[vtk-developers] Driving away your existing developers

Thu Aug 28 11:17:51 EDT 2014

Hi Berk -

This is a helpful discussion and I like that we are thinking through the
implications before making any decisions.

I also think we should look at development communities that work well, and
try to learn from those examples.  That's why I suggested node and npm as
an ecosystem that seems to really work in my experience.  There's lots of
sophisticated code that works well together and people use it to create
software that is of similar complexity to VTK and ParaView.  There are well
established testing and version control methodologies.  The core is very
small and stable, but there are packages for just about anything you can
imagine.  The community also attracts contributions from all the types
David mentioned (companies, academics, and sharp independent developers).

Do people have other community models that they think work well and would
be worth studying?  Or perhaps we could also learn from failed/failing
communities, so as not to repeat their mistakes.

-Steve

On Thu, Aug 28, 2014 at 10:44 AM, Berk Geveci <berk.geveci at kitware.com>
wrote:

> > I agree with Steve that vtk proper should get smaller.
>
> I agree with the spirit of this. In fact, I get very exciting thinking
> about how this would work and how nice and
> useful the infrastructure would be. However, I don't think it is very
> feasible to use such a mechanism to
> pull out parts of current VTK. It is a maintenance issue. We have had
> years of experience of maintaining
> tightly coupled projects that are in separate repositories (VTK and
> ParaView). And it is not easy. It requires
> team dedicated to maintaining each separately, with separate dashboard
> architecture, git branches, code
> review processes etc. If we had, say, 10 small projects that tightly
> depended on VTK and was developed
> by roughly the same team, there would be a significant overhead to
> development and maintenance. What
> would the development workflow be to keep these in sync? Having the
> "remote" modules depend on a release
> version of VTK is not tightly coupled enough and would hamper development
> significantly. So we would
> have to manage some version dependency mechanism that would be at best
> hard.
>
> I can see making a handful of modules that hardly change external. Or some
> modules that change more
> regularly but depend less on changes to core features. Some of the IO
> modules with external dependencies,
> maybe VTK Web, some filtering module such as Matlab, R, Reebgraph etc. But
> is it really worth the trouble?
> It would shrink VTK slightly but would make testing, integration etc.
> harder for everyone.
>
> I would rather focus on implementing this for new projects / modules.
> Specially those that are developed
> in a way that's less coupled with VTK. Some Slicer code could be organized
> this way for example. Slicer
> depends on a release version of VTK and has its own testing infrastructure
> so it should be easy. Things
> like ITK-VTK adapters could probably be organized the same way.
>
> What would the mechanism for indicating version dependencies, whether a
> remote module is tested
> regularly and things like that?
>
> Best,
> -berk
>
>
> On Wed, Aug 27, 2014 at 8:58 PM, Bill Lorensen <bill.lorensen at gmail.com>
> wrote:
>
>> ITK's  remote module mechanism is a great way to add domain specific
>> modules. I agree with Steve that vtk proper should get smaller.
>>  On Aug 27, 2014 5:04 PM, "Steve Pieper" <pieper at isomics.com> wrote:
>>
>>> I would love to see more of your excellent work, David, and I'd want to
>>> make use of it, but I'd argue that at least some of it, perhaps
>>> registration, may not belong in VTK itself.
>>>
>>> Speaking for myself from my experience in the slicer project, I can say
>>> that we end up putting a lot of our functionality in classes that don't
>>> belong back into VTK proper, such as vtkITK, vtkTeem, and vtkMRML, each of
>>> which adds functionality that is independent of our slicer application, but
>>> adds a layer of dependencies that nobody would want to see in VTK itself.
>>>
>>> On my wishlist for VTK would be a better way for new developers to
>>> contribute their work so that it's easily usable by others but doesn't have
>>> to pass the gauntlet of reviews and comments and be in VTK before it is
>>> easily usable by others.  I'd like to see an experience more like npm for
>>> node, where anyone can publish a domain-specific set of code and other
>>> people can use it or not as they see fit.  Node itself remains small and it
>>> rarely changes, but the ecosystem is quite vibrant.  It allows a lot of
>>> diversity, but amazing interoperability and reusability.
>>>
>>> In slicer we have something similar now with the Extension Catalog,
>>> where developers can offer code (either pure VTK or things that work at the
>>> VTK/ITK/Qt/Slicer level) that is optionally installed.  Since we perform
>>> nightly builds and test it on all platforms, this code is in reasonable
>>> shape for other people to draw from, either in slicer or elsewhere.
>>>  Typically these extensions are in github repositories, so they have their
>>> own wikis, issue lists, pull requests, etc.  We hope that slicer itself
>>> actually gets smaller over time and generally doesn't need to change very
>>> much or very often.
>>>
>>> I would love to see something similar in VTK proper.  Right now VTK has
>>> a lot of modules that are quite domain specific, such as Chemistry and
>>> Geovis which I do not envision using myself.  I would have preferred to see
>>> that code exist in a different repository where people who needed it would
>>> have the option of including it in their superbuild process.  Others could
>>> safely ignore it.  I would even argue that half (or more) of the current
>>> modules don't really belong in the 'actual' VTK and should be moved out.
>>>  If there were a good place to put some of this kind of code it might also
>>> be a good spot for libraries like vtkITK and vtkTeem.  In fact I would
>>> argue that if it weren't actually in VTK's repository, a project like
>>> vtkChemistry might attract more participants and have more features (but I
>>> can't prove that of course).
>>>
>>> So basically I'd say that VTK is currently too big and should get
>>> smaller, while the community is too small and should get bigger.
>>>
>>> -Steve
>>>
>>>
>>> On Wed, Aug 27, 2014 at 12:00 PM, David Gobbi <david.gobbi at gmail.com>
>>> wrote:
>>>
>>>> On Wed, Aug 27, 2014 at 7:10 AM, Berk Geveci <berk.geveci at kitware.com>
>>>> wrote:
>>>>
>>>> > One final note. Everyone please keep in mind that while there are
>>>> quite a
>>>> > lot of contributors to VTK, certain parts of it are developed by only
>>>> a few
>>>> > people. Core pipeline: I am it at this point. Rendering: by next
>>>> year, it
>>>> > will be Ken and Marcus. Imaging: hardly anyone in the scientific vis
>>>> team at
>>>> > Kitware does anything with imaging filters. We update them when we
>>>> make
>>>> > changes to the pipeline or data model but that's it. So some of those
>>>> are
>>>> > maintained and developed by others or they are some orphaned. This
>>>> can only
>>>> > be addressed by either encouraging new (or existing) developers to
>>>> take
>>>> > ownership or by removing those components all together.
>>>>
>>>> I have tons of plans for the VTK imaging classes, but unfortunately I
>>>> have
>>>> almost no resources so progress is slow.  Some examples:
>>>>
>>>> 1) Improving both the API and the performance for multi-threading.
>>>> Last year I worked with an intern to improve load sharing.  Never had
>>>> time to finish, because it wasn't the intern's main project.
>>>>
>>>> 2) Basic linear image registration in VTK.  I've finished it and it's
>>>> very robust (been using it for years), but refactoring so that it can
>>>> be contributed to VTK is an evening/weekend project and hence is
>>>> moving slowly.
>>>>
>>>> 3) Better image iterators.  I have a nice image iterator that iterates
>>>> over arbitrary shapes and also provides coordinate info while it
>>>> iterates.  I'd love to develop this further and contribute it, and I'd
>>>> especially love to wrap it, but it's out in left field as far as my
>>>> day job goes.
>>>>
>>>> 4) Better filters for going back and forth between image data and
>>>> polydata.  Lots of people use my vtkPolyDataToImageStencil filter,
>>>> but it has faults and I know exactly how to fix them, but I'll have to
>>>> wait until it comes up as a priority for one of my work projects.
>>>>
>>>> 5) Finally, a better overall structure for the imaging pipeline,
>>>> focused on the basics:
>>>>  a) scalar operation "functors" that can be plugged into filters
>>>>  b) interpolators, which I've already committed
>>>>  c) good image "region" descriptors (like my vtkImageStencilData)
>>>>  d) connectivity & morphological operations, I've got lots of
>>>> un-committed stuff
>>>>  e) a replacement for the lousy vtkImageMathematics filter
>>>>
>>>> Ideally I should write out a big roadmap (the items mentioned above
>>>> is just a fraction of the total), but it would be depressing because so
>>>> much of it feels like it's nothing more than a pipe dream.
>>>>
>>>> And, as Berk mentioned, there isn't anyone in Kitware for me to
>>>> directly work with on this stuff.  And all the other VTK imaging
>>>> developers outside of Kitware are content to keep their work to
>>>> themselves, without contributing back to the VTK core.  Or at least,
>>>> that's the way it looks from my vantage point.
>>>>
>>>>  - David
>>>> _______________________________________________
>>>> Powered by www.kitware.com
>>>>
>>>> Visit other Kitware open-source projects at
>>>> http://www.kitware.com/opensource/opensource.html
>>>>
>>>> Follow this link to subscribe/unsubscribe:
>>>> http://public.kitware.com/mailman/listinfo/vtk-developers
>>>>
>>>>
>>>>
>>>> The information in this e-mail is intended only for the person to whom
>>>> it is
>>>> addressed. If you believe this e-mail was sent to you in error and the
>>>> e-mail
>>>> contains patient information, please contact the Partners Compliance
>>>> HelpLine at
>>>> http://www.partners.org/complianceline . If the e-mail was sent to you
>>>> in error
>>>> but does not contain patient information, please contact the sender and
>>>> properly
>>>> dispose of the e-mail.
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://public.kitware.com/mailman/listinfo/vtk-developers
>>>
>>>
>>>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/vtk-developers
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/vtk-developers/attachments/20140828/b683d585/attachment-0002.html>