[vtk-developers] VTK and Qt5 / QML

[c.sell at byterefinery.de]

ok, thanks for the exchange, I think I have a better understanding  
now. I have managed to get the code running with QVTKInteractor and  
siblings - it was a matter of instantiating an (Q-)object at the wrong  
time and thus on the wrong thread => no rendering. I may try the  
multi-threaded approach as well, and will publish the sample code on  
github once I am happy with it.

chris

Zitat von Geoff Wright <gpwright at gmail.com>:

> Hi Chris,
>
> For the interactor there is a way to basically queue the event information
> you need e.g. from qmls onPressed and then pass it to an interactor that
> belongs to the particular viewport that was pressed/clicked so that it will
> be handled once that thread is being rendered.
>
> However, I found that I quickly wanted a more advanced interaction.  For
> example if your vtk qml item is inside a qml Flickable you can easily do
> very nice interactions for touch such as rotational momentum on flick that
> continues after release.
>
> I am not sure of the latest touch interactor in vtk but when I was working
> on this a couple of years ago it was easier to handle all of the interactor
> part in QML (e.g. utilizing Flickable) and then pass info to the rendering
> thread that would directly mutate the camera, effectively making the
> vtkCamera a slave to some qml properties.
>
> However if vtk has caught up with momentum based touch interactors it may
> be worthwhile to use these.  I am not caught up with what's been added to
> VTK in recent months.  Another advantage of keeping the camera master state
> in QML is that you can use Qt animation framework easily (e.g. they
> recently added an orientation animation that can be useful for controlling
> a camera).
>
> I think it depends how complex are the user interactions that you are
> planning for.
>
> G
>
>
>
>
>
> On Wed, May 17, 2017, 10:15 AM <c.sell at byterefinery.de> wrote:
>
>> I have to correct myself with regard to the "direct rendering" option.
>> There is, of course, an example that demonstrates this
>> (http://doc.qt.io/qt-5/qtquick-scenegraph-openglunderqml-example.html). I
>> doubt whether this could be implemented in conjunction with
>> thread-per-item, so it only adds one architectural option. I still
>> think this is the least versatile and generic one, though.
>>
>> chris
>>
>> Zitat von c.sell at byterefinery.de:
>>
>> > ok, things are clearing up a little more now. You ARE creating a
>> > separate thread for each QML item, which gives us these threads:
>> >
>> > - QML GUI thread (where the events come from)
>> > - QML rendering thread (aka scenegraph rendering thread aka
>> > QSGRenderThread), of which theres is only one
>> > - a dedicated thread for each item (RenderThread class in the example)
>> >
>> > the example also uses a QSGSimpleTextureNode subclass that does the
>> > blit'ing together. All of that decoupled via QueuedConnections and
>> > events signalled here and there. Not an easy beast, that.
>> >
>> > What remains unclear is how you deal with user input. For example,
>> > if the user were to drag something around in the scene (and your app
>> > were to support that), where do the MouseMove events go? This is
>> > really the main crux in this discussion IMO.
>> >
>> > To sum up the architectural choices:
>> >
>> > a) render into FBO
>> > a1)- do all rendering on the scenegraph thread (exemplified in
>> > http://doc.qt.io/qt-5/qtquick-scenegraph-textureinsgnode-example.html)
>> >
>> > a2) render on a dedicated thread for each item (exemplified in
>> > http://doc.qt.io/qt-5/qtquick-scenegraph-textureinthread-example.html)
>> >
>> > b) render into the QML context directly (not sure about this. Is
>> > there a Qt example demonstrating the practicability?)
>> > b?) undetermined
>> >
>> >
>> > The user intaction part is undetermined for me at this point. I am
>> > trying to get something going using QVTKInteractorAdapter, but
>> > havent had luck so far.
>> >
>> > In terms of ready usable code that demonstrates QML/VTK integration
>> > (including interaction and all) I see nothing yet on the horizon.
>> >
>> > chris
>> >
>> >
>> >
>> > Zitat von Geoff Wright <gpwright at gmail.com>:
>> >
>> >> Yes each vtk item has its own render thread that is why I would argue
>> this
>> >> approach is superior to the others since it scales with the number of
>> cores.
>> >>
>> >> I followed the approach described here:
>> >>
>> >> http://doc.qt.io/qt-5/qtquick-scenegraph-textureinthread-example.html
>> >>
>> >> So basically inside each vtk qml item there is a thread drawing a vtk
>> >> viewport into a texture, then the QSGRenderThread comes along and just
>> does
>> >> a single texture blit for each item (very cheap). The Qt main thread
>> takes
>> >> care of other event processing, models etc.
>> >>
>> >> This is sufficient if each item contains an independent scene (vtk
>> >> pipeline).  If you need to share state between two or more viewports
>> then
>> >> you need to be careful to control the timing of when any data objects
>> may
>> >> mutate since the vtk api is not inherently threadsafe.
>> >>
>> >> Does that make sense?
>> >>
>> >> G
>> >>
>> >>
>> >>
>> >> On Tue, May 16, 2017, 6:13 PM <c.sell at byterefinery.de> wrote:
>> >>
>> >>> Hello all,
>> >>>
>> >>> @Geoff: are you saying that each pair of QML item / vtkRenderWindow
>> >>> has its own render thread? Is that something QML does?
>> >>>
>> >>> My goal is to have QML/VTK items behave as normal QML items, i.e., I
>> >>> could have a dozen items on my screen each rendering something
>> >>> different through VTK without influencing each other (apart from
>> >>> system performance). I expect that the approach I have taken so far
>> >>> will already fulfill this requirement.
>> >>>
>> >>> @Sankesh: I object to the compile time flag. IMO both approaches
>> >>> should be usable at the same time, e.g. by extending different
>> >>> superclasses. As you say, the "underlay or overlay" aspect breaks QML
>> >>> concepts and makes any item built according to this approach an
>> >>> "alien" in the QML crowd that always requires special attention.
>> >>>
>> >>> I'll look into QVTKInteractorAdapter - that sounds like seomething
>> >>> useful. Should have come across it earlier.
>> >>>
>> >>> @both: is there a chance that usable code will come out of this
>> >>> exchange any time soon? I for my part would at least publish my
>> >>> prototypical implementations somewhere (GitHub?), unless they become
>> >>> obsolete by something "official"
>> >>>
>> >>> regards,
>> >>> Chris
>> >>>
>> >>> Zitat von Geoff Wright <gpwright at gmail.com>:
>> >>>
>> >>>> Hi Chris,
>> >>>>
>> >>>> Sorry for the confusion, yes I am doing what you describe.  The
>> >>> difference
>> >>>> may be that perhaps your app has only one viewport.  In my case there
>> >>> are a
>> >>>> number of different viewports (aka vtkRenderWindow)  each has its own
>> qml
>> >>>> item and is each rendering on it's own thread.  Plus the qml
>> composition
>> >>>> thread which is managed by Qt.
>> >>>>
>> >>>> I was just making the point that this architecture let's you draw
>> >>> different
>> >>>> viewports in parallel provided that each one has its own mappers and
>> >>> actors.
>> >>>>
>> >>>> G
>> >>>>
>> >>>>
>> >>>> On Tue, May 16, 2017, 4:13 PM <c.sell at byterefinery.de> wrote:
>> >>>>
>> >>>>> Hi Geoff,
>> >>>>>
>> >>>>> as I understand it, the blog post you linked to doesn't mention
>> >>>>> multiple threads beyond the QML UI thread and the scenegraph render
>> >>>>> thread.
>> >>>>>
>> >>>>> So far, I am using that approach with QQuickFrameBufferObject as
>> >>>>> starting point and vtkExternalOpenGLRenderWindow for the VTK side,
>> >>>>> which causes VTK to use QML's OpenGL context and thus render into the
>> >>>>> FBO maintained by QQuickFrameBufferObject. That seems to work
>> >>>>> flawlessly, as long as ALL rendering is kept on the scenegraph
>> thread,
>> >>>>> which is not always obvious to do at first, because VTK's API's don't
>> >>>>> separate between reading event values and rendering them (e.g.,
>> >>>>> SetSlice(2) will internally set the current slice and then do a
>> >>>>> Render()).
>> >>>>>
>> >>>>> But why maintain more threads?
>> >>>>>
>> >>>>> Anyway, please go ahead and explain your approach as detailed as
>> >>>>> possible, I am all ears.
>> >>>>>
>> >>>>> regards,
>> >>>>> Chris
>> >>>>>
>> >>>>> Zitat von Geoff Wright <gpwright at gmail.com>:
>> >>>>>
>> >>>>> > Hi guys,
>> >>>>> >
>> >>>>> > I would argue that for qml integration the best approach is for
>> each
>> >>> vtk
>> >>>>> > viewport to render into a texture on a dedicated thread, not the
>> qml
>> >>>>> > rendering thread.  Qml then draws the resulting texture which is
>> more
>> >>> in
>> >>>>> > line with the Qt design intention ( see
>> >>>>> >
>> >>>>>
>> >>>
>> http://blog.qt.io/blog/2015/05/11/integrating-custom-opengl-rendering-with-qt-quick-via-qquickframebufferobject/
>> >>>>> >  )
>> >>>>> >
>> >>>>> > I have been using this approach for a couple of years in a high
>> >>>>> performance
>> >>>>> > real time application.  The code is unfortunately proprietary but
>> very
>> >>>>> > happy to explain more about the approach.  It would be great to add
>> >>>>> support
>> >>>>> > for this to VTK.
>> >>>>> >
>> >>>>> > For the interactor part I do what Sankesh said, defer the events
>> and
>> >>>>> apply
>> >>>>> > them on the render thread of the particular vtk item.
>> >>>>> >
>> >>>>> > Regards,
>> >>>>> >
>> >>>>> > G
>> >>>>> >
>> >>>>> >
>> >>>>> >
>> >>>>> > On Tue, May 16, 2017, 3:06 PM Sankhesh Jhaveri <
>> >>>>> sankhesh.jhaveri at kitware.com>
>> >>>>> > wrote:
>> >>>>> >
>> >>>>> >> Hi Christian,
>> >>>>> >>
>> >>>>> >> Doing all VTK rendering on the QML render thread is the right
>> thing
>> >>> to
>> >>>>> do.
>> >>>>> >> As far as interaction is concerned, make sure that Qt events are
>> >>> queued
>> >>>>> on
>> >>>>> >> the main thread and processed when execution reaches the render
>> >>> thread.
>> >>>>> >>
>> >>>>> >> I have done some work on VTK and QtQuick integration that I’m
>> >>> planning
>> >>>>> to
>> >>>>> >> add to a VTK remote module. That way, it will be available as
>> part of
>> >>>>> VTK.
>> >>>>> >>
>> >>>>> >> Thanks,
>> >>>>> >> Sankhesh
>> >>>>> >> ​
>> >>>>> >>
>> >>>>> >> On Tue, May 16, 2017 at 4:57 AM <c.sell at byterefinery.de> wrote:
>> >>>>> >>
>> >>>>> >>> just for the record (nobody care about this subject?):
>> >>>>> >>>
>> >>>>> >>> I found that contrary to what I said I had NOT adhered to the
>> >>>>> >>> principle established with protoype 1 while implementing
>> protoype 2.
>> >>>>> >>> Rather, I had directly called into the VTK interactor from the
>> QML
>> >>> UI
>> >>>>> >>> thread. And because the VTK interactors immediately go into
>> >>> rendering,
>> >>>>> >>> all hell broke loose.
>> >>>>> >>>
>> >>>>> >>> After cleaning up example 2 according to the rule laid out
>> earlier
>> >>>>> >>> ("store event data in RenderDelegate, call update() on the QML
>> item,
>> >>>>> >>> pass the event to VTK inside RenderDelgate.Render()", all works
>> >>> well.
>> >>>>> >>>
>> >>>>> >>> What I would like to do is validate the solution against findings
>> >>> made
>> >>>>> >>> elsewhere, and then establish the "canonical" way of integrating
>> VTK
>> >>>>> >>> with QML. As I said, there are some architectural quirks with the
>> >>>>> >>> solution which I would also like to address.
>> >>>>> >>>
>> >>>>> >>> regards,
>> >>>>> >>> Christian
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>> Zitat von c.sell at byterefinery.de:
>> >>>>> >>>
>> >>>>> >>> > Hello all,
>> >>>>> >>> >
>> >>>>> >>> > I am looking into the possibility of replacing the 3D rendering
>> >>>>> >>> > engine in my Qt5 / QML based mobile tablet-oriented application
>> >>> with
>> >>>>> >>> > VTK. What I need is a first class QtQuick integration (not a
>> hack
>> >>> or
>> >>>>> >>> > workaround) that is 100% stable in every context (not an
>> unusual
>> >>>>> >>> > requirement, I admit). To my amazement, nothing of that kind
>> seems
>> >>>>> >>> > to exist (correct me if I'm wrong).
>> >>>>> >>> >
>> >>>>> >>> > I went on to investigate what had been done so far and to
>> >>> implement
>> >>>>> >>> > my first prototypes, using the QQuickFrameBufferObject
>> approach.
>> >>>>> >>> > From the very start this felt like an uphill battle, because
>> VTK
>> >>>>> >>> > seems to come from a Windowing background and is quite tightly
>> >>>>> >>> > integrated with concepts that are not valid in a QML context.
>> >>>>> >>> >
>> >>>>> >>> > I'll describe my findings together with the 2 prototypical QML
>> >>> items
>> >>>>> >>> > I implemented:
>> >>>>> >>> >
>> >>>>> >>> > 1st was an adaptation of the DICOM example which now runs as a
>> QML
>> >>>>> >>> > item. All user interaction is handled by QML and forwarded to
>> VTK
>> >>>>> >>> > (which is one thing that doesn't come naturally with VTK), and
>> >>> after
>> >>>>> >>> > some non-elegant tweaking I was able to have the UI move from
>> >>> slice
>> >>>>> >>> > to slice and re-render upon mouse wheel events as expected. The
>> >>>>> >>> > problem here was, that QML insists on keeping mouse event
>> handling
>> >>>>> >>> > and OpenGL rendering on separate threads, with one "rendering
>> >>>>> >>> > thread" dedicated to OpenGL. However, the pre-existing VTK
>> >>>>> >>> > Interactors directly call Render() after reconfiguring the UI
>> from
>> >>>>> >>> > the mouse event data, which is an absolute QML No-Go. I had to
>> >>>>> >>> > introduce a RenderDelegate that works somewhat like this:
>> >>>>> >>> >
>> >>>>> >>> > QML mouse event:
>> >>>>> >>> >     tell RenderDelegate about the event
>> >>>>> >>> >     call update() on the QML item, which triggers rer-endering
>> on
>> >>>>> >>> > the dedicated thread
>> >>>>> >>> > on the QML rendering thread:
>> >>>>> >>> >     call Render() on the RenderWindow
>> >>>>> >>> >     inside the RenderDelegate, look at whether a mouse event is
>> >>>>> >>> > pending, and call the corresponding VTK mouse handler
>> >>>>> >>> >     call the renderer's default Render() function while setting
>> >>> the
>> >>>>> >>> > delegate's "Used" flag to false
>> >>>>> >>> >
>> >>>>> >>> > 2nd was a QML item that displays a model loaded from a 3ds
>> file.
>> >>> My
>> >>>>> >>> > goal here was to move the model around using mouse drag
>> events. I
>> >>>>> >>> > took the lessons learned from the first example, threw in a
>> >>>>> >>> > vtkInteractorStyleTrackballCamera and hoped for the best. First
>> >>>>> >>> > thing I found was that I needed a specialized
>> >>>>> >>> > RenderWindowInteractor, which I provided. When I realized that
>> the
>> >>>>> >>> > most important requirement for that class was to provide
>> access to
>> >>>>> >>> > timer functionality, I already got wary, as throwing around
>> events
>> >>>>> >>> > and doing stuff offline while on the QML rendering thread is
>> >>> never a
>> >>>>> >>> > good idea. My fears came true when I finished wiring everything
>> >>>>> >>> > together: I was able to move the model using the mouse, but
>> after
>> >>> a
>> >>>>> >>> > few moments things got whacky with error output written to the
>> >>>>> >>> > console and finally a segemtation fault from deep inside VTK.
>> >>>>> >>> >
>> >>>>> >>> > I still need to investigate the second example, but would like
>> to
>> >>>>> >>> > synchronize with the community at this point in order to avoid
>> >>>>> >>> > errors/misconceptions on my side, to seek help if possible and
>> to
>> >>>>> >>> > offer my contribution to the VTK code base once this becomes
>> >>>>> >>> > functional. My first impression is that there are some issues
>> that
>> >>>>> >>> > lie on an architectural level and cannot be (elegantly) dealt
>> with
>> >>>>> >>> > on the QML side alone. Any comments?
>> >>>>> >>> >
>> >>>>> >>> > thanks,
>> >>>>> >>> > Christian Sell
>> >>>>> >>> >
>> >>>>> >>> > _______________________________________________
>> >>>>> >>> > Powered by www.kitware.com
>> >>>>> >>> >
>> >>>>> >>> > Visit other Kitware open-source projects at
>> >>>>> >>> > http://www.kitware.com/opensource/opensource.html
>> >>>>> >>> >
>> >>>>> >>> > Search the list archives at:
>> >>>>> >>> http://markmail.org/search/?q=vtk-developers
>> >>>>> >>> >
>> >>>>> >>> > Follow this link to subscribe/unsubscribe:
>> >>>>> >>> > http://public.kitware.com/mailman/listinfo/vtk-developers
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>> _______________________________________________
>> >>>>> >>> Powered by www.kitware.com
>> >>>>> >>>
>> >>>>> >>> Visit other Kitware open-source projects at
>> >>>>> >>> http://www.kitware.com/opensource/opensource.html
>> >>>>> >>>
>> >>>>> >>> Search the list archives at:
>> >>>>> http://markmail.org/search/?q=vtk-developers
>> >>>>> >>>
>> >>>>> >>> Follow this link to subscribe/unsubscribe:
>> >>>>> >>> http://public.kitware.com/mailman/listinfo/vtk-developers
>> >>>>> >>>
>> >>>>> >>> --
>> >>>>> >> Sankhesh Jhaveri *Sr. Research & Development Engineer* | Kitware
>> >>>>> >> <http://www.kitware.com/> | (518) 881-4417
>> >>>>> >> ​
>> >>>>> >> _______________________________________________
>> >>>>> >> Powered by www.kitware.com
>> >>>>> >>
>> >>>>> >> Visit other Kitware open-source projects at
>> >>>>> >> http://www.kitware.com/opensource/opensource.html
>> >>>>> >>
>> >>>>> >> Search the list archives at:
>> >>>>> http://markmail.org/search/?q=vtk-developers
>> >>>>> >>
>> >>>>> >> Follow this link to subscribe/unsubscribe:
>> >>>>> >> http://public.kitware.com/mailman/listinfo/vtk-developers
>> >>>>> >>
>> >>>>> >>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >
>> >
>> >
>> > _______________________________________________
>> > Powered by www.kitware.com
>> >
>> > Visit other Kitware open-source projects at
>> > http://www.kitware.com/opensource/opensource.html
>> >
>> > Search the list archives at:
>> http://markmail.org/search/?q=vtk-developers
>> >
>> > Follow this link to subscribe/unsubscribe:
>> > http://public.kitware.com/mailman/listinfo/vtk-developers
>>
>>
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Search the list archives at: http://markmail.org/search/?q=vtk-developers
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/vtk-developers
>>
>>