[vtk-developers] VTK and Qt5 / QML

Wed May 17 04:14:56 EDT 2017

I have to correct myself with regard to the "direct rendering" option.  
There is, of course, an example that demonstrates this  
(http://doc.qt.io/qt-5/qtquick-scenegraph-openglunderqml-example.html). I  
doubt whether this could be implemented in conjunction with  
thread-per-item, so it only adds one architectural option. I still  
think this is the least versatile and generic one, though.

chris

Zitat von c.sell at byterefinery.de:

> ok, things are clearing up a little more now. You ARE creating a  
> separate thread for each QML item, which gives us these threads:
>
> - QML GUI thread (where the events come from)
> - QML rendering thread (aka scenegraph rendering thread aka  
> QSGRenderThread), of which theres is only one
> - a dedicated thread for each item (RenderThread class in the example)
>
> the example also uses a QSGSimpleTextureNode subclass that does the  
> blit'ing together. All of that decoupled via QueuedConnections and  
> events signalled here and there. Not an easy beast, that.
>
> What remains unclear is how you deal with user input. For example,  
> if the user were to drag something around in the scene (and your app  
> were to support that), where do the MouseMove events go? This is  
> really the main crux in this discussion IMO.
>
> To sum up the architectural choices:
>
> a) render into FBO
> a1)- do all rendering on the scenegraph thread (exemplified in  
> http://doc.qt.io/qt-5/qtquick-scenegraph-textureinsgnode-example.html)
>
> a2) render on a dedicated thread for each item (exemplified in  
> http://doc.qt.io/qt-5/qtquick-scenegraph-textureinthread-example.html)
>
> b) render into the QML context directly (not sure about this. Is  
> there a Qt example demonstrating the practicability?)
> b?) undetermined
>
>
> The user intaction part is undetermined for me at this point. I am  
> trying to get something going using QVTKInteractorAdapter, but  
> havent had luck so far.
>
> In terms of ready usable code that demonstrates QML/VTK integration  
> (including interaction and all) I see nothing yet on the horizon.
>
> chris
>
>
>
> Zitat von Geoff Wright <gpwright at gmail.com>:
>
>> Yes each vtk item has its own render thread that is why I would argue this
>> approach is superior to the others since it scales with the number of cores.
>>
>> I followed the approach described here:
>>
>> http://doc.qt.io/qt-5/qtquick-scenegraph-textureinthread-example.html
>>
>> So basically inside each vtk qml item there is a thread drawing a vtk
>> viewport into a texture, then the QSGRenderThread comes along and just does
>> a single texture blit for each item (very cheap). The Qt main thread takes
>> care of other event processing, models etc.
>>
>> This is sufficient if each item contains an independent scene (vtk
>> pipeline).  If you need to share state between two or more viewports  then
>> you need to be careful to control the timing of when any data objects may
>> mutate since the vtk api is not inherently threadsafe.
>>
>> Does that make sense?
>>
>> G
>>
>>
>>
>> On Tue, May 16, 2017, 6:13 PM <c.sell at byterefinery.de> wrote:
>>
>>> Hello all,
>>>
>>> @Geoff: are you saying that each pair of QML item / vtkRenderWindow
>>> has its own render thread? Is that something QML does?
>>>
>>> My goal is to have QML/VTK items behave as normal QML items, i.e., I
>>> could have a dozen items on my screen each rendering something
>>> different through VTK without influencing each other (apart from
>>> system performance). I expect that the approach I have taken so far
>>> will already fulfill this requirement.
>>>
>>> @Sankesh: I object to the compile time flag. IMO both approaches
>>> should be usable at the same time, e.g. by extending different
>>> superclasses. As you say, the "underlay or overlay" aspect breaks QML
>>> concepts and makes any item built according to this approach an
>>> "alien" in the QML crowd that always requires special attention.
>>>
>>> I'll look into QVTKInteractorAdapter - that sounds like seomething
>>> useful. Should have come across it earlier.
>>>
>>> @both: is there a chance that usable code will come out of this
>>> exchange any time soon? I for my part would at least publish my
>>> prototypical implementations somewhere (GitHub?), unless they become
>>> obsolete by something "official"
>>>
>>> regards,
>>> Chris
>>>
>>> Zitat von Geoff Wright <gpwright at gmail.com>:
>>>
>>>> Hi Chris,
>>>>
>>>> Sorry for the confusion, yes I am doing what you describe.  The
>>> difference
>>>> may be that perhaps your app has only one viewport.  In my case there
>>> are a
>>>> number of different viewports (aka vtkRenderWindow)  each has its own qml
>>>> item and is each rendering on it's own thread.  Plus the qml composition
>>>> thread which is managed by Qt.
>>>>
>>>> I was just making the point that this architecture let's you draw
>>> different
>>>> viewports in parallel provided that each one has its own mappers and
>>> actors.
>>>>
>>>> G
>>>>
>>>>
>>>> On Tue, May 16, 2017, 4:13 PM <c.sell at byterefinery.de> wrote:
>>>>
>>>>> Hi Geoff,
>>>>>
>>>>> as I understand it, the blog post you linked to doesn't mention
>>>>> multiple threads beyond the QML UI thread and the scenegraph render
>>>>> thread.
>>>>>
>>>>> So far, I am using that approach with QQuickFrameBufferObject as
>>>>> starting point and vtkExternalOpenGLRenderWindow for the VTK side,
>>>>> which causes VTK to use QML's OpenGL context and thus render into the
>>>>> FBO maintained by QQuickFrameBufferObject. That seems to work
>>>>> flawlessly, as long as ALL rendering is kept on the scenegraph thread,
>>>>> which is not always obvious to do at first, because VTK's API's don't
>>>>> separate between reading event values and rendering them (e.g.,
>>>>> SetSlice(2) will internally set the current slice and then do a
>>>>> Render()).
>>>>>
>>>>> But why maintain more threads?
>>>>>
>>>>> Anyway, please go ahead and explain your approach as detailed as
>>>>> possible, I am all ears.
>>>>>
>>>>> regards,
>>>>> Chris
>>>>>
>>>>> Zitat von Geoff Wright <gpwright at gmail.com>:
>>>>>
>>>>> > Hi guys,
>>>>> >
>>>>> > I would argue that for qml integration the best approach is for each
>>> vtk
>>>>> > viewport to render into a texture on a dedicated thread, not the qml
>>>>> > rendering thread.  Qml then draws the resulting texture which is more
>>> in
>>>>> > line with the Qt design intention ( see
>>>>> >
>>>>>
>>> http://blog.qt.io/blog/2015/05/11/integrating-custom-opengl-rendering-with-qt-quick-via-qquickframebufferobject/
>>>>> >  )
>>>>> >
>>>>> > I have been using this approach for a couple of years in a high
>>>>> performance
>>>>> > real time application.  The code is unfortunately proprietary but very
>>>>> > happy to explain more about the approach.  It would be great to add
>>>>> support
>>>>> > for this to VTK.
>>>>> >
>>>>> > For the interactor part I do what Sankesh said, defer the events and
>>>>> apply
>>>>> > them on the render thread of the particular vtk item.
>>>>> >
>>>>> > Regards,
>>>>> >
>>>>> > G
>>>>> >
>>>>> >
>>>>> >
>>>>> > On Tue, May 16, 2017, 3:06 PM Sankhesh Jhaveri <
>>>>> sankhesh.jhaveri at kitware.com>
>>>>> > wrote:
>>>>> >
>>>>> >> Hi Christian,
>>>>> >>
>>>>> >> Doing all VTK rendering on the QML render thread is the right thing
>>> to
>>>>> do.
>>>>> >> As far as interaction is concerned, make sure that Qt events are
>>> queued
>>>>> on
>>>>> >> the main thread and processed when execution reaches the render
>>> thread.
>>>>> >>
>>>>> >> I have done some work on VTK and QtQuick integration that I’m
>>> planning
>>>>> to
>>>>> >> add to a VTK remote module. That way, it will be available as part of
>>>>> VTK.
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Sankhesh
>>>>> >> 
>>>>> >>
>>>>> >> On Tue, May 16, 2017 at 4:57 AM <c.sell at byterefinery.de> wrote:
>>>>> >>
>>>>> >>> just for the record (nobody care about this subject?):
>>>>> >>>
>>>>> >>> I found that contrary to what I said I had NOT adhered to the
>>>>> >>> principle established with protoype 1 while implementing protoype 2.
>>>>> >>> Rather, I had directly called into the VTK interactor from the QML
>>> UI
>>>>> >>> thread. And because the VTK interactors immediately go into
>>> rendering,
>>>>> >>> all hell broke loose.
>>>>> >>>
>>>>> >>> After cleaning up example 2 according to the rule laid out earlier
>>>>> >>> ("store event data in RenderDelegate, call update() on the QML item,
>>>>> >>> pass the event to VTK inside RenderDelgate.Render()", all works
>>> well.
>>>>> >>>
>>>>> >>> What I would like to do is validate the solution against findings
>>> made
>>>>> >>> elsewhere, and then establish the "canonical" way of integrating VTK
>>>>> >>> with QML. As I said, there are some architectural quirks with the
>>>>> >>> solution which I would also like to address.
>>>>> >>>
>>>>> >>> regards,
>>>>> >>> Christian
>>>>> >>>
>>>>> >>>
>>>>> >>> Zitat von c.sell at byterefinery.de:
>>>>> >>>
>>>>> >>> > Hello all,
>>>>> >>> >
>>>>> >>> > I am looking into the possibility of replacing the 3D rendering
>>>>> >>> > engine in my Qt5 / QML based mobile tablet-oriented application
>>> with
>>>>> >>> > VTK. What I need is a first class QtQuick integration (not a hack
>>> or
>>>>> >>> > workaround) that is 100% stable in every context (not an unusual
>>>>> >>> > requirement, I admit). To my amazement, nothing of that kind seems
>>>>> >>> > to exist (correct me if I'm wrong).
>>>>> >>> >
>>>>> >>> > I went on to investigate what had been done so far and to
>>> implement
>>>>> >>> > my first prototypes, using the QQuickFrameBufferObject approach.
>>>>> >>> > From the very start this felt like an uphill battle, because VTK
>>>>> >>> > seems to come from a Windowing background and is quite tightly
>>>>> >>> > integrated with concepts that are not valid in a QML context.
>>>>> >>> >
>>>>> >>> > I'll describe my findings together with the 2 prototypical QML
>>> items
>>>>> >>> > I implemented:
>>>>> >>> >
>>>>> >>> > 1st was an adaptation of the DICOM example which now runs as a QML
>>>>> >>> > item. All user interaction is handled by QML and forwarded to VTK
>>>>> >>> > (which is one thing that doesn't come naturally with VTK), and
>>> after
>>>>> >>> > some non-elegant tweaking I was able to have the UI move from
>>> slice
>>>>> >>> > to slice and re-render upon mouse wheel events as expected. The
>>>>> >>> > problem here was, that QML insists on keeping mouse event handling
>>>>> >>> > and OpenGL rendering on separate threads, with one "rendering
>>>>> >>> > thread" dedicated to OpenGL. However, the pre-existing VTK
>>>>> >>> > Interactors directly call Render() after reconfiguring the UI from
>>>>> >>> > the mouse event data, which is an absolute QML No-Go. I had to
>>>>> >>> > introduce a RenderDelegate that works somewhat like this:
>>>>> >>> >
>>>>> >>> > QML mouse event:
>>>>> >>> >     tell RenderDelegate about the event
>>>>> >>> >     call update() on the QML item, which triggers rer-endering on
>>>>> >>> > the dedicated thread
>>>>> >>> > on the QML rendering thread:
>>>>> >>> >     call Render() on the RenderWindow
>>>>> >>> >     inside the RenderDelegate, look at whether a mouse event is
>>>>> >>> > pending, and call the corresponding VTK mouse handler
>>>>> >>> >     call the renderer's default Render() function while setting
>>> the
>>>>> >>> > delegate's "Used" flag to false
>>>>> >>> >
>>>>> >>> > 2nd was a QML item that displays a model loaded from a 3ds file.
>>> My
>>>>> >>> > goal here was to move the model around using mouse drag events. I
>>>>> >>> > took the lessons learned from the first example, threw in a
>>>>> >>> > vtkInteractorStyleTrackballCamera and hoped for the best. First
>>>>> >>> > thing I found was that I needed a specialized
>>>>> >>> > RenderWindowInteractor, which I provided. When I realized that the
>>>>> >>> > most important requirement for that class was to provide access to
>>>>> >>> > timer functionality, I already got wary, as throwing around events
>>>>> >>> > and doing stuff offline while on the QML rendering thread is
>>> never a
>>>>> >>> > good idea. My fears came true when I finished wiring everything
>>>>> >>> > together: I was able to move the model using the mouse, but after
>>> a
>>>>> >>> > few moments things got whacky with error output written to the
>>>>> >>> > console and finally a segemtation fault from deep inside VTK.
>>>>> >>> >
>>>>> >>> > I still need to investigate the second example, but would like to
>>>>> >>> > synchronize with the community at this point in order to avoid
>>>>> >>> > errors/misconceptions on my side, to seek help if possible and to
>>>>> >>> > offer my contribution to the VTK code base once this becomes
>>>>> >>> > functional. My first impression is that there are some issues that
>>>>> >>> > lie on an architectural level and cannot be (elegantly) dealt with
>>>>> >>> > on the QML side alone. Any comments?
>>>>> >>> >
>>>>> >>> > thanks,
>>>>> >>> > Christian Sell
>>>>> >>> >
>>>>> >>> > _______________________________________________
>>>>> >>> > Powered by www.kitware.com
>>>>> >>> >
>>>>> >>> > Visit other Kitware open-source projects at
>>>>> >>> > http://www.kitware.com/opensource/opensource.html
>>>>> >>> >
>>>>> >>> > Search the list archives at:
>>>>> >>> http://markmail.org/search/?q=vtk-developers
>>>>> >>> >
>>>>> >>> > Follow this link to subscribe/unsubscribe:
>>>>> >>> > http://public.kitware.com/mailman/listinfo/vtk-developers
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> _______________________________________________
>>>>> >>> Powered by www.kitware.com
>>>>> >>>
>>>>> >>> Visit other Kitware open-source projects at
>>>>> >>> http://www.kitware.com/opensource/opensource.html
>>>>> >>>
>>>>> >>> Search the list archives at:
>>>>> http://markmail.org/search/?q=vtk-developers
>>>>> >>>
>>>>> >>> Follow this link to subscribe/unsubscribe:
>>>>> >>> http://public.kitware.com/mailman/listinfo/vtk-developers
>>>>> >>>
>>>>> >>> --
>>>>> >> Sankhesh Jhaveri *Sr. Research & Development Engineer* | Kitware
>>>>> >> <http://www.kitware.com/> | (518) 881-4417
>>>>> >> 
>>>>> >> _______________________________________________
>>>>> >> Powered by www.kitware.com
>>>>> >>
>>>>> >> Visit other Kitware open-source projects at
>>>>> >> http://www.kitware.com/opensource/opensource.html
>>>>> >>
>>>>> >> Search the list archives at:
>>>>> http://markmail.org/search/?q=vtk-developers
>>>>> >>
>>>>> >> Follow this link to subscribe/unsubscribe:
>>>>> >> http://public.kitware.com/mailman/listinfo/vtk-developers
>>>>> >>
>>>>> >>
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
>>>
>
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at  
> http://www.kitware.com/opensource/opensource.html
>
> Search the list archives at: http://markmail.org/search/?q=vtk-developers
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/vtk-developers