[Imstk-developers] high-speed thread in iMSTK

Fri Jun 23 16:13:11 EDT 2017

I can reproduce the synchronization issue in a Sandbox example (I don't see poor performance, but you can see various pieces of the mesh updating at different times):

testGeometryTransforms()

It looks like the vertices "jump," which might be caused by the physics/render thread interaction. The performance is probably tied to the physics thread though. This demo doesn't have any physics involved, which delays the mesh modification operations, making it easier for race conditions to occur. We need to talk about this next meeting because this is a serious issue.
________________________________
From: Imstk-developers [imstk-developers-bounces at imstk.org] on behalf of Milef, Nicholas Boris [milefn at rpi.edu]
Sent: Thursday, June 22, 2017 4:35 PM
To: Dzenan Zukic
Cc: Zhaohui Xia; imstk-developers at imstk.org
Subject: Re: [Imstk-developers] high-speed thread in iMSTK

That would lock the render thread though. That's unacceptable in VR simulations because we have to maintain a minimum of 90 FPS, otherwise it causes motion sickness. Even if this wasn't a problem, it causes too much latency, so feedback, whether it's haptic, visual, or something else, will be delayed. Plus, it ends up wasted too much thread time.

One way this could work is if we do double/triple buffering of mesh data, but that presents some other issues (multiple threads contending for mesh data at once, larger memory requirements, etc.).
________________________________
From: Dzenan Zukic [dzenan.zukic at kitware.com]
Sent: Thursday, June 22, 2017 4:26 PM
To: Milef, Nicholas Boris
Cc: Zhaohui Xia; imstk-developers at imstk.org
Subject: Re: [Imstk-developers] high-speed thread in iMSTK

"use trylock and come back later to the mesh which was already locked" - what I mean by this was to first do trylock on all the meshes a thread needs to update, and then do regular lock (which will put the thread into the wait queue) for each mesh for which trylock failed. Once the other thread releases the lock, this thread will awaken and start updating the mesh.

Regards

Dženan Zukić, PhD, Senior R&D Engineer, Kitware (Carrboro, N.C.)

On Thu, Jun 22, 2017 at 4:00 PM, Milef, Nicholas Boris <milefn at rpi.edu<mailto:milefn at rpi.edu>> wrote:
The problem with that is that if one thread runs faster than another, then it might not be updated on a certain thread. For instance, if the physics thread keep acquiring the mesh data, then the mesh data will never update on the render thread (or vice-versa).

What if we used a job-based approach? That way we would only need to lock the job submission and not the mesh data. I'm trying to think ahead for when we move normal computation or mesh interpolation to another thread (more times for contention). Logging is already on another thread, so that adds even more contention if you try to print mesh data.

http://seanmiddleditch.com/multi-threaded-game-engines/
________________________________
From: Dzenan Zukic [dzenan.zukic at kitware.com<mailto:dzenan.zukic at kitware.com>]
Sent: Thursday, June 22, 2017 3:50 PM
To: Milef, Nicholas Boris
Cc: Zhaohui Xia; imstk-developers at imstk.org<mailto:imstk-developers at imstk.org>

Subject: Re: [Imstk-developers] high-speed thread in iMSTK

Each mesh could have its own mutex, which needs to be obtained by the thread which wants to write it. Or if a thread can do something else, use trylock and come back later to the mesh which was already locked.

Regards

Dženan Zukić, PhD, Senior R&D Engineer, Kitware (Carrboro, N.C.)

On Thu, Jun 22, 2017 at 3:45 PM, Milef, Nicholas Boris <milefn at rpi.edu<mailto:milefn at rpi.edu>> wrote:
Ok, after talking to Zhaohui and profiling, it appears to be a thread synchronization issue (which is why it only shows up in debug mode).

This is the problem:
The render thread reads the mesh data. If the transform hasn't been applied, it gets applied.
The physics thread writes the mesh data. If the transform hasn't been applied, it gets applied.

The problem is that the render thread continually modifies the data as it's reading it. Meanwhile, the physics thread is setting data very quickly (or vice-versa, depending who wins the race condition). We saw earlier that the physics thread was less than 1 FPS, but now the render thread is less than 1 FPS. We don't synchronize between threads (which is another problem), so this is what's causing this.

Any ideas on how to proceed?
________________________________
From: Imstk-developers [imstk-developers-bounces at imstk.org<mailto:imstk-developers-bounces at imstk.org>] on behalf of Dzenan Zukic [dzenan.zukic at kitware.com<mailto:dzenan.zukic at kitware.com>]
Sent: Friday, June 09, 2017 11:53 AM
To: Zhaohui Xia

Cc: imstk-developers at imstk.org<mailto:imstk-developers at imstk.org>
Subject: Re: [Imstk-developers] high-speed thread in iMSTK

Hi Zhaohui,

something like this<https://stackoverflow.com/questions/41264827/setting-optimization-settings-in-visual-studio-through-cmake> should work, you need to modify debug mode instead of release though.

Regards

Dženan Zukić, PhD, Senior R&D Engineer, Kitware (Carrboro, N.C.)

On Fri, Jun 9, 2017 at 11:50 AM, Zhaohui Xia <unihui at gmail.com<mailto:unihui at gmail.com>> wrote:
Hi Dzenan,

Thanks for your advice. We need to set it for all the projects in the solution, is this correct?
Can we change it in the .CMakeList file to change the whole settings in an easy way?

Thanks,
Zhaohui
From: Dzenan Zukic [mailto:dzenan.zukic at kitware.com<mailto:dzenan.zukic at kitware.com>]
Sent: 2017年6月9日 8:57
To: Zhaohui Xia <unihui at gmail.com<mailto:unihui at gmail.com>>
Cc: Andinet Enquobahrie <andinet.enqu at kitware.com<mailto:andinet.enqu at kitware.com>>; imstk-developers at imstk.org<mailto:imstk-developers at imstk.org>

Subject: Re: [Imstk-developers] high-speed thread in iMSTK

Hi Zhaohui,

to enable a somewhat higher speed of debug mode, you can manually change the compiler settings for individual projects, see this<https://msdn.microsoft.com/en-us/library/8f8h5cxt.aspx> and this<https://msdn.microsoft.com/en-us/library/k1ack8f1.aspx>. Enabling optimizations increases difficulty of debugging, so you might enable more optimizations for the Visual Studio projects you are not trying to debug, and keep optimization disabled or at a low setting for the ones you are modifying.

Regards

Dženan Zukić, PhD, Senior R&D Engineer, Kitware (Carrboro, N.C.)

On Thu, Jun 8, 2017 at 3:33 PM, Zhaohui Xia <unihui at gmail.com<mailto:unihui at gmail.com>> wrote:
Hi Andinet,

I am afraid I made a mistake. The device thread is definitely a high-speed thread. I figure out the issue, because I forget to close something when I change the code in that thread.

I think there is no performance bottlenecks currently in the main thread under Release mode (60-70HZ), since the mesh size (1000 vertices) is not very big now.
But under Debug mode, the speed is too slow, it is just 0.16HZ for the same mesh. Do we have any ways to improve it, e.g. changing some settings? It would be very helpful for work if it can run faster.

Thanks,
Zhaohui
From: Andinet Enquobahrie [mailto:andinet.enqu at kitware.com<mailto:andinet.enqu at kitware.com>]
Sent: 2017年6月8日 13:29
To: Zhaohui Xia <unihui at gmail.com<mailto:unihui at gmail.com>>
Cc: imstk-developers at imstk.org<mailto:imstk-developers at imstk.org>
Subject: Re: [Imstk-developers] high-speed thread in iMSTK

Zhaohui

Can you profile your code and let us know the key performance bottlenecks in iMSTK?

This is something Dr. Des is also interested. We will be exploring ways of accelerating some of the modules.

thanks
Andinet

On Thu, Jun 8, 2017 at 12:40 PM, Zhaohui Xia <unihui at gmail.com<mailto:unihui at gmail.com>> wrote:
Hi All,

Do we have a high-speed thread (e.g. 1000HZ) in iMSTK? I tried different modules e.g VRPNDeviceServer (device) and SceneMange (main thread), but the maximum frame rates of them are 60-70HZ under Release mode.
I also tried to set the Loopdelay to 0.001 or smaller (from my understand, its default value is 0 should have the fastest speed, but it also 60-70HZ).

Thanks,
Zhaohui
-----------------------------------------------------------------------------------------------------------
Zhaohui Xia, Ph.D.
Postdoctoral Research Associate, Center for Modeling, Simulation and Imaging in Medicine (CeMSIM)
Rensselaer Polytechnic Institute

_______________________________________________
Imstk-developers mailing list
Imstk-developers at imstk.org<mailto:Imstk-developers at imstk.org>
http://public.kitware.com/mailman/listinfo/imstk-developers

--
Andinet Enquobahrie, Ph.D., MBA
Director of Medical Computing
Kitware, Inc.

http://www.kitware.com
(919) 969-6990 x311<tel:%28919%29%20969-6990%20x311>

_______________________________________________
Imstk-developers mailing list
Imstk-developers at imstk.org<mailto:Imstk-developers at imstk.org>
http://public.kitware.com/mailman/listinfo/imstk-developers

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/imstk-developers/attachments/20170623/ab39a379/attachment-0001.html>