[Rtk-users] rtkForwardProjections Segmentation fault...

[Steven Pollmann]

Hello Simon, 

Thanks for the update.  Definitely a large dataset, but nowhere near the largest we can generate on our high-res preclinical scanners.  I'm happy to report that the forward projection application no longer results in a segmentation fault, but rather a "Description: itk::ERROR: CUDA ERROR: out of memory" error.  This still puzzles me a little bit as the 1024x1024x680 Volume in memory (assuming 32bit float) is only 2.8GB, and all 1000 projections would also be around 2.8GB as well(1024x680 x1000projections) (5.6GB total, leaving around 400MB left over), but at least now it reports a Memory error rather than a seg fault.  

I've also tried forward projection through only 1 angle to see what the projection would look like (I've tried this before, but it always complained of memory issues.  Not sure why it works now.  Must have been on the old video card?).  Adding the --steps flag is definitely helpful, and (without doing a more rigorous test) I would have to say I can get the forward projection to look equivalent to the CPU forward projections, so thanks for that change!.  I am trying to understand what is meant by "steps", and in what units they would be in.  Is this the distance between sampling points in the ray going through the volume?  With it originally set to 1, was that interpreted as "1mm?".

Anyhow, I would definitely be interested in a streaming version of the forward projector.  It would definitely be useful with our extremely large datasets.

Thanks again for the help!

Steve  

________________________________________
From: simon.rit at gmail.com [simon.rit at gmail.com] On Behalf Of Simon Rit [simon.rit at creatis.insa-lyon.fr]
Sent: July 21, 2014 9:38 AM
To: Steven Pollmann
Cc: rtk-users at openrtk.org
Subject: Re: [Rtk-users] rtkForwardProjections Segmentation fault...

Hi Steven,
Big dataset. I unfortunately do not have a 6 GB card at hand to test it, the maximum I have is a 3 GB card. I still believe it is a simple memory issue. I noticed that there were some checks missing so I have added them<https://github.com/SimonRit/RTK/commit/6e040cbaf77a3105871f3f42721ef426806d000b>. Could you try again with the attached script instead of the one you have sent? It does only one projection which will require less memory and it works on my 3 GB card with half your volume. With the full volume, it fails but the new checks describe the memory issue as expected. There have been some changes in the management of the memory indeed and it will now create the full stack in the GPU memory with the example you sent. Streaming should be possible, we can work on it after you confirm it is a memory issue if you want.
About the aliasing, I noticed that your voxels are finely resolved. I have tried to change the hard-coded value "float t_step = 1" to 0.1 in rtkCudaForwardProjectionImageFilter.cxx and it removes them. I have now made that configurable<https://github.com/SimonRit/RTK/commit/fb62489657397df2c9f6d1d1bf0711bbc8fd7673> through the option --step of rtkforwardprojections.
Simon


On Thu, Jul 17, 2014 at 11:09 PM, Steven Pollmann <spollmann at robarts.ca<mailto:spollmann at robarts.ca>> wrote:
Hello RTK Users,

We have noticed some problems with RTK's forward projection software (rtkForwardProjections) using CUDA.  Originally, using a build we have been using from July 2013 (v1,0 rc3, commit ID 2dabe6a04cf33c0da45907efdf03d427b4473adc dated July 03, 2013), GPU Forward Projection would complete, but would leave some sort of aliasing artifacts in the projections that were not present when rtkForwardProjections was run using only the CPU methods (Joseph method, or RayCastInterpolator, changing --method commandline option, at the time).

The next time we tried using a newer build of openRTK to perform some forward projections using the GPU/CUDA was later in the year.  However, I believe some CUDA memory management changes were made, and a forward projection could no longer be attempted due to running out of memory (and rtkForwardProjections being unable to chunk. or split up the volume to forward project).  (Our GPU at the time was an Nvidia GTX 770 with 4GB vram).

More recently, we acquired an Nvidia GTX Titan Black with 6GB vram, running driver version 337.25 on Xubuntu 14.04, and I attempted a forward projection using the latest OPENRTK (v1.0.0, July 10, 2014 commit 8646fbdc3b01b439ae37c1d7084e2e68d155c3a5).  We no longer get the "out of memory" errors (this build still causes the GTX770 to complain of memory), but rather a Segmentation Fault when trying to do the forward projection.  The Segmentation Fault occurs after it has read in the 3D Volume, when it is "Projection Volume...".  The failure is immediate.  Again, if using just the CPU to do the forward projections, everything seems to work fine.

Anyhow, I was wondering if anyone had any insight into this issue?

Our scanners produce fairly large volumes (1024x1024x680) that we are interested in performing some iterative reconstructions on.

I've uploaded a sample dataset (a Rubiks Cube volume) onto Google drive at the following link.  Due to the rtkForwardProjections program being unable to "chunk", or "divide" the dataset, I think you would need a substantial amount of video memory (6GB or more) to test this dataset, and experience the Segmentation Fault.

https://drive.google.com/folderview?id=0B2rwn1Vax7UydVhudHlSZGlXVDQ&usp=sharing

There, you should be able to download the Rubiks MHD and Raw Volume file, the Geometry File, and a script file containing the forward projection commands we are using.

Let me know if any other information would be useful.

Thanks!

Steve
_______________________________________________
Rtk-users mailing list
Rtk-users at public.kitware.com<mailto:Rtk-users at public.kitware.com>
http://public.kitware.com/mailman/listinfo/rtk-users