[vtkusers] large data sets and memory
L.J. van Ruijven
L.J.vanRuijven at amc.uva.nl
Sat Nov 24 19:43:18 EST 2007
Hi Suzanne,
When I started the pipeline with a datafile of 100 MB I saw a quick
increase of the memory usage with 100 MB. This lasted about 20
seconds. Then it increased again quickly to about 600 MB and quickly
dropped back to about 100MB, after which the scenery appeared on the
screen. Moving the scenery on the screen again caused an increase and
decrease. With larger datafiles I got the memory error during the
second increase at about 1 GB.
I do not think the memory on a graphics card can increase the
available memory for win32. My computer was equipped with 4GB of RAM,
but the limit for one executable was still 3 GB. The amount of
available RAM does not determine this limit, but by the memory manager
of win32.
But if you insist, you might try to run your win32 program on a
computer with Windows XP x64 installed. This increases the maximum
memory available for your win32 executable to 4 GB. But more
importantly, these 4 GB should not be fragmented by the system drivers
that windows loads during startup, because these drivers should be
loaded in the 64 bit memory space. As far as I know this has not been
tested yet.
Leo.
----- Original Message -----
From: Suzanne Little <Suzanne.Little at ibai-institut.de>
Date: Friday, November 23, 2007 8:40 am
Subject: Re: [vtkusers] large data sets and memory
> Thanks Jon, XViz, Leo, others. That's quite helpful.
>
> Leo: You mentioned having problems when your dataset was between
> 100 and
> 200MB. Is this due to having multiple copies of the data generated
> during the pipeline? So the aggregate memory in use by VTK reaches
> 2GB?
> Am I right in thinking that using a specific accelerated graphics
> card
> will only improve the execution speed and responsiveness but
> doesn't
> change the memory issues? Or will on-card memory be added to
> Windows'
> 2GB process memory?
>
> Thanks,
> Suzanne
>
> Jon Johansson wrote:
> > I'm not sure this actually belongs on the VTK Wiki, but I've
> > seen the question come up before so I've added an entry at:
> >
> >
>
http://www.vtk.org/Wiki/VTK_FAQ#How_can_a_user_process_access_more_than
_2_GB> _of_ram_in_32-bit_Windows.3F
> >
> > I hope this helps,
> > Jon.
> >
> >
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > ~~ Jon I Johansson, Ph.D. * Tel.: (780) 492-9304
> ~~
> > ~~ jon.johansson at ualberta.ca * Fax.: (780) 492-1729
> ~~
> > ~~ * Office: G.S.B. 323C
> ~~
> > ~~ Programmer/Analyst http://sciviz.aict.ualberta.ca
> ~~
> > ~~
> ~~
> > ~~ Research Computing Support
> ~~
> > ~~ Room 352, General Services Building
> ~~
> > ~~ Academic Information and Communication Technologies (AICT)
> ~~
> > ~~ University of Alberta
> ~~
> > ~~ Edmonton, Alberta, CANADA, T6G 2H1
> ~~
> >
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > ! "This communication is intended for the use of the recipient
> to !
> > ! which it is addressed, and may contain confidential,
> personal, !
> > ! and/or privileged information. Please contact us
> immediately !
> > ! if you are not the intended recipient of this
> communication. !
> > ! If you are not the intended recipient of this
> communication, !
> > ! do not copy, distribute, or take action on it.
> Any !
> > ! communication received in error, or subsequent reply,
> should !
> > ! be deleted or destroyed."
> !
> > !----------------------------------------------------------------
> --!
> >
> > From: vtkusers-bounces+jon.johansson=ualberta.ca at vtk.org
> > [mailto:vtkusers-bounces+jon.johansson=ualberta.ca at vtk.org] On
> Behalf Of
> > XViz
> > Sent: Wednesday, November 21, 2007 8:30 AM
> > To: vtkusers
> > Subject: Re: [vtkusers] large data sets and memory
> >
> > Hello Thomas,
> >
> > Your problem was disscused many times in this mailing list. One
> process in
> > Windows x86 (NT based 32 bit) cannot use more than 2Gb of
> memory, and the
> > system itself - more than 4Gb (usually even less - near 3.2Gb).
> One possible
> > solution I know is using Windows x64 or some Linux. The other -
> reducing the
> > size of your dataset. :(
> >
> >
> > XViz, D_E at ukr.net
> > 2007-11-21
> > ----- ???????? ????????? -----
> > ??: thomas jetzfellner
> > ??????????: vtkusers
> > ?????: 2007-11-21, 16:17:07
> > ????: [vtkusers] large data sets and memory
> >
> > hi,
> >
> > this is the first time, I write to the list and I think my
> question
> > could sound a little bit "stupid". my current problem is, when i
> load
> > large datasets there is a problem with the memory allocation. my
> > programm crashes on the update call of the vtkImageData. I tried
> it in
> > different ways and none worked on large data sets. small data is
> > processed correctly. Attached you find some code snippeds, on
> the update
> > call my program dies.
> >
> > vtkImageData* volume= new vtkImageData ;
> > volume->SetInput( centerImage->GetOutput() ) ;
> > volume->Update();
> >
> > vtkImageDataStreamer *ids = vtkImageDataStreamer::New();
> > ids->SetInputConnection(centerImage->GetOutputPort());
> > ids->SetNumberOfStreamDivisions(200);
> > ids->UpdateInformation();
> > ids->GetExtentTranslator()->SetSplitModeToBlock();
> > ids->Update();
> >
> > vtkMemoryLimitImageDataStreamer* mlds =
> > vtkMemoryLimitImageDataStreamer::New();
> > mlds->SetInputConnection( centerImage->GetOutputPort() ) ;
> > mlds->SetMemoryLimit ( 10000 ) ;
> > mlds->Update();
> >
> > mybe it is a problem in my concept. i want to load a dataset
> with a
> > resolution of 512 x 512 x 1024, are there other ways to handle
> such a
> > load of data?
> > my dev environment is visual studio 2005 on winXP SP2
> >
> > thanks for any information and suggestions
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ljvanruijven.vcf
Type: text/x-vcard
Size: 262 bytes
Desc: Card for "L.J. van Ruijven" <L.J.vanRuijven at amc.uva.nl>
URL: <http://www.vtk.org/pipermail/vtkusers/attachments/20071125/c2e36e81/attachment.vcf>
More information about the vtkusers
mailing list