[Paraview] Convert Large CGNS Dataset to numpy

[Cory Quammen]

For file sizes in the several GB range, you'll definitely want to use
a 64bit version, particularly on Windows.

Cory

On Mon, Dec 5, 2016 at 1:34 PM, Thomas PEDOT <thomas.pedot at gmail.com> wrote:
> Thanks.
>
> So it might come from something else. Python crashed with a 1.4gb file and
> not with a smaller one.
>
> I'm on windows with a 32bits version of python and launching it with
> pycharm. I will test on a Linux cluster and/or with a 64bits version.
>
> Thanks for your fast and clear answer !
> Thomas
>
>
> Le lun. 5 déc. 2016 17:49, Cory Quammen <cory.quammen at kitware.com> a écrit :
>>
>> On Mon, Dec 5, 2016 at 11:04 AM, Thomas PEDOT <thomas.pedot at gmail.com>
>> wrote:
>> > Hi,
>> > I have some CGNS files I want to "convert" to numpy array. To write my
>> > script I first use a more direct method :
>> >
>> > Converting CGNS to VTK
>> > Use vtk_to_numpy method
>> > process numpy array
>> >
>> > This task is time consuming and I need a more direct option (if
>> > possible).
>> >
>> > For this, I tried to use paraview CGNSReader and simple module.
>> >
>> > I'm able to convert CGNS data to numpy with small data set (Some MB)
>> > with
>> > this approach :
>> >
>> > Case = CGNSSeriesReader(filename)
>> > Case.UpdatePipeline()
>> > rawData = servermanager.Fetch(Case)
>> > import vtk.numpy_interface.dataset_adapter as dsa
>> > # Wrap the raw data object to access NumPy friendly API
>> > data = dsa.WrapDataObject(rawData)
>> >
>> > But as it said here : Do not do this if data is large otherwise the
>> > client
>> > will run out of memory.
>> >
>> > So, I can I do this without running out of memory ? I don't need to load
>> > all
>> > Point Arrays at once.
>>
>> That warning is more for if you've got some very large data set loaded
>> into a ParaView server running on a large computing resource. If you
>> call this, it will load all that data onto the client, which may not
>> have enough RAM to hold the very large data set.
>>
>> If you are working locally with a data set that the ParaView client
>> can load, then you shouldn't have any problems with the script above.
>>
>> > Is it possible to select just 'Temperature' for example and only one
>> > block ?
>> > I guess it might be less memory consuming.
>>
>> To select just 'Temperature', you should be able to do something like:
>>
>> Case.PointVariables = ['Temperature']
>>
>> If you need to, you can add an ExtractBlock to your pipeline:
>>
>> extractBlock1 = ExtractBlock(Input=Case)
>> extractBlock1.BlockIndices = [1]
>>
>> I probably wouldn't do the above unless you try your original script
>> and run into out-of-memory errors.
>>
>> HTH,
>> Cory
>>
>>
>> --
>> Cory Quammen
>> Staff R&D Engineer
>> Kitware, Inc.



-- 
Cory Quammen
Staff R&D Engineer
Kitware, Inc.