[Paraview] Big data set very slow to load (over 500Mb, Gb...)

[Joachim Pouderoux]

Hi Cindy!

Could you please give us more information about the type of dataset you are
working with (structured or unstructured, multiblock or not) and the total
number of points & cells of those models?
Is your input stored in a paralle data format (eg.PVT*, xdmf, hdf5 etc)?
If not, and data is unstructured, there is pretty good chances that all the
data will be loaded only on the first rank of your server and then
processing only performed with real data on this rank (other will just pass
through empty dataset). You can check if data is distributed by applying
the "Process Id Scalars" and color the grid with this field, it will show
data partitionning over the processors (1 color per rank). If it is all
blue, data is only on first rank, bad news.
In this last case, you can indeed use D3 to redistribute the data over all
the processors, then all processors (apply Process Id Scalars) should show
better result.
Problem is that D3 takes time to perform and if your data is temporal, it
will be run at every timestep!

Hope it helps!

A+ et bon courage :)

Joachim


*Joachim Pouderoux*, PhD

*Technical Expert - Scientific Computing Team*
*Kitware SAS <http://www.kitware.fr>*


2016-11-15 13:13 GMT-04:00 Cindy Castro <cindy.castro at uvsq.fr>:

>
> Hello,
>>
>> I have the following question: commonly we do have large data set to be
>> run under paraview 5.0 installed over a cluster of 4 computers (64Gb RAM,
>> NVIDIA graphic cards for each, intel xeon CPU) connected to 8 screens
>> (interactive/immersive 3D wall with tracking).
>>
>> For instance  one data is 1.5- 2 Go (fortran files converted in VTK, hdf,
>> netcdf). We do have strong difficultues to manage such a large data.
>> Apparently the loading and the tracking is extremely slow; Any animation is
>> extremely slow too.
>>
>> I tried to use filters like D3 and Decimation which improve but only very
>> slightly the situtation; still it takes some time when applied. Maybe my
>> settings are not optimized?
>>
>> Could someone help me :
>>
>> - to understand why my final configuration is still so slow ?
>>
>> - is there any other solution to improve the situation ?
>>
>> With my anticipated thanks for any help.
>>
>> Sincerely.
>>
>>
>
> ---
> L'absence de virus dans ce courrier électronique a été vérifiée par le
> logiciel antivirus Avast.
> https://www.avast.com/antivirus
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensou
> rce/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at:
> http://paraview.org/Wiki/ParaView
>
> Search the list archives at: http://markmail.org/search/?q=ParaView
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/paraview
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/paraview/attachments/20161124/32b92f59/attachment.html>