[Paraview] Fluent files breaking ParaView?

[John Biddiscombe]

Following up on my own post. I'd like to also add that
1) The fluent file format is terrible. Everything is stored in a string 
like object which you have to parse to find out if it is useful, and 
since the length of it is not usually known, if it isn't useful, you 
can't easily skip it, you have to parse for the end of stream sequences.

2) Every node has to read the whole cas/dat file pair in order to find 
the stuff it needs. There are no hints anywhere, and the partition 
tables are stored in the file after the cell/face definitions, meaning 
you have to read the bloody thing twice - or waste memory on the first run.
Anyway, this means running our fluent reader on 16 nodes, each reading a 
1GB file, results in horrendous IO bottleneck. I recently ran an 
animation on our data using many nodes and looping over time, so files 
were read and then later re-read. It took days and days. The reader is 
dreadfully inefficient, its still an order of magnitude better than CVS, 
but the format itself is the main problem.

Interesting that these commercial CFD codes (CFX/Fluent) produce such 
awful files with badly documented APIs/formats and inefficient storage.

I'd choose HDF any day, given the choice.

JB
> Ian
>
> I can't comment much on the fluent reader in paraview/vtk cvs, but I 
> found it to be completely useless on anything we use.
>
> We had a number of simulations that were run on 16-64 processes using 
> fluent, so I created a fluent reader of our own from some code one of 
> my colleagues used with AVS. The main thing is that it parses the 
> Partition table before loading any cells/faces and does not load 
> anything that is not intended for a given processor. (Assuming the 
> original fluent run was done on a multiprocessor system). It also, 
> uses a much simpler memory allocation strategy, and only makes space 
> for the cells/faces that are actually read. If the number of pvserver 
> nodes is less than the number of original partitions in the fluent 
> data then it'll double up on some nodes.
>
> The downside is that I have stripped out all information from the 
> file, except volumes. Boundaries, surfaces, etc etc are all dropped. 
> We load 3D cells and nothing else.
>
> You can build it as a plugin from svn here 
> https://svn.cscs.ch/vtkContrib/trunk/vtkCSCS/vtkFluent/
>
> When loading in parallel, I am able to load models up to 20 million 
> cells (tets+hex+prism) (I do not have anything larger yet),
> NB. when testing on low memory machines, I set #define 
> LIMIT_PARTITIONS in the cxx file to some positive number, this forces 
> us to only load a fraction of the data. make sure this is commented 
> out if you only see part of the data.
>
> If it helps you, then wonderful. If not, I haven't got time to help! 
> (I've had far too many emails for other modules I've made available 
> and I'm fed up with it!).
>
> ttfn
>
> JB
>
>> Hi,
>>
>> I've just built the CVS wavefront (this morning, in fact) of ParaView 
>> on our
>> cluster (8 of dual-core dual-socket opteron, each with 8Gb memory and 
>> NVidia
>> fx4500 GPUs, running RHEL 4, update 6). I launched the latest Windows
>> client, and fed it a FLUENT datafile. I've built the MPI version (using
>> voltaire's stack), and run it across 8 nodes. Each node is launched as a
>> "pvserver". Started up fine, started reading the file...
>>
>> After a long delay, ParaView blew with an "out of memory" error. Oh my
>> goodness, it had tried using ~8Gb of memory on each node. Detail as 
>> follows:
>>
>> ----8<--------8<--------8<----
>> Process id: 0 >> ERROR: In
>> /sge/share/common/ParaView-cvs/ParaView3/Servers/Common/vtkProcessModule.cxx 
>>
>> , line 1281
>> vtkProcessModule (0x38c6d80): Exception: Insufficient memory exception.
>> ----8<--------8<--------8<----
>>
>> The original FLUENT file was (allegedly) ~400k points. Cas file was 
>> 38Mb,
>> dat file was ~300Mb. Also tried another large FLUENT file (47Mb CAS, 
>> 35Mb
>> DAT), similar result:
>>
>> ----8<--------8<--------8<----
>> Process id: 0 >> ERROR: In
>> /sge/share/common/ParaView-cvs/ParaView3/Servers/Common/vtkProcessModule.cxx 
>>
>> , line 1281
>> vtkProcessModule (0x38c6d80): Exception: Insufficient memory exception.
>>
>> terminate called after throwing an instance of 'std::bad_alloc'
>>   what():  St9bad_alloc
>> [mpirund] rank 5 has got signal 6
>> terminate called after throwing an instance of 'std::bad_alloc'
>>   what():  St9bad_alloc
>> ----8<--------8<--------8<----
>>
>> Has anyone else tried large FLUENT datasets? Any advice gratefully
>> appreciated, we're trying to view these files as a favour, so I have 
>> pretty
>> much 0 idea of what's in them until I can actually visualize them...
>>
>> Should I try running several render servers and separate data servers?
>>
>> Thanks!
>>
>> Ian
>>
>> -- 
>> Dr. Ian Grimstead, Senior Research Associate, School of Computer 
>> Science,
>> Cardiff University, 5 The Parade, Roath, Cardiff CF24 3AA, UK. Tel: 
>> +44 (0)
>> 29 2087 9091
>> _______________________________________________
>> ParaView mailing list
>> ParaView at paraview.org
>> http://www.paraview.org/mailman/listinfo/paraview
>>   
>
>


-- 
John Biddiscombe,                            email:biddisco @ cscs.ch
http://www.cscs.ch/about/BJohn.php
CSCS, Swiss National Supercomputing Centre  | Tel:  +41 (91) 610.82.07
Via Cantonale, 6928 Manno, Switzerland      | Fax:  +41 (91) 610.82.82