[vtkusers] different behavior between ascii and binary ensight gold reader
Georg Hammerl
hammerl at lnm.mw.tum.de
Mon Feb 10 11:55:36 EST 2014
Hi Robert,
it is been a while since we had this problem but we are running into it
again. Therefore I would like to reopen the discussion on making use of
the file index in the vtkEnSightGoldBinaryReader. I expect two
advantages by doing so, firstly it is more efficient and secondly, it
can be used to enable changing geometries over time (variable number of
points). At the moment, time steps are skipped (when reading vectors) by
assuming a constant number of points in the mesh over all time steps
which is of course not correct for variable number of points.
I implemented a preliminary version of reading the file index. It works
fine (and is fast :-)) and enables us to visualize changing geometries.
The implementation is not yet in a style that could be included in VTK,
I therefore wanted to get some pointers what is the best way to include
this feature into VTK.
I have thought of the following scenarios:
1)
when reading Scalars, Vectors, Tensors, ... the blocks of data until
timeStep-1 are being skipped one by one assuming a constant geometry. To
skip this we look for a file index and, if present, this->IFile is set
to the correct position in the file.
This has the advantage of being very easy to implement, however it has
to be implemented for every type of field separately. Also one has to
jump to the end of the file for every field in every file whenever a new
time step is requested.
2)
provide a method in the vtkEnSightGoldBinaryReader, that implements the
reading of the file index and setting this->IFile to the correct
position (for example a SkipTimestepsUsingFileindex(int timestep) ) or
that returns the address(es) of a(ll) time step(s), which can then be
used in the Read{Scalars,Vectors,..}.. methods.
This way only one implementation that actually reads the file index is
needed, but it still requires the file index to be read for every field
in every file whenever a new time step is requested.
3)
Read the file index of all files when the reader is initialized and
store the timestep positions per file in a member of the
vtkEnSightGoldBinaryReader. This way the index has to be read only once
and timesteps can be skipped by referring to the member. Drawback of
this method is, that I need some hints on how this could be implemented
consistent with the rest of the reader (i.e. map<std::string,
std::vectors<long> > vs. string[] and long[][] or anything in between).
There are some lines of code attached that show the proof of concept
implementation. I am looking forward to getting some suggestions.
Cheers,
Georg
On 19/04/13 14:51, Robert Maynard wrote:
> The file index is required by Ensight for all transient single file
> formats. The documentation also states that Ensight automatically
> writes out the file index information, so I would consider any
> transient single file format that doesn't have it be malformed.
>
>
> On Fri, Apr 19, 2013 at 7:57 AM, Georg Hammerl <hammerl at lnm.mw.tum.de
> <mailto:hammerl at lnm.mw.tum.de>> wrote:
>
> Hi Robert,
>
> thanks for the hint with the file index. The ensight gold files I
> have available come with a file index. I have tried to use it in
> Paraview, however it seems that the
> VTK/IO/Ensight/vtkEnSightGoldBinaryReader does not make use of the
> index.
> Unfortunately, I do not have enough insight into the inner
> workings of VTK, so I cannot see myself creating a clean
> implementation for using the file index.
>
> As the file index is just optional, I think it should be worth
> implementing an efficient way of reading ensight gold format even
> without the file index.
>
> Cheers,
>
> Georg
>
>
>
> On 18/04/13 20:06, Robert Maynard wrote:
>> Hi Georg,
>>
>> You are correct that a more efficient solution is possible. After
>> reading the Ensight Gold binary format (
>> http://www-vis.lbl.gov/NERSC/Software/ensight/doc/OnlineHelp/UM-C11.pdf
>> ) I have discovered how you can do this more efficiently.
>>
>> If you are start at 11-37 you will see that they state:
>>
>> "Note 3: Efficient reading of each file (especially binary)
>> is facilitated by
>> appending each file with a file index. A file index contains
>> appropriate
>> information to access the file byte positions of each time
>> step in the file. (EnSight
>> automatically appends a file index to each file when
>> exporting in transient single
>> file format.) If used, the file index must follow the last
>> END TIME STEP
>> wrapper in each file."
>>
>> File Index Usage:
>> ASCII, Binary, Item, Description
>> “%20d\n”, sizeof(int), n, Total number of data time steps in
>> the file.
>> “%20d\n”, sizeof(long), fb1, File byte loc for contents of
>> 1st time step*
>> “%20d\n”, sizeof(long), fb2, File byte loc for contents of
>> 2nd time step*
>> . . ., . . ., . . ., . . .
>> “%20d\n”, sizeof(long), fbn, File byte loc for contents of
>> nth time step*
>> “%20d\n”, sizeof(int), flag, Miscellaneous flag (= 0 for now)
>> “%20d\n”, sizeof(long), fb, of item n File byte loc for Item
>> n above
>> “%s\n”, sizeof(char)*80, “FILE_INDEX”, File index keyword
>> * Each file byte location is the first byte that follows the
>> BEGIN TIME STEP record
>>
>>
>> To me the best way is to read the file in reverse looking for
>> FILE_INDEX as the last 80 characters. From that you parse the
>> previous long as
>> the new seek position which jumps you to the start of the file
>> index records. After reading the file index records you have a
>> lookup table for all seek positions of the time steps in that
>> ensight file.
>>
>>
>> On Wed, Mar 6, 2013 at 4:23 AM, Georg Hammerl
>> <hammerl at lnm.mw.tum.de <mailto:hammerl at lnm.mw.tum.de>> wrote:
>>
>> Hello,
>>
>> I have already posted this issue on the paraview mailing
>> list, but then I realized that it is more of a VTK problem.
>> I can load the first time step of my results but when I
>> switch to the next step (in which 4 points are added),
>> paraview freezes and htop shows 100% load for this process.
>> Surprisingly, this only happens when I use binary ensight
>> gold format. The same results written in ascii ensight gold
>> format work. The ens_checker tells me for both cases that I
>> have valid output files.
>>
>> Meanwhile I had a look into the source code and discovered
>> the issue. Whenever the variable-files are read for a time
>> steps greater than 1, Paraview has to jump in the binary file
>> to the desired timestep. This is done by partially parsing
>> and skipping blocks until the correct "BEGIN TIME STEP" is
>> found. In order to skip the correct block length, Paraview
>> uses the number of points (numPts) from the current geometry
>> step. This breaks with a varying number of points per timestep.
>> As an example my first step contains 2 points and the second
>> step 6 points, therefore when Paraview skips the first time
>> step, it skips a block length equivalent to 6 points. The
>> correct number would be 2 points. It therefore skips past the
>> next "BEGIN TIME STEP".
>>
>> In order to verify my theory I have implemented a little hack
>> which looks for the next "BEGIN TIME STEP" by brute force. In
>> case someone wants to try this I have attached a patch which
>> can be used in conjunction with the attached examples.
>>
>> A clean solution would be to skip the correct number of
>> points for every given timestep. Unfortunately I do not have
>> enough insight into the VTK ensight reader to implement this
>> and would appreciate any help in finding a clean solution to
>> this.
>>
>> Cheers,
>>
>> Georg
>>
>> --
>> Dipl.-Ing. Georg Hammerl
>> Lehrstuhl für Numerische Mechanik
>> Technische Universität München
>> Boltzmannstrasse 15, D-85747 Garching b. München
>> phone +49 89 289 15237 <tel:%2B49%2089%20289%2015237>
>> fax +49 89 289 15301 <tel:%2B49%2089%20289%2015301>
>> hammerl at lnm.mw.tum.de <mailto:hammerl at lnm.mw.tum.de>
>> http://www.lnm.mw.tum.de
>>
>>
>> _______________________________________________
>> Powered by www.kitware.com <http://www.kitware.com>
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Please keep messages on-topic and check the VTK FAQ at:
>> http://www.vtk.org/Wiki/VTK_FAQ
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.vtk.org/mailman/listinfo/vtkusers
>>
>>
>>
>>
>> --
>> Robert Maynard
>
>
> --
> Dipl.-Ing. Georg Hammerl
> Lehrstuhl für Numerische Mechanik
> Technische Universität München
> Boltzmannstrasse 15, D-85747 Garching b. München
> phone+49 89 289 15237 <tel:%2B49%2089%20289%2015237>
> fax+49 89 289 15301 <tel:%2B49%2089%20289%2015301>
> hammerl at lnm.mw.tum.de <mailto:hammerl at lnm.mw.tum.de>
> http://www.lnm.mw.tum.de
>
>
>
>
> --
> Robert Maynard
--
Dipl.-Ing. Georg Hammerl
Lehrstuhl für Numerische Mechanik
Technische Universität München
Boltzmannstrasse 15, D-85747 Garching b. München
phone +49 89 289 15237
fax +49 89 289 15301
hammerl at lnm.mw.tum.de
http://www.lnm.mw.tum.de
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.vtk.org/pipermail/vtkusers/attachments/20140210/b81bbb9d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: file_index.patch
Type: text/x-patch
Size: 1493 bytes
Desc: not available
URL: <http://www.vtk.org/pipermail/vtkusers/attachments/20140210/b81bbb9d/attachment.bin>
More information about the vtkusers
mailing list