[vtkusers] different behavior between ascii and binary ensight gold reader
Georg Hammerl
hammerl at lnm.mw.tum.de
Mon Mar 10 09:33:16 EDT 2014
Hi Robert,
meanwhile we worked out an implementation solving the problem.
After discovering that the EnSightGoldReader (ASCII-version) uses a
caching system for the time steps we ported this in a slightly modified
way to the EnSightGoldBinaryReader. In addition we added functionality
to fill the cache in advance in case a file index is available.
This circumvents the problem with changing geometries.
Attached are two patch files, the first one implements the caching for
the binary reader and the second one implements the use of the file
index. It would be great if this could be included in VTK. If you find
any problems with the implementation, please let us know so we can find
a solution.
Thank you very much in advance
Karl & Georg
On 10/02/14 17:55, Georg Hammerl wrote:
> Hi Robert,
>
> it is been a while since we had this problem but we are running into
> it again. Therefore I would like to reopen the discussion on making
> use of the file index in the vtkEnSightGoldBinaryReader. I expect two
> advantages by doing so, firstly it is more efficient and secondly, it
> can be used to enable changing geometries over time (variable number
> of points). At the moment, time steps are skipped (when reading
> vectors) by assuming a constant number of points in the mesh over all
> time steps which is of course not correct for variable number of points.
>
> I implemented a preliminary version of reading the file index. It
> works fine (and is fast :-)) and enables us to visualize changing
> geometries. The implementation is not yet in a style that could be
> included in VTK, I therefore wanted to get some pointers what is the
> best way to include this feature into VTK.
>
> I have thought of the following scenarios:
>
> 1)
> when reading Scalars, Vectors, Tensors, ... the blocks of data until
> timeStep-1 are being skipped one by one assuming a constant geometry.
> To skip this we look for a file index and, if present, this->IFile is
> set to the correct position in the file.
> This has the advantage of being very easy to implement, however it has
> to be implemented for every type of field separately. Also one has to
> jump to the end of the file for every field in every file whenever a
> new time step is requested.
>
> 2)
> provide a method in the vtkEnSightGoldBinaryReader, that implements
> the reading of the file index and setting this->IFile to the correct
> position (for example a SkipTimestepsUsingFileindex(int timestep) ) or
> that returns the address(es) of a(ll) time step(s), which can then be
> used in the Read{Scalars,Vectors,..}.. methods.
> This way only one implementation that actually reads the file index is
> needed, but it still requires the file index to be read for every
> field in every file whenever a new time step is requested.
>
> 3)
> Read the file index of all files when the reader is initialized and
> store the timestep positions per file in a member of the
> vtkEnSightGoldBinaryReader. This way the index has to be read only
> once and timesteps can be skipped by referring to the member. Drawback
> of this method is, that I need some hints on how this could be
> implemented consistent with the rest of the reader (i.e.
> map<std::string, std::vectors<long> > vs. string[] and long[][] or
> anything in between).
>
> There are some lines of code attached that show the proof of concept
> implementation. I am looking forward to getting some suggestions.
>
> Cheers,
>
> Georg
>
>
> On 19/04/13 14:51, Robert Maynard wrote:
>> The file index is required by Ensight for all transient single file
>> formats. The documentation also states that Ensight automatically
>> writes out the file index information, so I would consider any
>> transient single file format that doesn't have it be malformed.
>>
>>
>> On Fri, Apr 19, 2013 at 7:57 AM, Georg Hammerl <hammerl at lnm.mw.tum.de
>> <mailto:hammerl at lnm.mw.tum.de>> wrote:
>>
>> Hi Robert,
>>
>> thanks for the hint with the file index. The ensight gold files I
>> have available come with a file index. I have tried to use it in
>> Paraview, however it seems that the
>> VTK/IO/Ensight/vtkEnSightGoldBinaryReader does not make use of
>> the index.
>> Unfortunately, I do not have enough insight into the inner
>> workings of VTK, so I cannot see myself creating a clean
>> implementation for using the file index.
>>
>> As the file index is just optional, I think it should be worth
>> implementing an efficient way of reading ensight gold format even
>> without the file index.
>>
>> Cheers,
>>
>> Georg
>>
>>
>>
>> On 18/04/13 20:06, Robert Maynard wrote:
>>> Hi Georg,
>>>
>>> You are correct that a more efficient solution is possible.
>>> After reading the Ensight Gold binary format (
>>> http://www-vis.lbl.gov/NERSC/Software/ensight/doc/OnlineHelp/UM-C11.pdf
>>> ) I have discovered how you can do this more efficiently.
>>>
>>> If you are start at 11-37 you will see that they state:
>>>
>>> "Note 3: Efficient reading of each file (especially binary)
>>> is facilitated by
>>> appending each file with a file index. A file index contains
>>> appropriate
>>> information to access the file byte positions of each time
>>> step in the file. (EnSight
>>> automatically appends a file index to each file when
>>> exporting in transient single
>>> file format.) If used, the file index must follow the last
>>> END TIME STEP
>>> wrapper in each file."
>>>
>>> File Index Usage:
>>> ASCII, Binary, Item, Description
>>> “%20d\n”, sizeof(int), n, Total number of data time steps in
>>> the file.
>>> “%20d\n”, sizeof(long), fb1, File byte loc for contents of
>>> 1st time step*
>>> “%20d\n”, sizeof(long), fb2, File byte loc for contents of
>>> 2nd time step*
>>> . . ., . . ., . . ., . . .
>>> “%20d\n”, sizeof(long), fbn, File byte loc for contents of
>>> nth time step*
>>> “%20d\n”, sizeof(int), flag, Miscellaneous flag (= 0 for now)
>>> “%20d\n”, sizeof(long), fb, of item n File byte loc for Item
>>> n above
>>> “%s\n”, sizeof(char)*80, “FILE_INDEX”, File index keyword
>>> * Each file byte location is the first byte that follows the
>>> BEGIN TIME STEP record
>>>
>>>
>>> To me the best way is to read the file in reverse looking for
>>> FILE_INDEX as the last 80 characters. From that you parse the
>>> previous long as
>>> the new seek position which jumps you to the start of the file
>>> index records. After reading the file index records you have a
>>> lookup table for all seek positions of the time steps in that
>>> ensight file.
>>>
>>>
>>> On Wed, Mar 6, 2013 at 4:23 AM, Georg Hammerl
>>> <hammerl at lnm.mw.tum.de <mailto:hammerl at lnm.mw.tum.de>> wrote:
>>>
>>> Hello,
>>>
>>> I have already posted this issue on the paraview mailing
>>> list, but then I realized that it is more of a VTK problem.
>>> I can load the first time step of my results but when I
>>> switch to the next step (in which 4 points are added),
>>> paraview freezes and htop shows 100% load for this process.
>>> Surprisingly, this only happens when I use binary ensight
>>> gold format. The same results written in ascii ensight gold
>>> format work. The ens_checker tells me for both cases that I
>>> have valid output files.
>>>
>>> Meanwhile I had a look into the source code and discovered
>>> the issue. Whenever the variable-files are read for a time
>>> steps greater than 1, Paraview has to jump in the binary
>>> file to the desired timestep. This is done by partially
>>> parsing and skipping blocks until the correct "BEGIN TIME
>>> STEP" is found. In order to skip the correct block length,
>>> Paraview uses the number of points (numPts) from the current
>>> geometry step. This breaks with a varying number of points
>>> per timestep.
>>> As an example my first step contains 2 points and the second
>>> step 6 points, therefore when Paraview skips the first time
>>> step, it skips a block length equivalent to 6 points. The
>>> correct number would be 2 points. It therefore skips past
>>> the next "BEGIN TIME STEP".
>>>
>>> In order to verify my theory I have implemented a little
>>> hack which looks for the next "BEGIN TIME STEP" by brute
>>> force. In case someone wants to try this I have attached a
>>> patch which can be used in conjunction with the attached
>>> examples.
>>>
>>> A clean solution would be to skip the correct number of
>>> points for every given timestep. Unfortunately I do not have
>>> enough insight into the VTK ensight reader to implement this
>>> and would appreciate any help in finding a clean solution to
>>> this.
>>>
>>> Cheers,
>>>
>>> Georg
>>>
>>> --
>>> Dipl.-Ing. Georg Hammerl
>>> Lehrstuhl für Numerische Mechanik
>>> Technische Universität München
>>> Boltzmannstrasse 15, D-85747 Garching b. München
>>> phone +49 89 289 15237 <tel:%2B49%2089%20289%2015237>
>>> fax +49 89 289 15301 <tel:%2B49%2089%20289%2015301>
>>> hammerl at lnm.mw.tum.de <mailto:hammerl at lnm.mw.tum.de>
>>> http://www.lnm.mw.tum.de
>>>
>>>
>>> _______________________________________________
>>> Powered by www.kitware.com <http://www.kitware.com>
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Please keep messages on-topic and check the VTK FAQ at:
>>> http://www.vtk.org/Wiki/VTK_FAQ
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://www.vtk.org/mailman/listinfo/vtkusers
>>>
>>>
>>>
>>>
>>> --
>>> Robert Maynard
>>
>>
>> --
>> Dipl.-Ing. Georg Hammerl
>> Lehrstuhl für Numerische Mechanik
>> Technische Universität München
>> Boltzmannstrasse 15, D-85747 Garching b. München
>> phone+49 89 289 15237 <tel:%2B49%2089%20289%2015237>
>> fax+49 89 289 15301 <tel:%2B49%2089%20289%2015301>
>> hammerl at lnm.mw.tum.de <mailto:hammerl at lnm.mw.tum.de>
>> http://www.lnm.mw.tum.de
>>
>>
>>
>>
>> --
>> Robert Maynard
>
>
> --
> Dipl.-Ing. Georg Hammerl
> Lehrstuhl für Numerische Mechanik
> Technische Universität München
> Boltzmannstrasse 15, D-85747 Garching b. München
> phone +49 89 289 15237
> fax +49 89 289 15301
> hammerl at lnm.mw.tum.de
> http://www.lnm.mw.tum.de
--
Dipl.-Ing. Georg Hammerl
Lehrstuhl für Numerische Mechanik
Technische Universität München
Boltzmannstrasse 15, D-85747 Garching b. München
phone +49 89 289 15237
fax +49 89 289 15301
hammerl at lnm.mw.tum.de
http://www.lnm.mw.tum.de
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.vtk.org/pipermail/vtkusers/attachments/20140310/f03040a4/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vtkEnSightGoldBinaryReader_caching.patch
Type: text/x-patch
Size: 10139 bytes
Desc: not available
URL: <http://www.vtk.org/pipermail/vtkusers/attachments/20140310/f03040a4/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vtkEnSightGoldBinaryReader_fileindex.patch
Type: text/x-patch
Size: 7082 bytes
Desc: not available
URL: <http://www.vtk.org/pipermail/vtkusers/attachments/20140310/f03040a4/attachment-0001.bin>
More information about the vtkusers
mailing list