[vtkusers] different behavior between ascii and binary ensight gold reader

[Robert Maynard]

I would consolidate the usage of the magic value of 80 into a single
place and document just once why you need to adjust the file positions
by that amount. You might think about storing the unadjusted
addresses, and subtracting 80 inside SeekToCachedTimeStep.

Inside SeekToCachedTimeStep I would lift the search for the fileName
outside the for loop, and store an iterator to the result of find,
that should improve performance of
the for loop, since you wont have to do multiple searches.

Otherwise everything looks pretty good.

On Mon, Mar 10, 2014 at 9:33 AM, Georg Hammerl <hammerl at lnm.mw.tum.de> wrote:
> Hi Robert,
>
> meanwhile we worked out an implementation solving the problem.
> After discovering that the EnSightGoldReader (ASCII-version) uses a caching
> system for the time steps we ported this in a slightly modified way to the
> EnSightGoldBinaryReader. In addition we added functionality to fill the
> cache in advance in case a file index is available.
> This circumvents the problem with changing geometries.
> Attached are two patch files, the first one implements the caching for the
> binary reader and the second one implements the use of the file index. It
> would be great if this could be included in VTK. If you find any problems
> with the implementation, please let us know so we can find a solution.
>
> Thank you very much in advance
>
> Karl & Georg
>
>
>
> On 10/02/14 17:55, Georg Hammerl wrote:
>
> Hi Robert,
>
> it is been a while since we had this problem but we are running into it
> again. Therefore I would like to reopen the discussion on making use of the
> file index in the vtkEnSightGoldBinaryReader. I expect two advantages by
> doing so, firstly it is more efficient and secondly, it can be used to
> enable changing geometries over time (variable number of points). At the
> moment, time steps are skipped (when reading vectors) by assuming a constant
> number of points in the mesh over all time steps which is of course not
> correct for variable number of points.
>
> I implemented a preliminary version of reading the file index. It works fine
> (and is fast :-)) and enables us to visualize changing geometries. The
> implementation is not yet in a style that could be included in VTK, I
> therefore wanted to get some pointers what is the best way to include this
> feature into VTK.
>
> I have thought of the following scenarios:
>
> 1)
>  when reading Scalars, Vectors, Tensors, ... the blocks of data until
> timeStep-1 are being skipped one by one assuming a constant geometry. To
> skip this we look for a file index and, if present, this->IFile is set to
> the correct position in the file.
> This has the advantage of being very easy to implement, however it has to be
> implemented for every type of field separately. Also one has to jump to the
> end of the file for every field in every file whenever a new time step is
> requested.
>
> 2)
> provide a method in the vtkEnSightGoldBinaryReader, that implements the
> reading of the file index and setting this->IFile to the correct position
> (for example a SkipTimestepsUsingFileindex(int timestep) ) or that returns
> the address(es) of a(ll) time step(s), which can then be used in the
> Read{Scalars,Vectors,..}.. methods.
> This way only one implementation that actually reads the file index is
> needed, but it still requires the file index to be read for every field in
> every file whenever a new time step is requested.
>
> 3)
> Read the file index of all files when the reader is initialized and store
> the timestep positions per file in a member of the
> vtkEnSightGoldBinaryReader. This way the index has to be read only once and
> timesteps can be skipped by referring to the member. Drawback of this method
> is, that I need some hints on how this could be implemented consistent with
> the rest of the reader (i.e. map<std::string, std::vectors<long> > vs.
> string[] and long[][] or anything in between).
>
> There are some lines of code attached that show the proof of concept
> implementation. I am looking forward to getting some suggestions.
>
> Cheers,
>
> Georg
>
>
> On 19/04/13 14:51, Robert Maynard wrote:
>
> The file index is required by Ensight for all transient single file formats.
> The documentation also states that Ensight automatically writes out the file
> index information, so I would consider any transient single file format that
> doesn't have it be malformed.
>
>
> On Fri, Apr 19, 2013 at 7:57 AM, Georg Hammerl <hammerl at lnm.mw.tum.de>
> wrote:
>>
>> Hi Robert,
>>
>> thanks for the hint with the file index. The ensight gold files I have
>> available come with a file index. I have tried to use it in Paraview,
>> however it seems that the VTK/IO/Ensight/vtkEnSightGoldBinaryReader does not
>> make use of the index.
>> Unfortunately, I do not have enough insight into the inner workings of
>> VTK, so I cannot see myself creating a clean implementation for using the
>> file index.
>>
>> As the file index is just optional, I think it should be worth
>> implementing an efficient way of reading ensight gold format even without
>> the file index.
>>
>> Cheers,
>>
>> Georg
>>
>>
>>
>> On 18/04/13 20:06, Robert Maynard wrote:
>>
>> Hi Georg,
>>
>> You are correct that a more efficient solution is possible. After reading
>> the Ensight Gold binary format (
>> http://www-vis.lbl.gov/NERSC/Software/ensight/doc/OnlineHelp/UM-C11.pdf ) I
>> have discovered how you can do this more efficiently.
>>
>> If you are start at 11-37 you will see that they state:
>>
>> "Note 3: Efficient reading of each file (especially binary) is facilitated
>> by
>> appending each file with a file index. A file index contains appropriate
>> information to access the file byte positions of each time step in the
>> file. (EnSight
>> automatically appends a file index to each file when exporting in
>> transient single
>> file format.) If used, the file index must follow the last END TIME STEP
>> wrapper in each file."
>>
>> File Index Usage:
>> ASCII, Binary, Item, Description
>> "%20d\n", sizeof(int), n, Total number of data time steps in the file.
>> "%20d\n", sizeof(long), fb1, File byte loc for contents of 1st time step*
>> "%20d\n", sizeof(long), fb2, File byte loc for contents of 2nd time step*
>> . . .,  . . .,  . . .,  . . .
>> "%20d\n", sizeof(long), fbn, File byte loc for contents of nth time step*
>> "%20d\n", sizeof(int), flag, Miscellaneous flag (= 0 for now)
>> "%20d\n", sizeof(long), fb, of item n File byte loc for Item n above
>> "%s\n", sizeof(char)*80, "FILE_INDEX", File index keyword
>> * Each file byte location is the first byte that follows the BEGIN TIME
>> STEP record
>>
>>
>> To me the best way is to read the file in reverse looking for FILE_INDEX
>> as the last 80 characters. From that you parse the previous long as
>> the new seek position which jumps you to the start of the file index
>> records. After reading the file index records you have a lookup table for
>> all seek positions of the time steps in that ensight file.
>>
>>
>> On Wed, Mar 6, 2013 at 4:23 AM, Georg Hammerl <hammerl at lnm.mw.tum.de>
>> wrote:
>>>
>>> Hello,
>>>
>>> I have already posted this issue on the paraview mailing list, but then I
>>> realized that it is more of a VTK problem.
>>> I can load the first time step of my results but when I switch to the
>>> next step (in which 4 points are added), paraview freezes and htop shows
>>> 100% load for this process. Surprisingly, this only happens when I use
>>> binary ensight gold format. The same results written in ascii ensight gold
>>> format work. The ens_checker tells me for both cases that I have valid
>>> output files.
>>>
>>> Meanwhile I had a look into the source code and discovered the issue.
>>> Whenever the variable-files are read for a time steps greater than 1,
>>> Paraview has to jump in the binary file to the desired timestep. This is
>>> done by partially parsing and skipping blocks until the correct "BEGIN TIME
>>> STEP" is found. In order to skip the correct block length, Paraview uses the
>>> number of points (numPts) from the current geometry step. This breaks with a
>>> varying number of points per timestep.
>>> As an example my first step contains 2 points and the second step 6
>>> points, therefore when Paraview skips the first time step, it skips a block
>>> length equivalent to 6 points. The correct number would be 2 points. It
>>> therefore skips past the next "BEGIN TIME STEP".
>>>
>>> In order to verify my theory I have implemented a little hack which looks
>>> for the next "BEGIN TIME STEP" by brute force. In case someone wants to try
>>> this I have attached a patch which can be used in conjunction with the
>>> attached examples.
>>>
>>> A clean solution would be to skip the correct number of points for every
>>> given timestep. Unfortunately I do not have enough insight into the VTK
>>> ensight reader to implement this and would appreciate any help in finding a
>>> clean solution to this.
>>>
>>> Cheers,
>>>
>>> Georg
>>>
>>> --
>>> Dipl.-Ing. Georg Hammerl
>>> Lehrstuhl für Numerische Mechanik
>>> Technische Universität München
>>> Boltzmannstrasse 15, D-85747 Garching b. München
>>> phone +49 89 289 15237
>>> fax +49 89 289 15301
>>> hammerl at lnm.mw.tum.de
>>> http://www.lnm.mw.tum.de
>>>
>>>
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Please keep messages on-topic and check the VTK FAQ at:
>>> http://www.vtk.org/Wiki/VTK_FAQ
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://www.vtk.org/mailman/listinfo/vtkusers
>>>
>>
>>
>>
>> --
>> Robert Maynard
>>
>>
>>
>> --
>> Dipl.-Ing. Georg Hammerl
>> Lehrstuhl für Numerische Mechanik
>> Technische Universität München
>> Boltzmannstrasse 15, D-85747 Garching b. München
>> phone +49 89 289 15237
>> fax +49 89 289 15301
>> hammerl at lnm.mw.tum.de
>> http://www.lnm.mw.tum.de
>
>
>
>
> --
> Robert Maynard
>
>
>
> --
> Dipl.-Ing. Georg Hammerl
> Lehrstuhl für Numerische Mechanik
> Technische Universität München
> Boltzmannstrasse 15, D-85747 Garching b. München
> phone +49 89 289 15237
> fax +49 89 289 15301
> hammerl at lnm.mw.tum.de
> http://www.lnm.mw.tum.de
>
>
>
> --
> Dipl.-Ing. Georg Hammerl
> Lehrstuhl für Numerische Mechanik
> Technische Universität München
> Boltzmannstrasse 15, D-85747 Garching b. München
> phone +49 89 289 15237
> fax +49 89 289 15301
> hammerl at lnm.mw.tum.de
> http://www.lnm.mw.tum.de