[Paraview] ParaView 5.0.0 for Windows x64 fails to open EnSight Gold files

Tue Mar 1 02:49:27 EST 2016

Sorry for many posts, but, I think, I found something interesting (or I
just think so :)). I digged a *stat*() usage in ParaView at official GitHub
source repository. If I get it right, there are two types of this function:
*stat*() and *stat64*(). My only program that deals with large files is a
disk editor & NTFS data recovery for Windows (it's free and was made at
home), this program uses WinAPI, not C/C++ calls for file operations. So I
don't know much about fopen/fclose/stat/e.t.c. on large files. So we have
*stat* and *stat64* + corresponding data structures. We can see an example
in this source:
https://github.com/Kitware/VTK/blob/eae7f4446c867ee9d29b466676fa11282d56dafc/IO/LSDyna/private/LSDynaFamily.cxx
For 64-bit configurations this code uses *stat64()* and *stat64* structure.

But if we look at the problematic place in *EnSight reader* we will only
see the "usual" *stat*() that is, to my current knowledge, is 32 bit:
https://github.com/Kitware/VTK/blob/master/IO/EnSight/vtkEnSightGoldBinaryReader.cxx
(Please see lines 100 and 101)
May it be the cause of the subj? Sorry if I'm wrong, I'm not a professional
programmer.

2016-03-01 9:38 GMT+03:00 Andrew <antech777 at gmail.com>:

> I took a look at the code. Te problematic fragment (line 114 in
> vtkEnSightGoldBinaryReader) just checks the file's existance and opens it
> in standard way. The problem occures in *stat*() function that is fed
> with the full file name and the reference to structure to receive file
> params. I tried to read the file from Python shell and the result is the
> same. I guessed that the reason is an additional slash before the short
> file name (see error message) but small files are processed OK so the only
> assumption I can make is that the problem occures in the *stat*() itself
> on large (> ~2GiB) files due to some strange 32-bit relicts (although
> itshould work perfectly in 64-bit builds).
> Can anybody test the opening of large (3...4 GiB or more) EnSight Gold
> binary files in ParaView 5.0.0 or 5.0.1? I cannot check any assumptions
> because I only have Linux compilators set up and cannot do the
> "hello-world"-like stat() test under Windows on a large file...
>
> 2016-03-01 8:53 GMT+03:00 Andrew <antech777 at gmail.com>:
>
>> Hello Samuel.
>>
>> Sorry, but I can't undersatand how can the file fragmentation affect it's
>> readability and parsing. Even if the file has thousands of fragments and
>> "brain-damaging" :) runlist in many extended FILE records on NTFS
>> partition, your program, if it uses Win API, will see it like a solid file
>> (althougt there will be very low performance of sequental access). So there
>> should be no any faults on reading highly fragmented files.
>> I have checked out the "direct" reading of EnSight case from the cluster
>> via samba with ParaView 5.0.0 on Windows machine. The same error appears as
>> in my first message, although there are *different files and filesystems*
>> .
>> I think the problem is not a fragmentation but a *file size*. Some
>> 32-bit-like relicts or so in EnSight reader (it's just a guess, sorry if
>> I'm wrong). But my hypothesis does not asks the question why does the bug
>> affect only Windows versions... Maybe there are differences between
>> versions concerning data types. Do you have any >4 GB binary EnSight 6 or
>> Gold case to test? I can't (not allowed) to publish my case and it's also
>> big enough for upload (although theoretically I can do it from my home PC).
>>
>>
>> 2016-02-29 20:27 GMT+03:00 Samuel Key <samuelkey at bresnan.net>:
>>
>>> Andrew,
>>>
>>> Let us focus on MS Windows for a moment. The NTS file system and the use
>>> of the file system by the O/S is not quite as it should be.
>>>
>>> In my experience, the above combination routinely creates an
>>> unbelievable number of file fragments. Think in terms of confetti. As time
>>> goes by, the fragmentation builds. Even though you have a respectable
>>> amount of RAM, the free space itself on the disk driver can get rather
>>> disorganized.
>>>
>>> You can start by using the MS-supplied defrager. Run this utility five
>>> to seven times in a row and things might start looking better. However the
>>> free-space can still remain disjointed. Personally, I prefer a 3rd party
>>> utility for organizing and defraging the NTS file system (for example,
>>> PerfectDisk from Raxco). Even this utility has to run a couple of times
>>> plus once during the O/S re-boot phase to get a decent amount of
>>> consolidated free space and a somewhat consolidate O/S paging file. All of
>>> this has to be done on the O/S partition (usually C:) and the partition
>>> where you read in the 5+ GB Ensight Geometry file.
>>>
>>> On my machine, the MS Win7 O/S has its own partition (100GB \C:) plus
>>> another smaller dedicated partition for a second paging file (28GB \F:).
>>> All of my simulation work including the EnSight results file-sets reside on
>>> an entirely separate disk drive (280GB \G: that now needs to be even
>>> larger). The 3rd party file consolidation software runs "day and night" on
>>> the \G: partition.
>>>
>>> Just to emphasize the point, copying the 5+ GB EnSight geometry file on
>>> to workstation can result in many scattered fragments even when the
>>> free-space starts out consolidated. If the free space is not consolidated,
>>> the file fragmentation will be even more.
>>>
>>> The attached screen capture, shows the O/S managed paging-file (brown)
>>> in the \C: partition is beginning to fragment.
>>>
>>> If you go down this path and it helps, please let me know the outcome.
>>>
>>> Regards,
>>>
>>> Sam Key
>>>
>>> On 2/29/2016 1:04 AM, Andrew wrote:
>>>
>>> Another "experimental data". I was able to load this big case in
>>> ParaView viat the CGNS format. It was neccesary to select CGNS 3.0 and HDF5
>>> options in CFX Converter. But when I apply the Extract Block filter I see
>>> no any boundaries, only fluid domain (that is displayed OK). So CGNS format
>>> is not a wayout...
>>>
>>> 2016-02-29 9:00 GMT+03:00 Andrew <antech777 at gmail.com>:
>>>
>>>> I performed another test with *EnSi.ght 6* (not Gold) format.
>>>> Conversion was made on Win-7 x64 machine. The same results: PV 4.3 crashed,
>>>> PV 5.0 "cannot stat" (full error message is in my first post on this
>>>> topic). Sorry, now I need to work on with my report...
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2016-02-29 8:47 GMT+03:00 Andrew < <antech777 at gmail.com>
>>>> antech777 at gmail.com>:
>>>>
>>>>> Hello. Sorry for silence, I'm usually offline on holydays, although I
>>>>> have internet connection and read your mail.
>>>>>
>>>>> Samuel Key
>>>>>
>>>>> Thanks for your attention.
>>>>>
>>>>> About transferring files from Linux to Windows. I use binary EnSight
>>>>> files because CFX converter fails to write so large files in ASCII format.
>>>>> But I tried to convert from CFX to EnSight (binary) on the same Win-7 x64
>>>>> machine with ParaView. Both installed versions (4.3 and 5.0) failed to open
>>>>> the geometry (I unchecked all fields). ParaView 4.3 crashed, ParaView 5.0
>>>>> just cannot open file...
>>>>>
>>>>> I think that the root cause may be a geometry file size. It is 4.9 GiB
>>>>> (5.3 GB), mesh have about 150 millions of cells, not 600 thusands like in
>>>>> your case. Its possible that ParaView for Windows cannot process so large
>>>>> EnSight files, although the Linux version reads in this geometry and
>>>>> displays outline (swapping is so intensive so I cannot displaye something
>>>>> else).
>>>>>
>>>>> 2016-02-26 19:31 GMT+03:00 Samuel Key < <samuelkey at bresnan.net>
>>>>> samuelkey at bresnan.net>:
>>>>>
>>>>>> Andrew,
>>>>>>
>>>>>> For what it is worth, I use ParaView
>>>>>> (ParaView-5.0.0-Qt4-OpenGL2-Windows-64bit) on Windows-7 64bit. I just now
>>>>>> tested two medium-sized, EnSight Gold-formatted simulation results files
>>>>>> (600,000 finite elements) and both worked.
>>>>>>
>>>>>> In the past, I have had difficulty with the EnSight ASCII-formatted
>>>>>> *.case file when moving the simulation results (*.case file and friends)
>>>>>> from MS Windows to EnSight running on a linux platform. It was necessary to
>>>>>> pass the ASCII *.case file through a 'dos2unix' filter. Transferring files
>>>>>> the other way might have the same problem???
>>>>>>
>>>>>> Hope this helps.
>>>>>>
>>>>>> Samuel Key
>>>>>> FMA Development, LLC
>>>>>> 1005 39th Ave NE
>>>>>> Great Falls, Montana 59404
>>>>>> USA
>>>>>>
>>>>>> On 2/25/2016 11:26 PM, Andrew wrote:
>>>>>>
>>>>>> Hello. We use Ansys CFX in our work but the standard post-processor
>>>>>> CFD-Post consumes one of Ansys licenses while Paraview is free. So I often
>>>>>> make pictures for reports with ParaView to preserve Ansys license for
>>>>>> another users/needs (because other users are not familiar with ParaView and
>>>>>> CFD-Post may consume general "acfd" license feature that may be used for
>>>>>> calculations). I convert CFX results to EnSight Gold format in CFX Solver
>>>>>> Manager (Export Results) and then open in ParaView. It was working well on
>>>>>> Linux (CentOS 6.7 x64, Ubuntu 12.04 x64 with updates) but now I work with
>>>>>> large files and I need to use another machine that has 128 GB of RAM and
>>>>>> runs Windows-7 x64.
>>>>>> I downloaded and installed the following version:
>>>>>> ParaView-5.0.0-Qt4-OpenGL2-Windows-64bit.exe
>>>>>> When I open my EnSight-format file I receive the error message that
>>>>>> geometry file cannot be read. I tried to move EnSight files to a simple
>>>>>> location (D:\Temp) and renamed them (test.case, test.geom + edited the case
>>>>>> to change geometry file name to "simple" one). The same result, it cannot
>>>>>> find geometry file... Full error message is as follows:
>>>>>>
>>>>>> ERROR: In
>>>>>> C:\bbd\df0abce0\source-paraview\VTK\IO\EnSight\vtkEnSightGoldBinaryReader.cxx,
>>>>>> line 114
>>>>>> vtkEnSightGoldBinaryReader (0000000009451230): stat failed.
>>>>>>
>>>>>>
>>>>>> ERROR: In
>>>>>> C:\bbd\df0abce0\source-paraview\VTK\IO\EnSight\vtkEnSightGoldBinaryReader.cxx,
>>>>>> line 220
>>>>>> vtkEnSightGoldBinaryReader (0000000009451230): Unable to open file:
>>>>>> D:\Storage\Media\Work\Gradirni\CFX\EnSight\C-G002L3S-R020-S000-14.5\/C-G002L3S-R020-S000.geom
>>>>>>
>>>>>>
>>>>>> ERROR: In
>>>>>> C:\bbd\df0abce0\source-paraview\VTK\IO\EnSight\vtkEnSightReader.cxx, line
>>>>>> 306
>>>>>> vtkEnSightGoldBinaryReader (0000000009451230): error reading geometry
>>>>>> file
>>>>>>
>>>>>>
>>>>>> ERROR: In
>>>>>> C:\bbd\df0abce0\source-paraview\VTK\Common\ExecutionModel\vtkExecutive.cxx,
>>>>>> line 784
>>>>>> vtkCompositeDataPipeline (0000000008E1A640): Algorithm
>>>>>> vtkEnSightGoldBinaryReader(0000000009451230) returned failure for request:
>>>>>> vtkInformation (00000000093C3920)
>>>>>>   Debug: Off
>>>>>>   Modified Time: 214873
>>>>>>   Reference Count: 1
>>>>>>   Registered Events: (none)
>>>>>>   Request: REQUEST_DATA
>>>>>>   ALGORITHM_AFTER_FORWARD: 1
>>>>>>   FORWARD_DIRECTION: 0
>>>>>>   FROM_OUTPUT_PORT: 0
>>>>>>
>>>>>> OK, I tried to open it in ParaView 4.3. It says nothing (only border
>>>>>> of the message window appears) and crashes. At that moment I see in the
>>>>>> status bar that EnSight reader progress is 100%.
>>>>>>
>>>>>> Both ParaView 5.0.0 and 4.3 was tested with EnSight files exported on
>>>>>> Linux and Windows machine. It didn't help.
>>>>>>
>>>>>> OK, I tried to export CFX results to CGNS format. ParaView 5.0.0
>>>>>> (without MPI) cannot read it, so I fed it to ParaView 4.3. It says that its
>>>>>> "unable to filed any meshes".
>>>>>>
>>>>>> I also performed an experiment on my Linux laptop. It has only 16 GB
>>>>>> of RAM so its unreal to work with such a big file on this machine but it
>>>>>> reads the same EnSight case and displays the domain! No arrays selected
>>>>>> (mesh only), swap file grows up to 8 GB but the file opens without errors
>>>>>> and crashes. It's only a test because I have not any Linux machine with
>>>>>> memory large enough (our Linux cluster has 128 GB of RAM on every node
>>>>>> including master but it has very poor video).
>>>>>>
>>>>>> I switched to CFD Post for this case but I want to know are there any
>>>>>> workarounds to open my EnSight-format results in ParaView for Windows.
>>>>>>
>>>>>> Thanks for your attention.
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Powered by www.kitware.com
>>>>>>
>>>>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>>>>>
>>>>>> Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView
>>>>>>
>>>>>> Search the list archives at: http://markmail.org/search/?q=ParaView
>>>>>>
>>>>>> Follow this link to subscribe/unsubscribe:http://public.kitware.com/mailman/listinfo/paraview
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Powered by <http://www.kitware.com>www.kitware.com
>>>>>>
>>>>>> Visit other Kitware open-source projects at
>>>>>> <http://www.kitware.com/opensource/opensource.html>
>>>>>> http://www.kitware.com/opensource/opensource.html
>>>>>>
>>>>>> Please keep messages on-topic and check the ParaView Wiki at:
>>>>>> <http://paraview.org/Wiki/ParaView>http://paraview.org/Wiki/ParaView
>>>>>>
>>>>>> Search the list archives at: <http://markmail.org/search/?q=ParaView>
>>>>>> http://markmail.org/search/?q=ParaView
>>>>>>
>>>>>> Follow this link to subscribe/unsubscribe:
>>>>>> http://public.kitware.com/mailman/listinfo/paraview
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>>
>>> Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView
>>>
>>> Search the list archives at: http://markmail.org/search/?q=ParaView
>>>
>>> Follow this link to subscribe/unsubscribe:http://public.kitware.com/mailman/listinfo/paraview
>>>
>>>
>>>
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Please keep messages on-topic and check the ParaView Wiki at:
>>> http://paraview.org/Wiki/ParaView
>>>
>>> Search the list archives at: http://markmail.org/search/?q=ParaView
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://public.kitware.com/mailman/listinfo/paraview
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/paraview/attachments/20160301/136fa120/attachment.html>