[Paraview] ParaView 5.0.0 for Windows x64 fails to open EnSight Gold files

Thu Mar 3 08:45:26 EST 2016

Andrew/ Sam,

Thanks for exploring this as much as you did. I am afraid the issue is
not _FILE_OFFSET_BITS flag, however. As far as I can tell, the flag
gets set "globally" when CMake determinees during configure time that
"large file support" is possible. To follow the code here's how it
works:

1. https://gitlab.kitware.com/vtk/vtk/blob/master/CMakeLists.txt#L335-340

At the top level, CMake sets up VTK_REQUIRE_LARGE_FILE_SUPPORT if
supported by the platform/compiler.

2. https://gitlab.kitware.com/vtk/vtk/blob/master/Common/Core/vtkWin32Header.h#L34-44

This header setups the appropriate values when
VTK_REQUIRE_LARGE_FILE_SUPPORT is defined (via vtkConfigure.h). This
header gets included in vtkSystemIncludes.h which gets included in
vtkObjectBase.h hence in all VTK classes. Despite the header name
(vtkWin32Header.h) it gets included on all platforms.

By putting out messages in vtkEnSightGoldBinaryReader.cxx, I confirmed
that _FILE_OFFSET_BITS is indeed set to 64 bit on my 64bit Mac
machine.

Something else is going on here. Does someone have a >2GB ensight gold
file to share to debug this or know where I can find one?

Utkarsh

On Tue, Mar 1, 2016 at 11:30 AM, Andrew <antech777 at gmail.com> wrote:
> Samuel, thanks for your experiment. File with size less than 2 GiB, of
> course, can't expose this issue.
> I read a bit more about stat() and it turned out that it should work as
> 64bit if FILE_OFFSET_BITS is set to 64. And it is set in other ParaView
> sources but, IMHO, not in EnSight reader. Dear ParaView team, please, take a
> look at this, I need to process my big case with Python script in
> ParaView...
>
> On Mar 1, 2016 18:52, "Samuel Key" <samuelkey at bresnan.net> wrote:
>>
>> Andrew,
>>
>> Regarding your question, "Do you have any EnSight Gold-Format files
>> greater than 4GB?" No, the largest EnSight *.geom file I have is 1 GB.
>>
>> I just tried it in PV v5.0.0.  Reading the datum set and running through
>> 67 time steps did not cause any problems. I don't think this *.geom file is
>> big enough to expose any of the PV reader reader problems you are
>> experiencing.
>>
>> Sam Key
>>
>> On 2/29/2016 10:53 PM, Andrew wrote:
>>
>> Hello Samuel.
>>
>> Sorry, but I can't undersatand how can the file fragmentation affect it's
>> readability and parsing. Even if the file has thousands of fragments and
>> "brain-damaging" :) runlist in many extended FILE records on NTFS partition,
>> your program, if it uses Win API, will see it like a solid file (althougt
>> there will be very low performance of sequental access). So there should be
>> no any faults on reading highly fragmented files.
>> I have checked out the "direct" reading of EnSight case from the cluster
>> via samba with ParaView 5.0.0 on Windows machine. The same error appears as
>> in my first message, although there are different files and filesystems.
>> I think the problem is not a fragmentation but a file size. Some
>> 32-bit-like relicts or so in EnSight reader (it's just a guess, sorry if I'm
>> wrong). But my hypothesis does not asks the question why does the bug affect
>> only Windows versions... Maybe there are differences between versions
>> concerning data types. Do you have any >4 GB binary EnSight 6 or Gold case
>> to test? I can't (not allowed) to publish my case and it's also big enough
>> for upload (although theoretically I can do it from my home PC).
>>
>>
>> 2016-02-29 20:27 GMT+03:00 Samuel Key <samuelkey at bresnan.net>:
>>>
>>> Andrew,
>>>
>>> Let us focus on MS Windows for a moment. The NTS file system and the use
>>> of the file system by the O/S is not quite as it should be.
>>>
>>> In my experience, the above combination routinely creates an unbelievable
>>> number of file fragments. Think in terms of confetti. As time goes by, the
>>> fragmentation builds. Even though you have a respectable amount of RAM, the
>>> free space itself on the disk driver can get rather disorganized.
>>>
>>> You can start by using the MS-supplied defrager. Run this utility five to
>>> seven times in a row and things might start looking better. However the
>>> free-space can still remain disjointed. Personally, I prefer a 3rd party
>>> utility for organizing and defraging the NTS file system (for example,
>>> PerfectDisk from Raxco). Even this utility has to run a couple of times plus
>>> once during the O/S re-boot phase to get a decent amount of consolidated
>>> free space and a somewhat consolidate O/S paging file. All of this has to be
>>> done on the O/S partition (usually C:) and the partition where you read in
>>> the 5+ GB Ensight Geometry file.
>>>
>>> On my machine, the MS Win7 O/S has its own partition (100GB \C:) plus
>>> another smaller dedicated partition for a second paging file (28GB \F:). All
>>> of my simulation work including the EnSight results file-sets reside on an
>>> entirely separate disk drive (280GB \G: that now needs to be even larger).
>>> The 3rd party file consolidation software runs "day and night" on the \G:
>>> partition.
>>>
>>> Just to emphasize the point, copying the 5+ GB EnSight geometry file on
>>> to workstation can result in many scattered fragments even when the
>>> free-space starts out consolidated. If the free space is not consolidated,
>>> the file fragmentation will be even more.
>>>
>>> The attached screen capture, shows the O/S managed paging-file (brown) in
>>> the \C: partition is beginning to fragment.
>>>
>>> If you go down this path and it helps, please let me know the outcome.
>>>
>>> Regards,
>>>
>>> Sam Key
>>>
>>> On 2/29/2016 1:04 AM, Andrew wrote:
>>>
>>> Another "experimental data". I was able to load this big case in ParaView
>>> viat the CGNS format. It was neccesary to select CGNS 3.0 and HDF5 options
>>> in CFX Converter. But when I apply the Extract Block filter I see no any
>>> boundaries, only fluid domain (that is displayed OK). So CGNS format is not
>>> a wayout...
>>>
>>> 2016-02-29 9:00 GMT+03:00 Andrew <antech777 at gmail.com>:
>>>>
>>>> I performed another test with EnSi.ght 6 (not Gold) format. Conversion
>>>> was made on Win-7 x64 machine. The same results: PV 4.3 crashed, PV 5.0
>>>> "cannot stat" (full error message is in my first post on this topic). Sorry,
>>>> now I need to work on with my report...
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2016-02-29 8:47 GMT+03:00 Andrew <antech777 at gmail.com>:
>>>>>
>>>>> Hello. Sorry for silence, I'm usually offline on holydays, although I
>>>>> have internet connection and read your mail.
>>>>>
>>>>> Samuel Key
>>>>>
>>>>> Thanks for your attention.
>>>>>
>>>>> About transferring files from Linux to Windows. I use binary EnSight
>>>>> files because CFX converter fails to write so large files in ASCII format.
>>>>> But I tried to convert from CFX to EnSight (binary) on the same Win-7 x64
>>>>> machine with ParaView. Both installed versions (4.3 and 5.0) failed to open
>>>>> the geometry (I unchecked all fields). ParaView 4.3 crashed, ParaView 5.0
>>>>> just cannot open file...
>>>>>
>>>>> I think that the root cause may be a geometry file size. It is 4.9 GiB
>>>>> (5.3 GB), mesh have about 150 millions of cells, not 600 thusands like in
>>>>> your case. Its possible that ParaView for Windows cannot process so large
>>>>> EnSight files, although the Linux version reads in this geometry and
>>>>> displays outline (swapping is so intensive so I cannot displaye something
>>>>> else).
>>>>>
>>>>>
>>>>> 2016-02-26 19:31 GMT+03:00 Samuel Key <samuelkey at bresnan.net>:
>>>>>>
>>>>>> Andrew,
>>>>>>
>>>>>> For what it is worth, I use ParaView
>>>>>> (ParaView-5.0.0-Qt4-OpenGL2-Windows-64bit) on Windows-7 64bit. I just now
>>>>>> tested two medium-sized, EnSight Gold-formatted simulation results files
>>>>>> (600,000 finite elements) and both worked.
>>>>>>
>>>>>> In the past, I have had difficulty with the EnSight ASCII-formatted
>>>>>> *.case file when moving the simulation results (*.case file and friends)
>>>>>> from MS Windows to EnSight running on a linux platform. It was necessary to
>>>>>> pass the ASCII *.case file through a 'dos2unix' filter. Transferring files
>>>>>> the other way might have the same problem???
>>>>>>
>>>>>> Hope this helps.
>>>>>>
>>>>>> Samuel Key
>>>>>> FMA Development, LLC
>>>>>> 1005 39th Ave NE
>>>>>> Great Falls, Montana 59404
>>>>>> USA
>>>>>>
>>>>>> On 2/25/2016 11:26 PM, Andrew wrote:
>>>>>>
>>>>>> Hello. We use Ansys CFX in our work but the standard post-processor
>>>>>> CFD-Post consumes one of Ansys licenses while Paraview is free. So I often
>>>>>> make pictures for reports with ParaView to preserve Ansys license for
>>>>>> another users/needs (because other users are not familiar with ParaView and
>>>>>> CFD-Post may consume general "acfd" license feature that may be used for
>>>>>> calculations). I convert CFX results to EnSight Gold format in CFX Solver
>>>>>> Manager (Export Results) and then open in ParaView. It was working well on
>>>>>> Linux (CentOS 6.7 x64, Ubuntu 12.04 x64 with updates) but now I work with
>>>>>> large files and I need to use another machine that has 128 GB of RAM and
>>>>>> runs Windows-7 x64.
>>>>>> I downloaded and installed the following version:
>>>>>> ParaView-5.0.0-Qt4-OpenGL2-Windows-64bit.exe
>>>>>> When I open my EnSight-format file I receive the error message that
>>>>>> geometry file cannot be read. I tried to move EnSight files to a simple
>>>>>> location (D:\Temp) and renamed them (test.case, test.geom + edited the case
>>>>>> to change geometry file name to "simple" one). The same result, it cannot
>>>>>> find geometry file... Full error message is as follows:
>>>>>>
>>>>>> ERROR: In
>>>>>> C:\bbd\df0abce0\source-paraview\VTK\IO\EnSight\vtkEnSightGoldBinaryReader.cxx,
>>>>>> line 114
>>>>>> vtkEnSightGoldBinaryReader (0000000009451230): stat failed.
>>>>>>
>>>>>>
>>>>>> ERROR: In
>>>>>> C:\bbd\df0abce0\source-paraview\VTK\IO\EnSight\vtkEnSightGoldBinaryReader.cxx,
>>>>>> line 220
>>>>>> vtkEnSightGoldBinaryReader (0000000009451230): Unable to open file:
>>>>>> D:\Storage\Media\Work\Gradirni\CFX\EnSight\C-G002L3S-R020-S000-14.5\/C-G002L3S-R020-S000.geom
>>>>>>
>>>>>>
>>>>>> ERROR: In
>>>>>> C:\bbd\df0abce0\source-paraview\VTK\IO\EnSight\vtkEnSightReader.cxx, line
>>>>>> 306
>>>>>> vtkEnSightGoldBinaryReader (0000000009451230): error reading geometry
>>>>>> file
>>>>>>
>>>>>>
>>>>>> ERROR: In
>>>>>> C:\bbd\df0abce0\source-paraview\VTK\Common\ExecutionModel\vtkExecutive.cxx,
>>>>>> line 784
>>>>>> vtkCompositeDataPipeline (0000000008E1A640): Algorithm
>>>>>> vtkEnSightGoldBinaryReader(0000000009451230) returned failure for request:
>>>>>> vtkInformation (00000000093C3920)
>>>>>>   Debug: Off
>>>>>>   Modified Time: 214873
>>>>>>   Reference Count: 1
>>>>>>   Registered Events: (none)
>>>>>>   Request: REQUEST_DATA
>>>>>>   ALGORITHM_AFTER_FORWARD: 1
>>>>>>   FORWARD_DIRECTION: 0
>>>>>>   FROM_OUTPUT_PORT: 0
>>>>>>
>>>>>> OK, I tried to open it in ParaView 4.3. It says nothing (only border
>>>>>> of the message window appears) and crashes. At that moment I see in the
>>>>>> status bar that EnSight reader progress is 100%.
>>>>>>
>>>>>> Both ParaView 5.0.0 and 4.3 was tested with EnSight files exported on
>>>>>> Linux and Windows machine. It didn't help.
>>>>>>
>>>>>> OK, I tried to export CFX results to CGNS format. ParaView 5.0.0
>>>>>> (without MPI) cannot read it, so I fed it to ParaView 4.3. It says that its
>>>>>> "unable to filed any meshes".
>>>>>>
>>>>>> I also performed an experiment on my Linux laptop. It has only 16 GB
>>>>>> of RAM so its unreal to work with such a big file on this machine but it
>>>>>> reads the same EnSight case and displays the domain! No arrays selected
>>>>>> (mesh only), swap file grows up to 8 GB but the file opens without errors
>>>>>> and crashes. It's only a test because I have not any Linux machine with
>>>>>> memory large enough (our Linux cluster has 128 GB of RAM on every node
>>>>>> including master but it has very poor video).
>>>>>>
>>>>>> I switched to CFD Post for this case but I want to know are there any
>>>>>> workarounds to open my EnSight-format results in ParaView for Windows.
>>>>>>
>>>>>> Thanks for your attention.
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Powered by www.kitware.com
>>>>>>
>>>>>> Visit other Kitware open-source projects at
>>>>>> http://www.kitware.com/opensource/opensource.html
>>>>>>
>>>>>> Please keep messages on-topic and check the ParaView Wiki at:
>>>>>> http://paraview.org/Wiki/ParaView
>>>>>>
>>>>>> Search the list archives at: http://markmail.org/search/?q=ParaView
>>>>>>
>>>>>> Follow this link to subscribe/unsubscribe:
>>>>>> http://public.kitware.com/mailman/listinfo/paraview
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Powered by www.kitware.com
>>>>>>
>>>>>> Visit other Kitware open-source projects at
>>>>>> http://www.kitware.com/opensource/opensource.html
>>>>>>
>>>>>> Please keep messages on-topic and check the ParaView Wiki at:
>>>>>> http://paraview.org/Wiki/ParaView
>>>>>>
>>>>>> Search the list archives at: http://markmail.org/search/?q=ParaView
>>>>>>
>>>>>> Follow this link to subscribe/unsubscribe:
>>>>>> http://public.kitware.com/mailman/listinfo/paraview
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Please keep messages on-topic and check the ParaView Wiki at:
>>> http://paraview.org/Wiki/ParaView
>>>
>>> Search the list archives at: http://markmail.org/search/?q=ParaView
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://public.kitware.com/mailman/listinfo/paraview
>>>
>>>
>>>
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Please keep messages on-topic and check the ParaView Wiki at:
>>> http://paraview.org/Wiki/ParaView
>>>
>>> Search the list archives at: http://markmail.org/search/?q=ParaView
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://public.kitware.com/mailman/listinfo/paraview
>>>
>>
>>
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Please keep messages on-topic and check the ParaView Wiki at:
>> http://paraview.org/Wiki/ParaView
>>
>> Search the list archives at: http://markmail.org/search/?q=ParaView
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/paraview
>>
>>
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Please keep messages on-topic and check the ParaView Wiki at:
>> http://paraview.org/Wiki/ParaView
>>
>> Search the list archives at: http://markmail.org/search/?q=ParaView
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/paraview
>>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at:
> http://paraview.org/Wiki/ParaView
>
> Search the list archives at: http://markmail.org/search/?q=ParaView
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/paraview
>