<div dir="ltr"><div><div><div>Sorry for many posts, but, I think, I found something interesting (or I just think so :)). I digged a <b>stat</b>() usage in ParaView at official GitHub source repository. If I get it right, there are two types of this function: <b>stat</b>() and <b>stat64</b>(). <font size="1">My only program that deals with large files is a disk editor & NTFS data recovery for Windows (it's free and was made at home), this program uses WinAPI, not C/C++ calls for file operations. So I don't know much about fopen/fclose/stat/e.t.c. on large files.</font> So we have <b>stat</b> and <b>stat64</b> + corresponding data structures. We can see an example in this source:<br><a href="https://github.com/Kitware/VTK/blob/eae7f4446c867ee9d29b466676fa11282d56dafc/IO/LSDyna/private/LSDynaFamily.cxx">https://github.com/Kitware/VTK/blob/eae7f4446c867ee9d29b466676fa11282d56dafc/IO/LSDyna/private/LSDynaFamily.cxx</a><br></div><div>For 64-bit configurations this code uses <b>stat64()</b> and <b>stat64</b> structure.<br></div><div><br></div>But if we look at the problematic place in <b>EnSight reader</b> we will only see the "usual" <b>stat</b>() that is, to my current knowledge, is 32 bit:<br><a href="https://github.com/Kitware/VTK/blob/master/IO/EnSight/vtkEnSightGoldBinaryReader.cxx">https://github.com/Kitware/VTK/blob/master/IO/EnSight/vtkEnSightGoldBinaryReader.cxx</a><br></div>(Please see lines 100 and 101)<br></div>May it be the cause of the subj? Sorry if I'm wrong, I'm not a professional programmer.<br><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-03-01 9:38 GMT+03:00 Andrew <span dir="ltr"><<a href="mailto:antech777@gmail.com" target="_blank">antech777@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I took a look at the code. Te problematic fragment (line 114 in vtkEnSightGoldBinaryReader) just checks the file's existance and opens it in standard way. The problem occures in <b>stat</b>() function that is fed with the full file name and the reference to structure to receive file params. I tried to read the file from Python shell and the result is the same. I guessed that the reason is an additional slash before the short file name (see error message) but small files are processed OK so the only assumption I can make is that the problem occures in the <b>stat</b>() itself on large (> ~2GiB) files due to some strange 32-bit relicts (although itshould work perfectly in 64-bit builds).<br></div>Can anybody test the opening of large (3...4 GiB or more) EnSight Gold binary files in ParaView 5.0.0 or 5.0.1? I cannot check any assumptions because I only have Linux compilators set up and cannot do the "hello-world"-like stat() test under Windows on a large file...<br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">2016-03-01 8:53 GMT+03:00 Andrew <span dir="ltr"><<a href="mailto:antech777@gmail.com" target="_blank">antech777@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hello <span name="Samuel Key">Samuel.<br><br></span></div><span name="Samuel Key">Sorry, but I can't undersatand how can the file fragmentation affect it's readability and parsing. Even if the file has thousands of fragments and "brain-damaging" :) runlist in many extended FILE records on NTFS partition, your program, if it uses Win API, will see it like a solid file (althougt there will be very low performance of sequental access). So there should be no any faults on reading highly fragmented files.<br></span></div><span name="Samuel Key">I have checked out the "direct" reading of EnSight case from the cluster via samba with ParaView 5.0.0 on Windows machine. The same error appears as in my first message, although there are <b>different files and filesystems</b>.<br></span></div><span name="Samuel Key">I think the problem is not a fragmentation but a <b>file size</b>. Some 32-bit-like relicts or so in EnSight reader (it's just a guess, sorry if I'm wrong). But my hypothesis does not asks the question why does the bug affect only Windows versions... Maybe there are differences between versions concerning data types. Do you have any >4 GB binary EnSight 6 or Gold case to test? I can't (not allowed) to publish my case and it's also big enough for upload (although theoretically I can do it from my home PC).<br><br></span></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">2016-02-29 20:27 GMT+03:00 Samuel Key <span dir="ltr"><<a href="mailto:samuelkey@bresnan.net" target="_blank">samuelkey@bresnan.net</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<tt>Andrew,<br>
<br>
Let us focus on MS Windows for a moment. The NTS file system and
the use of the file system by the O/S is not quite as it should
be.<br>
<br>
In my experience, the above combination routinely creates an
unbelievable number of file fragments. Think in terms of confetti.
As time goes by, the fragmentation builds. Even though you have a
respectable amount of RAM, the free space itself on the disk
driver can get rather disorganized.<br>
<br>
You can start by using the MS-supplied defrager. Run this utility
five to seven times in a row and things might start looking
better. However the free-space can still remain disjointed.
Personally, I prefer a 3rd party utility for organizing and
defraging the NTS file system (for example, PerfectDisk from
Raxco). Even this utility has to run a couple of times plus once
during the O/S re-boot phase to get a decent amount of
consolidated free space and a somewhat consolidate O/S paging
file. All of this has to be done on the O/S partition (usually C:)
and the partition where you read in the 5+ GB Ensight Geometry
file. <br>
<br>
On my machine, the MS Win7 O/S has its own partition (100GB \C:)
plus another smaller dedicated partition for a second paging file
(28GB \F:). All of my simulation work including the EnSight
results file-sets reside on an entirely separate disk drive (280GB
\G: that now needs to be even larger). The 3rd party file
consolidation software runs "day and night" on the \G: partition.<br>
<br>
Just to emphasize the point, copying the 5+ GB EnSight geometry
file on to workstation can result in many scattered fragments even
when the free-space starts out consolidated. If the free space is
not consolidated, the file fragmentation will be even more. <br>
<br>
The attached screen capture, shows the O/S managed paging-file
(brown) in the \C: partition is beginning to fragment.<br>
<br>
If you go down this path and it helps, please let me know the
outcome.<br>
</tt><br>
Regards,<br>
<br>
Sam Key<span><br>
<br>
<div>On 2/29/2016 1:04 AM, Andrew wrote:<br>
</div>
</span><blockquote type="cite"><span>
<div dir="ltr">
<div>Another "experimental data". I was able to load this big
case in ParaView viat the CGNS format. It was neccesary to
select CGNS 3.0 and HDF5 options in CFX Converter. But when I
apply the Extract Block filter I see no any boundaries, only
fluid domain (that is displayed OK). So CGNS format is not a
wayout...<br>
</div>
</div>
</span><div class="gmail_extra"><br>
<div class="gmail_quote"><span>2016-02-29 9:00 GMT+03:00 Andrew <span dir="ltr"><<a href="mailto:antech777@gmail.com" target="_blank">antech777@gmail.com</a>></span>:<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I performed another test with <b>EnSi.ght 6</b>
(not Gold) format. Conversion was made on Win-7 x64
machine. The same results: PV 4.3 crashed, PV 5.0 "cannot
stat" (full error message is in my first post on this
topic). Sorry, now I need to work on with my report...
<div><div><div>
<div><br>
<div class="gmail_extra"><br>
<br>
<br>
<br>
<div class="gmail_quote">2016-02-29 8:47 GMT+03:00
Andrew <span dir="ltr"><<a href="mailto:antech777@gmail.com" target="_blank"></a><a href="mailto:antech777@gmail.com" target="_blank">antech777@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hello. Sorry for silence, I'm
usually offline on holydays, although I have
internet connection and read your mail.<br>
<br>
<h3><span name="Samuel Key">Samuel Key</span></h3>
<p>Thanks for your attention.</p>
<p>About transferring files from Linux to
Windows. I use binary EnSight files because
CFX converter fails to write so large files
in ASCII format. But I tried to convert from
CFX to EnSight (binary) on the same Win-7
x64 machine with ParaView. Both installed
versions (4.3 and 5.0) failed to open the
geometry (I unchecked all fields). ParaView
4.3 crashed, ParaView 5.0 just cannot open
file...</p>
<p>I think that the root cause may be a
geometry file size. It is 4.9 GiB (5.3 GB),
mesh have about 150 millions of cells, not
600 thusands like in your case. Its possible
that ParaView for Windows cannot process so
large EnSight files, although the Linux
version reads in this geometry and displays
outline (swapping is so intensive so I
cannot displaye something else).<br>
</p>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2016-02-26
19:31 GMT+03:00 Samuel Key <span dir="ltr"><<a href="mailto:samuelkey@bresnan.net" target="_blank"></a><a href="mailto:samuelkey@bresnan.net" target="_blank">samuelkey@bresnan.net</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <tt>Andrew,<br>
<br>
For what it is worth, I use
ParaView
(ParaView-5.0.0-Qt4-OpenGL2-Windows-64bit)
on Windows-7 64bit. I just now
tested two medium-sized, EnSight
Gold-formatted simulation
results files (600,000 finite
elements) and both worked.<br>
<br>
In the past, I have had
difficulty with the EnSight
ASCII-formatted *.case file when
moving the simulation results
(*.case file and friends) from
MS Windows to EnSight running on
a linux platform. It was
necessary to pass the ASCII
*.case file through a 'dos2unix'
filter. Transferring files the
other way might have the same
problem???<br>
<br>
Hope this helps.<span><font color="#888888"><br>
<br>
Samuel Key<br>
FMA Development, LLC<br>
1005 39th Ave NE<br>
Great Falls, Montana 59404<br>
USA<br>
</font></span></tt>
<div>
<div><br>
<div>On 2/25/2016 11:26 PM,
Andrew wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>Hello. We
use Ansys CFX
in our work
but the
standard
post-processor
CFD-Post
consumes one
of Ansys
licenses while
Paraview is
free. So I
often make
pictures for
reports with
ParaView to
preserve Ansys
license for
another
users/needs
(because other
users are not
familiar with
ParaView and
CFD-Post may
consume
general "acfd"
license
feature that
may be used
for
calculations).
I convert CFX
results to
EnSight Gold
format in CFX
Solver Manager
(Export
Results) and
then open in
ParaView. It
was working
well on Linux
(CentOS 6.7
x64, Ubuntu
12.04 x64 with
updates) but
now I work
with large
files and I
need to use
another
machine that
has 128 GB of
RAM and runs
Windows-7 x64.<br>
</div>
I downloaded and
installed the
following
version:
ParaView-5.0.0-Qt4-OpenGL2-Windows-64bit.exe<br>
</div>
When I open my
EnSight-format
file I receive the
error message that
geometry file
cannot be read. I
tried to move
EnSight files to a
simple location
(D:\Temp) and
renamed them
(test.case,
test.geom + edited
the case to change
geometry file name
to "simple" one).
The same result,
it cannot find
geometry file...
Full error message
is as follows:<br>
<br>
ERROR: In
C:\bbd\df0abce0\source-paraview\VTK\IO\EnSight\vtkEnSightGoldBinaryReader.cxx,
line 114 <br>
vtkEnSightGoldBinaryReader
(0000000009451230):
stat failed. <br>
<br>
<br>
ERROR: In
C:\bbd\df0abce0\source-paraview\VTK\IO\EnSight\vtkEnSightGoldBinaryReader.cxx,
line 220 <br>
vtkEnSightGoldBinaryReader
(0000000009451230):
Unable to open
file:
D:\Storage\Media\Work\Gradirni\CFX\EnSight\C-G002L3S-R020-S000-14.5\/C-G002L3S-R020-S000.geom<br>
<br>
<br>
ERROR: In
C:\bbd\df0abce0\source-paraview\VTK\IO\EnSight\vtkEnSightReader.cxx,
line 306 <br>
vtkEnSightGoldBinaryReader
(0000000009451230):
error reading
geometry file <br>
<br>
<br>
ERROR: In
C:\bbd\df0abce0\source-paraview\VTK\Common\ExecutionModel\vtkExecutive.cxx,
line 784 <br>
vtkCompositeDataPipeline
(0000000008E1A640):
Algorithm
vtkEnSightGoldBinaryReader(0000000009451230)
returned failure
for request:
vtkInformation
(00000000093C3920)
<br>
Debug: Off <br>
Modified Time:
214873 <br>
Reference Count:
1 <br>
Registered
Events: (none) <br>
Request:
REQUEST_DATA <br>
ALGORITHM_AFTER_FORWARD:
1 <br>
FORWARD_DIRECTION:
0 <br>
FROM_OUTPUT_PORT:
0<br>
<br>
</div>
OK, I tried to open
it in ParaView 4.3.
It says nothing
(only border of the
message window
appears) and
crashes. At that
moment I see in the
status bar that
EnSight reader
progress is 100%.<br>
<br>
</div>
<div>Both ParaView
5.0.0 and 4.3 was
tested with EnSight
files exported on
Linux and Windows
machine. It didn't
help.<br>
</div>
<div><br>
</div>
OK, I tried to export
CFX results to CGNS
format. ParaView 5.0.0
(without MPI) cannot
read it, so I fed it
to ParaView 4.3. It
says that its "unable
to filed any meshes".<br>
<br>
</div>
<div>I also performed an
experiment on my Linux
laptop. It has only 16
GB of RAM so its
unreal to work with
such a big file on
this machine but it
reads the same EnSight
case and displays the
domain! No arrays
selected (mesh only),
swap file grows up to
8 GB but the file
opens without errors
and crashes. It's only
a test because I have
not any Linux machine
with memory large
enough (our Linux
cluster has 128 GB of
RAM on every node
including master but
it has very poor
video).<br>
</div>
<div><br>
</div>
I switched to CFD Post
for this case but I want
to know are there any
workarounds to open my
EnSight-format results
in ParaView for Windows.<br>
<br>
</div>
Thanks for your attention.<br>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<span>
<pre>_______________________________________________
Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a>
Please keep messages on-topic and check the ParaView Wiki at: <a href="http://paraview.org/Wiki/ParaView" target="_blank">http://paraview.org/Wiki/ParaView</a>
Search the list archives at: <a href="http://markmail.org/search/?q=ParaView" target="_blank">http://markmail.org/search/?q=ParaView</a>
Follow this link to subscribe/unsubscribe:
<a href="http://public.kitware.com/mailman/listinfo/paraview" target="_blank">http://public.kitware.com/mailman/listinfo/paraview</a>
</pre>
</span></blockquote>
<br>
</div>
<br>
_______________________________________________<br>
Powered by <a href="http://www.kitware.com" target="_blank"></a><a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
<br>
Visit other Kitware open-source
projects at <a href="http://www.kitware.com/opensource/opensource.html" target="_blank"></a><a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Please keep messages on-topic and
check the ParaView Wiki at: <a href="http://paraview.org/Wiki/ParaView" target="_blank"></a><a href="http://paraview.org/Wiki/ParaView" target="_blank">http://paraview.org/Wiki/ParaView</a><br>
<br>
Search the list archives at: <a href="http://markmail.org/search/?q=ParaView" target="_blank"></a><a href="http://markmail.org/search/?q=ParaView" target="_blank">http://markmail.org/search/?q=ParaView</a><br>
<br>
Follow this link to
subscribe/unsubscribe:<br>
<a href="http://public.kitware.com/mailman/listinfo/paraview" rel="noreferrer" target="_blank">http://public.kitware.com/mailman/listinfo/paraview</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div></div></div>
</blockquote>
</div>
<br>
</div><div><div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a>
Please keep messages on-topic and check the ParaView Wiki at: <a href="http://paraview.org/Wiki/ParaView" target="_blank">http://paraview.org/Wiki/ParaView</a>
Search the list archives at: <a href="http://markmail.org/search/?q=ParaView" target="_blank">http://markmail.org/search/?q=ParaView</a>
Follow this link to subscribe/unsubscribe:
<a href="http://public.kitware.com/mailman/listinfo/paraview" target="_blank">http://public.kitware.com/mailman/listinfo/paraview</a>
</pre>
</div></div></blockquote>
<br>
</div>
<br>_______________________________________________<br>
Powered by <a href="http://www.kitware.com" rel="noreferrer" target="_blank">www.kitware.com</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" rel="noreferrer" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Please keep messages on-topic and check the ParaView Wiki at: <a href="http://paraview.org/Wiki/ParaView" rel="noreferrer" target="_blank">http://paraview.org/Wiki/ParaView</a><br>
<br>
Search the list archives at: <a href="http://markmail.org/search/?q=ParaView" rel="noreferrer" target="_blank">http://markmail.org/search/?q=ParaView</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="http://public.kitware.com/mailman/listinfo/paraview" rel="noreferrer" target="_blank">http://public.kitware.com/mailman/listinfo/paraview</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>