<div dir="ltr"><div><div><div>Sorry for many posts, but, I think, I found something interesting (or I just think so :)). I digged a <b>stat</b>() usage in ParaView at official GitHub source repository. If I get it right, there are two types of this function: <b>stat</b>() and <b>stat64</b>(). <font size="1">My only program that deals with large files is a disk editor & NTFS data recovery for Windows (it's free and was made at home), this program uses WinAPI, not C/C++ calls for file operations. So I don't know much about fopen/fclose/stat/e.t.c. on large files.</font> So we have <b>stat</b> and <b>stat64</b> + corresponding data structures. We can see an example in this source:<br><a href="https://github.com/Kitware/VTK/blob/eae7f4446c867ee9d29b466676fa11282d56dafc/IO/LSDyna/private/LSDynaFamily.cxx">https://github.com/Kitware/VTK/blob/eae7f4446c867ee9d29b466676fa11282d56dafc/IO/LSDyna/private/LSDynaFamily.cxx</a><br></div><div>For 64-bit configurations this code uses <b>stat64()</b> and <b>stat64</b> structure.<br></div><div><br></div>But if we look at the problematic place in <b>EnSight reader</b> we will only see the "usual" <b>stat</b>() that is, to my current knowledge, is 32 bit:<br><a href="https://github.com/Kitware/VTK/blob/master/IO/EnSight/vtkEnSightGoldBinaryReader.cxx">https://github.com/Kitware/VTK/blob/master/IO/EnSight/vtkEnSightGoldBinaryReader.cxx</a><br></div>(Please see lines 100 and 101)<br></div>May it be the cause of the subj? Sorry if I'm wrong, I'm not a professional programmer.<br><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-03-01 9:38 GMT+03:00 Andrew <span dir="ltr"><<a href="mailto:antech777@gmail.com" target="_blank">antech777@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I took a look at the code. Te problematic fragment (line 114 in vtkEnSightGoldBinaryReader) just checks the file's existance and opens it in standard way. The problem occures in <b>stat</b>() function that is fed with the full file name and the reference to structure to receive file params. I tried to read the file from Python shell and the result is the same. I guessed that the reason is an additional slash before the short file name (see error message) but small files are processed OK so the only assumption I can make is that the problem occures in the <b>stat</b>() itself on large (> ~2GiB) files due to some strange 32-bit relicts (although itshould work perfectly in 64-bit builds).<br></div>Can anybody test the opening of large (3...4 GiB or more) EnSight Gold binary files in ParaView 5.0.0 or 5.0.1? I cannot check any assumptions because I only have Linux compilators set up and cannot do the "hello-world"-like stat() test under Windows on a large file...<br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">2016-03-01 8:53 GMT+03:00 Andrew <span dir="ltr"><<a href="mailto:antech777@gmail.com" target="_blank">antech777@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hello <span name="Samuel Key">Samuel.<br><br></span></div><span name="Samuel Key">Sorry, but I can't undersatand how can the file fragmentation affect it's readability and parsing. Even if the file has thousands of fragments and "brain-damaging" :) runlist in many extended FILE records on NTFS partition, your program, if it uses Win API, will see it like a solid file (althougt there will be very low performance of sequental access). So there should be no any faults on reading highly fragmented files.<br></span></div><span name="Samuel Key">I have checked out the "direct" reading of EnSight case from the cluster via samba with ParaView 5.0.0 on Windows machine. The same error appears as in my first message, although there are <b>different files and filesystems</b>.<br></span></div><span name="Samuel Key">I think the problem is not a fragmentation but a <b>file size</b>. Some 32-bit-like relicts or so in EnSight reader (it's just a guess, sorry if I'm wrong). But my hypothesis does not asks the question why does the bug affect only Windows versions... Maybe there are differences between versions concerning data types. Do you have any >4 GB binary EnSight 6 or Gold case to test? I can't (not allowed) to publish my case and it's also big enough for upload (although theoretically I can do it from my home PC).<br><br></span></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">2016-02-29 20:27 GMT+03:00 Samuel Key <span dir="ltr"><<a href="mailto:samuelkey@bresnan.net" target="_blank">samuelkey@bresnan.net</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF" text="#000000">
    <tt>Andrew,<br>
      <br>
      Let us focus on MS Windows for a moment. The NTS file system and
      the use of the file system by the O/S is not quite as it should
      be.<br>
      <br>
      In my experience, the above combination routinely creates an
      unbelievable number of file fragments. Think in terms of confetti.
      As time goes by, the fragmentation builds. Even though you have a
      respectable amount of RAM, the free space itself on the disk
      driver can get rather disorganized.<br>
      <br>
      You can start by using the MS-supplied defrager. Run this utility
      five to seven times in a row and things might start looking
      better. However the free-space can still remain disjointed.
      Personally, I prefer a 3rd party utility for organizing and
      defraging the NTS file system (for example, PerfectDisk from
      Raxco). Even this utility has to run a couple of times plus once
      during the O/S re-boot phase to get a decent amount of
      consolidated free space and a somewhat consolidate O/S paging
      file. All of this has to be done on the O/S partition (usually C:)
      and the partition where you read in the 5+ GB Ensight Geometry
      file. <br>
      <br>
      On my machine, the MS Win7 O/S has its own partition (100GB \C:)
      plus another smaller dedicated partition for a second paging file
      (28GB \F:). All of my simulation work including the EnSight
      results file-sets reside on an entirely separate disk drive (280GB
      \G: that now needs to be even larger). The 3rd party file
      consolidation software runs "day and night" on the \G: partition.<br>
      <br>
      Just to emphasize the point, copying the 5+ GB EnSight geometry
      file on to workstation can result in many scattered fragments even
      when the free-space starts out consolidated. If the free space is
      not consolidated, the file fragmentation will be even more. <br>
      <br>
      The attached screen capture, shows the O/S managed paging-file
      (brown) in the \C: partition is beginning to fragment.<br>
      <br>
      If you go down this path and it helps, please let me know the
      outcome.<br>
    </tt><br>
    Regards,<br>
    <br>
    Sam Key<span><br>
    <br>
    <div>On 2/29/2016 1:04 AM, Andrew wrote:<br>
    </div>
    </span><blockquote type="cite"><span>
      <div dir="ltr">
        <div>Another "experimental data". I was able to load this big
          case in ParaView viat the CGNS format. It was neccesary to
          select CGNS 3.0 and HDF5 options in CFX Converter. But when I
          apply the Extract Block filter I see no any boundaries, only
          fluid domain (that is displayed OK). So CGNS format is not a
          wayout...<br>
        </div>
      </div>
      </span><div class="gmail_extra"><br>
        <div class="gmail_quote"><span>2016-02-29 9:00 GMT+03:00 Andrew <span dir="ltr"><<a href="mailto:antech777@gmail.com" target="_blank">antech777@gmail.com</a>></span>:<br>
          </span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div dir="ltr">I performed another test with <b>EnSi.ght 6</b>
              (not Gold) format. Conversion was made on Win-7 x64
              machine. The same results: PV 4.3 crashed, PV 5.0 "cannot
              stat" (full error message is in my first post on this
              topic). Sorry, now I need to work on with my report...
              <div><div><div>
                <div><br>
                  <div class="gmail_extra"><br>
                    <br>
                    <br>
                    <br>
                    <div class="gmail_quote">2016-02-29 8:47 GMT+03:00
                      Andrew <span dir="ltr"><<a href="mailto:antech777@gmail.com" target="_blank"></a><a href="mailto:antech777@gmail.com" target="_blank">antech777@gmail.com</a>></span>:<br>
                      <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                        <div dir="ltr">Hello. Sorry for silence, I'm
                          usually offline on holydays, although I have
                          internet connection and read your mail.<br>
                          <br>
                          <h3><span name="Samuel Key">Samuel Key</span></h3>
                          <p>Thanks for your attention.</p>
                          <p>About transferring files from Linux to
                            Windows. I use binary EnSight files because
                            CFX converter fails to write so large files
                            in ASCII format. But I tried to convert from
                            CFX to EnSight (binary) on the same Win-7
                            x64 machine with ParaView. Both installed
                            versions (4.3 and 5.0) failed to open the
                            geometry (I unchecked all fields). ParaView
                            4.3 crashed, ParaView 5.0 just cannot open
                            file...</p>
                          <p>I think that the root cause may be a
                            geometry file size. It is 4.9 GiB (5.3 GB),
                            mesh have about 150 millions of cells, not
                            600 thusands like in your case. Its possible
                            that ParaView for Windows cannot process so
                            large EnSight files, although the Linux
                            version reads in this geometry and displays
                            outline (swapping is so intensive so I
                            cannot displaye something else).<br>
                          </p>
                          <div>
                            <div>
                              <div class="gmail_extra"><br>
                                <div class="gmail_quote">2016-02-26
                                  19:31 GMT+03:00 Samuel Key <span dir="ltr"><<a href="mailto:samuelkey@bresnan.net" target="_blank"></a><a href="mailto:samuelkey@bresnan.net" target="_blank">samuelkey@bresnan.net</a>></span>:<br>
                                  <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                    <div bgcolor="#FFFFFF" text="#000000"> <tt>Andrew,<br>
                                        <br>
                                        For what it is worth, I use
                                        ParaView
                                        (ParaView-5.0.0-Qt4-OpenGL2-Windows-64bit)
                                        on Windows-7 64bit. I just now
                                        tested two medium-sized, EnSight
                                        Gold-formatted simulation
                                        results files (600,000 finite
                                        elements) and both worked.<br>
                                        <br>
                                        In the past, I have had
                                        difficulty with the EnSight
                                        ASCII-formatted *.case file when
                                        moving the simulation results
                                        (*.case file and friends) from
                                        MS Windows to EnSight running on
                                        a linux platform. It was
                                        necessary to pass the ASCII
                                        *.case file through a 'dos2unix'
                                        filter. Transferring files the
                                        other way might have the same
                                        problem???<br>
                                        <br>
                                        Hope this helps.<span><font color="#888888"><br>
                                            <br>
                                            Samuel Key<br>
                                            FMA Development, LLC<br>
                                            1005 39th Ave NE<br>
                                            Great Falls, Montana 59404<br>
                                            USA<br>
                                          </font></span></tt>
                                      <div>
                                        <div><br>
                                          <div>On 2/25/2016 11:26 PM,
                                            Andrew wrote:<br>
                                          </div>
                                        </div>
                                      </div>
                                      <blockquote type="cite">
                                        <div>
                                          <div>
                                            <div dir="ltr">
                                              <div>
                                                <div>
                                                  <div>
                                                    <div>
                                                      <div>
                                                        <div>Hello. We
                                                          use Ansys CFX
                                                          in our work
                                                          but the
                                                          standard
                                                          post-processor
                                                          CFD-Post
                                                          consumes one
                                                          of Ansys
                                                          licenses while
                                                          Paraview is
                                                          free. So I
                                                          often make
                                                          pictures for
                                                          reports with
                                                          ParaView to
                                                          preserve Ansys
                                                          license for
                                                          another
                                                          users/needs
                                                          (because other
                                                          users are not
                                                          familiar with
                                                          ParaView and
                                                          CFD-Post may
                                                          consume
                                                          general "acfd"
                                                          license
                                                          feature that
                                                          may be used
                                                          for
                                                          calculations).
                                                          I convert CFX
                                                          results to
                                                          EnSight Gold
                                                          format in CFX
                                                          Solver Manager
                                                          (Export
                                                          Results) and
                                                          then open in
                                                          ParaView. It
                                                          was working
                                                          well on Linux
                                                          (CentOS 6.7
                                                          x64, Ubuntu
                                                          12.04 x64 with
                                                          updates) but
                                                          now I work
                                                          with large
                                                          files and I
                                                          need to use
                                                          another
                                                          machine that
                                                          has 128 GB of
                                                          RAM and runs
                                                          Windows-7 x64.<br>
                                                        </div>
                                                        I downloaded and
                                                        installed the
                                                        following
                                                        version:
                                                        ParaView-5.0.0-Qt4-OpenGL2-Windows-64bit.exe<br>
                                                      </div>
                                                      When I open my
                                                      EnSight-format
                                                      file I receive the
                                                      error message that
                                                      geometry file
                                                      cannot be read. I
                                                      tried to move
                                                      EnSight files to a
                                                      simple location
                                                      (D:\Temp) and
                                                      renamed them
                                                      (test.case,
                                                      test.geom + edited
                                                      the case to change
                                                      geometry file name
                                                      to "simple" one).
                                                      The same result,
                                                      it cannot find
                                                      geometry file...
                                                      Full error message
                                                      is as follows:<br>
                                                      <br>
                                                      ERROR: In
                                                      C:\bbd\df0abce0\source-paraview\VTK\IO\EnSight\vtkEnSightGoldBinaryReader.cxx,


                                                      line 114 <br>
                                                      vtkEnSightGoldBinaryReader

                                                      (0000000009451230):

                                                      stat failed. <br>
                                                       <br>
                                                       <br>
                                                      ERROR: In
                                                      C:\bbd\df0abce0\source-paraview\VTK\IO\EnSight\vtkEnSightGoldBinaryReader.cxx,


                                                      line 220 <br>
                                                      vtkEnSightGoldBinaryReader

                                                      (0000000009451230):

                                                      Unable to open
                                                      file:
D:\Storage\Media\Work\Gradirni\CFX\EnSight\C-G002L3S-R020-S000-14.5\/C-G002L3S-R020-S000.geom<br>
                                                       <br>
                                                       <br>
                                                      ERROR: In
                                                      C:\bbd\df0abce0\source-paraview\VTK\IO\EnSight\vtkEnSightReader.cxx,
                                                      line 306 <br>
                                                      vtkEnSightGoldBinaryReader

                                                      (0000000009451230):

                                                      error reading
                                                      geometry file <br>
                                                       <br>
                                                       <br>
                                                      ERROR: In
                                                      C:\bbd\df0abce0\source-paraview\VTK\Common\ExecutionModel\vtkExecutive.cxx,


                                                      line 784 <br>
                                                      vtkCompositeDataPipeline

                                                      (0000000008E1A640):

                                                      Algorithm
                                                      vtkEnSightGoldBinaryReader(0000000009451230)
                                                      returned failure
                                                      for request:
                                                      vtkInformation
                                                      (00000000093C3920)
                                                      <br>
                                                        Debug: Off <br>
                                                        Modified Time:
                                                      214873 <br>
                                                        Reference Count:
                                                      1 <br>
                                                        Registered
                                                      Events: (none) <br>
                                                        Request:
                                                      REQUEST_DATA <br>
                                                       
                                                      ALGORITHM_AFTER_FORWARD:
                                                      1 <br>
                                                       
                                                      FORWARD_DIRECTION:
                                                      0 <br>
                                                       
                                                      FROM_OUTPUT_PORT:
                                                      0<br>
                                                      <br>
                                                    </div>
                                                    OK, I tried to open
                                                    it in ParaView 4.3.
                                                    It says nothing
                                                    (only border of the
                                                    message window
                                                    appears) and
                                                    crashes. At that
                                                    moment I see in the
                                                    status bar that
                                                    EnSight reader
                                                    progress is 100%.<br>
                                                    <br>
                                                  </div>
                                                  <div>Both ParaView
                                                    5.0.0 and 4.3 was
                                                    tested with EnSight
                                                    files exported on
                                                    Linux and Windows
                                                    machine. It didn't
                                                    help.<br>
                                                  </div>
                                                  <div><br>
                                                  </div>
                                                  OK, I tried to export
                                                  CFX results to CGNS
                                                  format. ParaView 5.0.0
                                                  (without MPI) cannot
                                                  read it, so I fed it
                                                  to ParaView 4.3. It
                                                  says that its "unable
                                                  to filed any meshes".<br>
                                                  <br>
                                                </div>
                                                <div>I also performed an
                                                  experiment on my Linux
                                                  laptop. It has only 16
                                                  GB of RAM so its
                                                  unreal to work with
                                                  such a big file on
                                                  this machine but it
                                                  reads the same EnSight
                                                  case and displays the
                                                  domain! No arrays
                                                  selected (mesh only),
                                                  swap file grows up to
                                                  8 GB but the file
                                                  opens without errors
                                                  and crashes. It's only
                                                  a test because I have
                                                  not any Linux machine
                                                  with memory large
                                                  enough (our Linux
                                                  cluster has 128 GB of
                                                  RAM on every node
                                                  including master but
                                                  it has very poor
                                                  video).<br>
                                                </div>
                                                <div><br>
                                                </div>
                                                I switched to CFD Post
                                                for this case but I want
                                                to know are there any
                                                workarounds to open my
                                                EnSight-format results
                                                in ParaView for Windows.<br>
                                                <br>
                                              </div>
                                              Thanks for your attention.<br>
                                            </div>
                                            <br>
                                            <fieldset></fieldset>
                                            <br>
                                          </div>
                                        </div>
                                        <span>
                                          <pre>_______________________________________________
Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a>

Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a>

Please keep messages on-topic and check the ParaView Wiki at: <a href="http://paraview.org/Wiki/ParaView" target="_blank">http://paraview.org/Wiki/ParaView</a>

Search the list archives at: <a href="http://markmail.org/search/?q=ParaView" target="_blank">http://markmail.org/search/?q=ParaView</a>

Follow this link to subscribe/unsubscribe:
<a href="http://public.kitware.com/mailman/listinfo/paraview" target="_blank">http://public.kitware.com/mailman/listinfo/paraview</a>
</pre>
                                        </span></blockquote>
                                      <br>
                                    </div>
                                    <br>
_______________________________________________<br>
                                    Powered by <a href="http://www.kitware.com" target="_blank"></a><a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
                                    <br>
                                    Visit other Kitware open-source
                                    projects at <a href="http://www.kitware.com/opensource/opensource.html" target="_blank"></a><a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
                                    <br>
                                    Please keep messages on-topic and
                                    check the ParaView Wiki at: <a href="http://paraview.org/Wiki/ParaView" target="_blank"></a><a href="http://paraview.org/Wiki/ParaView" target="_blank">http://paraview.org/Wiki/ParaView</a><br>
                                    <br>
                                    Search the list archives at: <a href="http://markmail.org/search/?q=ParaView" target="_blank"></a><a href="http://markmail.org/search/?q=ParaView" target="_blank">http://markmail.org/search/?q=ParaView</a><br>
                                    <br>
                                    Follow this link to
                                    subscribe/unsubscribe:<br>
                                    <a href="http://public.kitware.com/mailman/listinfo/paraview" rel="noreferrer" target="_blank">http://public.kitware.com/mailman/listinfo/paraview</a><br>
                                    <br>
                                  </blockquote>
                                </div>
                                <br>
                              </div>
                            </div>
                          </div>
                        </div>
                      </blockquote>
                    </div>
                    <br>
                  </div>
                </div>
              </div>
            </div></div></div>
          </blockquote>
        </div>
        <br>
      </div><div><div>
      <br>
      <fieldset></fieldset>
      <br>
      <pre>_______________________________________________
Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a>

Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/opensource/opensource.html</a>

Please keep messages on-topic and check the ParaView Wiki at: <a href="http://paraview.org/Wiki/ParaView" target="_blank">http://paraview.org/Wiki/ParaView</a>

Search the list archives at: <a href="http://markmail.org/search/?q=ParaView" target="_blank">http://markmail.org/search/?q=ParaView</a>

Follow this link to subscribe/unsubscribe:
<a href="http://public.kitware.com/mailman/listinfo/paraview" target="_blank">http://public.kitware.com/mailman/listinfo/paraview</a>
</pre>
    </div></div></blockquote>
    <br>
  </div>

<br>_______________________________________________<br>
Powered by <a href="http://www.kitware.com" rel="noreferrer" target="_blank">www.kitware.com</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" rel="noreferrer" target="_blank">http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Please keep messages on-topic and check the ParaView Wiki at: <a href="http://paraview.org/Wiki/ParaView" rel="noreferrer" target="_blank">http://paraview.org/Wiki/ParaView</a><br>
<br>
Search the list archives at: <a href="http://markmail.org/search/?q=ParaView" rel="noreferrer" target="_blank">http://markmail.org/search/?q=ParaView</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="http://public.kitware.com/mailman/listinfo/paraview" rel="noreferrer" target="_blank">http://public.kitware.com/mailman/listinfo/paraview</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>