[Paraview] RAW (binary) file import issues
Burlen Loring
bloring at lbl.gov
Fri Nov 25 13:05:30 EST 2016
> File sizes and linear offsets use a type MPI_Offset, which according
> to the specification should be large enough to hold the size of any
> file supported by the file system.
hmm, that may be but there are places in MPI I/O that limit to size of
signed int, here is cut direct from the mpi 3 standard:
int MPI_File_read_all(MPI_File fh, void *buf, int count,
MPI_Datatype datatype, MPI_Status *status)
the count argument is a signed int, so one can't read more than INT_MAX
in one go, and here is the VTK code where that is used
291 // Figure out how many bytes to read.
292 vtkIdType length = this->GetDataScalarTypeSize();
293 length *= this->NumberOfScalarComponents;
294 length *= extent[1]-extent[0]+1;
295 length *= extent[3]-extent[2]+1;
296 if (this->GetFileDimensionality() == 3) length *=
extent[5]-extent[4]+1;
297
298 if (length > VTK_INT_MAX)
299 {
300 vtkErrorMacro(<< "Cannot read more than " << VTK_INT_MAX
301 << " bytes at a time.");
302 }
303
304 // Do the read. This is a coordinated parallel operation for
efficiency.
305 MPICall(MPI_File_read_all(file.Handle, buffer,
static_cast<int>(length),
306 MPI_BYTE, MPI_STATUS_IGNORE));
the VTK code checks for over flow. however as that is not the error
reported by the user, I have to admit that I am wrong about Sam's
problem. Sorry about that.
Burlen
On 11/22/2016 02:11 PM, Moreland, Kenneth wrote:
>
> Although we have certainly run into issues with communicating large
> messages, I’m not sure this is the issue with MPI-I/O. File sizes and
> linear offsets use a type MPI_Offset, which according to the
> specification should be large enough to hold the size of any file
> supported by the file system. Although it is true that creating the 3D
> subarray uses 32-bit indices, those are for each of the separate
> dimensions, and 2000 falls well below that limit.
>
> Besides, it looks like that error happens when checking an ifstream
> object, so it looks like the reader is bypassing the MPI-I/O path anyway.
>
> I would follow Berk’s suggesting of looking into the filesystem before
> diving into the guts of MPI-I/O.
>
> -Ken
>
> *From:*ParaView [mailto:paraview-bounces at paraview.org] *On Behalf Of
> *Burlen Loring
> *Sent:* Tuesday, November 22, 2016 12:55 PM
> *To:* Geveci, Berk (External Contact) <berk.geveci at kitware.com>; Keyes
> S.D. <S.D.Keyes at soton.ac.uk>
> *Cc:* paraview at paraview.org
> *Subject:* [EXTERNAL] Re: [Paraview] RAW (binary) file import issues
>
> I've hit that before. the RAW reader uses MPI-I/O, and MPI uses signed
> int everywhere in its API, so there are limits that you end up
> hitting. For instance to partition the data among processors one must
> give MPI-I/O the starting offset of the data in the file as a signed
> int, and that is limited to 2^31.
>
> Burlen
>
> On 11/22/2016 11:20 AM, Berk Geveci wrote:
>
> Hmmm that happens to be around 4 GB which makes me wonder if there
> is a limit somewhere in that reader... Any chance you can try this
> on a Linux machine?
>
> On Thu, Nov 17, 2016 at 10:30 AM, Keyes S.D.
> <S.D.Keyes at soton.ac.uk <mailto:S.D.Keyes at soton.ac.uk>> wrote:
>
> Dear all,
>
> I am encountering issues when trying to import large 8 bit RAW
> volumes to Paraview 5.1.2 (64 bit).
>
> The file I wish to import has xyz dimensions of 2000x2000x1920
> voxels (Unsigned char), 7.5GB. I am running a machine with
> 192 GB RAM and 12 CPU cores.
>
> I can import up to a 2000x2000x1000 crop of this file, but I
> encounter the following message when I attempt larger z
> dimensions:
>
> ERROR: In
> C:\bbd\df0abce0\source-paraview\VTK\IO\Image\vtkImageReader2.cxx,
> line 592
>
> vtkPVImageReader (0000000007767F90): Initialize: Could not
> open file
> \\cseg_2\ERC\SLS_201606\Myco_2\CT2PpR3h1_\Hyphalseg_Raw_2000_2000_1920_8bit_SKELETON_FILT_PARAVIEW_TEST.raw
> <file://cseg_2/ERC/SLS_201606/Myco_2/CT2PpR3h1_/Hyphalseg_Raw_2000_2000_1920_8bit_SKELETON_FILT_PARAVIEW_TEST.raw>
>
>
> I cannot find an obvious memory ceiling within the paraview
> settings - any ideas as to whether there is some effective
> upper limit to import file size?
>
> Sam
>
> Samuel D Keyes, MEng, PhD
> New Frontiers Fellow
> Bioengineering / μVIS Centre for Computed Tomography
> University of Southampton
> E: S.D.Keyes at soton.ac.uk
> <mailto:S.D.Keyes at soton.ac.uk><https://www.outlook.soton.ac.uk/owa/
> <http://www.outlook.soton.ac.uk/owa/>redir.aspx?SURL=dhUg3da8XR77LeItLDzJQwhAwmH0Wx6PHrXeKfxx2tOD3LCr737TCG0AYQBpAGwAdABvADoAUwAuAEQALgBLAGUAeQBlAHMAQABzAG8AdABvAG4ALgBhAGMALgB1AGsA&URL=mailto%3aS.D.Keyes%40soton.ac.uk>
> T: 07898720248
> _______________________________________________
> Powered by www.kitware.com <http://www.kitware.com>
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at:
> http://paraview.org/Wiki/ParaView
> <http://paraview.org/Wiki/ParaView>
>
> Search the list archives at:
> http://markmail.org/search/?q=ParaView
> <http://markmail.org/search/?q=ParaView>
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/paraview
>
>
>
>
> _______________________________________________
>
> Powered bywww.kitware.com <http://www.kitware.com>
>
> Visit other Kitware open-source projects athttp://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at:http://paraview.org/Wiki/ParaView
>
> Search the list archives at:http://markmail.org/search/?q=ParaView
>
> Follow this link to subscribe/unsubscribe:
>
> http://public.kitware.com/mailman/listinfo/paraview
>
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView
>
> Search the list archives at: http://markmail.org/search/?q=ParaView
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/paraview
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/paraview/attachments/20161125/36d9fe4f/attachment.html>
More information about the ParaView
mailing list