<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Although we have certainly run into issues with communicating large messages, I’m not sure this is the issue with MPI-I/O. File sizes and linear offsets use a
type MPI_Offset, which according to the specification should be large enough to hold the size of any file supported by the file system. Although it is true that creating the 3D subarray uses 32-bit indices, those are for each of the separate dimensions, and
2000 falls well below that limit.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Besides, it looks like that error happens when checking an ifstream object, so it looks like the reader is bypassing the MPI-I/O path anyway.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I would follow Berk’s suggesting of looking into the filesystem before diving into the guts of MPI-I/O.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Ken<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext"> ParaView [mailto:paraview-bounces@paraview.org]
<b>On Behalf Of </b>Burlen Loring<br>
<b>Sent:</b> Tuesday, November 22, 2016 12:55 PM<br>
<b>To:</b> Geveci, Berk (External Contact) <berk.geveci@kitware.com>; Keyes S.D. <S.D.Keyes@soton.ac.uk><br>
<b>Cc:</b> paraview@paraview.org<br>
<b>Subject:</b> [EXTERNAL] Re: [Paraview] RAW (binary) file import issues<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">I've hit that before. the RAW reader uses MPI-I/O, and MPI uses signed int everywhere in its API, so there are limits that you end up hitting. For instance to partition the data among processors one must give
MPI-I/O the starting offset of the data in the file as a signed int, and that is limited to 2^31.
<br>
<br>
Burlen<o:p></o:p></p>
<div>
<p class="MsoNormal">On 11/22/2016 11:20 AM, Berk Geveci wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Hmmm that happens to be around 4 GB which makes me wonder if there is a limit somewhere in that reader... Any chance you can try this on a Linux machine?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Thu, Nov 17, 2016 at 10:30 AM, Keyes S.D. <<a href="mailto:S.D.Keyes@soton.ac.uk" target="_blank">S.D.Keyes@soton.ac.uk</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Dear all,<br>
<br>
I am encountering issues when trying to import large 8 bit RAW volumes to Paraview 5.1.2 (64 bit).<br>
<br>
The file I wish to import has xyz dimensions of 2000x2000x1920 voxels (Unsigned char), 7.5GB. I am running a machine with 192 GB RAM and 12 CPU cores.<br>
<br>
I can import up to a 2000x2000x1000 crop of this file, but I encounter the following message when I attempt larger z dimensions:<br>
<br>
ERROR: In C:\bbd\df0abce0\source-paraview\VTK\IO\Image\vtkImageReader2.cxx, line 592<br>
<br>
vtkPVImageReader (0000000007767F90): Initialize: Could not open file <a href="file://cseg_2/ERC/SLS_201606/Myco_2/CT2PpR3h1_/Hyphalseg_Raw_2000_2000_1920_8bit_SKELETON_FILT_PARAVIEW_TEST.raw">
\\cseg_2\ERC\SLS_201606\Myco_2\CT2PpR3h1_\Hyphalseg_Raw_2000_2000_1920_8bit_SKELETON_FILT_PARAVIEW_TEST.raw</a><br>
<br>
<br>
I cannot find an obvious memory ceiling within the paraview settings - any ideas as to whether there is some effective upper limit to import file size?<br>
<br>
Sam<br>
<br>
Samuel D Keyes, MEng, PhD<br>
New Frontiers Fellow<br>
Bioengineering / μVIS Centre for Computed Tomography<br>
University of Southampton<br>
E: <a href="mailto:S.D.Keyes@soton.ac.uk">S.D.Keyes@soton.ac.uk</a><<a href="https://">https://</a><a href="http://www.outlook.soton.ac.uk/owa/">www.outlook.soton.ac.uk/owa/</a>redir.aspx?SURL=dhUg3da8XR77LeItLDzJQwhAwmH0Wx6PHrXeKfxx2tOD3LCr737TCG0AYQBpAGwAdABvADoAUwAuAEQALgBLAGUAeQBlAHMAQABzAG8AdABvAG4ALgBhAGMALgB1AGsA&URL=mailto%3aS.D.Keyes%40soton.ac.uk><br>
T: 07898720248<br>
_______________________________________________<br>
Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/">http://www.kitware.com/</a>opensource/opensource.html<br>
<br>
Please keep messages on-topic and check the ParaView Wiki at: <a href="http://paraview.org/Wiki/ParaView" target="_blank">
http://paraview.org/Wiki/ParaView</a><br>
<br>
Search the list archives at: <a href="http://markmail.org/search/?q=ParaView" target="_blank">
http://markmail.org/search/?q=ParaView</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="http://public.kitware.com/mailman/listinfo/paraview" target="_blank">http://public.kitware.com/mailman/listinfo/paraview</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><br>
<br>
<br>
<o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Powered by <a href="http://www.kitware.com">www.kitware.com</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html">http://www.kitware.com/opensource/opensource.html</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Please keep messages on-topic and check the ParaView Wiki at: <a href="http://paraview.org/Wiki/ParaView">http://paraview.org/Wiki/ParaView</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Search the list archives at: <a href="http://markmail.org/search/?q=ParaView">http://markmail.org/search/?q=ParaView</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Follow this link to subscribe/unsubscribe:<o:p></o:p></pre>
<pre><a href="http://public.kitware.com/mailman/listinfo/paraview">http://public.kitware.com/mailman/listinfo/paraview</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>