[Paraview] Memory seeking problems in using the xdmf2 reader
Pulido, Jesus
pulido at lanl.gov
Fri Aug 14 14:52:13 EDT 2015
Hi David,
Here's the output from the code changes (parsed output of one scalar field), focusing on XdmfValuesBinary.cxx.
XDMF Debug : ..\..\..\..\..\..\VTK\ThirdParty\xdmf2\vtkxdmf2\libsrc\XdmfValuesBinary.cxx line 231 (Accessing Binary CDATA)
Seek as string is 3221225664
XDMF Debug : ..\..\..\..\..\..\VTK\ThirdParty\xdmf2\vtkxdmf2\libsrc\XdmfValuesBinary.cxx line 267 (Data Size : 134217728)
XDMF Debug : ..\..\..\..\..\..\VTK\ThirdParty\xdmf2\vtkxdmf2\libsrc\XdmfValuesBinary.cxx line 268 (Size[Byte]: 1073741824)
XDMF Debug : ..\..\..\..\..\..\VTK\ThirdParty\xdmf2\vtkxdmf2\libsrc\XdmfValuesBinary.cxx line 269 ( Byte 8)
XDMF Debug : ..\..\..\..\..\..\VTK\ThirdParty\xdmf2\vtkxdmf2\libsrc\XdmfValuesBinary.cxx line 287 (Opening Binary Data for Reading : F:/rstrt//r
SEEK as size_t is 2147483647
XDMF Debug : ..\..\..\..\..\..\VTK\ThirdParty\xdmf2\vtkxdmf2\libsrc\XdmfValuesBinary.cxx line 322 (Seek: 2147483647)
XDMF Debug : ..\..\..\..\..\..\VTK\ThirdParty\xdmf2\vtkxdmf2\libsrc\XdmfValuesBinary.cxx line 331 (Hyperslab data)
XDMF Debug : ..\..\..\..\..\..\VTK\ThirdParty\xdmf2\vtkxdmf2\libsrc\XdmfValuesBinary.cxx line 161 (Reduce Rank: 3 to 1)
XDMF Debug : ..\..\..\..\..\..\VTK\ThirdParty\xdmf2\vtkxdmf2\libsrc\XdmfValuesBinary.cxx line 178 (Contiguous byte: 1073741824)
As a string, it looks like the seek location is being read right. Somewhere when it's converted to size_t, it defaults to 2147483647.
Jesus
________________________________
From: David E DeMarle [dave.demarle at kitware.com]
Sent: Friday, August 14, 2015 6:33 AM
To: Pulido, Jesus
Cc: paraview at paraview.org
Subject: Re: [Paraview] Memory seeking problems in using the xdmf2 reader
Mind trying this?
diff --git a/ThirdParty/xdmf2/vtkxdmf2/libsrc/XdmfValuesBinary.cxx b/ThirdParty/xdmf2/vtkxdmf2/libsrc/XdmfValuesBinary.cxx
index 04ea0a7..d4cb2ed 100644
--- a/ThirdParty/xdmf2/vtkxdmf2/libsrc/XdmfValuesBinary.cxx
+++ b/ThirdParty/xdmf2/vtkxdmf2/libsrc/XdmfValuesBinary.cxx
@@ -187,7 +187,7 @@ public:
size_t XdmfValuesBinary::getSeek(){
if(this->Seek==NULL)return 0;
- return static_cast<size_t>(atoi(this->Seek));
+ return static_cast<size_t>(atol(this->Seek));
}
@@ -244,6 +244,7 @@ XdmfValuesBinary::Read(XdmfArray *anArray){
}else{
this->Seek = NULL;
}
+ cerr << "Seek as string is " << Value << endl;
}
{
XdmfConstString Value = this->Get("Compression");
@@ -291,6 +292,7 @@ XdmfValuesBinary::Read(XdmfArray *anArray){
//strcpy(path+strlen(this->DOM->GetWorkingDirectory()), DataSetName);
try{
size_t seek = this->getSeek();
+ cerr << "SEEK as size_t is " << seek << endl;
switch(getCompressionType()){
case Zlib:
XdmfDebug("Compression: Zlib");
David E DeMarle
Kitware, Inc.
R&D Engineer
21 Corporate Drive
Clifton Park, NY 12065-8662
Phone: 518-881-4909
On Thu, Jul 30, 2015 at 5:55 PM, Pulido, Jesus <pulido at lanl.gov<mailto:pulido at lanl.gov>> wrote:
Hi David,
The patch did not work, the file is still not read correctly after the second scalar component.
After applying your patch, I did a bit of debugging myself. I enabled XDMF Debug in the code and found that the seek values are incorrect for scalar fields 3, 4, and 6.
They seem to be stuck at the value 2147483647<tel:2147483647>. On the plus side, it looks like the array size (array size in bytes) is being computed correctly.
Please take a look at the parsed debug log file that I've attached. Maybe the seek value is being malformed somewhere before it reaches XdmfValuesBinary.cxx?
Thanks,
Jesus
________________________________
From: David E DeMarle [dave.demarle at kitware.com<mailto:dave.demarle at kitware.com>]
Sent: Wednesday, July 29, 2015 10:36 PM
To: Pulido, Jesus
Cc: paraview at paraview.org<mailto:paraview at paraview.org>
Subject: Re: [Paraview] Memory seeking problems in using the xdmf2 reader
Would you mind trying this Jesus?
diff --git a/ThirdParty/xdmf2/vtkxdmf2/libsrc/XdmfValuesBinary.cxx b/ThirdParty/xdmf2/vtkxdmf2/libsrc/XdmfValuesBinary.cxx
index 04ea0a7..6fe37d0 100644
--- a/ThirdParty/xdmf2/vtkxdmf2/libsrc/XdmfValuesBinary.cxx
+++ b/ThirdParty/xdmf2/vtkxdmf2/libsrc/XdmfValuesBinary.cxx
@@ -187,7 +187,7 @@ public:
size_t XdmfValuesBinary::getSeek(){
if(this->Seek==NULL)return 0;
- return static_cast<size_t>(atoi(this->Seek));
+ return static_cast<size_t>(atol(this->Seek));
}
Once you confirm it fixes it for you too I'll check it in.
Thanks,
David E DeMarle
Kitware, Inc.
R&D Engineer
21 Corporate Drive
Clifton Park, NY 12065-8662
Phone: 518-881-4909<tel:518-881-4909>
On Fri, Jul 24, 2015 at 12:43 PM, Pulido, Jesus <pulido at lanl.gov<mailto:pulido at lanl.gov>> wrote:
Hello,
I have written a custom xmf file used by the xdmf2 reader to properly read in (raw) binary data into Paraview. This normally works but only for small datasets. The file requires the input of a memory "seek" location to start reading the different scalar components of the data. A problem arises when there is a 512^3 dataset, with 5 scalar components that are required to be read in. The first two scalar components are read in correctly, then the next two components receive the error:
ERROR: In /Users/kitware/Dashboards/MyTests/NightlyMaster/ParaViewSuperbuild-Release-Python27/paraview/src/paraview/VTK/IO/Xdmf2/vtkXdmfHeavyData.cxx, line 1128
vtkXdmfReader (0x7f8614672af0): Failed to read attribute data
The 5th component then becomes a duplicate of the 1st component.
What I believe that's happening is that the "seek" value is only being represented as a signed integer, this would explain the memory location rolling from MAX_INT to -MAX_INT, and failing to read in scalar components 3 and 4. Their seek locations fall within this case. By the time the 5th component is read, the seek value may have rolled over back to positive values (seek > 0) and then duplicates the same data as the 1st component.
Is this assumption correct and intended? Is there fix for this or an alternative way around reading large, raw datasets using this format? I'm attaching a sample xmf file
Thank you,
Jesus
_______________________________________________
Powered by www.kitware.com<http://www.kitware.com>
Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView
Search the list archives at: http://markmail.org/search/?q=ParaView
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/paraview
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/paraview/attachments/20150814/e0b30e30/attachment.html>
More information about the ParaView
mailing list