[Paraview] loading nc formatted data

Moreland, Kenneth kmorel at sandia.gov
Tue Mar 22 14:22:34 EDT 2011 

Simon,

After taking a look at your zos_*.nc file, I believe the problem is that the bounds variables (specifically lon_vertices and lat_vertices) are incorrect and causing malformed cells.  The proper format for these variables according to the CF convention is documented here – http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/ch07.html#cell-boundaries – under "Bounds for 2-D coordinate variables with 4-sided cells".  These variables are referred to as lonbnd and latbnd in the CF documentation, but otherwise the name is inconsequential.

These bounds variables are supposed to contain a 4-tuple for each cell in the grid.  The 4 values in each tuple specify the longitude or latitude coordinates in counterclockwise order starting with the "lower left" value.  Adjacent cells should have duplicate longitude and latitude coordinates where appropriate.

So, assuming that the bounds of the first cell (which I am extrapolating from other coordinate variables in the zos file) is –280 to –279 in the longitude direction and –82 to –81 in the latitude direction, then the first 4-tuple in lon_vertices should be [–280, –279, –279, –280] and the first tuple in lat_vertices should be [–82, –82, –81, –81].  The next entries in the lon_vertices and lat_vertices arrays are for the adjacent cell in the horizontal direction and should be [–279, –278, –278, –279] and [–82, –82, –81, –81], respectively.

In this regard, the values stored in zon*.nc's lon_vertices and lat_vertices are nonsensical.  ncdump shows the first two entries of each as:

 lon_vertices =
  -280, -279, -278, -277,
  -276, -275, -274, -273,

 lat_vertices =
  -82, -82, -82, -82,
  -82, -82, -82, -82,

As you can see, these bounds do not form proper quadrilaterals.  They are all degenerate polygons on the –82 latitude coordinate.  My guess is that whatever wrote out this file mistook the format for boundary variables for coordinate variables.

-Ken

   ****      Kenneth Moreland
    ***      Sandia National Laboratories
***********
*** *** ***  email: kmorel at sandia.gov
**  ***  **  phone: (505) 844-8919
    ***      web:   http://www.cs.unm.edu/~kmorel

From: Simon Su <newsgroup4ssu at gmail.com<mailto:newsgroup4ssu at gmail.com>>
Date: Tue, 22 Mar 2011 11:40:46 -0400
To: Kenneth Moreland <kmorel at sandia.gov<mailto:kmorel at sandia.gov>>
Cc: Andy Bauer <andy.bauer at kitware.com<mailto:andy.bauer at kitware.com>>, "paraview at paraview.org<mailto:paraview at paraview.org>" <paraview at paraview.org<mailto:paraview at paraview.org>>
Subject: Re: [Paraview] loading nc formatted data

Hi Ken,

Thank you again for your help.

Is there another wiki page that also describe all the xml used syntax in the http://www.vtk.org/Wiki/ParaView/Plugin_HowTo ? It may help for me to have an overall picture of the xml syntax used? thanks. And is there a complete example out there that show the whole process on how one can create a custom data loader plugin into ParaView? Not sure if I am saying this correctly but something like, say I have this data file in my own format that has grid information and a scalar variable. How do I write a data loader plugin so that I can use ParaView to visualize the data. Then if I have to guess, first you write this xml file to describe the data loader plugin to ParaView and then use use sort of utility to generate the plugin standard code, and then implement the function in the code to load and create your vtk data structure (grid data type and scalar variable data type) and then pass it to ParaView.  something like that?  thanks...

yes. it make sense that it is a part of VTK. ParaView is to expose VTK functionality after all.

good thing is, I think we settle on a single large nc file that has all the information needed to plot the visualization, including tripolar grid information. Hopefully that will simplify things for me.

I am sorry that the ftp site is not behaving. I have placed the files at http://www.cs.uh.edu/~ssu/4ParaView together with the nc file I used to generate the image 4.png (what looks like a double grid to me) and 5.png (coloring by the variable zos  which I don't thin it should look like that )  Hopefully vtkNetCDFCFReader will do the job for me.

thanks
-simon



On Mon, Mar 21, 2011 at 7:12 PM, Moreland, Kenneth <kmorel at sandia.gov<mailto:kmorel at sandia.gov>> wrote:
It looks like the documentation for a reader plugin is slightly messed up.  I'll leave it to Andy (or anyone who is not me) to fix.  The example in the ParaView source does not list a file in SERVER_MANAGER_SOURCES because it uses a source (vtkPNGReader.cxx) that is already compiled as part of VTK.  The Wiki is trying to capture the fact that usually you are building a plugin with your own reader, and you will have to list your source code for the reader there.  vtkMyPNGReader.cxx is just a stand in for some reader that you wrote.  It probably does not really exist.  I noticed, however, that the Wiki documentation is also wrong in that the XML file is pointing to the vtkPNGReader class when the CMake configuration is implying that it should be pointing to the (imaginary) vtkMyPNGReader class.

The vtkNetCDFCFReader.cxx file is located in ParaView/VTK/IO/vtkNetCDFCFReader.cxx.  Don't get too wrapped up around the complexity of the NetCDF reader definition in readers.xml.  It's complicated because it supports defining a time series as a collection of files (http://www.vtk.org/Wiki/Animating_legacy_VTK_file_series#Making_custom_readers_work_with_file_series).  If you collect all times in a single netCDF file (or don't have time steps), you can skip all that and just define a new object much like that of netCDFReaderCore in reader.xml (in the sources ProxyGroup of course).

That said, I think this should all be unnecessary since the CF convention (and ParaView reader) already contains a means of defining cells that bridge across these seems.  I suspect it has nothing to do with the CF convention version.  It is probably that your data files with openings at the seems are not defining a bounds attribute, which is optional in CF.  If you could post some example data (or even the listing of the file header from ncdump) I might be able to verify that.

I'm not sure what is going on with the new data set (4.png).  I can't comment on it much because your ftp server seems to be broken for me and I don't remember the image too well.  At any rate, I would probably say something like I couldn't tell you what was going on without seeing the nc file.

-Ken

   ****      Kenneth Moreland
    ***      Sandia National Laboratories
***********
*** *** ***  email: kmorel at sandia.gov<http://kmorel@sandia.gov>
**  ***  **  phone: (505) 844-8919<tel:%28505%29%20844-8919>
    ***      web:   http://www.cs.unm.edu/~kmorel<http://www.cs.unm.edu/%7Ekmorel>

From: Simon Su <newsgroup4ssu at gmail.com<mailto:newsgroup4ssu at gmail.com>>
Date: Mon, 21 Mar 2011 15:19:24 -0400
To: Kenneth Moreland <kmorel at sandia.gov<mailto:kmorel at sandia.gov>>
Cc: Andy Bauer <andy.bauer at kitware.com<mailto:andy.bauer at kitware.com>>, "paraview at paraview.org<mailto:paraview at paraview.org>" <paraview at paraview.org<mailto:paraview at paraview.org>>

Subject: Re: [Paraview] loading nc formatted data

Hi Ken,

Thank you for explaining.

The reader that is being used is vtkNetCDFCFReader.  It is defined in ParaView/Servers/ServerManager/Resources/readers.xml, although it is not obvious.


Can you (or Andy) point me to documentation more documentation beside http://www.vtk.org/Wiki/ParaView/Plugin_HowTo ? Andy also metioned readers.xml file you mentioned. But after going through ParaView/Servers/
ServerManager/Resources/readers.xml, I am still fuzzy about where the file vtkNetCDFCFReader.cxx ( if any -  I would like to know how it is implemented and maybe start by modifying it in my learning process). The plugin wiki page on "Adding a Reader" section mentioned CMakeList.txt file


FIND_PACKAGE(ParaView REQUIRED)
INCLUDE(${PARAVIEW_USE_FILE})
 ADD_PARAVIEW_PLUGIN(MyReader "1.0"


   SERVER_MANAGER_XML MyPNGReader.xml
   SERVER_MANAGER_SOURCES vtkMyPNGReader.cxx
   GUI_RESOURCE_FILES MyReaderGUI.xml)

but the one on my source tree ParaView-3.10.0/Examples/Plugins/Reader/CMakeList.txt looks like

  ADD_PARAVIEW_PLUGIN(MyPNGReader "1.0"
    SERVER_MANAGER_XML readers.xml
    GUI_RESOURCE_FILES pqReader.xml
    )

note no vtkMyPNGReader.cxx file mention and the file is also not in the directory.  I also did a search on source tree for vtkNetCDFCFReader.cxx file but can't find it. I think I am missing a big piece of something  since I am not getting the point that I should be getting after looking at readers.xml. In fact, I can't find any of the netcdf file readers (other than CF reader). It is also not listed under my Plugin Manager GUI on the ParaView that I compiled but it is loading the CF convention nc file. Are they (the *.cxx files) generated on the fly during compile time?

Another thing, besides http://www.paraview.org/Wiki/VisIt_Database_Bridge , is there a place that I can find how to use that VisIt bridge in ParaView. Maybe an example of how to load silo file? I actually have a plugin of my own in VisIt that can load the climate modeling data that I have. I would like to see if I can use my VisIt plugin in ParaView and not write another plugin for ParaView? :)


As the name implies, this reader reads netCDF files using the CF convention.  (As the default netCDF reader, it also gracefully handles files that do not follow this convention.) With this assumption, I will try to explain what it does.  It reads arrays as regular 1, 2, or 3D arrays, possibly with time.  The CF convention also provides a means to assign coordinates to each grid point and to identify the coordinates as longitude or latitude.


The published nc files we have do adhere to CF convention we also use CMOR in one of the pipeline. I noticed  Data_tos_O1_2001-2002.nc is at CF-1.0 and since I am new at CF Convention, I can only speculate that what I see at ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/4.png with what looks like double grid is due to difference between CF-1.0 and CF-1.4? Will vtkNetCDFCFReader handle CF-1.4 data?



Even though coordinates are defined as longitude and latitude, the topology of the grid itself is still a grid.  Thus, the grid gets wrapped around, but still has these seems that you see because topologically the one end of the regular grid is not attached to the other.  I can't think of any filter that will identify and close these seems.  In fact, it's not straightforward to do at all.  If you look at your topology, it is not lain out on a simple spherical grid.

The "right" way to solve your problem, which may or may not be in your control, is to create netCDF files that specify cell boundaries for a closed topology.  Your netCDF file must be following at least some parts of the CF convention; your data would not show up as a sphere if it were not.  The CF convention provides a way of defining cells that are not constrained by a regular grid topology.  It is done through a "bounds" attribute on the dimension descriptor variables.  You need either 1D or 2D bounds.  They are described in this section of the CF convention documentation: http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/cf-conventions.html#cell-boundaries.


Agreed. The hack we have in VisIt loader (replace the last grid value with the 1st grid value to create a "closed topology" ) is not "generalizable" and will blow up in my face if I hand it to the user.


thanks
-simon




The VTK test data has examples of both 1D bounds (tos_O1_2001-2002.nc) and 2D bounds (sampleCurveGrid4.nc).  You can get the VTK test data from git (http://vtk.org/VTKData.git), or download them directly from the gitourious server:

http://vtk.org/gitweb?p=VTKData.git;a=blob;f=Data/tos_O1_2001-2002.nc;h=30aa4a9b5e08b9bdf64540f2b144d83b279cca6c;hb=HEAD

http://vtk.org/gitweb?p=VTKData.git;a=blob;f=Data/sampleCurveGrid4.nc;h=0ab89c27a25f92c047b58dca8b3057ca8d4df017;hb=HEAD

-Ken

   ****      Kenneth Moreland
    ***      Sandia National Laboratories
***********
*** *** ***  email: kmorel at sandia.gov<http://kmorel@sandia.gov>
**  ***  **  phone: (505) 844-8919<tel:%28505%29%20844-8919>
    ***      web:   http://www.cs.unm.edu/~kmorel<http://www.cs.unm.edu/%7Ekmorel>

From: Simon Su <newsgroup4ssu at gmail.com<mailto:newsgroup4ssu at gmail.com>>
Date: Fri, 18 Mar 2011 16:43:42 -0400
To: Andy Bauer <andy.bauer at kitware.com<mailto:andy.bauer at kitware.com>>
Cc: <paraview at paraview.org<mailto:paraview at paraview.org>>
Subject: Re: [Paraview] loading nc formatted data

Hi Andy,

python trace gave me


zos_Omon_GFDLESM2M_historical_r1i1p1_186101188012_nc = NetCDFReader( FileName=['/work/sms/data/cmor-20110128/mon/ocean/zos/r1i1p1/zos_Omon_GFDL-ESM2M_historical_r1i1p1_186101-188012.nc'] )


and there are tons of netcdf reader in ParaView and it is not in the plugin directory

sms:/local/home/build/paraview/ParaView-3.10.0/Plugins> pwd
/local/home/build/paraview/ParaView-3.10.0/Plugins
sms:/local/home/build/paraview/ParaView-3.10.0/Plugins> ll
total 84
drwxr-xr-x 3 sms t 4096 Mar 10 12:37 AdiosReader/
drwxr-xr-x 2 sms t 4096 Mar 10 12:37 AnalyzeNIfTIReaderWriter/
-rw-r--r-- 1 sms t 3261 Mar  9 13:31 CMakeLists.txt
drwxr-xr-x 3 sms t 4096 Mar 10 12:37 CoProcessingScriptGenerator/
drwxr-xr-x 2 sms t 4096 Mar 10 12:37 CosmoFilters/
drwxr-xr-x 4 sms t 4096 Mar 10 12:37 EyeDomeLighting/
drwxr-xr-x 2 sms t 4096 Mar 10 12:37 ForceTime/
drwxr-xr-x 3 sms t 4096 Mar 10 12:37 H5PartReader/
drwxr-xr-x 4 sms t 4096 Mar 10 12:37 Manta/
drwxr-xr-x 2 sms t 4096 Mar 10 12:37 Moments/
drwxr-xr-x 2 sms t 4096 Mar 10 12:37 NetDMFReader/
drwxr-xr-x 6 sms t 4096 Mar 10 12:37 PointSprite/
drwxr-xr-x 4 sms t 4096 Mar 10 12:37 PrismPlugins/
drwxr-xr-x 3 sms t 4096 Mar 10 12:37 pvblot/
drwxr-xr-x 3 sms t 4096 Mar 10 12:37 SierraPlotTools/
drwxr-xr-x 3 sms t 4096 Mar 10 12:37 SLACTools/
drwxr-xr-x 4 sms t 4096 Mar 10 12:37 StreamingView/
drwxr-xr-x 2 sms t 4096 Mar 10 12:37 SurfaceLIC/
drwxr-xr-x 2 sms t 4096 Mar 10 12:37 Vapor/
drwxr-xr-x 2 sms t 4096 Mar 10 12:37 VRPN/
drwxr-xr-x 2 sms t 4096 Mar  9 13:31 VRUI/
sms:/local/home/build/paraview/ParaView-3.10.0/Plugins>


Can you help describe how Netcdf files plugins are done in ParaView. Do they have a super class of Netcdf that they all derive from to write the different flavors of netcdf readers? If so, where can the code be found?

thanks
-simon



On Fri, Mar 18, 2011 at 10:30 AM, Andy Bauer <andy.bauer at kitware.com<mailto:andy.bauer at kitware.com>> wrote:
Hi Simon,

Replies below...

On Thu, Mar 17, 2011 at 12:41 PM, Simon Su <newsgroup4ssu at gmail.com<mailto:newsgroup4ssu at gmail.com>> wrote:
Hi All,

ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/1.png
ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/2.png
ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/3.png

the above are screen shot from older version of *.nc files that I have that I loaded up with Paraview 3.10.0 64-bit which I compiled myself. As you can see, the grid is correctly loaded. But in the data, there is a crack. Is there a filter that can fix this in ParaView? :)

I'm not aware of any filter that will fix this automatically.   Can you describe the grid a bit more?  It kind of looks like a multiblock of structured grids.

ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/4.png is the latest nc file that I have of similar simulation preprocessed output. When I loaded it up, it is clearly doing making assumption on the grid that is not correct and hence, the double looking grid.

The questions now are:

1. ParaView has lots of *.nc file loader. How do I know which loader is ParaView using to load the data? If I pick a type for the Files of type option in the open File window, will that gurantees that ParaView will be using that particular file loader?

If there is an ambiguity for which file loader to use (i.e. multiple readers assume the same extension), then the GUI should pop up a dialog for you to specify which one to use.  You can use the python trace to figure out exactly what reader is being used to load the file.


2. If I know which loader is used, I would like to see if I can modify the existing loader to create a new type of *.nc file loader to fix the grid of my data. Where is the code in the source tree of ParaView is the loader plugin placed?
Based on the name of the name of the reader from the python script, you can look up the actual class name in the ParaView/Servers/ServerManager/Resources/readers.xml file.


or if there is an easier solution to fix my grid problem that doesn't involve developing a new data loader plugin that would be better.... :)  please let me know...

thanks
-simon


_______________________________________________
Powered by www.kitware.com<http://www.kitware.com>

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView

Follow this link to subscribe/unsubscribe:
http://www.paraview.org/mailman/listinfo/paraview



_______________________________________________ Powered by www.kitware.com<http://www.kitware.com> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView Follow this link to subscribe/unsubscribe: http://www.paraview.org/mailman/listinfo/paraview


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.paraview.org/pipermail/paraview/attachments/20110322/7fd0a24c/attachment-0001.htm>

More information about the ParaView mailing list