[Paraview] loading nc formatted data
Simon Su
newsgroup4ssu at gmail.com
Tue Mar 22 15:07:57 EDT 2011
Hi Ken,
This is great. I will work with the scientific programming team that came up
with the file to see if we can go anywhere with it. Thank you so much for
taking the time to look into the file.
best
-simon
On Tue, Mar 22, 2011 at 2:22 PM, Moreland, Kenneth <kmorel at sandia.gov>wrote:
> Simon,
>
> After taking a look at your zos_*.nc file, I believe the problem is that
> the bounds variables (specifically lon_vertices and lat_vertices) are
> incorrect and causing malformed cells. The proper format for these
> variables according to the CF convention is documented here –
> http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/ch07.html#cell-boundaries –
> under "Bounds for 2-D coordinate variables with 4-sided cells". These
> variables are referred to as lonbnd and latbnd in the CF documentation, but
> otherwise the name is inconsequential.
>
> These bounds variables are supposed to contain a 4-tuple for each cell in
> the grid. The 4 values in each tuple specify the longitude or latitude
> coordinates in counterclockwise order starting with the "lower left" value.
> Adjacent cells should have duplicate longitude and latitude coordinates
> where appropriate.
>
> So, assuming that the bounds of the first cell (which I am extrapolating
> from other coordinate variables in the zos file) is –280 to –279 in the
> longitude direction and –82 to –81 in the latitude direction, then the first
> 4-tuple in lon_vertices should be [–280, –279, –279, –280] and the first
> tuple in lat_vertices should be [–82, –82, –81, –81]. The next entries in
> the lon_vertices and lat_vertices arrays are for the adjacent cell in the
> horizontal direction and should be [–279, –278, –278, –279]
> and [–82, –82, –81, –81], respectively.
>
> In this regard, the values stored in zon*.nc's lon_vertices and
> lat_vertices are nonsensical. ncdump shows the first two entries of each
> as:
>
> lon_vertices =
> -280, -279, -278, -277,
> -276, -275, -274, -273,
>
> lat_vertices =
> -82, -82, -82, -82,
> -82, -82, -82, -82,
>
>
> As you can see, these bounds do not form proper quadrilaterals. They are
> all degenerate polygons on the –82 latitude coordinate. My guess is that
> whatever wrote out this file mistook the format for boundary variables for
> coordinate variables.
>
> -Ken
>
> **** Kenneth Moreland
> *** Sandia National Laboratories
> ***********
> *** *** *** email: kmorel at sandia.gov
> ** *** ** phone: (505) 844-8919
> *** web: http://www.cs.unm.edu/~kmorel
>
> From: Simon Su <newsgroup4ssu at gmail.com>
> Date: Tue, 22 Mar 2011 11:40:46 -0400
>
> To: Kenneth Moreland <kmorel at sandia.gov>
> Cc: Andy Bauer <andy.bauer at kitware.com>, "paraview at paraview.org" <
> paraview at paraview.org>
> Subject: Re: [Paraview] loading nc formatted data
>
> Hi Ken,
>
> Thank you again for your help.
>
> Is there another wiki page that also describe all the xml used syntax in
> the http://www.vtk.org/Wiki/ParaView/Plugin_HowTo ? It may help for me to
> have an overall picture of the xml syntax used? thanks. And is there a
> complete example out there that show the whole process on how one can create
> a custom data loader plugin into ParaView? Not sure if I am saying this
> correctly but something like, say I have this data file in my own format
> that has grid information and a scalar variable. How do I write a data
> loader plugin so that I can use ParaView to visualize the data. Then if I
> have to guess, first you write this xml file to describe the data loader
> plugin to ParaView and then use use sort of utility to generate the plugin
> standard code, and then implement the function in the code to load and
> create your vtk data structure (grid data type and scalar variable data
> type) and then pass it to ParaView. something like that? thanks...
>
> yes. it make sense that it is a part of VTK. ParaView is to expose VTK
> functionality after all.
>
> good thing is, I think we settle on a single large nc file that has all the
> information needed to plot the visualization, including tripolar grid
> information. Hopefully that will simplify things for me.
>
> I am sorry that the ftp site is not behaving. I have placed the files at
> http://www.cs.uh.edu/~ssu/4ParaView together with the nc file I used to
> generate the image 4.png (what looks like a double grid to me) and 5.png
> (coloring by the variable zos which I don't thin it should look like that
> ) Hopefully vtkNetCDFCFReader will do the job for me.
>
> thanks
> -simon
>
>
>
> On Mon, Mar 21, 2011 at 7:12 PM, Moreland, Kenneth <kmorel at sandia.gov>wrote:
>
>> It looks like the documentation for a reader plugin is slightly messed
>> up. I'll leave it to Andy (or anyone who is not me) to fix. The example in
>> the ParaView source does not list a file in SERVER_MANAGER_SOURCES because
>> it uses a source (vtkPNGReader.cxx) that is already compiled as part of VTK.
>> The Wiki is trying to capture the fact that usually you are building a
>> plugin with your own reader, and you will have to list your source code for
>> the reader there. vtkMyPNGReader.cxx is just a stand in for some reader
>> that you wrote. It probably does not really exist. I noticed, however,
>> that the Wiki documentation is also wrong in that the XML file is pointing
>> to the vtkPNGReader class when the CMake configuration is implying that it
>> should be pointing to the (imaginary) vtkMyPNGReader class.
>>
>> The vtkNetCDFCFReader.cxx file is located
>> in ParaView/VTK/IO/vtkNetCDFCFReader.cxx. Don't get too wrapped up around
>> the complexity of the NetCDF reader definition in readers.xml. It's
>> complicated because it supports defining a time series as a collection of
>> files (
>> http://www.vtk.org/Wiki/Animating_legacy_VTK_file_series#Making_custom_readers_work_with_file_series).
>> If you collect all times in a single netCDF file (or don't have time
>> steps), you can skip all that and just define a new object much like that of
>> netCDFReaderCore in reader.xml (in the sources ProxyGroup of course).
>>
>> That said, I think this should all be unnecessary since the CF
>> convention (and ParaView reader) already contains a means of defining cells
>> that bridge across these seems. I suspect it has nothing to do with the CF
>> convention version. It is probably that your data files with openings at
>> the seems are not defining a bounds attribute, which is optional in CF. If
>> you could post some example data (or even the listing of the file header
>> from ncdump) I might be able to verify that.
>>
>> I'm not sure what is going on with the new data set (4.png). I can't
>> comment on it much because your ftp server seems to be broken for me and I
>> don't remember the image too well. At any rate, I would probably say
>> something like I couldn't tell you what was going on without seeing the nc
>> file.
>>
>> -Ken
>>
>> **** Kenneth Moreland
>> *** Sandia National Laboratories
>> ***********
>> *** *** *** email: kmorel at sandia.gov
>> ** *** ** phone: (505) 844-8919
>> *** web: http://www.cs.unm.edu/~kmorel
>>
>> From: Simon Su <newsgroup4ssu at gmail.com>
>> Date: Mon, 21 Mar 2011 15:19:24 -0400
>> To: Kenneth Moreland <kmorel at sandia.gov>
>> Cc: Andy Bauer <andy.bauer at kitware.com>, "paraview at paraview.org" <
>> paraview at paraview.org>
>>
>> Subject: Re: [Paraview] loading nc formatted data
>>
>> Hi Ken,
>>
>> Thank you for explaining.
>>
>> The reader that is being used is vtkNetCDFCFReader. It is defined in
>>> ParaView/Servers/ServerManager/Resources/readers.xml, although it is not
>>> obvious.
>>>
>>>
>> Can you (or Andy) point me to documentation more documentation beside
>> http://www.vtk.org/Wiki/ParaView/Plugin_HowTo ? Andy also metioned
>> readers.xml file you mentioned. But after going through ParaView/Servers/
>> ServerManager/Resources/readers.xml, I am still fuzzy about where the file
>> vtkNetCDFCFReader.cxx ( if any - I would like to know how it is implemented
>> and maybe start by modifying it in my learning process). The plugin wiki
>> page on "Adding a Reader" section mentioned CMakeList.txt file
>>
>> FIND_PACKAGE(ParaView REQUIRED)
>> INCLUDE(${PARAVIEW_USE_FILE})
>> ADD_PARAVIEW_PLUGIN(MyReader "1.0"
>>
>>
>> SERVER_MANAGER_XML MyPNGReader.xml
>> SERVER_MANAGER_SOURCES vtkMyPNGReader.cxx
>> GUI_RESOURCE_FILES MyReaderGUI.xml)
>>
>>
>> but the one on my source tree
>> ParaView-3.10.0/Examples/Plugins/Reader/CMakeList.txt looks like
>>
>> ADD_PARAVIEW_PLUGIN(MyPNGReader "1.0"
>> SERVER_MANAGER_XML readers.xml
>> GUI_RESOURCE_FILES pqReader.xml
>> )
>>
>> note no vtkMyPNGReader.cxx file mention and the file is also not in the
>> directory. I also did a search on source tree for vtkNetCDFCFReader.cxx
>> file but can't find it. I think I am missing a big piece of something since
>> I am not getting the point that I should be getting after looking at
>> readers.xml. In fact, I can't find any of the netcdf file readers (other
>> than CF reader). It is also not listed under my Plugin Manager GUI on the
>> ParaView that I compiled but it is loading the CF convention nc file. Are
>> they (the *.cxx files) generated on the fly during compile time?
>>
>> Another thing, besides http://www.paraview.org/Wiki/VisIt_Database_Bridge, is there a place that I can find how to use that VisIt bridge in ParaView.
>> Maybe an example of how to load silo file? I actually have a plugin of my
>> own in VisIt that can load the climate modeling data that I have. I would
>> like to see if I can use my VisIt plugin in ParaView and not write another
>> plugin for ParaView? :)
>>
>>
>>
>>> As the name implies, this reader reads netCDF files using the CF
>>> convention. (As the default netCDF reader, it also gracefully handles files
>>> that do not follow this convention.) With this assumption, I will try to
>>> explain what it does. It reads arrays as regular 1, 2, or 3D arrays,
>>> possibly with time. The CF convention also provides a means to assign
>>> coordinates to each grid point and to identify the coordinates as longitude
>>> or latitude.
>>>
>>>
>> The published nc files we have do adhere to CF convention we also use CMOR
>> in one of the pipeline. I noticed Data_tos_O1_2001-2002.nc is at CF-1.0 and
>> since I am new at CF Convention, I can only speculate that what I see at
>> ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/4.png with what looks like
>> double grid is due to difference between CF-1.0 and CF-1.4? Will
>> vtkNetCDFCFReader handle CF-1.4 data?
>>
>>
>>
>>
>>> Even though coordinates are defined as longitude and latitude, the
>>> topology of the grid itself is still a grid. Thus, the grid gets wrapped
>>> around, but still has these seems that you see because topologically the one
>>> end of the regular grid is not attached to the other. I can't think of any
>>> filter that will identify and close these seems. In fact, it's not
>>> straightforward to do at all. If you look at your topology, it is not lain
>>> out on a simple spherical grid.
>>>
>>> The "right" way to solve your problem, which may or may not be in your
>>> control, is to create netCDF files that specify cell boundaries for a closed
>>> topology. Your netCDF file must be following at least some parts of the CF
>>> convention; your data would not show up as a sphere if it were not. The CF
>>> convention provides a way of defining cells that are not constrained by a
>>> regular grid topology. It is done through a "bounds" attribute on the
>>> dimension descriptor variables. You need either 1D or 2D bounds. They are
>>> described in this section of the CF convention documentation:
>>> http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/cf-conventions.html#cell-boundaries
>>> .
>>>
>>>
>> Agreed. The hack we have in VisIt loader (replace the last grid value with
>> the 1st grid value to create a "closed topology" ) is not "generalizable"
>> and will blow up in my face if I hand it to the user.
>>
>>
>> thanks
>> -simon
>>
>>
>>
>>
>>
>>> The VTK test data has examples of both 1D bounds
>>> (tos_O1_2001-2002.nc) and 2D bounds (sampleCurveGrid4.nc). You can get the
>>> VTK test data from git (http://vtk.org/VTKData.git), or download them
>>> directly from the gitourious server:
>>>
>>>
>>> http://vtk.org/gitweb?p=VTKData.git;a=blob;f=Data/tos_O1_2001-2002.nc;h=30aa4a9b5e08b9bdf64540f2b144d83b279cca6c;hb=HEAD
>>>
>>>
>>> http://vtk.org/gitweb?p=VTKData.git;a=blob;f=Data/sampleCurveGrid4.nc;h=0ab89c27a25f92c047b58dca8b3057ca8d4df017;hb=HEAD
>>>
>>>
>>> -Ken
>>>
>>> **** Kenneth Moreland
>>> *** Sandia National Laboratories
>>> ***********
>>> *** *** *** email: *kmorel at sandia.gov
>>> *** *** ** phone: (505) 844-8919
>>> *** web: *http://www.cs.unm.edu/~kmorel*
>>>
>>> From: Simon Su <newsgroup4ssu at gmail.com>
>>> Date: Fri, 18 Mar 2011 16:43:42 -0400
>>> To: Andy Bauer <andy.bauer at kitware.com>
>>> Cc: <paraview at paraview.org>
>>> Subject: Re: [Paraview] loading nc formatted data
>>>
>>> Hi Andy,
>>>
>>> python trace gave me
>>>
>>> zos_Omon_GFDLESM2M_historical_r1i1p1_186101188012_nc = NetCDFReader(
>>> FileName=['/work/sms/data/cmor-20110128/mon/ocean/zos/r1i1p1/zos_Omon_GFDL-ESM2M_historical_r1i1p1_186101-188012.nc']
>>> )
>>>
>>> and there are tons of netcdf reader in ParaView and it is not in the
>>> plugin directory
>>>
>>> sms:/local/home/build/paraview/ParaView-3.10.0/Plugins> pwd
>>> /local/home/build/paraview/ParaView-3.10.0/Plugins
>>> sms:/local/home/build/paraview/ParaView-3.10.0/Plugins> ll
>>> total 84
>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 AdiosReader/
>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 AnalyzeNIfTIReaderWriter/
>>> -rw-r--r-- 1 sms t 3261 Mar 9 13:31 CMakeLists.txt
>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 CoProcessingScriptGenerator/
>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 CosmoFilters/
>>> drwxr-xr-x 4 sms t 4096 Mar 10 12:37 EyeDomeLighting/
>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 ForceTime/
>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 H5PartReader/
>>> drwxr-xr-x 4 sms t 4096 Mar 10 12:37 Manta/
>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 Moments/
>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 NetDMFReader/
>>> drwxr-xr-x 6 sms t 4096 Mar 10 12:37 PointSprite/
>>> drwxr-xr-x 4 sms t 4096 Mar 10 12:37 PrismPlugins/
>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 pvblot/
>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 SierraPlotTools/
>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 SLACTools/
>>> drwxr-xr-x 4 sms t 4096 Mar 10 12:37 StreamingView/
>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 SurfaceLIC/
>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 Vapor/
>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 VRPN/
>>> drwxr-xr-x 2 sms t 4096 Mar 9 13:31 VRUI/
>>> sms:/local/home/build/paraview/ParaView-3.10.0/Plugins>
>>>
>>>
>>> Can you help describe how Netcdf files plugins are done in ParaView. Do
>>> they have a super class of Netcdf that they all derive from to write the
>>> different flavors of netcdf readers? If so, where can the code be found?
>>>
>>> thanks
>>> -simon
>>>
>>>
>>>
>>> On Fri, Mar 18, 2011 at 10:30 AM, Andy Bauer <andy.bauer at kitware.com>wrote:
>>>
>>>> Hi Simon,
>>>>
>>>> Replies below...
>>>>
>>>> On Thu, Mar 17, 2011 at 12:41 PM, Simon Su <newsgroup4ssu at gmail.com>wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/1.png
>>>>> ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/2.png
>>>>> ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/3.png
>>>>>
>>>>> the above are screen shot from older version of *.nc files that I have
>>>>> that I loaded up with Paraview 3.10.0 64-bit which I compiled myself. As you
>>>>> can see, the grid is correctly loaded. But in the data, there is a crack. Is
>>>>> there a filter that can fix this in ParaView? :)
>>>>>
>>>>
>>>> I'm not aware of any filter that will fix this automatically. Can you
>>>> describe the grid a bit more? It kind of looks like a multiblock of
>>>> structured grids.
>>>>
>>>>>
>>>>> ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/4.png is the latest nc file
>>>>> that I have of similar simulation preprocessed output. When I loaded it up,
>>>>> it is clearly doing making assumption on the grid that is not correct and
>>>>> hence, the double looking grid.
>>>>>
>>>>> The questions now are:
>>>>>
>>>>> 1. ParaView has lots of *.nc file loader. How do I know which loader is
>>>>> ParaView using to load the data? If I pick a type for the Files of type
>>>>> option in the open File window, will that gurantees that ParaView will be
>>>>> using that particular file loader?
>>>>>
>>>>
>>>> If there is an ambiguity for which file loader to use (i.e. multiple
>>>> readers assume the same extension), then the GUI should pop up a dialog for
>>>> you to specify which one to use. You can use the python trace to figure out
>>>> exactly what reader is being used to load the file.
>>>>
>>>>
>>>>>
>>>>> 2. If I know which loader is used, I would like to see if I can modify
>>>>> the existing loader to create a new type of *.nc file loader to fix the grid
>>>>> of my data. Where is the code in the source tree of ParaView is the loader
>>>>> plugin placed?
>>>>>
>>>> Based on the name of the name of the reader from the python script,
>>>> you can look up the actual class name in the
>>>> ParaView/Servers/ServerManager/Resources/readers.xml file.
>>>>
>>>>
>>>>>
>>>>> or if there is an easier solution to fix my grid problem that doesn't
>>>>> involve developing a new data loader plugin that would be better.... :)
>>>>> please let me know...
>>>>>
>>>>> thanks
>>>>> -simon
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Powered by www.kitware.com
>>>>>
>>>>> Visit other Kitware open-source projects at
>>>>> http://www.kitware.com/opensource/opensource.html
>>>>>
>>>>> Please keep messages on-topic and check the ParaView Wiki at:
>>>>> http://paraview.org/Wiki/ParaView
>>>>>
>>>>> Follow this link to subscribe/unsubscribe:
>>>>> http://www.paraview.org/mailman/listinfo/paraview
>>>>>
>>>>>
>>>>
>>> _______________________________________________ Powered by
>>> www.kitware.com Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html Please keep messages
>>> on-topic and check the ParaView Wiki at:
>>> http://paraview.org/Wiki/ParaView Follow this link to
>>> subscribe/unsubscribe: http://www.paraview.org/mailman/listinfo/paraview
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.paraview.org/pipermail/paraview/attachments/20110322/ae656f54/attachment-0001.htm>
More information about the ParaView
mailing list