[Paraview] loading nc formatted data
Simon Su
newsgroup4ssu at gmail.com
Thu Mar 24 13:24:33 EDT 2011
Hi Andy, thanks.... :)
Ken, the newer version produced by the scientific programming team has some
improvement. But the crack along the north pole (tripolar poles between Asia
and America continent) is still there. I saved a screenshot at
http://www.cs.uh.edu/~ssu/4ParaView/20110324/1.png
the data file is a little more than 800MB if you can get to
ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/20110324/thetao_Omon_GFDL-ESM2M_historical_r1i1p1_194601-195012.nc
I don't think it is right but I have don't have a point of reference. If you
have the time, can you point me to the data file which is proper or a screen
shot of a properly rendered vis of the CF convention file?
thank you very much
-simon
On Wed, Mar 23, 2011 at 10:47 PM, Andy Bauer <andy.bauer at kitware.com> wrote:
> FYI: I will take care of the wrong documentation. Not today and probably
> not tomorrow but soon :)
>
> Andy
>
>
> On Tue, Mar 22, 2011 at 3:07 PM, Simon Su <newsgroup4ssu at gmail.com> wrote:
>
>> Hi Ken,
>>
>> This is great. I will work with the scientific programming team that came
>> up with the file to see if we can go anywhere with it. Thank you so much
>> for taking the time to look into the file.
>>
>> best
>> -simon
>>
>>
>> On Tue, Mar 22, 2011 at 2:22 PM, Moreland, Kenneth <kmorel at sandia.gov>wrote:
>>
>>> Simon,
>>>
>>> After taking a look at your zos_*.nc file, I believe the problem is
>>> that the bounds variables (specifically lon_vertices and lat_vertices) are
>>> incorrect and causing malformed cells. The proper format for these
>>> variables according to the CF convention is documented here –
>>> http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/ch07.html#cell-boundaries –
>>> under "Bounds for 2-D coordinate variables with 4-sided cells". These
>>> variables are referred to as lonbnd and latbnd in the CF documentation, but
>>> otherwise the name is inconsequential.
>>>
>>> These bounds variables are supposed to contain a 4-tuple for each cell
>>> in the grid. The 4 values in each tuple specify the longitude or latitude
>>> coordinates in counterclockwise order starting with the "lower left" value.
>>> Adjacent cells should have duplicate longitude and latitude coordinates
>>> where appropriate.
>>>
>>> So, assuming that the bounds of the first cell (which I am
>>> extrapolating from other coordinate variables in the zos file) is –280 to
>>> –279 in the longitude direction and –82 to –81 in the latitude direction,
>>> then the first 4-tuple in lon_vertices should be [–280, –279, –279, –280]
>>> and the first tuple in lat_vertices should be [–82, –82, –81, –81]. The
>>> next entries in the lon_vertices and lat_vertices arrays are for the
>>> adjacent cell in the horizontal direction and should be
>>> [–279, –278, –278, –279] and [–82, –82, –81, –81], respectively.
>>>
>>> In this regard, the values stored in zon*.nc's lon_vertices and
>>> lat_vertices are nonsensical. ncdump shows the first two entries of each
>>> as:
>>>
>>> lon_vertices =
>>> -280, -279, -278, -277,
>>> -276, -275, -274, -273,
>>>
>>> lat_vertices =
>>> -82, -82, -82, -82,
>>> -82, -82, -82, -82,
>>>
>>>
>>> As you can see, these bounds do not form proper quadrilaterals. They
>>> are all degenerate polygons on the –82 latitude coordinate. My guess is
>>> that whatever wrote out this file mistook the format for boundary variables
>>> for coordinate variables.
>>>
>>> -Ken
>>>
>>> **** Kenneth Moreland
>>> *** Sandia National Laboratories
>>> ***********
>>> *** *** *** email: kmorel at sandia.gov
>>> ** *** ** phone: (505) 844-8919
>>> *** web: http://www.cs.unm.edu/~kmorel
>>>
>>> From: Simon Su <newsgroup4ssu at gmail.com>
>>> Date: Tue, 22 Mar 2011 11:40:46 -0400
>>>
>>> To: Kenneth Moreland <kmorel at sandia.gov>
>>> Cc: Andy Bauer <andy.bauer at kitware.com>, "paraview at paraview.org" <
>>> paraview at paraview.org>
>>> Subject: Re: [Paraview] loading nc formatted data
>>>
>>> Hi Ken,
>>>
>>> Thank you again for your help.
>>>
>>> Is there another wiki page that also describe all the xml used syntax in
>>> the http://www.vtk.org/Wiki/ParaView/Plugin_HowTo ? It may help for me
>>> to have an overall picture of the xml syntax used? thanks. And is there a
>>> complete example out there that show the whole process on how one can create
>>> a custom data loader plugin into ParaView? Not sure if I am saying this
>>> correctly but something like, say I have this data file in my own format
>>> that has grid information and a scalar variable. How do I write a data
>>> loader plugin so that I can use ParaView to visualize the data. Then if I
>>> have to guess, first you write this xml file to describe the data loader
>>> plugin to ParaView and then use use sort of utility to generate the plugin
>>> standard code, and then implement the function in the code to load and
>>> create your vtk data structure (grid data type and scalar variable data
>>> type) and then pass it to ParaView. something like that? thanks...
>>>
>>> yes. it make sense that it is a part of VTK. ParaView is to expose VTK
>>> functionality after all.
>>>
>>> good thing is, I think we settle on a single large nc file that has all
>>> the information needed to plot the visualization, including tripolar grid
>>> information. Hopefully that will simplify things for me.
>>>
>>> I am sorry that the ftp site is not behaving. I have placed the files at
>>> http://www.cs.uh.edu/~ssu/4ParaView together with the nc file I used to
>>> generate the image 4.png (what looks like a double grid to me) and 5.png
>>> (coloring by the variable zos which I don't thin it should look like that
>>> ) Hopefully vtkNetCDFCFReader will do the job for me.
>>>
>>> thanks
>>> -simon
>>>
>>>
>>>
>>> On Mon, Mar 21, 2011 at 7:12 PM, Moreland, Kenneth <kmorel at sandia.gov>wrote:
>>>
>>>> It looks like the documentation for a reader plugin is slightly
>>>> messed up. I'll leave it to Andy (or anyone who is not me) to fix. The
>>>> example in the ParaView source does not list a file in
>>>> SERVER_MANAGER_SOURCES because it uses a source (vtkPNGReader.cxx) that is
>>>> already compiled as part of VTK. The Wiki is trying to capture the fact
>>>> that usually you are building a plugin with your own reader, and you will
>>>> have to list your source code for the reader there. vtkMyPNGReader.cxx is
>>>> just a stand in for some reader that you wrote. It probably does not really
>>>> exist. I noticed, however, that the Wiki documentation is also wrong in
>>>> that the XML file is pointing to the vtkPNGReader class when the CMake
>>>> configuration is implying that it should be pointing to the (imaginary)
>>>> vtkMyPNGReader class.
>>>>
>>>> The vtkNetCDFCFReader.cxx file is located
>>>> in ParaView/VTK/IO/vtkNetCDFCFReader.cxx. Don't get too wrapped up around
>>>> the complexity of the NetCDF reader definition in readers.xml. It's
>>>> complicated because it supports defining a time series as a collection of
>>>> files (
>>>> http://www.vtk.org/Wiki/Animating_legacy_VTK_file_series#Making_custom_readers_work_with_file_series).
>>>> If you collect all times in a single netCDF file (or don't have time
>>>> steps), you can skip all that and just define a new object much like that of
>>>> netCDFReaderCore in reader.xml (in the sources ProxyGroup of course).
>>>>
>>>> That said, I think this should all be unnecessary since the CF
>>>> convention (and ParaView reader) already contains a means of defining cells
>>>> that bridge across these seems. I suspect it has nothing to do with the CF
>>>> convention version. It is probably that your data files with openings at
>>>> the seems are not defining a bounds attribute, which is optional in CF. If
>>>> you could post some example data (or even the listing of the file header
>>>> from ncdump) I might be able to verify that.
>>>>
>>>> I'm not sure what is going on with the new data set (4.png). I can't
>>>> comment on it much because your ftp server seems to be broken for me and I
>>>> don't remember the image too well. At any rate, I would probably say
>>>> something like I couldn't tell you what was going on without seeing the nc
>>>> file.
>>>>
>>>> -Ken
>>>>
>>>> **** Kenneth Moreland
>>>> *** Sandia National Laboratories
>>>> ***********
>>>> *** *** *** email: kmorel at sandia.gov
>>>> ** *** ** phone: (505) 844-8919
>>>> *** web: http://www.cs.unm.edu/~kmorel
>>>>
>>>> From: Simon Su <newsgroup4ssu at gmail.com>
>>>> Date: Mon, 21 Mar 2011 15:19:24 -0400
>>>> To: Kenneth Moreland <kmorel at sandia.gov>
>>>> Cc: Andy Bauer <andy.bauer at kitware.com>, "paraview at paraview.org" <
>>>> paraview at paraview.org>
>>>>
>>>> Subject: Re: [Paraview] loading nc formatted data
>>>>
>>>> Hi Ken,
>>>>
>>>> Thank you for explaining.
>>>>
>>>> The reader that is being used is vtkNetCDFCFReader. It is defined
>>>>> in ParaView/Servers/ServerManager/Resources/readers.xml, although it is not
>>>>> obvious.
>>>>>
>>>>>
>>>> Can you (or Andy) point me to documentation more documentation beside
>>>> http://www.vtk.org/Wiki/ParaView/Plugin_HowTo ? Andy also metioned
>>>> readers.xml file you mentioned. But after going through ParaView/Servers/
>>>> ServerManager/Resources/readers.xml, I am still fuzzy about where the
>>>> file vtkNetCDFCFReader.cxx ( if any - I would like to know how it is
>>>> implemented and maybe start by modifying it in my learning process). The
>>>> plugin wiki page on "Adding a Reader" section mentioned CMakeList.txt file
>>>>
>>>> FIND_PACKAGE(ParaView REQUIRED)
>>>> INCLUDE(${PARAVIEW_USE_FILE})
>>>> ADD_PARAVIEW_PLUGIN(MyReader "1.0"
>>>>
>>>>
>>>>
>>>>
>>>> SERVER_MANAGER_XML MyPNGReader.xml
>>>> SERVER_MANAGER_SOURCES vtkMyPNGReader.cxx
>>>> GUI_RESOURCE_FILES MyReaderGUI.xml)
>>>>
>>>>
>>>> but the one on my source tree
>>>> ParaView-3.10.0/Examples/Plugins/Reader/CMakeList.txt looks like
>>>>
>>>> ADD_PARAVIEW_PLUGIN(MyPNGReader "1.0"
>>>> SERVER_MANAGER_XML readers.xml
>>>> GUI_RESOURCE_FILES pqReader.xml
>>>> )
>>>>
>>>> note no vtkMyPNGReader.cxx file mention and the file is also not in the
>>>> directory. I also did a search on source tree for vtkNetCDFCFReader.cxx
>>>> file but can't find it. I think I am missing a big piece of something since
>>>> I am not getting the point that I should be getting after looking at
>>>> readers.xml. In fact, I can't find any of the netcdf file readers (other
>>>> than CF reader). It is also not listed under my Plugin Manager GUI on the
>>>> ParaView that I compiled but it is loading the CF convention nc file. Are
>>>> they (the *.cxx files) generated on the fly during compile time?
>>>>
>>>> Another thing, besides
>>>> http://www.paraview.org/Wiki/VisIt_Database_Bridge , is there a place
>>>> that I can find how to use that VisIt bridge in ParaView. Maybe an example
>>>> of how to load silo file? I actually have a plugin of my own in VisIt that
>>>> can load the climate modeling data that I have. I would like to see if I can
>>>> use my VisIt plugin in ParaView and not write another plugin for ParaView?
>>>> :)
>>>>
>>>>
>>>>
>>>>> As the name implies, this reader reads netCDF files using the CF
>>>>> convention. (As the default netCDF reader, it also gracefully handles files
>>>>> that do not follow this convention.) With this assumption, I will try to
>>>>> explain what it does. It reads arrays as regular 1, 2, or 3D arrays,
>>>>> possibly with time. The CF convention also provides a means to assign
>>>>> coordinates to each grid point and to identify the coordinates as longitude
>>>>> or latitude.
>>>>>
>>>>>
>>>> The published nc files we have do adhere to CF convention we also use
>>>> CMOR in one of the pipeline. I noticed Data_tos_O1_2001-2002.nc is at
>>>> CF-1.0 and since I am new at CF Convention, I can only speculate that what I
>>>> see at ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/4.png with what looks
>>>> like double grid is due to difference between CF-1.0 and CF-1.4? Will
>>>> vtkNetCDFCFReader handle CF-1.4 data?
>>>>
>>>>
>>>>
>>>>
>>>>> Even though coordinates are defined as longitude and latitude, the
>>>>> topology of the grid itself is still a grid. Thus, the grid gets wrapped
>>>>> around, but still has these seems that you see because topologically the one
>>>>> end of the regular grid is not attached to the other. I can't think of any
>>>>> filter that will identify and close these seems. In fact, it's not
>>>>> straightforward to do at all. If you look at your topology, it is not lain
>>>>> out on a simple spherical grid.
>>>>>
>>>>> The "right" way to solve your problem, which may or may not be in
>>>>> your control, is to create netCDF files that specify cell boundaries for a
>>>>> closed topology. Your netCDF file must be following at least some parts of
>>>>> the CF convention; your data would not show up as a sphere if it were not.
>>>>> The CF convention provides a way of defining cells that are not constrained
>>>>> by a regular grid topology. It is done through a "bounds" attribute on the
>>>>> dimension descriptor variables. You need either 1D or 2D bounds. They are
>>>>> described in this section of the CF convention documentation:
>>>>> http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/cf-conventions.html#cell-boundaries
>>>>> .
>>>>>
>>>>>
>>>> Agreed. The hack we have in VisIt loader (replace the last grid value
>>>> with the 1st grid value to create a "closed topology" ) is not
>>>> "generalizable" and will blow up in my face if I hand it to the user.
>>>>
>>>>
>>>> thanks
>>>> -simon
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> The VTK test data has examples of both 1D bounds
>>>>> (tos_O1_2001-2002.nc) and 2D bounds (sampleCurveGrid4.nc). You can get the
>>>>> VTK test data from git (http://vtk.org/VTKData.git), or download them
>>>>> directly from the gitourious server:
>>>>>
>>>>>
>>>>> http://vtk.org/gitweb?p=VTKData.git;a=blob;f=Data/tos_O1_2001-2002.nc;h=30aa4a9b5e08b9bdf64540f2b144d83b279cca6c;hb=HEAD
>>>>>
>>>>>
>>>>> http://vtk.org/gitweb?p=VTKData.git;a=blob;f=Data/sampleCurveGrid4.nc;h=0ab89c27a25f92c047b58dca8b3057ca8d4df017;hb=HEAD
>>>>>
>>>>>
>>>>> -Ken
>>>>>
>>>>> **** Kenneth Moreland
>>>>> *** Sandia National Laboratories
>>>>> ***********
>>>>> *** *** *** email: *kmorel at sandia.gov
>>>>> *** *** ** phone: (505) 844-8919
>>>>> *** web: *http://www.cs.unm.edu/~kmorel*
>>>>>
>>>>> From: Simon Su <newsgroup4ssu at gmail.com>
>>>>> Date: Fri, 18 Mar 2011 16:43:42 -0400
>>>>> To: Andy Bauer <andy.bauer at kitware.com>
>>>>> Cc: <paraview at paraview.org>
>>>>> Subject: Re: [Paraview] loading nc formatted data
>>>>>
>>>>> Hi Andy,
>>>>>
>>>>> python trace gave me
>>>>>
>>>>> zos_Omon_GFDLESM2M_historical_r1i1p1_186101188012_nc =
>>>>> NetCDFReader(
>>>>> FileName=['/work/sms/data/cmor-20110128/mon/ocean/zos/r1i1p1/zos_Omon_GFDL-ESM2M_historical_r1i1p1_186101-188012.nc']
>>>>> )
>>>>>
>>>>> and there are tons of netcdf reader in ParaView and it is not in the
>>>>> plugin directory
>>>>>
>>>>> sms:/local/home/build/paraview/ParaView-3.10.0/Plugins> pwd
>>>>> /local/home/build/paraview/ParaView-3.10.0/Plugins
>>>>> sms:/local/home/build/paraview/ParaView-3.10.0/Plugins> ll
>>>>> total 84
>>>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 AdiosReader/
>>>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 AnalyzeNIfTIReaderWriter/
>>>>> -rw-r--r-- 1 sms t 3261 Mar 9 13:31 CMakeLists.txt
>>>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 CoProcessingScriptGenerator/
>>>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 CosmoFilters/
>>>>> drwxr-xr-x 4 sms t 4096 Mar 10 12:37 EyeDomeLighting/
>>>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 ForceTime/
>>>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 H5PartReader/
>>>>> drwxr-xr-x 4 sms t 4096 Mar 10 12:37 Manta/
>>>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 Moments/
>>>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 NetDMFReader/
>>>>> drwxr-xr-x 6 sms t 4096 Mar 10 12:37 PointSprite/
>>>>> drwxr-xr-x 4 sms t 4096 Mar 10 12:37 PrismPlugins/
>>>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 pvblot/
>>>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 SierraPlotTools/
>>>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 SLACTools/
>>>>> drwxr-xr-x 4 sms t 4096 Mar 10 12:37 StreamingView/
>>>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 SurfaceLIC/
>>>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 Vapor/
>>>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 VRPN/
>>>>> drwxr-xr-x 2 sms t 4096 Mar 9 13:31 VRUI/
>>>>> sms:/local/home/build/paraview/ParaView-3.10.0/Plugins>
>>>>>
>>>>>
>>>>> Can you help describe how Netcdf files plugins are done in ParaView. Do
>>>>> they have a super class of Netcdf that they all derive from to write the
>>>>> different flavors of netcdf readers? If so, where can the code be found?
>>>>>
>>>>> thanks
>>>>> -simon
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Mar 18, 2011 at 10:30 AM, Andy Bauer <andy.bauer at kitware.com>wrote:
>>>>>
>>>>>> Hi Simon,
>>>>>>
>>>>>> Replies below...
>>>>>>
>>>>>> On Thu, Mar 17, 2011 at 12:41 PM, Simon Su <newsgroup4ssu at gmail.com>wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/1.png
>>>>>>> ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/2.png
>>>>>>> ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/3.png
>>>>>>>
>>>>>>> the above are screen shot from older version of *.nc files that I
>>>>>>> have that I loaded up with Paraview 3.10.0 64-bit which I compiled myself.
>>>>>>> As you can see, the grid is correctly loaded. But in the data, there is a
>>>>>>> crack. Is there a filter that can fix this in ParaView? :)
>>>>>>>
>>>>>>
>>>>>> I'm not aware of any filter that will fix this automatically. Can
>>>>>> you describe the grid a bit more? It kind of looks like a multiblock of
>>>>>> structured grids.
>>>>>>
>>>>>>>
>>>>>>> ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/4.png is the latest nc
>>>>>>> file that I have of similar simulation preprocessed output. When I loaded it
>>>>>>> up, it is clearly doing making assumption on the grid that is not correct
>>>>>>> and hence, the double looking grid.
>>>>>>>
>>>>>>> The questions now are:
>>>>>>>
>>>>>>> 1. ParaView has lots of *.nc file loader. How do I know which loader
>>>>>>> is ParaView using to load the data? If I pick a type for the Files of type
>>>>>>> option in the open File window, will that gurantees that ParaView will be
>>>>>>> using that particular file loader?
>>>>>>>
>>>>>>
>>>>>> If there is an ambiguity for which file loader to use (i.e. multiple
>>>>>> readers assume the same extension), then the GUI should pop up a dialog for
>>>>>> you to specify which one to use. You can use the python trace to figure out
>>>>>> exactly what reader is being used to load the file.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> 2. If I know which loader is used, I would like to see if I can
>>>>>>> modify the existing loader to create a new type of *.nc file loader to fix
>>>>>>> the grid of my data. Where is the code in the source tree of ParaView is the
>>>>>>> loader plugin placed?
>>>>>>>
>>>>>> Based on the name of the name of the reader from the python script,
>>>>>> you can look up the actual class name in the
>>>>>> ParaView/Servers/ServerManager/Resources/readers.xml file.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> or if there is an easier solution to fix my grid problem that doesn't
>>>>>>> involve developing a new data loader plugin that would be better.... :)
>>>>>>> please let me know...
>>>>>>>
>>>>>>> thanks
>>>>>>> -simon
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Powered by www.kitware.com
>>>>>>>
>>>>>>> Visit other Kitware open-source projects at
>>>>>>> http://www.kitware.com/opensource/opensource.html
>>>>>>>
>>>>>>> Please keep messages on-topic and check the ParaView Wiki at:
>>>>>>> http://paraview.org/Wiki/ParaView
>>>>>>>
>>>>>>> Follow this link to subscribe/unsubscribe:
>>>>>>> http://www.paraview.org/mailman/listinfo/paraview
>>>>>>>
>>>>>>>
>>>>>>
>>>>> _______________________________________________ Powered by
>>>>> www.kitware.com Visit other Kitware open-source projects at
>>>>> http://www.kitware.com/opensource/opensource.html Please keep messages
>>>>> on-topic and check the ParaView Wiki at:
>>>>> http://paraview.org/Wiki/ParaView Follow this link to
>>>>> subscribe/unsubscribe:
>>>>> http://www.paraview.org/mailman/listinfo/paraview
>>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.paraview.org/pipermail/paraview/attachments/20110324/46c7c543/attachment-0001.htm>
More information about the ParaView
mailing list