[Paraview] loading nc formatted data
Andy Bauer
andy.bauer at kitware.com
Wed Mar 23 22:47:17 EDT 2011
FYI: I will take care of the wrong documentation. Not today and probably
not tomorrow but soon :)
Andy
On Tue, Mar 22, 2011 at 3:07 PM, Simon Su <newsgroup4ssu at gmail.com> wrote:
> Hi Ken,
>
> This is great. I will work with the scientific programming team that came
> up with the file to see if we can go anywhere with it. Thank you so much
> for taking the time to look into the file.
>
> best
> -simon
>
>
> On Tue, Mar 22, 2011 at 2:22 PM, Moreland, Kenneth <kmorel at sandia.gov>wrote:
>
>> Simon,
>>
>> After taking a look at your zos_*.nc file, I believe the problem is that
>> the bounds variables (specifically lon_vertices and lat_vertices) are
>> incorrect and causing malformed cells. The proper format for these
>> variables according to the CF convention is documented here –
>> http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/ch07.html#cell-boundaries –
>> under "Bounds for 2-D coordinate variables with 4-sided cells". These
>> variables are referred to as lonbnd and latbnd in the CF documentation, but
>> otherwise the name is inconsequential.
>>
>> These bounds variables are supposed to contain a 4-tuple for each cell
>> in the grid. The 4 values in each tuple specify the longitude or latitude
>> coordinates in counterclockwise order starting with the "lower left" value.
>> Adjacent cells should have duplicate longitude and latitude coordinates
>> where appropriate.
>>
>> So, assuming that the bounds of the first cell (which I am extrapolating
>> from other coordinate variables in the zos file) is –280 to –279 in the
>> longitude direction and –82 to –81 in the latitude direction, then the first
>> 4-tuple in lon_vertices should be [–280, –279, –279, –280] and the first
>> tuple in lat_vertices should be [–82, –82, –81, –81]. The next entries in
>> the lon_vertices and lat_vertices arrays are for the adjacent cell in the
>> horizontal direction and should be [–279, –278, –278, –279]
>> and [–82, –82, –81, –81], respectively.
>>
>> In this regard, the values stored in zon*.nc's lon_vertices and
>> lat_vertices are nonsensical. ncdump shows the first two entries of each
>> as:
>>
>> lon_vertices =
>> -280, -279, -278, -277,
>> -276, -275, -274, -273,
>>
>> lat_vertices =
>> -82, -82, -82, -82,
>> -82, -82, -82, -82,
>>
>>
>> As you can see, these bounds do not form proper quadrilaterals. They
>> are all degenerate polygons on the –82 latitude coordinate. My guess is
>> that whatever wrote out this file mistook the format for boundary variables
>> for coordinate variables.
>>
>> -Ken
>>
>> **** Kenneth Moreland
>> *** Sandia National Laboratories
>> ***********
>> *** *** *** email: kmorel at sandia.gov
>> ** *** ** phone: (505) 844-8919
>> *** web: http://www.cs.unm.edu/~kmorel
>>
>> From: Simon Su <newsgroup4ssu at gmail.com>
>> Date: Tue, 22 Mar 2011 11:40:46 -0400
>>
>> To: Kenneth Moreland <kmorel at sandia.gov>
>> Cc: Andy Bauer <andy.bauer at kitware.com>, "paraview at paraview.org" <
>> paraview at paraview.org>
>> Subject: Re: [Paraview] loading nc formatted data
>>
>> Hi Ken,
>>
>> Thank you again for your help.
>>
>> Is there another wiki page that also describe all the xml used syntax in
>> the http://www.vtk.org/Wiki/ParaView/Plugin_HowTo ? It may help for me to
>> have an overall picture of the xml syntax used? thanks. And is there a
>> complete example out there that show the whole process on how one can create
>> a custom data loader plugin into ParaView? Not sure if I am saying this
>> correctly but something like, say I have this data file in my own format
>> that has grid information and a scalar variable. How do I write a data
>> loader plugin so that I can use ParaView to visualize the data. Then if I
>> have to guess, first you write this xml file to describe the data loader
>> plugin to ParaView and then use use sort of utility to generate the plugin
>> standard code, and then implement the function in the code to load and
>> create your vtk data structure (grid data type and scalar variable data
>> type) and then pass it to ParaView. something like that? thanks...
>>
>> yes. it make sense that it is a part of VTK. ParaView is to expose VTK
>> functionality after all.
>>
>> good thing is, I think we settle on a single large nc file that has all
>> the information needed to plot the visualization, including tripolar grid
>> information. Hopefully that will simplify things for me.
>>
>> I am sorry that the ftp site is not behaving. I have placed the files at
>> http://www.cs.uh.edu/~ssu/4ParaView together with the nc file I used to
>> generate the image 4.png (what looks like a double grid to me) and 5.png
>> (coloring by the variable zos which I don't thin it should look like that
>> ) Hopefully vtkNetCDFCFReader will do the job for me.
>>
>> thanks
>> -simon
>>
>>
>>
>> On Mon, Mar 21, 2011 at 7:12 PM, Moreland, Kenneth <kmorel at sandia.gov>wrote:
>>
>>> It looks like the documentation for a reader plugin is slightly messed
>>> up. I'll leave it to Andy (or anyone who is not me) to fix. The example in
>>> the ParaView source does not list a file in SERVER_MANAGER_SOURCES because
>>> it uses a source (vtkPNGReader.cxx) that is already compiled as part of VTK.
>>> The Wiki is trying to capture the fact that usually you are building a
>>> plugin with your own reader, and you will have to list your source code for
>>> the reader there. vtkMyPNGReader.cxx is just a stand in for some reader
>>> that you wrote. It probably does not really exist. I noticed, however,
>>> that the Wiki documentation is also wrong in that the XML file is pointing
>>> to the vtkPNGReader class when the CMake configuration is implying that it
>>> should be pointing to the (imaginary) vtkMyPNGReader class.
>>>
>>> The vtkNetCDFCFReader.cxx file is located
>>> in ParaView/VTK/IO/vtkNetCDFCFReader.cxx. Don't get too wrapped up around
>>> the complexity of the NetCDF reader definition in readers.xml. It's
>>> complicated because it supports defining a time series as a collection of
>>> files (
>>> http://www.vtk.org/Wiki/Animating_legacy_VTK_file_series#Making_custom_readers_work_with_file_series).
>>> If you collect all times in a single netCDF file (or don't have time
>>> steps), you can skip all that and just define a new object much like that of
>>> netCDFReaderCore in reader.xml (in the sources ProxyGroup of course).
>>>
>>> That said, I think this should all be unnecessary since the CF
>>> convention (and ParaView reader) already contains a means of defining cells
>>> that bridge across these seems. I suspect it has nothing to do with the CF
>>> convention version. It is probably that your data files with openings at
>>> the seems are not defining a bounds attribute, which is optional in CF. If
>>> you could post some example data (or even the listing of the file header
>>> from ncdump) I might be able to verify that.
>>>
>>> I'm not sure what is going on with the new data set (4.png). I can't
>>> comment on it much because your ftp server seems to be broken for me and I
>>> don't remember the image too well. At any rate, I would probably say
>>> something like I couldn't tell you what was going on without seeing the nc
>>> file.
>>>
>>> -Ken
>>>
>>> **** Kenneth Moreland
>>> *** Sandia National Laboratories
>>> ***********
>>> *** *** *** email: kmorel at sandia.gov
>>> ** *** ** phone: (505) 844-8919
>>> *** web: http://www.cs.unm.edu/~kmorel
>>>
>>> From: Simon Su <newsgroup4ssu at gmail.com>
>>> Date: Mon, 21 Mar 2011 15:19:24 -0400
>>> To: Kenneth Moreland <kmorel at sandia.gov>
>>> Cc: Andy Bauer <andy.bauer at kitware.com>, "paraview at paraview.org" <
>>> paraview at paraview.org>
>>>
>>> Subject: Re: [Paraview] loading nc formatted data
>>>
>>> Hi Ken,
>>>
>>> Thank you for explaining.
>>>
>>> The reader that is being used is vtkNetCDFCFReader. It is defined in
>>>> ParaView/Servers/ServerManager/Resources/readers.xml, although it is not
>>>> obvious.
>>>>
>>>>
>>> Can you (or Andy) point me to documentation more documentation beside
>>> http://www.vtk.org/Wiki/ParaView/Plugin_HowTo ? Andy also metioned
>>> readers.xml file you mentioned. But after going through ParaView/Servers/
>>> ServerManager/Resources/readers.xml, I am still fuzzy about where the
>>> file vtkNetCDFCFReader.cxx ( if any - I would like to know how it is
>>> implemented and maybe start by modifying it in my learning process). The
>>> plugin wiki page on "Adding a Reader" section mentioned CMakeList.txt file
>>>
>>> FIND_PACKAGE(ParaView REQUIRED)
>>> INCLUDE(${PARAVIEW_USE_FILE})
>>> ADD_PARAVIEW_PLUGIN(MyReader "1.0"
>>>
>>>
>>>
>>> SERVER_MANAGER_XML MyPNGReader.xml
>>> SERVER_MANAGER_SOURCES vtkMyPNGReader.cxx
>>> GUI_RESOURCE_FILES MyReaderGUI.xml)
>>>
>>>
>>> but the one on my source tree
>>> ParaView-3.10.0/Examples/Plugins/Reader/CMakeList.txt looks like
>>>
>>> ADD_PARAVIEW_PLUGIN(MyPNGReader "1.0"
>>> SERVER_MANAGER_XML readers.xml
>>> GUI_RESOURCE_FILES pqReader.xml
>>> )
>>>
>>> note no vtkMyPNGReader.cxx file mention and the file is also not in the
>>> directory. I also did a search on source tree for vtkNetCDFCFReader.cxx
>>> file but can't find it. I think I am missing a big piece of something since
>>> I am not getting the point that I should be getting after looking at
>>> readers.xml. In fact, I can't find any of the netcdf file readers (other
>>> than CF reader). It is also not listed under my Plugin Manager GUI on the
>>> ParaView that I compiled but it is loading the CF convention nc file. Are
>>> they (the *.cxx files) generated on the fly during compile time?
>>>
>>> Another thing, besides
>>> http://www.paraview.org/Wiki/VisIt_Database_Bridge , is there a place
>>> that I can find how to use that VisIt bridge in ParaView. Maybe an example
>>> of how to load silo file? I actually have a plugin of my own in VisIt that
>>> can load the climate modeling data that I have. I would like to see if I can
>>> use my VisIt plugin in ParaView and not write another plugin for ParaView?
>>> :)
>>>
>>>
>>>
>>>> As the name implies, this reader reads netCDF files using the CF
>>>> convention. (As the default netCDF reader, it also gracefully handles files
>>>> that do not follow this convention.) With this assumption, I will try to
>>>> explain what it does. It reads arrays as regular 1, 2, or 3D arrays,
>>>> possibly with time. The CF convention also provides a means to assign
>>>> coordinates to each grid point and to identify the coordinates as longitude
>>>> or latitude.
>>>>
>>>>
>>> The published nc files we have do adhere to CF convention we also use
>>> CMOR in one of the pipeline. I noticed Data_tos_O1_2001-2002.nc is at
>>> CF-1.0 and since I am new at CF Convention, I can only speculate that what I
>>> see at ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/4.png with what looks
>>> like double grid is due to difference between CF-1.0 and CF-1.4? Will
>>> vtkNetCDFCFReader handle CF-1.4 data?
>>>
>>>
>>>
>>>
>>>> Even though coordinates are defined as longitude and latitude, the
>>>> topology of the grid itself is still a grid. Thus, the grid gets wrapped
>>>> around, but still has these seems that you see because topologically the one
>>>> end of the regular grid is not attached to the other. I can't think of any
>>>> filter that will identify and close these seems. In fact, it's not
>>>> straightforward to do at all. If you look at your topology, it is not lain
>>>> out on a simple spherical grid.
>>>>
>>>> The "right" way to solve your problem, which may or may not be in your
>>>> control, is to create netCDF files that specify cell boundaries for a closed
>>>> topology. Your netCDF file must be following at least some parts of the CF
>>>> convention; your data would not show up as a sphere if it were not. The CF
>>>> convention provides a way of defining cells that are not constrained by a
>>>> regular grid topology. It is done through a "bounds" attribute on the
>>>> dimension descriptor variables. You need either 1D or 2D bounds. They are
>>>> described in this section of the CF convention documentation:
>>>> http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/cf-conventions.html#cell-boundaries
>>>> .
>>>>
>>>>
>>> Agreed. The hack we have in VisIt loader (replace the last grid value
>>> with the 1st grid value to create a "closed topology" ) is not
>>> "generalizable" and will blow up in my face if I hand it to the user.
>>>
>>>
>>> thanks
>>> -simon
>>>
>>>
>>>
>>>
>>>
>>>> The VTK test data has examples of both 1D bounds
>>>> (tos_O1_2001-2002.nc) and 2D bounds (sampleCurveGrid4.nc). You can get the
>>>> VTK test data from git (http://vtk.org/VTKData.git), or download them
>>>> directly from the gitourious server:
>>>>
>>>>
>>>> http://vtk.org/gitweb?p=VTKData.git;a=blob;f=Data/tos_O1_2001-2002.nc;h=30aa4a9b5e08b9bdf64540f2b144d83b279cca6c;hb=HEAD
>>>>
>>>>
>>>> http://vtk.org/gitweb?p=VTKData.git;a=blob;f=Data/sampleCurveGrid4.nc;h=0ab89c27a25f92c047b58dca8b3057ca8d4df017;hb=HEAD
>>>>
>>>>
>>>> -Ken
>>>>
>>>> **** Kenneth Moreland
>>>> *** Sandia National Laboratories
>>>> ***********
>>>> *** *** *** email: *kmorel at sandia.gov
>>>> *** *** ** phone: (505) 844-8919
>>>> *** web: *http://www.cs.unm.edu/~kmorel*
>>>>
>>>> From: Simon Su <newsgroup4ssu at gmail.com>
>>>> Date: Fri, 18 Mar 2011 16:43:42 -0400
>>>> To: Andy Bauer <andy.bauer at kitware.com>
>>>> Cc: <paraview at paraview.org>
>>>> Subject: Re: [Paraview] loading nc formatted data
>>>>
>>>> Hi Andy,
>>>>
>>>> python trace gave me
>>>>
>>>> zos_Omon_GFDLESM2M_historical_r1i1p1_186101188012_nc = NetCDFReader(
>>>> FileName=['/work/sms/data/cmor-20110128/mon/ocean/zos/r1i1p1/zos_Omon_GFDL-ESM2M_historical_r1i1p1_186101-188012.nc']
>>>> )
>>>>
>>>> and there are tons of netcdf reader in ParaView and it is not in the
>>>> plugin directory
>>>>
>>>> sms:/local/home/build/paraview/ParaView-3.10.0/Plugins> pwd
>>>> /local/home/build/paraview/ParaView-3.10.0/Plugins
>>>> sms:/local/home/build/paraview/ParaView-3.10.0/Plugins> ll
>>>> total 84
>>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 AdiosReader/
>>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 AnalyzeNIfTIReaderWriter/
>>>> -rw-r--r-- 1 sms t 3261 Mar 9 13:31 CMakeLists.txt
>>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 CoProcessingScriptGenerator/
>>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 CosmoFilters/
>>>> drwxr-xr-x 4 sms t 4096 Mar 10 12:37 EyeDomeLighting/
>>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 ForceTime/
>>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 H5PartReader/
>>>> drwxr-xr-x 4 sms t 4096 Mar 10 12:37 Manta/
>>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 Moments/
>>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 NetDMFReader/
>>>> drwxr-xr-x 6 sms t 4096 Mar 10 12:37 PointSprite/
>>>> drwxr-xr-x 4 sms t 4096 Mar 10 12:37 PrismPlugins/
>>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 pvblot/
>>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 SierraPlotTools/
>>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 SLACTools/
>>>> drwxr-xr-x 4 sms t 4096 Mar 10 12:37 StreamingView/
>>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 SurfaceLIC/
>>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 Vapor/
>>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 VRPN/
>>>> drwxr-xr-x 2 sms t 4096 Mar 9 13:31 VRUI/
>>>> sms:/local/home/build/paraview/ParaView-3.10.0/Plugins>
>>>>
>>>>
>>>> Can you help describe how Netcdf files plugins are done in ParaView. Do
>>>> they have a super class of Netcdf that they all derive from to write the
>>>> different flavors of netcdf readers? If so, where can the code be found?
>>>>
>>>> thanks
>>>> -simon
>>>>
>>>>
>>>>
>>>> On Fri, Mar 18, 2011 at 10:30 AM, Andy Bauer <andy.bauer at kitware.com>wrote:
>>>>
>>>>> Hi Simon,
>>>>>
>>>>> Replies below...
>>>>>
>>>>> On Thu, Mar 17, 2011 at 12:41 PM, Simon Su <newsgroup4ssu at gmail.com>wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/1.png
>>>>>> ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/2.png
>>>>>> ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/3.png
>>>>>>
>>>>>> the above are screen shot from older version of *.nc files that I have
>>>>>> that I loaded up with Paraview 3.10.0 64-bit which I compiled myself. As you
>>>>>> can see, the grid is correctly loaded. But in the data, there is a crack. Is
>>>>>> there a filter that can fix this in ParaView? :)
>>>>>>
>>>>>
>>>>> I'm not aware of any filter that will fix this automatically. Can you
>>>>> describe the grid a bit more? It kind of looks like a multiblock of
>>>>> structured grids.
>>>>>
>>>>>>
>>>>>> ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/4.png is the latest nc file
>>>>>> that I have of similar simulation preprocessed output. When I loaded it up,
>>>>>> it is clearly doing making assumption on the grid that is not correct and
>>>>>> hence, the double looking grid.
>>>>>>
>>>>>> The questions now are:
>>>>>>
>>>>>> 1. ParaView has lots of *.nc file loader. How do I know which loader
>>>>>> is ParaView using to load the data? If I pick a type for the Files of type
>>>>>> option in the open File window, will that gurantees that ParaView will be
>>>>>> using that particular file loader?
>>>>>>
>>>>>
>>>>> If there is an ambiguity for which file loader to use (i.e. multiple
>>>>> readers assume the same extension), then the GUI should pop up a dialog for
>>>>> you to specify which one to use. You can use the python trace to figure out
>>>>> exactly what reader is being used to load the file.
>>>>>
>>>>>
>>>>>>
>>>>>> 2. If I know which loader is used, I would like to see if I can modify
>>>>>> the existing loader to create a new type of *.nc file loader to fix the grid
>>>>>> of my data. Where is the code in the source tree of ParaView is the loader
>>>>>> plugin placed?
>>>>>>
>>>>> Based on the name of the name of the reader from the python script,
>>>>> you can look up the actual class name in the
>>>>> ParaView/Servers/ServerManager/Resources/readers.xml file.
>>>>>
>>>>>
>>>>>>
>>>>>> or if there is an easier solution to fix my grid problem that doesn't
>>>>>> involve developing a new data loader plugin that would be better.... :)
>>>>>> please let me know...
>>>>>>
>>>>>> thanks
>>>>>> -simon
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Powered by www.kitware.com
>>>>>>
>>>>>> Visit other Kitware open-source projects at
>>>>>> http://www.kitware.com/opensource/opensource.html
>>>>>>
>>>>>> Please keep messages on-topic and check the ParaView Wiki at:
>>>>>> http://paraview.org/Wiki/ParaView
>>>>>>
>>>>>> Follow this link to subscribe/unsubscribe:
>>>>>> http://www.paraview.org/mailman/listinfo/paraview
>>>>>>
>>>>>>
>>>>>
>>>> _______________________________________________ Powered by
>>>> www.kitware.com Visit other Kitware open-source projects at
>>>> http://www.kitware.com/opensource/opensource.html Please keep messages
>>>> on-topic and check the ParaView Wiki at:
>>>> http://paraview.org/Wiki/ParaView Follow this link to
>>>> subscribe/unsubscribe:
>>>> http://www.paraview.org/mailman/listinfo/paraview
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.paraview.org/pipermail/paraview/attachments/20110323/0d355693/attachment-0001.htm>
More information about the ParaView
mailing list