PyNIO

Important announcement: NCAR has made the decision to adopt Python as the scripting language platform of choice for future development of analysis and visualization tools. As part of this decision, both NCL and PyNIO are being placed into maintenance mode. Please read this open letter to NCL users to understand what kind of impact this will have on the future of NCL and PyNIO.

PyNIO is a Python package that allows read and/or write access to a variety of data formats using an interface modeled on netCDF. PyNIO is composed of a C library called libnio along with a Python module based on and with an interface similar to the Scientific.IO.NetCDF module written by Konrad Hinsen. The C library contains the same data I/O code used in NCL, a scripting language developed for analysis and visualization of geo-scientific data. Currently supported formats include:

More detailed information about PyNIO's support for these formats is available at Data Formats Supported by the PyNIO module.

PyNIO interfaces with the Python environment using the multi-dimensional array module, NumPy. The NumPy version used to build PyNIO is available as the value of the module-level attribute __array_module_version__. PyNIO supports an extended selection mechanism for specifying subsets of file variables using coordinate values. This facility also has support for interpolation, dimension reordering (transposition), and indirect selection via auxiliary 2D coordinate variables. For information about this feature see PyNIO Extended Selection.

Note that while the package name is PyNIO, the module name that is imported into a Python script is Nio.


Installation instructions

For more details on installing packages via "conda", see the download page.

You can download the latest version of PyNIO using conda. We recommend installing it to a separate environment:

  conda create --name pynio_env --channel conda-forge pynio
  source activate pynio_env

Note, you can name the pynio_env environment whatever you like.

To quickly test if Nio installed correctly, you can print the version and inline documentation:

  import Nio
  print Nio.__version__
  print Nio.__doc__


Class NioFile: Access supported-format files for reading and/or writing

Constructor: open_file(filepath, mode='r', options=None, history='', format='')

filepath

The full or relative path of a file with data in a supported format. The file path may contain a home directory indicator starting with "~". A filepath extension (a suffix consisting of the final characters following the last dot) indicates the expected format. The extension can be added to specify the format regardless of whether it is part of the actual filename. Valid extensions include:

  • .nc, .cdf, .netcdf, .nc3, .nc4 (classic) (NetCDF)
  • .nc4 (NetCDF-4) (1.5.0 only)
  • .gr, .gr1, .grb, .grib, .grb1, .grib1, .gr2, .grb2, .grib2 (GRIB 1 and GRIB 2)
  • .hd, .h4, .hdf (HDF)
  • .he2, .he4, .hdfeos (HDF-EOS2)
  • .he5, .hdfeos5 (HDF-EOS5)
  • .h5, .hdf5 (HDF5) (1.5.0 only)
  • .ccm (CCM history files)
  • .shp, .mif, .gmt, .rt1 (OGR Simple Feature formats, such as ESRI Shapefiles)

PyNIO handles these extensions in a case-insensitive manner: .grib, .Grib, and .GRIB all indicate a GRIB file.

Rather than appending a format-specifying suffix to the actual filename, you may instead explicitly specify the file format using the optional argument format, described below.

mode

This optional argument specifies the access mode. If not specified it defaults to 'r' -- read-only. PyNIO has three access modes, some of which can be specified in more than one way:

'r'
Open an existing file for reading. PyNIO returns an error if the file does not exist. Attempts to modify the data will fail.
'w','r+','rw','a'
Open an existing file for modification or if the file does not exist create it. PyNIO returns an error if the file cannot be created because of file system access issues, or if a non-writable format is specified. Note that PyNIO never overwrites an existing file with an entirely new file.
'c'
Create a file open for writing. PyNIO returns an error if the file exists, if the file cannot be created because of file system access issues, or if a non-writable format is specified.
options

This optional argument must be an instance of the NioOptions class created prior to calling the open_file constructor. It is used to set a number of options that affect how the data is read and/or written.

history

This optional argument is a user-specified string that is appended to the global "history" attribute of a file that is opened for writing. If no "history" attribute currently exists in the file, it is created.

format

This optional argument provides an alternate means of explicitly indicating the file format without the need to specify an artificial name for the data file. Instead of tacking on a suffix indicating the file format to the filepath argument, any valid extension documented under filepath, not including the initial period (.), may be supplied as the string value to the format argument. This argument or the tacked-on extension is only required if the file name itself does not end with a recognized extension. It can also be used in certain cases to handle a file as a different format than the extension indicates. For instance, HDF-EOS files often simply have an ".hdf" extension and can be read either as HDF or as HDF-EOS.

A NioFile object has the following standard attributes:

You should never modify any of these dictionary attributes directly. Even though no error is raised if you do change them, the data structures will not be updated correctly and your changes will not be propagated to the file.

Methods:

Class NioOptions: Specify options

Constructor: options()

PyNIO enables you to specify a number of options that affect how data is read and/or written. Some are specific to a supported file format while others apply to all formats. No options need to be set for basic access to any of the supported file formats. There are several mechanisms for setting options. You can establish default option values that apply to all files subsequently opened during the execution of the script or the interactive session. This mechanism is discussed in the section option_defaults below. Conversely, using the NioFile method set_option, you can change certain options repeatedly after an open NioFile object has been created. However, the traditional method for setting options is to create an instance of the NioOptions class.

You create an instance of the NioOptions class prior to calling the NioFile constructor (open_file). Define attributes using the names of the options to be set along with their desired values, and then pass the NioOptions instance as the options argument to the NioFile constructor. All option names and string-typed values are handled in a case-insensitive manner. Options set in this manner override the values contained in the option_defaults dictionary. The same NioOptions instance can be used for multiple calls to the NioFile constructor. There are a number of format-specific options for NetCDF and GRIB files. In addition, there are several options that apply generically to all formats.

Format-specific options

The format-specific options are documented along with the details of the interface to each supported format.

Generic options

FileStructure
If this option is set to "advanced", then it allows you to read advance features like groups and compound data in NetCDF-4 and HDF5 files.
ExplicitFillValues
This option, whose default value is None, allows you to set a single value or a sequence of values that will be masked in any data that is returned from a NioVariable object into a masked array. If a sequence is specified, its first value is set as the fill_value attribute of the array. When this option is set to any value other than None, the value of MaskedArrayMode is set to MaskedExplicit, meaning that any fill value attribute attached to the NioVariable will be ignored. This option can be set for an open NioFile instance, using the set_option method.
MaskAboveValue
This option, whose default value is None, allows you to set a single value above which all values will be masked in any data that is returned from a NioVariable object into a masked array. However, if a value is also set for the MaskBelowValue option that is greater than the value set for MaskAboveValue, then the two values together define a range between which the values will be masked in the returned array. In any case, values specified using the ExplicitFillValues option are also masked. Setting this option to a value other than None causes the value of MaskedArrayMode to be set to MaskedExplicit, meaning that any fill value attribute attached to the NioVariable will be ignored. This option can be set for an open NioFile instance, using the set_option method.
MaskBelowValue
This option, whose default value is None, allows you to set a single value below which all values will be masked in any data that is returned from a NioVariable object into a masked array. However, if a value is also set for the MaskAboveValue option that is less that the value set for MaskBelowValue, then the two values together define a range between which the values will be masked in the returned array. In any case, values specified using the ExplicitFillValues option are also masked. Setting this option to a value other than None causes the value of MaskedArrayMode to be set to MaskedExplicit, meaning that any fill value attribute attached to the NioVariable will be ignored. This option can be set for an open NioFile instance, using the set_option method.
MaskedArrayMode
This option allows the user to control PyNIO's behavior with respect to masked arrays. There are four possible settings that control how PyNIO decides whether to return a masked array or a normal NumPy array:
'MaskedIfFillAtt' (default)
Return a masked array if the file variable has a fill value attribute. The recognized fill value attributes are _FillValue and missing_value. The fill value is assigned to the MaskedArray fill_value attribute. missing_value is not used if _FillValue exists.
'MaskedAlways'
Always return a masked array regardless of whether there is any fill value attribute.
'MaskedNever'
Always return a normal NumPy array regardless of the existence of a fill value attribute.
'MaskedIfFillAttAndValue'
Return a masked array only if there is a fill value attribute and the subset selected from the variable actually contains one or more elements equal to the fill value.
'MaskedExplicit'
Ignore the fill value attributes associated with the file variable and only masked values based on the set values of the options ExplicitFillValues, MaskAboveValue, and MaskBelowValue. Any time any of these options are set to a value other than None, MaskedArrayMode is automatically set to this value (although it may subsequently be given another value).
UseAxisAttribute
This boolean option applies only when extended selection syntax is used for subscripting a PyNIO variable. By default it is False. When set True it specifies that if a coordinate variable associated with a dimension of the variable has the CF-compliant axis attribute, then its value must be used instead of the actual dimension name in the selection string. In order to be valid the axis attribute must have one of the values 'T', 'Z', 'Y', or 'X'. These values are CF-compliant shorthand usually representing the time, height, latitude, and longitude dimensions of a variable, although they can be used to identify other temporal-spatial coordinate systems as well. If the coordinate variables in a file have the axis attribute, then setting this option True allows you to specify an extended selection more compactly.

option_defaults

PyNIO now supplies a module-level dictionary called option_defaults that contains the names of all available options and their default values. Changing the value associated with an option name in the dictionary changes the default value that will apply for all subsequently opened files. You can retrieve the keys and values of this dictionary to discover the current default for any available option. Be aware, however, that if you set an option to an invalid value using this method, there will be no warning until the option's value is needed, generally upon opening a file that supports the option. Also note that if you access individual options by name using this method, the option names are not case-insensitive. They should be specified exactly as listed in the documentation above. However, string-valued option values are case-insensitive.

Class NioVariable: Variable contained in a NioFile object

PyNIO creates a NioVariable object for each variable in the file when the open_file constructor is called. PyNIO also creates a NioVariable object when you call the create_variable method on a NioFile object for writing.

NioVariable objects behave much like array objects defined in the NumPy module, except that their data resides in a file. Subscripting a NioVariable using standard NumPy indexing syntax causes data to be read as a NumPy array. Assigning to a subscripted NioVariable causes data to be written if the file is open in write mode. You can access the complete contents of the variable using the [:] index notation or using the methods get_value and assign_value. An extended selection mechanism using a quoted string inside the selection brackets allows for the specification of variable subsets using coordinate variables associated with named dimensions in the file. This new mechanism supports transposition of dimensions based on the order of the named dimensions in the specification string, as well as interpolation to intermediate coordinate values, selection along an axis using a vector of coordinate values, and indirect specification using an intermediate multidimensional variable. For more information about the extended selection feature, see PyNIO Extended Selection.

NioVariable objects have the following standard attributes:

Like the NioFile object attributes, these attributes should be considered read-only.

Methods:


Usage guide

This guide uses small snippets of sample code to document the basic capabilities of the PyNIO module. Operations that can be performed in read-only mode are listed first.
The following operations require the file to be opened for writing.

Note: the preceding usage information was modeled after this guide to the Python NetCDF module