How to build PyNIO 1.4.1 from source code

PyNIO can only be built on UNIX systems. It has been successfully built on systems running Linux (32 or 64-bit), MacOSX (PPC or Intel), Windows/Cygwin, and IBMs running AIX (64-bit).

This version of PyNIO was tested with Python versions 2.6.x and 2.7.x, and NumPy versions 1.5.1 and 1.6.0. We have some precompiled binaries you can try.

PyNIO has a number of file formats that it supports for reading and/or writing. Some of these file formats are required, and some are optional:

GRIB 2 - optional
HDFEOS 2 - optional
HDFEOS 5 - optional
HDF 4 - required
HDF 5 - optional
NetCDF - required
OGR - optional

For more information on these formats, see the main PyNIO page.

Table of Contents

Download and build non-optional external software
- JPEG
- ZLIB
- NetCDF
Download and build optional external software
- GRIB2
- GDAL/PROJ.4
  - PROJ.4
  - GDAL
- HDF-4
- HDF-EOS 2
  - gctp
- HDF-EOS 5
  - szip
  - HDF-5
Download the source code for PyNIO
Set environment variables if necessary
Execute the 'setup.py' script
Test PyNIO
Where to get help

Download and build non-optional external software

The software listed below is not optional if you plan to build PyNIO. Some of this software may already be installed on your system. You can use the "locate" command to find it.

Once you download and install any external software to use with PyNIO, you must comply with the license of that software, regardless of what PyNIO's license is.

JPEG
Download version 6b (jpeg-7 supposedly has some problems)
Once you have the jpeg source code, you can build and install it with:
```
  ./configure --prefix=/usr/local
  make all install
```
If you downloaded version 6b, you will additionally need to do this:
```
  make install-lib
  make install-headers
```
zlib
Download version 1.2.5 or later.
Once you have the zlib source code, you can build and install it with:
```
  ./configure --prefix=/usr/local
  make all install
```
NetCDF
Download version 4.1.2 or later.
Please note that NetCDF-3 and NetCDF-4 are different file formats and have nothing to do with the version of the NetCDF software. NetCDF software version 4.x supports both NetCDF-3 and NetCDF-4 file formats, but as noted below, you may have to set some flags in order to get support for both. NetCDF-4 files are actually written as HDF5 under the hood.

Currently, PyNIO supports NetCDF-3 files and the "classic" version of NetCDF-4 files. "Classic" NetCDF-4 files look like NetCDF-3 files to the user, but they allow for compression for potentially smaller files. Classic NetCDF-4 files cannot be read by programs that only support NetCDF-3.
NetCDF versions 4.1.3 and later support OPenDAP, and this is built by default.
This software can be built different ways: with or without "classic" NetCDF-4 support, and with or without OPeNDAP support:
- Classic NetCDF-4 support gives you file compression capabilities.
- OPeNDAP support allows you to read NetCDF files served on the web by OPeNDAP servers.
We recommend at least building with NetCDF-4 support turned on.
Decide which version you want to build and follow one of the four set of instructions:
1. Without "classic" NetCDF-4 support, without OPeNDAP support
```
  ./configure --disable-f90 --disable-dap --prefix=/usr/local
  make all install
```
2. Without "classic" NetCDF-4 support, with OPeNDAP support
```
  ./configure --disable-f90 --prefix=/usr/local
  make all install
```
3. With "classic" NetCDF-4 support, with OPeNDAP support
You first need to download and install the following packages:
- zlib - see instructions above.
- szip - Download version 2.0 or later. (Click on "External Libraries Used By HDF5".)
  The szip 2.x software is available from same page where you download HDF-5. Use this version, rather than the one on the official szip page.
```
  ./configure --prefix=/usr/local
  make all install >& make-output
```
- HDF-5 - Download version 1.8.6 or later.
```
  ./configure --with-zlib=/usr/local --with-szlib=/usr/local --prefix=/usr/local
  make all install >& make-output
```
- libcurl
  This software may already be on your system. You can use "locate" to look for "libcurl*" type of files before you install this software yourself.
  Download version 7.16.4 or later. Once you have the source code, you can build and install it with:
```
  ./configure --prefix=/usr/local --with-zlib=/usr/local --with-pic
  make all install >& make-output
```
Then, build NetCDF as follows and be sure to include the "--enable-netcdf-4" option:
```
  ./configure --with-hdf5=/usr/local --with-zlib=/usr/local --with-szlib=/usr/local \
       --prefix=/usr/local --disable-f90 --enable-netcdf-4
  make all install >& make-output
```
Note that OPeNDAP support will be built in by default. This means that you may need the "curl" library later when building PyNIO. This library may already be on your system.
4. With "classic" NetCDF-4 support, without OPeNDAP support
Follow the instructions above for building HDF5, szip, and zlib. You don't need "libcurl". Be sure to include "--disable-dap" on the configure line:
```
  ./configure --disable-dap --with-hdf5=/usr/local --with-zlib=/usr/local --with-szlib=/usr/local \
       --prefix=/usr/local --disable-f90 --enable-netcdf-4
  make all install >& make-output
```
Troubleshooting for either NetCDF build:
Note: if during the "make all install" phase of building the NetCDF software you see an error message like:
```
"cfortran.h", line 138.3: 1506-205 (S) #error cfortran.h:  Can't find your environment among:
    - MIPS cc and f77 2.0. (e.g. Silicon Graphics, DECstations, ...)
 . . .
make[3]: *** [fort-attio.lo] Error 1
```
then you need to "help" the NetCDF build recognize your system by adding the appropriate "-D" macro definition to your CPPFLAGS line. The error message above suggests some of them for you, or you can go to the "Other Builds of the NetCDF Package" page for some tips.
For example, if you are on an IBM/AIX system, you might need the "-DIBMR2Fortran" flag, and you can set it with:
[if CPPFLAGS is already set]:
```
setenv CPPFLAGS  '$CPPFLAGS -DIBMR2Fortran'
```
or
```
export CPPFLAGS='$CPPFLAGS -DIBMR2Fortran'
```
[if CPPFLAGS is not already set]:
```
setenv CPPFLAGS  -DIBMR2Fortran
```
or
```
export CPPFLAGS=-DIBMR2Fortran
```
Download and build optional external software, if needed

GRIB2 software
If you need support for reading GRIB2 files, then you will need to download and install the following packages.
- Jasper - Download version 1.900 or later.
  Once you have the Jasper source code, you can build and install it with:
```
  ./configure --prefix=/usr/local
  make all install
```
  libpng
  Download version 1.2.27 (earlier versions had a serious bug) or later
  Once you have the libpng source code, you can build and install it with:
```
  ./configure  --with-pic --prefix=/usr/local 
  make all install
```
- g2clib - normally you would download this software from http://www.nco.ncep.noaa.gov/pmb/codes/GRIB2/, but there were a few bugs we had to fix. Download the g2clib_with_changes-1.2.1.tar.gz file instead.
  Once you have this modified g2clib source code, make changes to "makefile" to:
  1. change the compiler (CC line) and flags (CFLAGS lines)
  2. make sure the DEFS line has both "-DUSE_JPEG2000" and "-DUSE_PNG"
  3. if on a 64-bit system (x86_64, ia64, etc), make sure the DEFS line also has "-D__64BIT__" (that's two underscores before and after)
  4. add the appropriate location of the Jasper/libpng include files to the "INC" line
  Now type:
```
  make all
  mv libgrib2c.a /usr/local/lib
  cp grib2.h /usr/local/include
```
GDAL/PROJ.4 software
If you need support for reading shapefile, mapinfo, TIGER, etc, file formats, then you will need to download and install the following two packages.
- PROJ.4 - Cartographic Projections Library - Download version 4.7.0 or later.
  Once you have the PROJ.4 source code, you can build and install it with:
```
  ./configure --prefix=/usr/local
  make all install
```
- GDAL - Geospatial Data Abstraction Library - Download version 1.7.x or later.
  Once you have the GDAL source code, you can build and install it with:
```
  ./configure --prefix=/usr/local --with-static-proj4=/usr/local
  make all install
```
HDF-4
Download version 4.2.5 or later. (HDF version 5.x.y is a different software package).
Important note: By default, the HDF-4 include files get installed to $prefix/include. Since this could cause some netCDF include files to get overwritten, you must use the "--includedir" option to install the HDF include files instead to $prefix/include/hdf.
Build and install HDF-4 with:
```
 ./configure --prefix=/usr/local --with-zlib=/usr/local \
       --with-jpeg=/usr/local --includedir=/usr/local/include/hdf \
       --disable-netcdf
  make all install
```
Note: if you are having problems with "make", one user reported that he used "gmake" instead.
The paths used for the "--with-zlib" and "--with-jpeg" options must be set to whatever you used above for these two software packages. If you choose to build HDF-4 with the szip compression library, be sure to add "--with-szip=/usr/local" (or whatever location you used when building it) to the configuration process.

HDF-EOS 2 software

If you need support for reading HDF-EOS 2 files, then you will need to download and install the following packages:

HDF-EOS 2 - Download version 2.18.v1 (HDF-EOS2.18v1.00) or later.
You may need to edit the file "bin/INSTALL-HDFEOS" to make sure it uses the correct compilers and compile options. Then run:
```
  bin/INSTALL-HDFEOS
```
Answer questions about location of HDF-4 lib and include files. You must include the "hdf" subdir for the HDF4 include directory (for example, "/usr/local/include/hdf").
Pay attention to where "libhdfeos.a" file gets installed under the hdfeos/lib directory (for example, "hdfeos/lib/linux64"), and copy or move this file to your library installation directory. Do the same for the include files. Do not copy libGctp.a! For example, on a 64-bit linux machine, you would type:
```
  mv lib/linux64/libhdfeos.a /usr/local/lib/.
  cp include/*.h /usr/local/include/.
```
gctp - this source should be part of the HDF-EOS2 software you just downloaded, in a directory "gctp/src".
- cd gctp/src
- Copy the appropriate makexxxx file to "makefile".
- Make "makefile" writable:
```
  chmod u+rw makefile
```
- Edit "makefile" and change LIBDIR appropriately (/usr/local/lib), and change the library name from "libgctpxxxx.a" to "libGctp.a" in all locations in this file (there's more than one reference).
- Copy some include files to the same directory you're in:
```
  cp ../include/*.h .
```
- Type:
```
  make
```

HDF-EOS 5 software

If you need support for reading HDF-EOS 5 files, then you will need to download and install the following packages:

szip - see instructions above.
HDF5 - see instructions above.

HDF-EOS 5 - Download version 1.11.v1 (HDF-EOS5.1.13) or later.

Once you have the HDF-EOS 5 source code, you can build and install it with:

  ./configure --with-hdf5=/usr/local --with-zlib=/usr/local --with-szlib=/usr/local --prefix=/usr/local
  make all install

You will also have to hand-copy two include files:

  cp include/HE5_GctpFunc.h include/HE5_HdfEosDef.h /usr/local/include/.

Download the source code for PyNIO

Accessing, downloading, and/or using PyNIO implies acceptance of the PyNIO source code license.

To download the PyNIO source code, follow the instructions at:

http://www.pyngl.ucar.edu/Download/

The source code you download will be a single compressed tar file called something like "PyNIO-1.4.1.tar.gz". Move this file to a temporary directory where you have plenty of disk space (around 250 megabytes to hold all of the source code, object files, binaries, and so on). Then, uncompress and untar the file as follows:

  gunzip PyNIO-1.4.1.tar.gz
  tar -xvf PyNIO-1.4.1.tar

The above steps will create a directory called "PyNIO-1.4.1".

Set environment variables if necessary

To build PyNIO, you may need to set some environment variables.

First, you need to set one or more of the following environment variables to "1" if you build this software package and want to include it in PyNIO:

  setenv HAS_GDAL 1
  setenv HAS_GRIB2 1
  setenv HAS_HDFEOS 1
  setenv HAS_HDFEOS5 1

In addition, if NetCDF was built with "classic" NetCDF-4 support:

  setenv HAS_NETCDF4 1

In addition, if NetCDF was built without OPeNDAP support:

  setenv HAS_OPENDAP 0

Furthermore, to help PyNIO locate the installed software, you must set the corresponding environment variables to the root directory of that installed software:

  GDAL_PREFIX
  GRIB2_PREFIX
  HDFEOS_PREFIX
  HDFEOS5_PREFIX
  NETCDF_PREFIX

You only need to set one of these if the paths are all the same.

Finally, you may need to help the PyNIO installation find the location of any system Fortran libraries needed to resolve symbols between C and Fortran code. For example, "-lgfortran" is needed if you built the software with gfortran, "-lg2c" if g77 was used, and "-lf95" for g95. Use F2CLIBS to indicate the library name (don't include the "-l"), and F2CLIBS_PREFIX to point to the location. For example:

  setenv F2CLIBS gfortran
  setenv F2CLIBS_PREFIX /usr/local/lib

Execute the 'setup.py' script

Type:

python setup.py install

to build and install PyNIO. To test PyNIO, see the "test PyNIO" section.

If you have problems, send email to pyngl-talk (you must be a member to post). We will add a trouble-shooting guide once we see what kind of problems people have.

Test PyNIO

To quickly test if PyNIO can be imported without errors, type "python" at the UNIX prompt and then "import Nio" at the python prompt. You should get the python prompt back with no errors.

To more thorougly test PyNIO, cd to the "test" directory and execute one of:

  ./test2.6.sh

  ./test2.7.sh

depending on whether you are using Python 2.6.x or 2.7.x.

The output will be sent to a file called "test.yyyy-mm-dd.py2n" which you can compare with the "test_compare" file.

You can further test PyNIO by installing PyNGL and running some of the examples available through pynglex.

Where to get help

If you are having problems building or installing PyNIO, send email to pyngl-talk@ucar.edu. You need to be a member in order to post.