Re: Problem creating hdf file from PyNio

From: David Brown <dbrown_at_nyahnyahspammersnyahnyah>
Date: Tue, 25 Aug 2009 18:17:11 -0600

Hi Andrew,

The problem seems to be that, unlike with NetCDF, the HDF 4 library
(or at least the HDF 4 NetCDF emulation code that the NIO library
relies on) does not like attributes to be set with an empty string --
(actually a 0-length character array). So this code in your
'replacevar' function:

         if hasattr(fvar.variables[varname], 'units'):
             setattr(fvar.variables[varname], 'units', '')

puts the HDF library into an error state causing all the subsequent
attempts to set an attribute in the 'replacevar' function to fail.

Changing the code to (e.g.):

         if hasattr(fvar.variables[varname], 'units'):
             setattr(fvar.variables[varname], 'units', 'none')

results in error-free execution of your script.

I will put it on my list to modify the HDF 4 code in the NIO library
to skip trying to set any attribute that is assigned an empty string
and then to return a meaningful warning message.

FYI, note that changing the _FillValue attribute does not change the
fill values in the data itself. You will have to do that yourself
using something like:

#replace the data -- assuming floating point data
vardata = np.where(np.isnan(vardata), 255, vardata)
fvar.variables[varname].assign_value(vardata)

(For awhile I thought maybe this had something to do with the problem)

Hope this helps,
  -dave

On Aug 25, 2009, at 10:22 AM, Andrew Meredith wrote:

> That's promising that you can reproduce the problem on your end.
> Thanks for the update.
>
> Andrew
>
> David Brown wrote:
>> Hi Andrew,
>>
>> I have your script and data and have managed to reproduce the
>> error. Hopefully it won't take too long to figure out what the
>> problem is. So far it has mystified me, but I am sure there is an
>> explanation.
>> -dave
>>
>> On Aug 24, 2009, at 6:37 AM, Andrew Meredith wrote:
>>
>>> Dave,
>>>
>>> I attached a sample script and an hdf input file that
>>> demonstrates the problem (at least on my end). You'll probably
>>> need to edit the anomfile variable to point to the correct
>>> location of the hdf file.
>>>
>>> Andrew
>>>
>>> David Brown wrote:
>>>> Hi Andrew,
>>>>
>>>> For awhile I seemed to be getting some errors similar (though
>>>> not the same) as you errors, but after rebuilding and making
>>>> sure everything is up-to-date I could no longer reproduce these
>>>> errors. It is somewhat mystifying because I don't know anything
>>>> in the re-build process that should have changed the behavior. I
>>>> did find one bug today:
>>>> PyNIO was not recognizing the '~' character or environment
>>>> variables specified in the filepath argument if (and only if)
>>>> the file was opened in write mode.
>>>> f = Nio.open_file('~/test.hdf','w') # produced an error.
>>>> I have checked in a fix for this, but it does not sound like it
>>>> is your problem.
>>>>
>>>> If possible can you send me a script along with (a) sample data
>>>> file(s)?
>>>>
>>>> You can upload as follows:
>>>>
>>>> ftp ftp.cgd.ucar.edu
>>>> login: anonymous
>>>> password: your email
>>>> cd incoming
>>>> put file1
>>>> put file2
>>>> ...etc.
>>>>
>>>> Then let me know the file names, because the ftp server does not
>>>> allow directory listings.
>>>> Thanks,
>>>> -dave
>>>>
>>>>
>>>> On Aug 21, 2009, at 2:02 PM, Andrew Meredith wrote:
>>>>
>>>>> Thanks.
>>>>>
>>>>> David Brown wrote:
>>>>>> Hi Andrew,
>>>>>> I am looking into this. If I need more information from you I
>>>>>> may need to contact you again. Either way I will let you know
>>>>>> what I discover.
>>>>>> -dave
>>>>>>
>>>>>> On Aug 21, 2009, at 10:37 AM, Andrew Meredith wrote:
>>>>>>
>>>>>>> Mary,
>>>>>>>
>>>>>>> I downloaded and installed the updated binaries. I was able
>>>>>>> to successfully create an hdf file although i noticed a
>>>>>>> couple of problems that don't occur if i create a netcdf file
>>>>>>> with the same code.
>>>>>>>
>>>>>>> 1) Updating some variable attributes reported fatal errors
>>>>>>> (see msg below). Some variable attributes didn't appear to
>>>>>>> cause problems.
>>>>>>>
>>>>>>> fatal:HDF: An error occurred while attempting to write the
>>>>>>> attribute (_FillValue) to variable (chlorANOMALY) in file (/
>>>>>>> work/AndrewM/HABensemble/products/
>>>>>>> MODSCW_P2009222_190560_D61_P2009138_P2009198_GM03_closest_chlorA
>>>>>>> NOMALY.hdf)
>>>>>>> fatal:HDF: An error occurred while attempting to write the
>>>>>>> attribute (missing_value) to variable (chlorANOMALY) in file
>>>>>>> (/work/AndrewM/HABensemble/products/
>>>>>>> MODSCW_P2009222_190560_D61_P2009138_P2009198_GM03_closest_chlorA
>>>>>>> NOMALY.hdf)
>>>>>>> fatal:HDF: An error occurred while attempting to write the
>>>>>>> attribute (fraction_digits) to variable (chlorANOMALY) in
>>>>>>> file (/work/AndrewM/HABensemble/products/
>>>>>>> MODSCW_P2009222_190560_D61_P2009138_P2009198_GM03_closest_chlorA
>>>>>>> NOMALY.hdf
>>>>>>>
>>>>>>> 2) I tried to update a global attribute, but the change
>>>>>>> wasn't reflected in the updated file. Code snippet:
>>>>>>> k = 'history'
>>>>>>> attrib = fvar.attributes[k]
>>>>>>> attrib += '\nReplaced the chlorAnomaly product
>>>>>>> with the HAB Ensemble product.'
>>>>>>> setattr(fvar, k, attrib)
>>>>>>>
>>>>>>> Thank very much for your help.
>>>>>>> Andrew
>>>>>>>
>>>>>>> Mary Haley wrote:
>>>>>>>>
>>>>>>>> Hi Andrew,
>>>>>>>>
>>>>>>>> We don't have a 32-bit system with gcc 3.x, so I built it
>>>>>>>> with gcc
>>>>>>>> 4.1, and we'll see if that works for you.
>>>>>>>>
>>>>>>>> The file is on ftp:
>>>>>>>>
>>>>>>>> ftp ftp.ucar.edu
>>>>>>>> <log in as "anonymous">
>>>>>>>> <Use email address as password>
>>>>>>>> cd /pub/scd/haley
>>>>>>>> get PyNIO-1.3.0b3.linux-i686.tar.gz
>>>>>>>> quit
>>>>>>>>
>>>>>>>> The instructions for installing the software are at:
>>>>>>>>
>>>>>>>> http://www.pyngl.ucar.edu/Download/install_from_binary.shtml
>>>>>>>>
>>>>>>>> Let me know if you have any problems with it.
>>>>>>>>
>>>>>>>> --Mary
>>>>>>>>
>>>>>>>> On Fri, 21 Aug 2009, Andrew Meredith wrote:
>>>>>>>>
>>>>>>>>> Hi Mary,
>>>>>>>>>
>>>>>>>>> Here's the info. you requested.
>>>>>>>>>
>>>>>>>>> :uname -a;gcc --version
>>>>>>>>> Linux nccos-s-vmatic.nos.noaa 2.6.9-89.0.3.ELsmp #1 SMP Sat
>>>>>>>>> Jun 13 07:05:54 EDT 2009 i686 i686 i386 GNU/Linux
>>>>>>>>> gcc (GCC) 3.4.6 20060404 (Red Hat 3.4.6-11)
>>>>>>>>> Copyright (C) 2006 Free Software Foundation, Inc.
>>>>>>>>> This is free software; see the source for copying
>>>>>>>>> conditions. There is NO warranty; not even for
>>>>>>>>> MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>>>>>>>>>
>>>>>>>>> Andrew
>>>>>>>>>
>>>>>>>>> Mary Haley wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Andrew,
>>>>>>>>>>
>>>>>>>>>> We think we have a fix.
>>>>>>>>>>
>>>>>>>>>> Can you send me the output of:
>>>>>>>>>>
>>>>>>>>>> uname -a
>>>>>>>>>> gcc --version
>>>>>>>>>>
>>>>>>>>>> and I'll give you a new PyNIO binary.
>>>>>>>>>>
>>>>>>>>>> --Mary
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, 17 Aug 2009, Andrew Meredith wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Mary,
>>>>>>>>>>>
>>>>>>>>>>> Just wondering if you have made any progress on this issue?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Andrew
>>>>>>>>>>>
>>>>>>>>>>> Mary Haley wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Andrew,
>>>>>>>>>>>>
>>>>>>>>>>>> We're looking into this. You should be able to do this.
>>>>>>>>>>>>
>>>>>>>>>>>> We're getting conflicting results when we try to run
>>>>>>>>>>>> this example. It
>>>>>>>>>>>> may be due to the way I compiled the b2 version.
>>>>>>>>>>>>
>>>>>>>>>>>> I'll keep you posted.
>>>>>>>>>>>>
>>>>>>>>>>>> --Mary
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, 10 Aug 2009, Andrew Meredith wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I recently installed PyNio/PyNgl from binaries provided
>>>>>>>>>>>>> by Mary Haley
>>>>>>>>>>>>> (PyNGL-1.3.0b2.linux-i686-gcc412.tar.gz and
>>>>>>>>>>>>> PyNIO-1.3.0b2.linux-i686-gcc412.tar.gz). I installed
>>>>>>>>>>>>> on an i386
>>>>>>>>>>>>> platform with RHEL 4, gcc 3.4.6, and python 2.6.2.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I can successfully read hdf4 files, but I am unable to
>>>>>>>>>>>>> create a new hdf
>>>>>>>>>>>>> file.
>>>>>>>>>>>>>
>>>>>>>>>>>>> f = Nio.open_file('test.hdf', 'c') fails with a
>>>>>>>>>>>>> 'nccreate: Bad Flag' error.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Should i be able to create an hdf file? I can create a
>>>>>>>>>>>>> netCDF file.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Andrew Meredith
>>>>>>>>>>>>> CSS, Inc. Contractor
>>>>>>>>>>>>> NOAA National Centers for Coastal Ocean Science
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2234 South Hobson Ave
>>>>>>>>>>>>> Charleston, SC 29405-2413
>>>>>>>>>>>>> (843) 740-1309
>>>>>>>>>>>>> Andrew.Meredith_at_noaa.gov
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> pyngl-talk mailing list
>>>>>>>>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Andrew Meredith
>>>>>>>>>>> CSS, Inc. Contractor
>>>>>>>>>>> NOAA National Centers for Coastal Ocean Science
>>>>>>>>>>>
>>>>>>>>>>> 2234 South Hobson Ave
>>>>>>>>>>> Charleston, SC 29405-2413
>>>>>>>>>>> (843) 740-1309 Andrew.Meredith_at_noaa.gov
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Andrew Meredith
>>>>>>>>> CSS, Inc. Contractor
>>>>>>>>> NOAA National Centers for Coastal Ocean Science
>>>>>>>>>
>>>>>>>>> 2234 South Hobson Ave
>>>>>>>>> Charleston, SC 29405-2413
>>>>>>>>> (843) 740-1309 Andrew.Meredith_at_noaa.gov
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Andrew Meredith
>>>>>>> CSS, Inc. Contractor
>>>>>>> NOAA National Centers for Coastal Ocean Science
>>>>>>>
>>>>>>> 2234 South Hobson Ave
>>>>>>> Charleston, SC 29405-2413
>>>>>>> (843) 740-1309 Andrew.Meredith_at_noaa.gov
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Andrew Meredith
>>>>> CSS, Inc. Contractor
>>>>> NOAA National Centers for Coastal Ocean Science
>>>>>
>>>>> 2234 South Hobson Ave
>>>>> Charleston, SC 29405-2413
>>>>> (843) 740-1309 Andrew.Meredith_at_noaa.gov
>>>>>
>>>>
>>>
>>> --
>>> Andrew Meredith
>>> CSS, Inc. Contractor
>>> NOAA National Centers for Coastal Ocean Science
>>>
>>> 2234 South Hobson Ave
>>> Charleston, SC 29405-2413
>>> (843) 740-1309 Andrew.Meredith_at_noaa.gov
>>>
>>> <MODSCW_P2009222_1905_D61_P2009138_P2009198_GM03_closest_chlorANOMAL
>>> Y.hdf.bz2>#!/usr/bin/env python
>>> '''
>>> Decription: Sample script to demonstrate PyNIO problem with
>>> updating file & variable attributes
>>>
>>> '''
>>> import sys, os, glob, re
>>> import numpy as np
>>> import numpy.ma as ma
>>> import Nio
>>> import Ngl
>>> from datetime import datetime
>>>
>>>
>>> def copyhdffile(outfname, in_fvar):
>>> '''Creates an hdf file based on another file. Copies all
>>> global attributes, variables
>>> (including var. attributes and data).'''
>>>
>>> #map Numpy array types to PyNIO variable types
>>> pynio_types = {'uint8':'b', 'int8':'b', 'int16':'h',
>>> 'int32':'i', 'long':'l', 'float32':'f', 'float64':'d'}
>>>
>>> out_fvar = Nio.open_file(outfname, 'c')
>>>
>>> #copy the dimensions
>>> for k in in_fvar.dimensions.keys():
>>> out_fvar.create_dimension(k, in_fvar.dimensions[k])
>>>
>>> #copy the file attributes
>>> for k in in_fvar.attributes.keys():
>>> attrib = in_fvar.attributes[k]
>>> setattr(out_fvar, k, attrib)
>>>
>>> #copy variable including attibutes and data
>>> for k in in_fvar.variables.keys():
>>> v = in_fvar.variables[k]
>>> data = v.get_value()
>>>
>>> if not pynio_types[str(data.dtype)]:
>>> print 'Unsupported data type "%s". %s variable not
>>> copied to output fil' % (data.dtype, k)
>>> continue
>>> var = out_fvar.create_variable(k, pynio_types[str
>>> (data.dtype)], getattr(v, 'dimensions'))
>>>
>>> #copy attributes
>>> for attrkey in v.attributes.keys():
>>> if attrkey == 'hdf_name': continue #created
>>> automatically by PyNIO for hdf files
>>>
>>> attrib = v.attributes[attrkey]
>>> setattr(out_fvar.variables[k], attrkey, attrib)
>>>
>>> #copy data
>>> var.assign_value(data)
>>>
>>> out_fvar.close()
>>>
>>> def replacevar(fname, vardata, varname='chlorANOMALY',
>>> longname='HAB ensemble product - replaced chlorANOMALY'):
>>> '''Replace variable "varname" within file "fname" with data
>>> contained within "vardata" array.
>>> It is assumed the vardata array matches the dimensions of the
>>> file variable.'''
>>>
>>> print 'in replacevar: fname=%s varname=%s' % (fname, varname)
>>>
>>> fvar = Nio.open_file(fname, 'w')
>>>
>>> #check variable exists within file
>>> if varname in fvar.variables:
>>> print 'in replacevar: %s found in %s' % (varname, fname)
>>>
>>> #replace the data
>>> fvar.variables[varname].assign_value(vardata)
>>>
>>> #update variable metadata
>>> if hasattr(fvar.variables[varname], 'long_name'):
>>> setattr(fvar.variables[varname], 'long_name', longname)
>>>
>>> if hasattr(fvar.variables[varname], 'units'):
>>> setattr(fvar.variables[varname], 'units', '')
>>>
>>> if hasattr(fvar.variables[varname], '_FillValue'):
>>> setattr(fvar.variables[varname], '_FillValue', 255.0)
>>>
>>> if hasattr(fvar.variables[varname], 'missing_value'):
>>> setattr(fvar.variables[varname], 'missing_value', 255.0)
>>>
>>> if hasattr(fvar.variables[varname], 'fraction_digits'):
>>> setattr(fvar.variables[varname], 'fraction_digits', 1)
>>>
>>> #update global metadata
>>> k = 'history'
>>> if hasattr(fvar, k):
>>> # --include timestamp
>>> date = datetime.utcnow().strftime('%Y%m%d %H:%M:%S')
>>> attrib = fvar.attributes[k]
>>> attrib += '\n%s - Replaced the %s product with the
>>> HAB Ensemble product.\n%s - Modified the filename where the SS
>>> component of the YYYYDDD_HHMMSS filename was set to "60". If no
>>> SS component within filename a SS component was added.\n' %
>>> (date, varname, date)
>>> setattr(fvar, k, attrib)
>>> else:
>>> print 'Error in replacevar - Replace failed.: %s variable
>>> not found within %s.' % (varname, fname)
>>>
>>> fvar.close()
>>>
>>> if __name__ == '__main__':
>>>
>>> #noaa coastwatch HDF file
>>> anomfile = '/data/MODIS/GM/cwhdf/
>>> MODSCW_P2009222_1905_D61_P2009138_P2009198_GM03_closest_chlorANOMALY
>>> .hdf'
>>> anom_fvar = Nio.open_file(anomfile, 'r')
>>>
>>> #processing here that generates a new data layer, but for
>>> simplicity we'll just grab
>>> # an existing file variable.
>>> var_data = anom_fvar.variables['chlor_a'].get_value()
>>>
>>> outfname = 'test.hdf'
>>>
>>> ##copy hdf file to new hdf file
>>> # #try creating a netcdf file for now - since PyNIO can't
>>> currently write the hdf
>>> # outfname = outfname.replace('.hdf', '.nc')
>>> os.system('rm -f %s' % outfname)
>>> print 'invoking copyhdffile'
>>> copyhdffile(outfname, anom_fvar)
>>>
>>> #replace the anomaly product with habensemble product
>>> print 'invoking replacevar(%s, %s, %s)' % (outfname,
>>> var_data, "varname='chlorANOMALY'")
>>> replacevar(outfname, var_data, varname='chlorANOMALY')
>>>
>>> #close files
>>> anom_fvar.close()
>>>
>>> Ngl.end()
>>
>
> --
> Andrew Meredith
> CSS, Inc. Contractor
> NOAA National Centers for Coastal Ocean Science
>
> 2234 South Hobson Ave
> Charleston, SC 29405-2413
> (843) 740-1309 Andrew.Meredith_at_noaa.gov
>

_______________________________________________
pyngl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
Received on Tue Aug 25 2009 - 18:17:11 MDT

This archive was generated by hypermail 2.2.0 : Thu Sep 10 2009 - 15:06:39 MDT